Speaking Out

Testmaker: What went wrong with TNReady

The head of the company that created TNReady accepts blame for this year’s botched rollout of Tennessee’s new standardized online assessment, but says the subsequent delays in delivering printed testing materials were unavoidable.

Measurement Inc. president and founder Henry Scherich says Tennessee Education Commissioner Candice McQueen’s decision to scrap the online assessment on the first day of testing in February set in motion a chain of logistical quagmires that were impossible to overcome.

Once McQueen ordered districts to switch back to paper tests, his company found the sudden task of printing and delivering up to 5 million documents this spring overwhelming, if not impossible.

Henry Scherich
Henry Scherich

“I understand the frustration of superintendents and the state department,” Scherich said. “Having said all of that, this was a huge job that we took on and there’s been no testing company in the country — in the world, probably — who has taken on the task of printing and shipping this many tests in this short a period of time, and we really struggled with it.”

Last week, the Tennessee Department of Education informed district leaders that many of the testing materials wouldn’t arrive in time for the opening of this week’s final TNReady testing window — the latest in a series of delivery delays that has wreaked havoc in districts and classrooms across the state. State leaders placed the blame squarely on Measurement Inc.

In an interview this week with Chalkbeat, Scherich acknowledged that developing and delivering TNReady in a new online platform was the biggest job that his 36-year-old Durham, N.C.-based company has ever undertaken — perhaps too big given the one-year deadline.

He offered a behind-the-scenes look at the snafus and challenges. At the same time, he insisted that TNReady is a strong test and — once its delivery platform is fixed — the assessment can can help Tennessee reach its accountability goals.

Measurement Inc. won the bid to create Tennessee’s test for grades 3-11 math and English language arts in October of 2014, only months after a vote by the Tennessee legislature prompted the Department of Education to pull out of PARCC, a consortium of other states with a shared Common Core-aligned assessment. The company would have a year to develop a test for Tennessee. A small number of high school students on block schedules would take the test in the fall of 2015, with the bulk of students in grades 3-11 taking it the following spring.

TNReady marked an unprecedented shift for Tennessee and, like PARCC, was supposed to be online and aligned with the current Common Core State Standards.

"It was a failure in some respects because we were supposed to design a system that would take 100,000 students in at one time."Henry Scherich

It was also an unprecedented task for Measurement Inc., which had never before developed and delivered a state’s entire online testing program.

But on Feb. 8, the very first day of statewide online testing, the test buckled as more and more students logged on. Even so, leaders of Measurement Inc. were surprised when McQueen quickly pulled the plug on the online assessment, and announced that the state would switch to paper-and-pencil versions.

Here’s what happened, according to Scherich:

Online ‘crash’

Scherich says that, first of all, the system never “crashed” on the first day. Students’ screens never went blank. Instead, he calls what happened “infrastructure saturation.” As more and more students logged on, their cursors began to spin, signaling that the test was taking longer to load than it should have.

What was the problem? Ultimately, Scherich says, there weren’t enough servers for the volume of students online, causing the system to clog up as more and more students logged in. He declined to speculate on how long it would have taken to fix the problem and add more primary servers, but said that it would have been possible to get back on track.

“We could have duplicated the system,” he said. “We would have said to half of the state, you work on these 64 servers and the other half work on another set.”

About 48,000 students logged on that day, and about 18,000 submitted assessments. It’s unknown the number of students who weren’t having troubles with the test, but stopped after McQueen sent an email instructing districts to halt testing.

“It was a failure in some respects because we were supposed to design a system that would take 100,000 students in at one time… We had a problem with 48,000,” Scherich said.

Printing delays

Scherich says the subsequent delays come down to this: There were a lot of tests to be printed, and not a lot of printers available on short notice. Overall, the switch to printing meant Measurement Inc. had to scramble to print answer sheets and test booklets for grades 3-11 amounting to 5 million documents — when only weeks before, they hadn’t planned on printing any.

"There’s been no testing company in the country — in the world, probably — who has taken on the task of printing and shipping this many tests in this short a period of time ..."

Measurement Inc. worked with the Department of Education to transfer different versions of the tests from computer to paper. Each test had several versions with different field test items embedded within.

“You can’t just push the button on the computer and have the test be printed out,” he said. “The formatting is all different.”

In the meantime, Measurement Inc. sought out printers who were able to fulfill the large order quickly. Through 36 years in testing, the company had a lot of connections, but only three printing plant operators said they were up to the task. Eventually, two backed out, leaving Measurement Inc. with one: RR Donnelley based in Chicago.

“It’s a large printing company, and they had plants all over the U.S. They printed one or more of the tests or the answer documents in 11 different printing plants around the country,” he said. “So we were getting tests from Minnesota, Missouri. They ran a lot of night shifts to do that for us.”

Once the tests and answer sheets arrived at Measurement Inc., they had to be sorted and distributed to schools. That’s 5 million tests, spread across nearly 1,000 schools.

The last documents arrived from the printer last Saturday, and Measurement Inc. is rushing to get them out in the next two to three days, Scherich said.

Tight timeline

Measurement Inc. had about a year to develop the test and roll out an online system for the entire state. In comparison, PARCC, the online assessment that Tennessee originally was slated to use, was developed in about five years.

Though Measurement Inc. had been working on its online platform for six years and used it previously in other states, including Tennessee for its writing test, the company had never before undertaken a state’s entire testing program.

Measurement Inc. not only developed the TNReady tests for math and English language arts, but also put the content for science and social studies on its online platform, known as MIST.

He said a lot was done right in developing TNReady, including the recruitment of 400 Tennessee teachers to help write test questions designed to measure critical thinking skills.

“I think that our staff and the state of Tennessee staff did an excellent job in building an assessment,” he said. “The math test is a good test. (English language arts) is a good test. Tennessee has a good catalog, a good library of test items for the future.”

good news bad news

Most Tennessee districts are showing academic growth, but districts with the farthest to go improved the least

PHOTO: Alan Petersime

It’s not just Memphis: Across Tennessee, districts with many struggling schools posted lower-than-expected growth scores on this year’s state exams, according to data released Tuesday.

The majority of Tennessee’s 147 districts did post scores that suggest students are making or exceeding expected progress, with over a third earning the top growth score.

But most students in three of the state’s four largest districts — in Memphis, Nashville and Chattanooga — aren’t growing academically as they should, and neither are those in most of their “priority schools” in the state’s bottom 5 percent.

The divide prompted Education Commissioner Candice McQueen to send a “good news, bad news” email to superintendents.

“These results point to the ability for all students to grow,” she wrote of the top-performing districts, many of which have a wide range of academic achievement and student demographics.

Of those in the bottom, she said the state would analyze the latest data to determine “critical next steps,” especially for priority schools, which also are located in high-poverty communities.

“My message to the leaders of Priority schools … is that this level of growth will never get kids back on track, so we have to double-down on what works – strong instruction and engagement, every day, with no excuses,” McQueen said.

Growth scores are supposed to take poverty into account, so the divide suggests that either the algorithm didn’t work as it’s supposed to or, in fact, little has happened to change conditions at the state’s lowest-performing schools, despite years of aggressive efforts in many places.

The results are bittersweet for Tennessee, which has pioneered growth measures for student learning and judging the effectiveness of its teachers and schools under its Tennessee Value-Added Assessment System, known as TVAAS.

On the one hand, the latest TVAAS data shows mostly stable growth through the transition to TNReady, the state’s new test aligned to new academic standards, in the first year of full testing for grades 3-11. On the other hand, Tennessee has invested tens of millions of dollars and years of reforms toward improving struggling schools — all part of its massive overhaul of K-12 education fueled by its 2009 federal Race to the Top award.

The state-run Achievement School District, which launched in the Race to the Top era to turn around the lowest-performing schools, saw a few bright spots, but almost two-thirds of schools in its charter-reliant portfolio scored in the bottom levels of student growth.

Shelby County’s own turnaround program, the Innovation Zone, fared poorly too, with a large percentage of its Memphis schools scoring 1 on a scale of 1 to 5, after years of scoring 4s and 5s.


District profile: Most Memphis schools score low on student growth


Superintendent Dorsey Hopson called the results a “wakeup call” for the state’s biggest district in Memphis.

“When you have a population of kids in high poverty that were already lagging behind on the old, much easier test, it’s not surprising that we’ve got a lot of work to do here,” he said, citing the need to support teachers in mastering the state’s new standards.

“The good part is that we’ve seen the test now and we know what’s expected. The bad part is we’ve seen the test … and it’s a different monster,” he told Chalkbeat.

You can find district composite scores below. (A TVAAS score of 3 represents average growth for a student in one school year.) For a school-by-school list, visit the state’s website.

exclusive

Most Memphis schools score low on student growth under new state test

PHOTO: Stephanie Snyder

More than half of Memphis schools received the lowest possible score for student growth on Tennessee’s new test last school year, according to data obtained by Chalkbeat for Shelby County Schools.

On a scale of 1 to 5, with 1 being the lowest measure, about 54 percent of the district’s 187 schools scored in the bottom rung of the Tennessee Value-Added Assessment System, known as TVAAS.

That includes most schools in the Innovation Zone, a reversal after years of showing high growth in the district’s prized turnaround program.

Charter schools fared poorly as well, as did schools that were deemed among the state’s fastest-improving in 2015.

Superintendent Dorsey Hopson called the scores a “huge wakeup call.”

“It shows that we’ve got a tremendous amount of work to do,” Hopson told Chalkbeat on Monday. “It’s going to be hard and it’s going to be frustrating. … It starts with making sure we’re supporting teachers around mastering the new standards.”

District leaders across Tennessee have been trying to wrap their heads around the latest growth scores since receiving the data in late August from the State Department of Education. Only two years earlier, the Memphis district garnered the highest possible overall growth score. But since then, the state has switched to a harder test called TNReady that is aligned for the first time to more rigorous academic standards.

TVAAS results are scheduled to be released publicly this week, but Chalkbeat obtained a copy being circulated within Shelby County Schools, Tennessee’s largest district.

The data is prompting questions from some Memphis educators — and assurances from state officials — over the validity of TVAAS, the state’s system for measuring learning and judging the effectiveness of its teachers and schools.

This is the first year of issuing district-wide TVAAS scores since 2015. That’s because of the state’s cancellation of 2016 testing for grades 3-8 due mostly to failures in the switch to online testing.

Some educators wonder whether the bumpy switch to TNReady is a factor in this year’s nosedive, along with changes in how the scores are calculated.

For example, data for fourth-graders is missing since there is no prior state testing in third grade for comparison. Elementary and middle schools also don’t have growth scores for social studies, since the 2017 questions were a trial run and the results don’t count toward a school’s score.

Hopson acknowledged concerns over how the state compares results from “two very different tests which clearly are apples and oranges,” but he added that the district won’t use that as an excuse.

“Notwithstanding those questions, it’s the system upon which we’re evaluated on and judged,” he said.

State officials stand by TVAAS. They say drops in proficiency rates resulting from a harder test have no impact on the ability of teachers, schools and districts to earn strong TVAAS scores, since all students are experiencing the same change.

“Because TVAAS always looks at relative growth from year to year, not absolute test scores, it can be stable through transitions,” said Sara Gast, a spokeswoman for the State Department of Education.

Shelby County Schools is not the only district with disappointing TVAAS results. In Chattanooga, Hamilton County Schools logged low growth scores. But Gast said that more districts earned average or high growth scores of 3, 4 or 5 last school year than happened in 2015.

Want to help us understand this issue? Send your observations to [email protected]

Below is a breakdown of Shelby County’s TVAAS scores. A link to a school-by-school list of scores is at the bottom of this story.

Districtwide

School-wide scores are a combination of growth in each tested subject: literacy, math, science and social studies.

Fifty three schools saw high growth in literacy, an area where Shelby County Schools has doubled down, especially in early grades. And 51 schools saw high growth in math.

Note: A TVAAS score of 3 represents average growth for a student in one school year. A score of 1 represents significantly lower academic growth compared to peers across the state.

2017

School-wide composite Number of schools Percent of schools
1 101 54%
2 19 10%
3 20 11%
4 10 5%
5 37 20%

2015

School-wide composite Number of schools Percent of schools
1 58 28%
2 16 8%
3 38 19%
4 18 9%
5 75 37%

Innovation Zone

Out of the 23 schools in the district’s program to turn around low-performing schools, most received a growth score of 1 in 2017. That stands in stark contrast to prior years since the program opened in 2012, when most schools were on a fast growth track.

School-wide composite Number of iZone schools
1 14
2 2
3 2
4 0
5 5

Reward schools

Nearly half of 32 schools deemed 2015 Tennessee reward schools for high growth saw a major drop in TVAAS scores in 2017:

  • Central High
  • Cherokee Elementary
  • Germanshire Elementary
  • KIPP Memphis Middle Academy
  • Kirby High
  • Memphis Business Academy Elementary
  • Power Center Academy High
  • Power Center Academy Middle
  • Ross Elementary
  • Sheffield High
  • South Park Elementary
  • Southwind High
  • Treadwell Middle
  • Westside Elementary

Charter schools

Charter schools authorized by Shelby County Schools fared similarly to district-run schools in growth scores, with nearly half receiving a TVAAS of 1 compared to 26 percent of charter schools receiving the same score in 2015.

2017

School-wide composite Number of iZone schools
1 18
2 6
3 7
4 2
5 7

2015

School-wide composite Number of iZone schools
1 10
2 2
3 7
4 3
5 16

Optional schools

Half of the the district’s optional schools, which are special studies schools that require students to test into its programs, received a 1 on TVAAS. That’s compared to just 19 percent in 2015.

2017

School-wide composite Number of iZone schools
1 23
2 6
3 5
4 2
5 10

2015

School-wide composite Number of iZone schools
2 5
3 6
4 5
5 14

You can sort through a full list of TVAAS scores for Shelby County Schools here.