Speaking Out

Testmaker: What went wrong with TNReady

The head of the company that created TNReady accepts blame for this year’s botched rollout of Tennessee’s new standardized online assessment, but says the subsequent delays in delivering printed testing materials were unavoidable.

Measurement Inc. president and founder Henry Scherich says Tennessee Education Commissioner Candice McQueen’s decision to scrap the online assessment on the first day of testing in February set in motion a chain of logistical quagmires that were impossible to overcome.

Once McQueen ordered districts to switch back to paper tests, his company found the sudden task of printing and delivering up to 5 million documents this spring overwhelming, if not impossible.

Henry Scherich
Henry Scherich

“I understand the frustration of superintendents and the state department,” Scherich said. “Having said all of that, this was a huge job that we took on and there’s been no testing company in the country — in the world, probably — who has taken on the task of printing and shipping this many tests in this short a period of time, and we really struggled with it.”

Last week, the Tennessee Department of Education informed district leaders that many of the testing materials wouldn’t arrive in time for the opening of this week’s final TNReady testing window — the latest in a series of delivery delays that has wreaked havoc in districts and classrooms across the state. State leaders placed the blame squarely on Measurement Inc.

In an interview this week with Chalkbeat, Scherich acknowledged that developing and delivering TNReady in a new online platform was the biggest job that his 36-year-old Durham, N.C.-based company has ever undertaken — perhaps too big given the one-year deadline.

He offered a behind-the-scenes look at the snafus and challenges. At the same time, he insisted that TNReady is a strong test and — once its delivery platform is fixed — the assessment can can help Tennessee reach its accountability goals.

Measurement Inc. won the bid to create Tennessee’s test for grades 3-11 math and English language arts in October of 2014, only months after a vote by the Tennessee legislature prompted the Department of Education to pull out of PARCC, a consortium of other states with a shared Common Core-aligned assessment. The company would have a year to develop a test for Tennessee. A small number of high school students on block schedules would take the test in the fall of 2015, with the bulk of students in grades 3-11 taking it the following spring.

TNReady marked an unprecedented shift for Tennessee and, like PARCC, was supposed to be online and aligned with the current Common Core State Standards.

"It was a failure in some respects because we were supposed to design a system that would take 100,000 students in at one time."Henry Scherich

It was also an unprecedented task for Measurement Inc., which had never before developed and delivered a state’s entire online testing program.

But on Feb. 8, the very first day of statewide online testing, the test buckled as more and more students logged on. Even so, leaders of Measurement Inc. were surprised when McQueen quickly pulled the plug on the online assessment, and announced that the state would switch to paper-and-pencil versions.

Here’s what happened, according to Scherich:

Online ‘crash’

Scherich says that, first of all, the system never “crashed” on the first day. Students’ screens never went blank. Instead, he calls what happened “infrastructure saturation.” As more and more students logged on, their cursors began to spin, signaling that the test was taking longer to load than it should have.

What was the problem? Ultimately, Scherich says, there weren’t enough servers for the volume of students online, causing the system to clog up as more and more students logged in. He declined to speculate on how long it would have taken to fix the problem and add more primary servers, but said that it would have been possible to get back on track.

“We could have duplicated the system,” he said. “We would have said to half of the state, you work on these 64 servers and the other half work on another set.”

About 48,000 students logged on that day, and about 18,000 submitted assessments. It’s unknown the number of students who weren’t having troubles with the test, but stopped after McQueen sent an email instructing districts to halt testing.

“It was a failure in some respects because we were supposed to design a system that would take 100,000 students in at one time… We had a problem with 48,000,” Scherich said.

Printing delays

Scherich says the subsequent delays come down to this: There were a lot of tests to be printed, and not a lot of printers available on short notice. Overall, the switch to printing meant Measurement Inc. had to scramble to print answer sheets and test booklets for grades 3-11 amounting to 5 million documents — when only weeks before, they hadn’t planned on printing any.

"There’s been no testing company in the country — in the world, probably — who has taken on the task of printing and shipping this many tests in this short a period of time ..."

Measurement Inc. worked with the Department of Education to transfer different versions of the tests from computer to paper. Each test had several versions with different field test items embedded within.

“You can’t just push the button on the computer and have the test be printed out,” he said. “The formatting is all different.”

In the meantime, Measurement Inc. sought out printers who were able to fulfill the large order quickly. Through 36 years in testing, the company had a lot of connections, but only three printing plant operators said they were up to the task. Eventually, two backed out, leaving Measurement Inc. with one: RR Donnelley based in Chicago.

“It’s a large printing company, and they had plants all over the U.S. They printed one or more of the tests or the answer documents in 11 different printing plants around the country,” he said. “So we were getting tests from Minnesota, Missouri. They ran a lot of night shifts to do that for us.”

Once the tests and answer sheets arrived at Measurement Inc., they had to be sorted and distributed to schools. That’s 5 million tests, spread across nearly 1,000 schools.

The last documents arrived from the printer last Saturday, and Measurement Inc. is rushing to get them out in the next two to three days, Scherich said.

Tight timeline

Measurement Inc. had about a year to develop the test and roll out an online system for the entire state. In comparison, PARCC, the online assessment that Tennessee originally was slated to use, was developed in about five years.

Though Measurement Inc. had been working on its online platform for six years and used it previously in other states, including Tennessee for its writing test, the company had never before undertaken a state’s entire testing program.

Measurement Inc. not only developed the TNReady tests for math and English language arts, but also put the content for science and social studies on its online platform, known as MIST.

He said a lot was done right in developing TNReady, including the recruitment of 400 Tennessee teachers to help write test questions designed to measure critical thinking skills.

“I think that our staff and the state of Tennessee staff did an excellent job in building an assessment,” he said. “The math test is a good test. (English language arts) is a good test. Tennessee has a good catalog, a good library of test items for the future.”

more digging

Kingsbury High added to list of Memphis schools under investigation for grade changing

PHOTO: Shelby County Schools
Kingsbury High School was added to a list of schools being investigated by an outside firm for improper grade changes. Here, Principal Terry Ross was featured in a Shelby County Schools video about a new school budget tool.

Another Memphis high school has been added to the list of schools being investigated to determine if they made improper changes to student grades.

Adding Kingsbury High School to seven others in Shelby County Schools will further delay the report initially expected to be released in mid-June.

But from what school board Chairwoman Shante Avant has heard so far, “there haven’t been any huge irregularities.”

“Nothing has surfaced that gives me pause at this point,” Avant told Chalkbeat on Thursday.

The accounting firm Dixon Hughes Goodman is conducting the investigation.

This comes about three weeks after a former Kingsbury teacher, Alesia Harris, told school board members that Principal Terry Ross instructed someone to change 17 student exam grades to 100 percent — against her wishes.

Shelby County Schools said the allegations were “inaccurate” and that the grade changes were a mistake that was self-reported by an employee.

“The school administration immediately reported, and the central office team took the necessary actions and promptly corrected the errors,” the district said in a statement.

Chalkbeat requested a copy of the district’s own initial investigation the day after Harris spoke at the board’s June meeting, but district officials said they likely would not have a response for Chalkbeat until July 27.

Harris said that no one from Dixon Hughes Goodman has contacted her regarding the investigation as of Thursday.

The firm’s investigation initially included seven schools. Kingsbury was not among them. Those seven schools are:

  • Kirby High
  • Raleigh-Egypt High
  • Bolton High
  • Westwood High
  • White Station High
  • Trezevant High
  • Memphis Virtual School

The firm’s first report found as many as 2,900 failing grades changed during four years at nine Memphis-area schools. At the request of the board, two schools were eliminated: one a charter managed by a nonprofit, and a school outside the district. The firm said at the time that further investigation was warranted to determine if the grade changes were legitimate.

The $145,000 investigation includes interviews with teachers and administrators, comparing teachers’ paper grade books to electronic versions, accompanying grade change forms, and inspecting policies and procedures for how school employees track and submit grades.

Since the controversy started last year, the district has restricted the number of employees authorized to make changes to a student’s report card or transcript, and also requires a monthly report from principals detailing any grade changes.

Silver Lining Playbook

Memphis’ youngest students show reading gains on 2018 state tests — and that’s a big deal

PHOTO: Caroline Bauman
A student works on reading comprehension skills at Lucie E Campbell Elementary School in Memphis and Shelby County Schools.

Those working to improve early literacy rates in Shelby County Schools got a small morale boost Thursday as newly released scores show the district’s elementary school students improved their reading on 2018 state tests.

The percentage of Memphis elementary-age students considered proficient in reading rose by 3 points to almost one-fourth of the district’s children in grades 3 through 5. That’s still well below the state average, and Superintendent Dorsey Hopson said “we obviously have a long way to go.”

PHOTO: Caroline Bauman
Superintendent Dorsey Hopson has overseen Tennessee’s largest public school district since 2013.

Strengthening early literacy has been a priority for the Memphis district, which views better reading skills as crucial to predicting high school graduation and career success. To that end, Shelby County Schools has expanded access to pre-K programs, adjusted reading curriculum, and made investments in literacy training for teachers.

Hopson said the payoff on this year’s TNReady scores was a jump of almost 5 percentage points in third-grade reading proficiency.

“It was about five years ago when we really, really, really started pushing pre-K, and those pre-K kids are now in the third grade. I think that’s something that’s really positive,” Hopson said of the gains, adding that third-grade reading levels are an important indicator of future school performance.

TNReady scores for Shelby County Schools, which has a high concentration of low-performing schools and students living in poverty, were a mixed bag, as they were statewide.

Math scores went up in elementary, middle, and high schools in Tennessee’s largest district. But science scores went down across the board, and the percentage of high school students who scored proficient in reading dropped by 4 percentage points.

The three charts below illustrate, by subject, the percentages of students who performed on track or better in elementary, middle, and high schools within Shelby County Schools. The blue bars reflect the district’s most recent scores, the black bars show last year’s scores, and the yellow bars depict this year’s statewide averages.

Hopson said he was unsure how much the scores of older students — all of whom tested online — were affected by technical problems that hampered Tennessee’s return this year to computerized testing.

“From what people tell me, kids either didn’t try as hard in some instances or didn’t take it seriously,” Hopson told reporters. “We’ll never know what the real impact is, but we have to accept the data that came from these tests.”

But students in two of the district’s school improvement initiatives — the Innovation Zone and the Empowerment Zone — showed progress. “We’re going to double down on these strategies,” Hopson said of the extra investments and classroom supports.

In the state-run Achievement School District, or ASD, which oversees 30 low-performing schools in Memphis, grades 3 through 8 saw an uptick in scores in both reading and math. But high schoolers scored more than 3 percentage points lower in reading and also took a step back in science.

The ASD takes over schools in the state’s bottom 5 percent and assigns them to charter operators to improve. But in the five years that the ASD has been in Memphis, its scores have been mostly stagnant.

Tennessee Education Commissioner Candice McQueen said she and new ASD Superintendent Sharon Griffin are reviewing the new data to determine next steps.

“We are seeing some encouraging momentum shifts,” McQueen said.

Chalkbeat illustrator Sam Park contributed to this story.