Bumpy runway

Emails reveal months of missteps leading up to Tennessee’s disastrous online testing debut

PHOTO: Chalkbeat Photo Illustration

Tennessee education officials allowed students and teachers to go ahead with a new online testing system that had failed repeatedly in classrooms across the state, according to emails obtained by Chalkbeat.

After local districts spent millions of dollars on new computers, iPads, and upgraded internet service, teachers and students practiced for months taking the tests using MIST, an online testing system run by North Carolina-based test maker Measurement Inc.

They encountered myriad problems: Sometimes, the test questions took three minutes each to load, or wouldn’t load at all. At other times, the test wouldn’t work on iPads. And in some cases, the system even saved the wrong answers.

When students in McMinnville, a town southeast of Nashville, logged on to take their practice tests, they found some questions already filled in — incorrectly — and that they couldn’t change the answers. The unsettling implication: Even if students could take the exam, the scores would not reflect their skills.

“That is a HUGE issue to me,” Warren County High School assistant principal Penny Shockley wrote to Measurement Inc.

Tennessee Education Commissioner Candice McQueen speaks with reporters in February about technical problems with the state's new online assessment.
PHOTO: Grace Tatter
Tennessee Education Commissioner Candice McQueen speaks with reporters in February about technical problems with the state’s new online assessment.

The emails contain numerous alarming reports about practice tests gone awry. They also show that miscommunication between officials with the Tennessee Department of Education and Measurement Inc. made it difficult to fix problems in time for launch.

And they suggest that even as problems continued to emerge as the test date neared, state officials either failed to understand or downplayed the widespread nature of the problems to schools. As a result, district leaders who could have chosen to have students take the test on paper instead moved forward with the online system.

The messages span from October until Feb. 10, two days after the online test’s debut and cancellation hours later. Together, they offer a peek into how Tennessee wound up with a worst-case scenario: countless hours wasted by teachers and students preparing for tests that could not be taken.

October: ‘Frustration … is definitely peaking’

Leaders with the Education Department, local districts and Measurement Inc. all knew that Tennessee’s transition to online tests wouldn’t be easy. So the test maker and the department developed a plan to identify weaknesses: stress tests they called “Break MIST” to tax and troubleshoot the online system.

They all had a lot riding on a smooth rollout. Tennessee was counting on the scores to assess whether students are measuring up to new and more challenging standards, to evaluate teachers, and to decide which schools to close. Districts, even the most cash-strapped, had invested millions of dollars on new technology. And Measurement Inc., a small company headquartered in Durham, was looking to prove that it belonged in the multibillion-dollar testing industry’s top tier.

The first “Break MIST” day on Oct. 1 was a mess — as expected. Students in the eastern part of the state logged on without issue, but the system stumbled as the majority of students started their tests an hour later.

That morning, emails show that Measurement Inc. received 105 calls reporting problems. The company noted particular problems in districts using iPads. Officials from the testing company assured the state that the bugs could be fixed, and the education department passed the message on to the public.

Department officials said nearly 1.5 million practice tests were completed successfully over the course of the fall. But emails show that even on days that weren’t meant to tax the system, problems emerged.

On Oct. 20, students in some districts were taking practice tests when “everything quit,” according to a state official who summarized complaints that local technology coordinators were swapping by email.

“Not very reassuring,” wrote Randy Damewood, the IT coordinator in Coffee County.

“Not good news,” agreed John Payne, director of technology for Kingsport City Schools, who suggested that his own district’s tests were working that day.

“The frustration among teachers and central office staff is definitely peaking,” wrote Eric Brown, a state official.

But there was more frustration to come, much of it behind the scenes at the Education Department.

December to January: Communication falters

Even after Measurement Inc. and department officials worked together to address problems during practice tests, the department still wasn’t confident in the online system. They weren’t sure whether problems were due to local infrastructure or something bigger. Officials planned two more “Break MIST” days in January to find out.

But they didn’t involve Measurement Inc. in the planning, at least according to company officials who wrote to the department to say they learned of those plans only after being copied on an email sent to local superintendents by Education Commissioner Candice McQueen.

That message was one of many in which officials with the state or the testing company expressed frustration about communication in the weeks leading up to the testing period.

One tense exchange dealt with the problems faced by students taking practice tests on iPads. “Will the iPad platform be ready for primetime in the spring?” Assistant Commissioner Nakia Towns asked Measurement Inc. officials on Dec. 3. “I feel like we need to be honest on this one.”

The test maker did not email a response, and Towns raised the issue again a month later and indicated that she was still waiting for an answer. “I had asked the question very directly in December,” she wrote Measurement Inc. on Jan. 6. “We urgently need an update.”

It took five more days, until Jan. 11, for her to get an answer. A reply from a Measurement Inc. testing expert blamed the problem on Apple but suggested the company had a “workaround.”

The next day, 504 students in Dyer County, about 80 miles north of Memphis, attempted to take the exam, many of them using iPads. Not one was able to complete the test because questions took too long to load, according to a report from Measurement Inc.’s call center. (Another half-million tests were completed successfully during January, according to department officials.)

Henry Scherich
Henry Scherich

In an interview this week, McQueen told Chalkbeat that Measurement Inc. never fixed the iPad problem and that state officials called Apple themselves looking for a solution. She was still looking for an answer on Jan. 21, when she tried to speak directly with Measurement Inc. President Harry Scherich.

“She is wondering if there is any way for you to find even 15 minutes today for a call,” McQueen’s chief of staff wrote. “Commissioner will make herself available. We need to speak to someone who would be able to make a decision concerning technology in an effort to get communication to directors of schools today.”

Scherich, who was in Michigan meeting with that state’s education department, initially said he did not have time to speak with McQueen. (Measurement Inc. is one of two companies producing Michigan’s new exam.) Later that day, he agreed to speak.

McQueenEmail

McQueen said she and her team came to a conclusion the next day: The test wouldn’t work on iPads. They emailed and called districts that had purchased tablets for testing and recommended a switch to paper.

February: A last-minute warning gets too little attention

Even as tensions mounted and glitches piled up, both the department and Measurement Inc. projected confidence about what would happen on Feb. 8, when the test would go live for most Tennessee schools. State officials even invited reporters to Department of Education offices on Feb. 3 to say they were optimistic about the rollout.

But behind the scenes, they were preparing for the worst. McQueen asked the test maker’s call centers to prepare for a major outage, something a Measurement Inc. employee told her was “very unlikely.”

She also emailed districts telling them they should consider switching to paper tests if their students were waiting too long for questions to load. She gave them three days to decide.

Just 15 of Tennessee’s nearly 150 districts took her up on the offer, McQueen told Chalkbeat.

But emails show that the state knew that most districts were having difficulties. When one district’s technology coordinator asked the state for a list of districts ready for the online exam, officials came up short.

“I don’t think I can answer that with any confidence,” the department’s top technology officer wrote.

Five days later, on Monday, Feb. 8, the test officially began. Again, the system handled the first set of test takers but broke down when the rest of the state’s students logged on.

As students stopped being able to connect or saw their tests freeze, emails show that technology directors began frantically contacting each other.

“Has anyone else had MIST drop out on them?” the director from Houston County Schools asked. A chorus of technology directors from other districts replied in the affirmative.

Within hours, Tennessee had ended its foray into online testing. First, McQueen told districts to suspend the exams, then directed them to give up on the online platform altogether.

“We are not confident in the system’s ability to perform consistently,” she wrote in an email to school superintendents that afternoon.

McQueen told Chalkbeat that officials started the day “in good faith,” with an assumption that Measurement Inc. had resolved problems adequately. Scherich told Chalkbeat that he’s still unconvinced that the problems were the company’s fault. He suggested that Tennessee’s decision to cancel testing came too soon.

Either way, the department’s top technology official put it simply when he emailed McQueen on the day of the failure. “It appears that greater procedural and operational rigor could have prevented the network outage,” Cliff Lloyd wrote to McQueen.

The debacle was just what Ravi Gupta, the CEO of a Nashville-based charter school, was worried about when he pressed the state in January for more transparency about the status of the online platform.

“It would be a betrayal of our students’ hard work if adult technical failures stood in the way of their success,” Gupta wrote to McQueen.

In the end, that’s exactly what happened.

Clarification (June 28, 2016): This story has also been revised to clarify the impact of the department’s communications on district testing decisions. It has also been updated to include new information about successful practice tests.

good news bad news

Most Tennessee districts are showing academic growth, but districts with the farthest to go improved the least

PHOTO: Alan Petersime

It’s not just Memphis: Across Tennessee, districts with many struggling schools posted lower-than-expected growth scores on this year’s state exams, according to data released Tuesday.

The majority of Tennessee’s 147 districts did post scores that suggest students are making or exceeding expected progress, with over a third earning the top growth score.

But most students in three of the state’s four largest districts — in Memphis, Nashville and Chattanooga — aren’t growing academically as they should, and neither are those in most of their “priority schools” in the state’s bottom 5 percent.

The divide prompted Education Commissioner Candice McQueen to send a “good news, bad news” email to superintendents.

“These results point to the ability for all students to grow,” she wrote of the top-performing districts, many of which have a wide range of academic achievement and student demographics.

Of those in the bottom, she said the state would analyze the latest data to determine “critical next steps,” especially for priority schools, which also are located in high-poverty communities.

“My message to the leaders of Priority schools … is that this level of growth will never get kids back on track, so we have to double-down on what works – strong instruction and engagement, every day, with no excuses,” McQueen said.

Growth scores are supposed to take poverty into account, so the divide suggests that either the algorithm didn’t work as it’s supposed to or, in fact, little has happened to change conditions at the state’s lowest-performing schools, despite years of aggressive efforts in many places.

The results are bittersweet for Tennessee, which has pioneered growth measures for student learning and judging the effectiveness of its teachers and schools under its Tennessee Value-Added Assessment System, known as TVAAS.

On the one hand, the latest TVAAS data shows mostly stable growth through the transition to TNReady, the state’s new test aligned to new academic standards, in the first year of full testing for grades 3-11. On the other hand, Tennessee has invested tens of millions of dollars and years of reforms toward improving struggling schools — all part of its massive overhaul of K-12 education fueled by its 2009 federal Race to the Top award.

The state-run Achievement School District, which launched in the Race to the Top era to turn around the lowest-performing schools, saw a few bright spots, but almost two-thirds of schools in its charter-reliant portfolio scored in the bottom levels of student growth.

Shelby County’s own turnaround program, the Innovation Zone, fared poorly too, with a large percentage of its Memphis schools scoring 1 on a scale of 1 to 5, after years of scoring 4s and 5s.


District profile: Most Memphis schools score low on student growth


Superintendent Dorsey Hopson called the results a “wakeup call” for the state’s biggest district in Memphis.

“When you have a population of kids in high poverty that were already lagging behind on the old, much easier test, it’s not surprising that we’ve got a lot of work to do here,” he said, citing the need to support teachers in mastering the state’s new standards.

“The good part is that we’ve seen the test now and we know what’s expected. The bad part is we’ve seen the test … and it’s a different monster,” he told Chalkbeat.

You can find district composite scores below. (A TVAAS score of 3 represents average growth for a student in one school year.) For a school-by-school list, visit the state’s website.

exclusive

Most Memphis schools score low on student growth under new state test

PHOTO: Stephanie Snyder

More than half of Memphis schools received the lowest possible score for student growth on Tennessee’s new test last school year, according to data obtained by Chalkbeat for Shelby County Schools.

On a scale of 1 to 5, with 1 being the lowest measure, about 54 percent of the district’s 187 schools scored in the bottom rung of the Tennessee Value-Added Assessment System, known as TVAAS.

That includes most schools in the Innovation Zone, a reversal after years of showing high growth in the district’s prized turnaround program.

Charter schools fared poorly as well, as did schools that were deemed among the state’s fastest-improving in 2015.

Superintendent Dorsey Hopson called the scores a “huge wakeup call.”

“It shows that we’ve got a tremendous amount of work to do,” Hopson told Chalkbeat on Monday. “It’s going to be hard and it’s going to be frustrating. … It starts with making sure we’re supporting teachers around mastering the new standards.”

District leaders across Tennessee have been trying to wrap their heads around the latest growth scores since receiving the data in late August from the State Department of Education. Only two years earlier, the Memphis district garnered the highest possible overall growth score. But since then, the state has switched to a harder test called TNReady that is aligned for the first time to more rigorous academic standards.

TVAAS results are scheduled to be released publicly this week, but Chalkbeat obtained a copy being circulated within Shelby County Schools, Tennessee’s largest district.

The data is prompting questions from some Memphis educators — and assurances from state officials — over the validity of TVAAS, the state’s system for measuring learning and judging the effectiveness of its teachers and schools.

This is the first year of issuing district-wide TVAAS scores since 2015. That’s because of the state’s cancellation of 2016 testing for grades 3-8 due mostly to failures in the switch to online testing.

Some educators wonder whether the bumpy switch to TNReady is a factor in this year’s nosedive, along with changes in how the scores are calculated.

For example, data for fourth-graders is missing since there is no prior state testing in third grade for comparison. Elementary and middle schools also don’t have growth scores for social studies, since the 2017 questions were a trial run and the results don’t count toward a school’s score.

Hopson acknowledged concerns over how the state compares results from “two very different tests which clearly are apples and oranges,” but he added that the district won’t use that as an excuse.

“Notwithstanding those questions, it’s the system upon which we’re evaluated on and judged,” he said.

State officials stand by TVAAS. They say drops in proficiency rates resulting from a harder test have no impact on the ability of teachers, schools and districts to earn strong TVAAS scores, since all students are experiencing the same change.

“Because TVAAS always looks at relative growth from year to year, not absolute test scores, it can be stable through transitions,” said Sara Gast, a spokeswoman for the State Department of Education.

Shelby County Schools is not the only district with disappointing TVAAS results. In Chattanooga, Hamilton County Schools logged low growth scores. But Gast said that more districts earned average or high growth scores of 3, 4 or 5 last school year than happened in 2015.

Want to help us understand this issue? Send your observations to [email protected]

Below is a breakdown of Shelby County’s TVAAS scores. A link to a school-by-school list of scores is at the bottom of this story.

Districtwide

School-wide scores are a combination of growth in each tested subject: literacy, math, science and social studies.

Fifty three schools saw high growth in literacy, an area where Shelby County Schools has doubled down, especially in early grades. And 51 schools saw high growth in math.

Note: A TVAAS score of 3 represents average growth for a student in one school year. A score of 1 represents significantly lower academic growth compared to peers across the state.

2017

School-wide composite Number of schools Percent of schools
1 101 54%
2 19 10%
3 20 11%
4 10 5%
5 37 20%

2015

School-wide composite Number of schools Percent of schools
1 58 28%
2 16 8%
3 38 19%
4 18 9%
5 75 37%

Innovation Zone

Out of the 23 schools in the district’s program to turn around low-performing schools, most received a growth score of 1 in 2017. That stands in stark contrast to prior years since the program opened in 2012, when most schools were on a fast growth track.

School-wide composite Number of iZone schools
1 14
2 2
3 2
4 0
5 5

Reward schools

Nearly half of 32 schools deemed 2015 Tennessee reward schools for high growth saw a major drop in TVAAS scores in 2017:

  • Central High
  • Cherokee Elementary
  • Germanshire Elementary
  • KIPP Memphis Middle Academy
  • Kirby High
  • Memphis Business Academy Elementary
  • Power Center Academy High
  • Power Center Academy Middle
  • Ross Elementary
  • Sheffield High
  • South Park Elementary
  • Southwind High
  • Treadwell Middle
  • Westside Elementary

Charter schools

Charter schools authorized by Shelby County Schools fared similarly to district-run schools in growth scores, with nearly half receiving a TVAAS of 1 compared to 26 percent of charter schools receiving the same score in 2015.

2017

School-wide composite Number of iZone schools
1 18
2 6
3 7
4 2
5 7

2015

School-wide composite Number of iZone schools
1 10
2 2
3 7
4 3
5 16

Optional schools

Half of the the district’s optional schools, which are special studies schools that require students to test into its programs, received a 1 on TVAAS. That’s compared to just 19 percent in 2015.

2017

School-wide composite Number of iZone schools
1 23
2 6
3 5
4 2
5 10

2015

School-wide composite Number of iZone schools
2 5
3 6
4 5
5 14

You can sort through a full list of TVAAS scores for Shelby County Schools here.