Bumpy runway

Emails reveal months of missteps leading up to Tennessee’s disastrous online testing debut

PHOTO: Chalkbeat Photo Illustration

Tennessee education officials allowed students and teachers to go ahead with a new online testing system that had failed repeatedly in classrooms across the state, according to emails obtained by Chalkbeat.

After local districts spent millions of dollars on new computers, iPads, and upgraded internet service, teachers and students practiced for months taking the tests using MIST, an online testing system run by North Carolina-based test maker Measurement Inc.

They encountered myriad problems: Sometimes, the test questions took three minutes each to load, or wouldn’t load at all. At other times, the test wouldn’t work on iPads. And in some cases, the system even saved the wrong answers.

When students in McMinnville, a town southeast of Nashville, logged on to take their practice tests, they found some questions already filled in — incorrectly — and that they couldn’t change the answers. The unsettling implication: Even if students could take the exam, the scores would not reflect their skills.

“That is a HUGE issue to me,” Warren County High School assistant principal Penny Shockley wrote to Measurement Inc.

Tennessee Education Commissioner Candice McQueen speaks with reporters in February about technical problems with the state's new online assessment.
PHOTO: Grace Tatter
Tennessee Education Commissioner Candice McQueen speaks with reporters in February about technical problems with the state’s new online assessment.

The emails contain numerous alarming reports about practice tests gone awry. They also show that miscommunication between officials with the Tennessee Department of Education and Measurement Inc. made it difficult to fix problems in time for launch.

And they suggest that even as problems continued to emerge as the test date neared, state officials either failed to understand or downplayed the widespread nature of the problems to schools. As a result, district leaders who could have chosen to have students take the test on paper instead moved forward with the online system.

The messages span from October until Feb. 10, two days after the online test’s debut and cancellation hours later. Together, they offer a peek into how Tennessee wound up with a worst-case scenario: countless hours wasted by teachers and students preparing for tests that could not be taken.

October: ‘Frustration … is definitely peaking’

Leaders with the Education Department, local districts and Measurement Inc. all knew that Tennessee’s transition to online tests wouldn’t be easy. So the test maker and the department developed a plan to identify weaknesses: stress tests they called “Break MIST” to tax and troubleshoot the online system.

They all had a lot riding on a smooth rollout. Tennessee was counting on the scores to assess whether students are measuring up to new and more challenging standards, to evaluate teachers, and to decide which schools to close. Districts, even the most cash-strapped, had invested millions of dollars on new technology. And Measurement Inc., a small company headquartered in Durham, was looking to prove that it belonged in the multibillion-dollar testing industry’s top tier.

The first “Break MIST” day on Oct. 1 was a mess — as expected. Students in the eastern part of the state logged on without issue, but the system stumbled as the majority of students started their tests an hour later.

That morning, emails show that Measurement Inc. received 105 calls reporting problems. The company noted particular problems in districts using iPads. Officials from the testing company assured the state that the bugs could be fixed, and the education department passed the message on to the public.

Department officials said nearly 1.5 million practice tests were completed successfully over the course of the fall. But emails show that even on days that weren’t meant to tax the system, problems emerged.

On Oct. 20, students in some districts were taking practice tests when “everything quit,” according to a state official who summarized complaints that local technology coordinators were swapping by email.

“Not very reassuring,” wrote Randy Damewood, the IT coordinator in Coffee County.

“Not good news,” agreed John Payne, director of technology for Kingsport City Schools, who suggested that his own district’s tests were working that day.

“The frustration among teachers and central office staff is definitely peaking,” wrote Eric Brown, a state official.

But there was more frustration to come, much of it behind the scenes at the Education Department.

December to January: Communication falters

Even after Measurement Inc. and department officials worked together to address problems during practice tests, the department still wasn’t confident in the online system. They weren’t sure whether problems were due to local infrastructure or something bigger. Officials planned two more “Break MIST” days in January to find out.

But they didn’t involve Measurement Inc. in the planning, at least according to company officials who wrote to the department to say they learned of those plans only after being copied on an email sent to local superintendents by Education Commissioner Candice McQueen.

That message was one of many in which officials with the state or the testing company expressed frustration about communication in the weeks leading up to the testing period.

One tense exchange dealt with the problems faced by students taking practice tests on iPads. “Will the iPad platform be ready for primetime in the spring?” Assistant Commissioner Nakia Towns asked Measurement Inc. officials on Dec. 3. “I feel like we need to be honest on this one.”

The test maker did not email a response, and Towns raised the issue again a month later and indicated that she was still waiting for an answer. “I had asked the question very directly in December,” she wrote Measurement Inc. on Jan. 6. “We urgently need an update.”

It took five more days, until Jan. 11, for her to get an answer. A reply from a Measurement Inc. testing expert blamed the problem on Apple but suggested the company had a “workaround.”

The next day, 504 students in Dyer County, about 80 miles north of Memphis, attempted to take the exam, many of them using iPads. Not one was able to complete the test because questions took too long to load, according to a report from Measurement Inc.’s call center. (Another half-million tests were completed successfully during January, according to department officials.)

Henry Scherich
Henry Scherich

In an interview this week, McQueen told Chalkbeat that Measurement Inc. never fixed the iPad problem and that state officials called Apple themselves looking for a solution. She was still looking for an answer on Jan. 21, when she tried to speak directly with Measurement Inc. President Harry Scherich.

“She is wondering if there is any way for you to find even 15 minutes today for a call,” McQueen’s chief of staff wrote. “Commissioner will make herself available. We need to speak to someone who would be able to make a decision concerning technology in an effort to get communication to directors of schools today.”

Scherich, who was in Michigan meeting with that state’s education department, initially said he did not have time to speak with McQueen. (Measurement Inc. is one of two companies producing Michigan’s new exam.) Later that day, he agreed to speak.

McQueenEmail

McQueen said she and her team came to a conclusion the next day: The test wouldn’t work on iPads. They emailed and called districts that had purchased tablets for testing and recommended a switch to paper.

February: A last-minute warning gets too little attention

Even as tensions mounted and glitches piled up, both the department and Measurement Inc. projected confidence about what would happen on Feb. 8, when the test would go live for most Tennessee schools. State officials even invited reporters to Department of Education offices on Feb. 3 to say they were optimistic about the rollout.

But behind the scenes, they were preparing for the worst. McQueen asked the test maker’s call centers to prepare for a major outage, something a Measurement Inc. employee told her was “very unlikely.”

She also emailed districts telling them they should consider switching to paper tests if their students were waiting too long for questions to load. She gave them three days to decide.

Just 15 of Tennessee’s nearly 150 districts took her up on the offer, McQueen told Chalkbeat.

But emails show that the state knew that most districts were having difficulties. When one district’s technology coordinator asked the state for a list of districts ready for the online exam, officials came up short.

“I don’t think I can answer that with any confidence,” the department’s top technology officer wrote.

Five days later, on Monday, Feb. 8, the test officially began. Again, the system handled the first set of test takers but broke down when the rest of the state’s students logged on.

As students stopped being able to connect or saw their tests freeze, emails show that technology directors began frantically contacting each other.

“Has anyone else had MIST drop out on them?” the director from Houston County Schools asked. A chorus of technology directors from other districts replied in the affirmative.

Within hours, Tennessee had ended its foray into online testing. First, McQueen told districts to suspend the exams, then directed them to give up on the online platform altogether.

“We are not confident in the system’s ability to perform consistently,” she wrote in an email to school superintendents that afternoon.

McQueen told Chalkbeat that officials started the day “in good faith,” with an assumption that Measurement Inc. had resolved problems adequately. Scherich told Chalkbeat that he’s still unconvinced that the problems were the company’s fault. He suggested that Tennessee’s decision to cancel testing came too soon.

Either way, the department’s top technology official put it simply when he emailed McQueen on the day of the failure. “It appears that greater procedural and operational rigor could have prevented the network outage,” Cliff Lloyd wrote to McQueen.

The debacle was just what Ravi Gupta, the CEO of a Nashville-based charter school, was worried about when he pressed the state in January for more transparency about the status of the online platform.

“It would be a betrayal of our students’ hard work if adult technical failures stood in the way of their success,” Gupta wrote to McQueen.

In the end, that’s exactly what happened.

Clarification (June 28, 2016): This story has also been revised to clarify the impact of the department’s communications on district testing decisions. It has also been updated to include new information about successful practice tests.

Reading revisited

McQueen ends her Tennessee tenure the same way she started — focused on reading

Tennessee Education Commissioner Candice McQueen reads to students during one of her classroom tours. (Photo courtesy of Tennessee Department of Education)

When then-newly appointed Education Commissioner Candice McQueen began touring Tennessee schools in 2015, she was “ashamed” of the dearth of strong reading materials available for many students and their teachers.

“Depending on what districts and classrooms you were in, some people had resources and curriculum and some did not,” recalls McQueen, a former classroom teacher and university dean of education.

The shortcoming was just one of several that helped explain Tennessee’s stagnant reading scores and why only one in three students was considered proficient in reading, based on national tests.

There also was a gap in how teachers or teacher candidates were being equipped to teach reading, a lack of attention to fostering reading skills in students’ early years, and little to no public education programming to address “summer slide,” the tendency for especially low-income students to regress in academic skills during their summer break from school.

McQueen has sought to address all of those weaknesses through various investments and supports under Read to be Ready, which was her first sweeping initiative under Gov. Bill Haslam.

Now, as she winds down her four-year tenure this month, the outgoing commissioner considers that work — launched in 2016 with the support of Haslam and his wife, Chrissy — among her most important legacies as education chief.

Last week, as a fitting bookend to her statewide leadership before starting her new job as CEO of a national education organization, McQueen put reading front and center during three days of regional gatherings of teachers and literacy coaches in Memphis, Nashville, and Knoxville.

“We’re just now beginning to see progress on TNReady,” she said of last year’s reading gains in grades 3-5 on the state’s standardized test.

“It’s progress we’re proud of, even though it’s not as much as we want,” she added.

Indeed, the climb ahead is steep, despite this year’s 2.3 percent increase to almost 37 percent of third-graders reading on or above grade level. To reach Tennessee’s lofty goal of 75 percent by 2025, the state will have to move 5 to 6 percent more third-graders to proficiency every year.

McQueen says reaching the goal is “absolutely doable” and cites the groundwork laid through Read to be Ready. Since 2016, Tennessee has launched a statewide coaching network for elementary reading teachers, offered new training for educators, and made investments in better resources for students. There are also new standards and expectations in teacher training and summer reading camps for first- through third-graders who are furthest behind.

McQueen is especially encouraged by summer camps that have shown statistically significant reading improvements for participating students during the past two years. She recently announced $8.9 million in state grants to 218 public schools to host even more camps next summer.

PHOTO: TDOE
Children participate in a 2016 summer reading program in Lauderdale County in West Tennessee as part of the new grant-based literacy program overseen by the Tennessee Department of Education.

As for the lack of high-quality textbooks and materials she first encountered in 2015, the state has identified texts that align with Tennessee’s new academic standards, and McQueen is urging districts to plan now to budget more for them.

“We’re building in this idea that you don’t just adopt; you purchase,” she told Chalkbeat. “Sometimes we see adoption where you have a set that all teachers are sharing. We feel like every teacher needs their own sets of books, their own curriculums, so they can adequately support all their students.”

Recognizing that strong reading skills are the foundation for learning and success in all subject areas, most Tennessee’s districts have embraced some or all parts of Read to be Ready. It’s popular as well with teachers, who say they like having both guidance and flexibility to help their students learn to decode letters and words, expand vocabularies, and deepen comprehension skills.

“This makes concrete resources available, but we’re also empowered to use our own teacher resources,” said Emily Townsend, who teaches kindergarteners in Coffee County.

Others are concerned that the focus on young children is coming at the expense of struggling middle and high school readers. “These are not throwaway kids,” said Stephanie Love, a board member for Shelby County Schools.

Love said the effects of poverty are also at play and require a deeper look at illiteracy in large cities like Memphis.

“I don’t think we need more initiatives; I think we need to reevaluate and see what’s preventing so many of our students from reading well,” said Love, a proponent of more state funding for schools. “Do they need glasses? Are they dyslexic? Did they not attend a pre-K or Head Start program?”

McQueen agrees that illiteracy is a “true equity issue.”

“Reading skills are a predictor of so many things across a lifetime,” she said of navigating school and jobs and avoiding crime and poverty. “We know that if you’re not reading proficiently by the third grade, you can still catch up, but it gets harder over time. Our passion for this work comes from what we know happens when kids are not reading.”

more money more learning

Does money matter for schools? Why one researcher says the question is ‘essentially settled’

PHOTO: AAron Ontiveroz/The Denver Post
Educators wearing red and holding signs rally for more education funding at the Colorado Capitol on April 26, 2018.

“Throwing money at the problem” has long gotten a bad rap in education.

“The notion that spending more money is going to bring about different results is ill-placed and ill-advised,” U.S. Secretary of Education Betsy DeVos said last year.

But a string of recent studies have undermined that perspective. Now, a new review of research drives another nail into the argument’s coffin.

The review looks closely at 13 studies focused on schools nationwide or in multiple states. Twelve found that spending more money meant statistically significant benefits for students, including rising test scores and high school graduation rates.

“By and large, the question of whether money matters is essentially settled,” Northwestern economist Kirabo Jackson concludes. “Researchers should now focus on understanding what kinds of spending increases matter the most.”

In the paper, which was released Monday through the National Bureau of Economic Research and has not been peer-reviewed, Jackson looks at attempts to pin down the effects of school spending. This is critical, because policymakers like DeVos often focus on correlations between spending and test scores.

The results of the 13 studies are remarkably consistent, even though they span different time periods.

For instance, students saw big gains in school districts where spending jumped between 1972 and 1990, one study found. A 10 percent increase in spending across a student’s 12 years in public school led students to complete an additional one-third of a year of school and boosted their adult wages by 7 percent. The gains were largest for low-income kids.

Studies of more recent changes tell a similarly encouraging story. States that increased school funding between 1990 and 2011 saw substantial gains on federal exams soon after, another analysis found.

A separate paper found that 12 percent increases in school spending boosted graduation rates by several percentage points

And another study found that cutting funding in the wake of the Great Recession hurt student test scores and graduation rates.

Jackson identifies just one national paper without clear positive effects.

“Money used wisely clearly matters,” said Lori Taylor, a Texas A&M school finance researcher  who praised Jackson’s study. “One of the takeaways from this newer literature might be that schools are more wise than we thought.”

Studies looking at single states have also found largely encouraging results. One recent study in New York took advantage of a quirk in the state’s funding formula that allowed certain districts with falling enrollment to get extra funding. Those extra dollars led to higher scores on state exams, it found.

Another New York study found that a 2 to 3 percent increase in funding led to a 0.5 to 0.8 percentage point decline in the high school dropout rate.

Head over to Ohio, and the results look similar: passing a funding ballot measure caused a boost in test scores. Three separate papers in Michigan, as well as a study in Massachusetts, found positive results, too. And Jackson’s overview may actually understate the evidence, as it does not include recent research in California and Texas, which also found gains from additional funding.

The only state study that showed unrestricted funding increases did not result in any improvements was a 2003 paper looking at Kentucky.

The pattern is consistent with other recent research overviews, but it’s a sharp departure from an older one by Eric Hanushek, a Stanford economist who has frequently testified on behalf of states defending against lawsuits aimed at increasing school funding. His 1997 review looked at studies conducted before 1995, and found that only 27 percent of the results showed statistically gains from additional school spending.

Jackson argues that Hanushek’s review — which was vigorously challenged even at the time — is dated and relies on studies with crude methodologies.

Hanushek concedes that, but says his view on the matter is largely unchanged. The gains shown in the studies in Jackson’s paper differ in size, he said. And he noted a similar correlation to ones that DeVos cites: as spending has increased over the past several decades, scores on 12th grade federal tests have remained largely stagnant.

“The variation in the results that you get indicate quite clearly if I want to fix [a school district] and I just drop money on them, they may or may not get better,” Hanushek said. “It’s how the money is spent more than how much.”

Still, even Hanushek acknowledges there is a case for spending more money in schools.

“I think we’re underinvesting in education in the U.S. and I think it’s pretty serious,” he said. “But I don’t want to just do what we’ve done in the past and hope for something different.”

Jackson’s results are a bit murkier when examining state spending that is earmarked for specific uses. School construction spending, for example, led to gains in some cases but no clear effects in others. A trio of New York City studies found that federal Title I funds targeted at disadvantaged students did not have clear positive effects.

Jackson’s paper also does not review research on spending increases to pay for smaller class sizes, teacher salary increases, tutoring programs, or school turnaround efforts. A number of turnaround initiatives with big price tags have yielded disappointing results.

On balance, Taylor of Texas A&M says that the research points in a clear direction — though it still may not persuade skeptics.

“There were some circles that never bought the premise that money doesn’t matter,” she said. “There are other circles that will never accept the premise that money does matter.”