What went down

‘There was no cyber attack,’ investigator says of Tennessee’s online testing shutdown

PHOTO: Manuel Breva Colmeiro/Getty Images

Questar’s unauthorized change of an online testing tool — not a possible cyber attack, as earlier reported by the company — was responsible for shutting down Tennessee’s computerized exams on their second day this spring, the state’s chief investigator reported Wednesday.

An independent probe determined that “there was no cyber attack,” nor was any student data compromised, when thousands of students could not log onto the online exam known as TNReady on April 17.

Instead, investigators said, Questar was mostly responsible for this year’s testing miscues. The main culprit was a combination of “bugs in the software” and the slowness of a computerized tool designed to let students turn text into speech if they need audible instructions.

Comptroller Justin P. Wilson reviewed early findings of his office’s internal review and the external investigation by a company hired by the Education Department during a legislative hearing in Nashville.

Education Commissioner Candice McQueen also told lawmakers that Tennessee is docking Questar about $2.5 million this year out of its $30 million contract because of the online problems that plagued many students and schools during the three-week testing window.

Payments being withheld are punitive, as well as to cover the state’s costs to address the problems, she said, adding that other discounts could follow.

Last week, McQueen announced that the state plans to launch a new search this fall for one or more testing companies to take over TNReady beginning in the 2019-20 school year. She said a track record of successful online testing is a must.

The text-to-speech tool worked fine last fall when a smaller number of high school students tested online. But the state said Questar made a “significant and unauthorized change” to that feature before the launch of spring testing that affects the vast majority of Tennessee students.  

“We now know this decision led to the severity of other issues we experienced during online testing,” the Education Department said in a statement.

House Speaker Beth Harwell and Rep. Jeremy Faison asked the comptroller to review the state’s contract with Questar, particularly related to reports of a possible cyber attack. Wilson’s office also looked into other technical snafus that disrupted student testing for days, prompting the legislature to pass emergency laws that make this year’s scores inconsequential.

“We believe that the student testing issues occurred primarily because of how Questar set the student assessment system up to work,” said Brent Rumbley, the comptroller’s information systems audit manager.

Education Commissioner Candice McQueen testifies during Wednesday’s hearing, where specialists in the state comptroller’s office also testified.

On the second day of exams, Rumbley said, those issues manifested themselves in a suspiciously high volume of internet traffic to the testing platform.

“That’s what led the Department of Education and Questar to believe that there may have been a cyber attack,” he told lawmakers. “This traffic eventually shut the system down.”

Even though Questar upgraded the processing capability of its equipment in response, students and educators continued to report problems logging in, staying online, and submitting tests until Questar turned off the text-to-speech tool beginning May 1.

The comptroller’s office also found that Questar was ill-prepared to handle the fallout from the technical glitches. For instance, the company struggled to manually recover the high number of tests that students couldn’t submit online. And school personnel calling the customer service line experienced wait times as long as 60 minutes, prompting many to just hang up.

New details emerged Wednesday about other testing problems, too.

On April 25, a Questar employee “inadvertently overrode” custom rosters statewide that allowed schools to match students with available testing devices. “As a result, teachers and test coordinators had to scramble to get students the tests they should take,” Rumbley said.

The next day, more problems erupted when an internet cable was severed by a dump truck in a traffic accident in Hawkins County.

“According to the vendor that manages the fiber optic line, 21 districts were without internet from approximately two to four hours,” said Rumbley, adding that neither Questar nor the department could have prevented the outage that day.

Lawmakers will get an expanded look at the Education Department and its testing program in November when Wilson’s office presents the results of a year-long performance audit, along with findings from a massive survey of Tennessee educators about TNReady.

The two-hour hearing gave lawmakers a platform to take jabs at McQueen and her department for their handling of testing.

Rep. Bo Mitchell admonished the Education Department for tweeting on the second day of testing that Questar “may have experienced a deliberate attack” that morning.

“This gets into the public trust and throwing out information to the public from the Department of Education that the failure was a hack … Whose decision was that to put that out into the public domain without any proof?” asked Mitchell, a Democrat from Nashville.

McQueen clarified that the department never used the word “hack,” but reported that the testing system was experiencing a “pattern of data that was consistent with a cyber attack.” The description was based on what was known as the time, she said.

Sen. Janice Bowling, a Republican from Tullahoma, said Questar’s $2.5 million penalty “seems like a smack on the wrist” given the disruption caused by the company’s mistakes.

McQueen responded that the state is withholding almost $11 million invoiced by Questar for online testing as it continues negotiations. She added that the state’s biggest testing expenses stem from printing and transit costs for paper materials used by about half of its students this year. The state is transitioning to computerized testing and has decided to slow the switch for a second time in the wake of this year’s challenges.

Justin P. Wilson

Questar officials told Chalkbeat last week that the company plans to pursue the state’s new contract next year, but Rep. Craig Fitzhugh told McQueen that he doesn’t want the Minnesota-based company involved after it completes its current contract.

“I don’t think we can let Questar get in the ballgame again,” said the Ripley Democrat.

The proposal will be competitively bid, said Wilson, adding that Questar’s past performance will be taken into account.

For more on how Tennessee got here, read why state lawmakers share blame, too, for TNReady testing headaches.

Testing ESSA

One state, three lists of troubled schools — another consequence of Tennessee’s testing mess

PHOTO: Sue Barr/Getty Images

After months of talks, federal and Tennessee education officials have come to terms on how to identify and address the state’s lowest-performing schools in light of last spring’s problem-plagued student assessment program.

Their agreement, which navigates conflicting state and federal laws over the use of test results, means the state Education Department will release three lists of challenged schools in the coming weeks.

One list — Tennessee’s highly anticipated roster of “priority schools,” which perform in the bottom 5 percent — will exclude scores from last school year’s beleaguered TNReady assessment. Issued every three years, this roster will serve as the basis for determining state interventions and supports for at least the next year.

Two other lists — both of which are new — will include those results in deciding which schools will receive additional federal funding.

The revised framework, which is designed to hold schools and districts accountable on student achievement, is more complex than initially planned for meeting the standards of the federal Every Student Succeeds Act.

Under the 2015 law, Tennessee is required to factor in its most recent standardized test results when it comes to school accountability. But technical problems that disrupted computerized exams last spring prompted the Legislature’s order to disregard 2017-18 TNReady scores when compiling Tennessee’s priority school list, the state’s highest-stakes vehicle for improving underperforming schools by making them eligible for interventions such as state takeover and charter conversion.

The dueling laws have required thoughtful discussions and additional footwork, according to Sara Gast, spokeswoman for the state Education Department.

“The U.S. Department of Education was clear in our conversations that the last-minute legislation did not change the federal requirement for states to use 2017-18 data in accountability determinations,” Gast told Chalkbeat last week.

As a result, to comply with the emergency state laws, the upcoming priority list will be based mostly on test scores from 2015-16 and 2016-17 — not last school year — when charting the state’s school improvement strategies, interventions, and investments.

To satisfy federal law, a new CSI list, which stands for Comprehensive Support and Improvement, will identify the state’s bottom 5 percent of schools based on test results from all three years. This roster will determine opportunities for additional federal funding through several school improvement grants.

State officials emphasize that the CSI list, which will number about 85 schools, will have significant overlap with the state’s priority school list.

The other new accountability list, called ATSI for Additional Targeted Support and Improvement, will be based solely on last school year’s TNReady data. This list will identify schools with the lowest performance across student groups such as black, Hispanic, or Native Americans, or those who are economically disadvantaged, English learners, or have disabilities. ATSI replaces, for now, the state’s previously planned “focus school” list under its original ESSA plan.

“There is no adverse action that comes from being on the ATSI or CSI list,” Gast said. “The only changes for these schools is they will now be eligible for additional federal funding to support their students, and the department will be available to provide guidance to support improvement.”

"I think one of the frustrations for everyone is that the federal DOE is less flexible when adjustments need to be made based on real-world situations in each state."Rep. Eddie Smith, R-Knoxville

The state began rolling out its new school accountability details a few weeks ago, catching some district leaders off guard with the news of additional lists to keep up with.

“This kind of came at us out of the blue,” said Paul Changas, who heads research and assessment for Metropolitan Nashville Public Schools. “Given the complexity of the new accountability system, the addition of new categories will likely make our training efforts even more challenging.”

Rep. Eddie Smith, who sponsored the emergency TNReady laws, acknowledged that multiple lists may cause some confusion. But the Knoxville Republican said it’s the best way to keep fallout from the state’s testing problems from disrupting the flow of federal resources to struggling schools.

“While we do not want schools punished, we do want to make sure that any support needed by these schools is received,” Smith said.

He added: “I think one of the frustrations for everyone is that the federal DOE is less flexible when adjustments need to be made based on real-world situations in each state.”

Tennessee’s amended ESSA accountability plan, which federal officials signed off on in mid-August, will be in effect for the next year and possibly for several years to come. Because state law now prevents 2017-18 TNReady results from being factored in to most state accountability work going forward, Tennessee will have to submit another ESSA amendment for 2018-19 and beyond.

Below, find details about the lists of schools being released this month by the state, including the reward list that recognizes schools achieving the highest performance or progress.

Tennessee School Accountability Designations

List Purpose Eligibility What will happen
Priority Schools Identify and improve schools struggling with overall student achievement, in compliance with state law Ranked in state’s bottom 5 percent on student achievement based on 2015-16 and 2016-17 tests and did not earn overall TVAAS growth score of 4 or 5 in both 2015-16 and 2016-17 or 2016-17 and 2017-18; or has a graduation rate of less than 67 percent for the 2017-18 school year. Schools previously on the 2015 priority list will remain on the intervention track assigned to them based on their 2016-17 performance, and no schools will be assigned to the Achievement School District based on 2017-18 data. All schools new to the 2018 list will be placed on the Delta track, which requires da local school improvement plan in collaboration with the state. Locally authorized charter schools on the list are subject to charter revocation and closure at the end of the 2018-19 school year. State-authorized charter schools making the 2018 and 2021 priority lists will close at the end of the 2021-22 school year.
CSI Schools (Comprehensive Support and Improvement) Identify and improve schools struggling with overall student achievement, in compliance with federal law Ranked in state’s bottom 5 percent on student achievement based on 2015-16, 2016-17, and 2017-18 tests and earned a score of 3 or less in 2016-17 and/or 2017-18 under the state’s new school rating system; or had a graduation rate of less than 67 percent for the 2017-18 school year. (All schools in the Achievement School District are designated for CSI.) Newly named CSI schools will work with the state’s Office of School Improvement and are eligible for additional funding through federal school improvement grants.
ATSI Schools (Additional Targeted Support and Improvement) Identify and improve schools struggling with student achievement among one or more student groups, in compliance with federal law An overall score of 1 or less under the state’s new school rating system and ranks in state’s bottom 5 percent for at least one student group (i.e., black, Hispanic, or Native American; economically disadvantaged; English learners; students with disabilities); or ranks in bottom 5 percent for two or more accountability subgroups or racial/ethnic groups. Newly named ATSI schools will work with the state’s Office of School Improvement and its Centers for Regional Excellence and will be eligible for additional federal support. (Priority and/or CSI schools are not eligible for ATSI.)
Reward Schools Identify and reward schools with the highest performance or extraordinary progress in student achievement An overall score of 3 or higher on the state’s new school rating system. Schools are not eligible if any student group performs in the state’s bottom 5 percent for that group. Special recognition

listening tour

Haslam will hit the road to troubleshoot Tennessee’s testing problems

PHOTO: Marta W. Aldrich
Flanked by 37 educators serving as Tennessee's new "TNReady ambassadors," Gov. Bill Haslam announces the kickoff of a statewide "listening tour" aimed at improving administration of the state's standardized assessment.

Gov. Bill Haslam said Tuesday he won’t pause state testing this school year and instead will launch a statewide “listening tour” aimed at fixing problems that have hampered Tennessee’s TNReady assessment in its first three years.

Responding to calls for a break in testing from school superintendents in Memphis and Nashville and from 18 state legislators, the Republican governor said he’s committed to getting TNReady right before he leaves office in January.

“Throwing in the towel on the policies instrumental to our progress should not be an option,” Haslam said during a news conference at the state Capitol.

Critics quickly countered that the listening tour is really just a road show with a predetermined outcome.

“We are in the middle of election season and the governor is in his final days. What more can he add to the education debate after eight years, that he hasn’t already tried?” wrote JC Bowman, executive director of the Professional Educators of Tennessee, in a column following the announcement.

Haslam acknowledged “significant problems” with TNReady, which this spring was marred by technical disruptions during a second attempt in three years at statewide computerized testing. But he added that now is not the time to point fingers.

“Without aligned assessments, we don’t know where our students stand and where we need to improve,” he said.

Declaring that they have “no confidence” in the test, Dorsey Hopson and Shawn Joseph — leaders of Shelby County Schools and Metro Nashville Public Schools, respectively — called earlier this month for a testing moratorium to let the next governor address the problems.

Now Haslam, who is term-limited after eight years in office, is trying to keep intact the linchpin of Tennessee’s blueprint for student improvement, which began under the administration of Democratic Gov. Phil Bredesen in the Race to the Top era. Haslam has stood by that sweeping overhaul — including a state test designed to measure how students are learning Tennessee’s new academic standards and to hold teachers accountable for the results. He believes passionately that the policies have led to Tennessee’s gains on national tests since 2011.

"I am committed to doing everything I can as governor before I leave to getting this right. "Gov. Bill Haslam

The listening tour will launch Friday in Knoxville, and will bring together teachers, testing and technology coordinators, and school administrators. Other stops are planned in Hamilton, Shelby, Williamson, Greene, and Gibson counties.

Haslam and his education chief, Candice McQueen, will attend the meetings, which will be facilitated by Wayne Miller, a long-time educator and retired director of the Tennessee Organization of School Superintendents.

The governor also named a three-member advisory team to help guide the feedback sessions and develop recommendations for him and his successor. Those advisers are Cicely Woodard of Franklin, the state’s current teacher of the year; Hamblen County teacher Derek Voiles, named the state’s top teacher in 2017; and Mike Winstead; director of Maryville City Schools and this year’s superintendent of the year.

The new president of the state’s largest teacher organization expressed cautious optimism on Wednesday about the tour, especially given Miller’s involvement.

“TEA has worked previously with Wayne Miller on many issues, and we believe he can bring a practitioner’s perspective to the dialogue,” said Beth Brown of the Tennessee Education Association, which has been an outspoken critic of TNReady. “I think transparency is a big issue. There’s some good things that can come out of this if we can talk honestly about letting parents and teachers have access to large portions of questions from the state test.”

The state already has conducted multiple surveys with educators about this year’s testing experience and recently named 37 teachers and test coordinators to serve as “TNReady ambassadors,” advising the state Education Department and its testing companies. McQueen also meets frequently with an educator-laden task force to confer about testing matters.

In addition, Tennessee is developing its request for proposals for one or more testing companies to take the reins from Questar, the state’s current vendor. That request is scheduled to go out late this year for testing administration that would begin in the fall of 2019.

Editor’s note: This story has been updated to include reaction from two teachers groups.