survey says

How did Tennessee testing go this year? An investigator is about to ask 64,000 educators

PHOTO: Ian Lishman/Getty Images
Questar's administration of Tennessee's TNReady exam, especially the online version, is being reviewed by the state comptroller.

Tennessee’s chief internal investigator is turning to educators for a review of the state’s troubled student testing system and the company overseeing the exam.

The state comptroller’s office will email separate surveys this week to public school teachers and superintendents. It also seeks to connect with testing coordinators for every school district, according to an email sent late last week to district superintendents.

“The results of all of these surveys are truly anonymous,” wrote Dan Willis, the comptroller’s assistant director for training and information systems.

Inquiries into TNReady and testing company Questar come as the state Department of Education winds down more than three weeks of standardized testing that’s been beset with problems, particularly malfunctions with the computerized version taken by high schoolers statewide and in middle grades in some districts.

The survey represents the first public action taken by the state’s chief auditor since a review was requested April 24 by Tennessee House Speaker Beth Harwell and Rep. Jeremy Faison, chairman of the House Government Operations Committee.

The comptroller’s office expects to collect the surveys through this month and present early results in late June to the Government Operations Committee, spokesman John Dunn said Monday.

In all, about 64,000 educators will be invited to share their TNReady experience, said Dunn, adding that the information will go directly to the comptroller’s office.

Dunn declined to share questions from the survey. But a request from House leaders offers some insights into the motivation behind the review.

Harwell and Faison asked if the state’s $30 million-a-year contract with Minnesota-based Questar includes provisions that would allow the state to recover money due to testing failures this spring. Their letter also asks if Questar is contractually required to protect all student testing data — a reference to the company’s report that an April 17 cyber attack disrupted Tennessee students taking TNReady online on the second day of testing. Officials with Questar and the state Department of Education maintain that no student information was compromised.

Tennessee’s two-year contract with Questar ends on Nov. 30, and the state either will have to stick with the firm or search for its third testing vendor in four years. Education Commissioner Candice McQueen fired Measurement Inc., the first overseer of the TNReady era, after technical problems ruined the test’s online rollout in 2016.

Leaders of the state’s largest teachers union said Monday that they welcome the chance for teachers to weigh in on testing challenges in 2018.

“We’ve been getting tons and tons of calls and emails with different anecdotes across the state of testing inconsistencies,” said Amanda Chaney, spokeswoman for the Tennessee Education Association.

Added TEA president Barbara Gray: “We have concerns about the reliability and the validity of the test scores after so many stops and starts, so maybe this survey will help answer some of our questions.”

TNReady testing continues online through Wednesday. As of the beginning of this week, more than 80 percent of computerized exams were complete, state officials reported.

The letter requesting the comptroller’s review is below:

Regrouping

After another bumpy testing year, Tennessee likely will slow its switch to online exams

PHOTO: Marta W. Aldrich
Members of Tennessee's testing task force listen to a presentation by Mary Batilwalla, deputy commissioner over assessment for Tennessee's Department of Education. The group offered feedback on options for transitioning to online testing after more problems occurred this year.

Tennessee education leaders are rethinking their timeline for adopting computerized testing after a parade of technical problems bedeviled students taking the state’s TNReady exam for a third straight year.

Most students are scheduled to test online next school year under a three-year transition plan. But since keyboard testing had significant challenges this year with half that number of students, Education Commissioner Candice McQueen is backing off from that timetable.

And while there’s disagreement over exactly how to move ahead, there’s consensus about one thing.

“We have a credibility issue,” said state Rep. John Forgety, “and we need to get it right one time.”

McQueen floated three options for the 2018-19 school year to members of her testing task force during its Wednesday meeting in Nashville:

  •     Returning to paper testing across all grades for one year;
  •     Computer testing for high school students; paper testing for grades 3-8;
  •     Computer testing for grade 6 through high school; paper testing for grades 3-5

Off the table, however, is the option that districts had this year to give computer tests to more grades than required by the state.

The state ordered that all high school students take the test by computer this year, but about 40 percent of districts also chose to go digital for at least some of their students in grades 5-8.

The early thinking had been that letting districts test more students than required would expedite Tennessee’s online switch if local leaders felt ready. But state officials now believe the piecemeal approach only complicated the process.

“We feel very strongly” about this decision, Deputy Education Commissioner Mary Batilwalla told the task force. “The complexity is really too great for us to overcome in ensuring that we have a seamless delivery.”

The 30-member task force of educators and advocates has been McQueen’s sounding board on TNReady and other testing issues, and she sought the group’s feedback one week after the state’s messy testing season ended.

“We don’t want to introduce any additional complexity. We want to eliminate complexity, eliminate risk,” said McQueen, who also is turning to superintendents and upcoming focus groups for advice about how to improve their TNReady experience.

PHOTO: Marta W. Aldrich
Education Commissioner Candice McQueen speaks at a 2017 event as Gov. Bill Haslam looks on.

McQueen will decide about digital vs. paper — and for which grades — by late June. She is leaning toward keeping high schools online and putting all lower grades on paper tests, but it’s not a done deal, she told Chalkbeat.

“The feedback we’re getting is for more to go online than not, and that’s very meaningful to hear,” she said.

Her boss, Gov. Bill Haslam, has made it clear that Tennessee is committed to eventually adopting computerized testing.

“It’s not just that’s where the world is going; that’s where the world is,” Haslam said earlier in the week.

About 300,000 students took TNReady online this year — the most ever since a wholesale switch to computers failed in 2016 under Measurement Inc. McQueen fired that testing company, hired Questar as its successor, and unveiled a new game plan to gradually wade back in. That approach worked well last year for the 24 districts that did a trial run for high schools, although later scoring errors detracted from Questar’s debut.

This year marked the return to statewide online assessments, beginning with Tennessee’s oldest students. But challenges included a cyber attack and lousy internet service when a dump truck cut a main fiber optic cable — examples that demonstrate the risks of computerized testing.

There are benefits, too, however. Digital exams are quicker to score, offer more flexibility in the types of questions asked, and ultimately cost less. Returning to all paper testing would cost an extra $11 million in printing and shipping costs.

One big advantage of paper-and-pencil testing is a shorter testing period. Three weeks were allotted to TNReady this spring because schools had to rotate their students in and out of testing labs to use a limited number of computers. That requires a lot of coordination and time.

Task force members agreed that reverting to paper would be a step backward, especially with the state’s focus on the technical skills needed for college and careers and the significant investments made by school districts to prepare for online testing.

But they were adamant that Tennessee needs a win next time around to rebuild trust in a test that many consider broken.

“There has been a serious erosion in confidence in state testing, whether it’s online or on paper,” said Shawn Kimble, director of schools in Lauderdale County. “If we fail again, where does that leave us as a state?”

Another pause

It’s official! Results from Tennessee’s ugly testing year won’t count for much of anything

PHOTO: Marta W. Aldrich
Gov. Bill Haslam and his education chief, Candice McQueen, speak with reporters Monday about how Tennessee will handle standardized test results this year because of technical problems administering the exams by computer.

Tennessee teachers and school districts can decide how to use this year’s standardized test results and won’t be penalized for low growth scores after another year of problems with the state’s computerized exam.

The state also will shift some responsibilities to a different testing company while deciding whether to extend Minnesota-based Questar’s contract to give the test past November, Education Commissioner Candice McQueen announced on Monday.

In addition, the state has hired two outside groups to sort through all that went wrong with TNReady during almost a month of testing that ended last week. One will look at the validity of the results to see if frequent online interruptions made the scores unreliable. The other will scrutinize Questar’s technology systems to determine why so much went awry.

The actions were announced as McQueen and Gov. Bill Haslam faced reporters together for the first time in the wake of this year’s sloppy return to statewide computerized testing.

After weeks of being on the defensive, Haslam’s administration sought to take control of the situation and emphasize that testing — done correctly — is critical to improving student achievement across the state.

I still have full confidence that testing is the right thing to do,” said the Republican governor, finishing the last year of an eight-year term. I’m frustrated like everybody else that we had issues with the online portion of this. But having said that, do I think the test is a good test? I do.”

Haslam also said Tennessee must forge ahead with computerized testing.

We’re one of only 10 states that has not already moved [completely] to online testing. And so it’s not just that’s where the world is going; that’s where the world is. And our students have to be prepared,” he said.

The decision to shield students, teachers, and schools from accountability for poor growth scores falls in line with emergency legislation passed by state lawmakers last month as reports of TNReady’s technical problems escalated. After weeks of studying the two new laws, McQueen and her team offered their first analysis of what the legislation means:

Teacher evaluations. The state still plans to include student growth scores in evaluations, but each teacher will have “complete control to nullify” that portion if they choose to rely solely on other measures, McQueen said.

Student grades. Local school boards will decide whether to incorporate TNReady scores into this year’s final grades. Many districts already have begun that process, and most are opting to exclude the results this year.

School ratings. Tennessee’s A-F rating system will not launch this fall as scheduled, although the state still will publish the achievement results that would have gone into them.

Priority schools. As planned, the state will release its “priority list” this fall of the 5 percent of lowest-performing schools, but this year’s test results will not be a factor. Instead, the list will be based on two years of previous scores for high schools and one year for lower-grade schools. “We will not be moving any schools based on that data into the Achievement School District,” McQueen said of the state-run turnaround program that takes over local schools and assigns them to charter operators.

Whether the adjustments put Tennesseee out of compliance with federal law remains to be seen, though. The 2015 Every Student Succeeds Act requires that student achievement — as measured by tests like TNReady — be part of each state’s plan for holding struggling schools accountable.

“We’ve been in lots of conversations with the U.S. Department of Education, and they are continuing to work with us on that,” McQueen acknowledged. “So what we will do is create something called a comprehensive support list, which is required under the Every Student Succeeds Act. That will include this year’s data, but there will not be any adverse action taken based on that comprehensive school list.

"It's important for all of us that we get this right. "Gov. Bill Haslam

After three years of testing problems, McQueen announced that Tennessee will create a “TNReady Ambassadors” program to improve customer service and will hire a full-time overseer to work with testing coordinators at the district level.

We did not have our expectations met in terms of customer service from Questar,” she said.

To that end, the state is reviewing its annual $30 million, two-year contract with Questar that expires on Nov. 30.

Haslam said changing companies in the middle of a school year wouldn’t be seamless because Questar will begin testing high school students on non-traditional block schedules this fall. There’s a little bit of a practical problem switching vendors right in the middle of that, so it’s part of negotiations we’re in the middle of,” he said.

McQueen praised Educational Testing Services, the New Jersey-based company that will take over the test’s design work while Questar focuses on test delivery. Also known as ETS, the vendor has had contracts with Tennessee since 2015 to create the state’s social studies and science tests, and to design many of its teacher certification exams. (ETS also owns Questar. Read the details here.)

It is a vendor that is well-known. It has a reputation for very high-quality work in terms of how they design tests,” she said.

Deeper dive: By getting testing wrong again, will Tennessee undo what it may be getting right?