Growing pains

Latest Colorado test results provide long-awaited glimpse at how students are growing academically

Students at Mrachek Middle School in Aurora work to solve a math problem. (Photo by Nicholas Garcia, Chalkbeat)

Newly released state test results measuring students’ academic growth show strong progress for Denver Public Schools in English, slow going for Aurora Public Schools in math and a potentially alarming achievement gap for students with disabilities statewide.

Unlike earlier math and English results that showed students’ proficiency in meeting academic standards, Colorado’s growth report measures how much students learn year-to-year compared to their academic peers. The information, which is the primary component of a school’s and district’s quality rating, is often heralded as providing a more complete picture of how much students, especially those who are less likely to be proficient on state tests, are faring in school.

Put simply, the growth numbers provide a picture of how students are progressing and how fast compared to their peers, not taking into account where they are proficiency-wise.

“If we can take our most struggling kids as far as they can go, and the highest (performing) kids who don’t get the attention they deserve as far as they can go, we’re succeeding,” said Leslie Nichols, superintendent of the tiny rural Hinsdale School District in southwest Colorado. Hinsdale students posted higher growth rates in English than any other school district in the state.

In 2009, Colorado began using the growth measure, which relies on results from the state’s English and math standardized tests, to supplement basic achievement data.

A student’s growth percentile, which ranges from 1 to 99, indicates how that student’s performance changed over time, relative to students with similar performances on state assessments. School and district growth rates, which make up the greatest share in their quality ratings, are determined by the median growth score from all students in that school or district.

Tuesday’s release marks the end of a multi-year transition from the state’s previous testing system, TCAP, to its current system that includes PARCC English and math tests.

Because of the transition to new tests, Colorado neither released growth data nor school quality ratings last year. While hiccups remain, state education officials say confidence is growing in the exams and the data they provide. But members of the State Board of Education last week signaled they were prepared to upend the entire system all over again.

“We’re glad to have a growth metric again,” said Alyssa Pearson, the state education department’s associate commissioner for accountability. “We believe it provides a really important dimension to understand the quality of a school to go along with achievement.”

State growth results

The state’s median growth percentile is always about 50. Groups of students, schools and districts that have a percentile score higher than 50 are on average learning at a faster rate than their peers. Conversely, a percentile score lower than 50 means on average students are learning at a slower rate than their peers.

Hitting the 50 mark represents about a year’s worth of academic growth.

Like under the old system, growth gaps exist between the state’s white students from middle-income households and their more at-risk peers. The gap between students with disabilities, those with individualized education plans, and their non-disabled peers was the largest on the state’s English and math tests. The gap between boys and girls also grew, state officials acknowledged, with girls learning at a faster rate.

Only English language learners demonstrated equal growth to their native-English speaking peers on the state’s English test.


While state education officials called attention to the yawning gap between students with disabilities and their peers without disabilities, officials were hesitant to prescribe cause.

“I don’t know if we can jump to conclusions yet,” Pearson said. “But it’s important to make it visible.”

Similar gaps did not exist with other student groups, including English learners and low-income students.

Growth results from Colorado’s 10 largest school districts were mostly in line with the state’s. Denver Public Schools posted the highest growth rate on the English tests, while the Adams 12 Five Star district posted the highest growth rate on math tests.

Aurora Public Schools posted the lowest growth rate on the state’s math tests. APS also tied the Douglas County School District for the lowest rate on the state’s English tests.


Like previously released achievement data measuring how well students are meeting academic expectations, state officials cautioned growth results had less reliability at schools with low participation rates.

Tracking growth in high schools was made more difficult by state lawmakers eliminating PARCC 10th and 11th grade tests last school year, leaving only 9th grade growth data.

To get a full picture of high school performance requires looking at PARCC in 9th grade, the PSAT in 10th grade, the ACT in 11th grade (and starting this academic year, the SAT) and some measure of postsecondary readiness for 12th graders, said Chris Gibbons, CEO of the Denver-based STRIVE Prep charter school network.

“It’s important our view of the performance of a high school – of any school – is informed by the entirety of the school and not just a single grade,” Gibbons said.

Tracking growth in math in higher grades also poses challenges — and in some circumstances, it’s impossible. Starting in the seventh grade, students may take any one of five math tests. Students who took math tests two grade levels higher than their actual grade level did not have growth results in math, said Pearson, of the education department.

‘Critical data’

Nichols, superintendent of the Hinsdale County School District in Lake City, has long been awaiting the state’s release of growth data.

“I’ve been holding my breath for this release of data,” she said, adding that her schools usually has too few students to publicly disclose achievement results.

Hinsdale posted the state’s highest median growth percentile on the English test. On average, the 32 students who took the state’s English test learned at a quicker rate than 82 percent of their academic peers.

Nichols immediately credited her teachers.

“Their expertise in writing and reading instruction is obviously shining through in these results,” she said.

Unlike some other rural superintendents who have been vocal critics of the PARCC tests, Nichols said she and her school district value the critical data the multi-state tests provide.

“I really need that connection to the larger world of education,” she said, adding another change in assessment would prove difficult for her small school district. “I could not do [standards and testing] by myself. I could not write my own. I get tired of everyone saying local is better all the time. It’s OK to measure my kids against something a little bigger.”

Find your school and district’s growth rate

Correction: An earlier version of this article incorrectly reported the Adams 12 Five Star district’s growth rate on the math tests. It is 55, making it the highest growth rate of the 10 largest school district’s in the state. An earlier version reported Cherry Creek had the highest rate.

What went down

‘There was no cyber attack,’ investigator says of Tennessee’s online testing shutdown

PHOTO: Manuel Breva Colmeiro/Getty Images

Questar’s unauthorized change of an online testing tool — not a possible cyber attack, as earlier reported by the company — was responsible for shutting down Tennessee’s computerized exams on their second day this spring, the state’s chief investigator reported Wednesday.

An independent probe determined that “there was no cyber attack,” nor was any student data compromised, when thousands of students could not log onto the online exam known as TNReady on April 17.

Instead, investigators said, Questar was mostly responsible for this year’s testing miscues. The main culprit was a combination of “bugs in the software” and the slowness of a computerized tool designed to let students turn text into speech if they need audible instructions.

Comptroller Justin P. Wilson reviewed early findings of his office’s internal review and the external investigation by a company hired by the Education Department during a legislative hearing in Nashville.

Education Commissioner Candice McQueen also told lawmakers that Tennessee is docking Questar about $2.5 million this year out of its $30 million contract because of the online problems that plagued many students and schools during the three-week testing window.

Payments being withheld are punitive, as well as to cover the state’s costs to address the problems, she said, adding that other discounts could follow.

Last week, McQueen announced that the state plans to launch a new search this fall for one or more testing companies to take over TNReady beginning in the 2019-20 school year. She said a track record of successful online testing is a must.

The text-to-speech tool worked fine last fall when a smaller number of high school students tested online. But the state said Questar made a “significant and unauthorized change” to that feature before the launch of spring testing that affects the vast majority of Tennessee students.  

“We now know this decision led to the severity of other issues we experienced during online testing,” the Education Department said in a statement.

House Speaker Beth Harwell and Rep. Jeremy Faison asked the comptroller to review the state’s contract with Questar, particularly related to reports of a possible cyber attack. Wilson’s office also looked into other technical snafus that disrupted student testing for days, prompting the legislature to pass emergency laws that make this year’s scores inconsequential.

“We believe that the student testing issues occurred primarily because of how Questar set the student assessment system up to work,” said Brent Rumbley, the comptroller’s information systems audit manager.

Education Commissioner Candice McQueen testifies during Wednesday’s hearing, where specialists in the state comptroller’s office also testified.

On the second day of exams, Rumbley said, those issues manifested themselves in a suspiciously high volume of internet traffic to the testing platform.

“That’s what led the Department of Education and Questar to believe that there may have been a cyber attack,” he told lawmakers. “This traffic eventually shut the system down.”

Even though Questar upgraded the processing capability of its equipment in response, students and educators continued to report problems logging in, staying online, and submitting tests until Questar turned off the text-to-speech tool beginning May 1.

The comptroller’s office also found that Questar was ill-prepared to handle the fallout from the technical glitches. For instance, the company struggled to manually recover the high number of tests that students couldn’t submit online. And school personnel calling the customer service line experienced wait times as long as 60 minutes, prompting many to just hang up.

New details emerged Wednesday about other testing problems, too.

On April 25, a Questar employee “inadvertently overrode” custom rosters statewide that allowed schools to match students with available testing devices. “As a result, teachers and test coordinators had to scramble to get students the tests they should take,” Rumbley said.

The next day, more problems erupted when an internet cable was accidentally severed by a dump truck in a traffic accident in Hawkins County.

“According to the vendor that manages the fiber optic line, 21 districts were without internet from approximately two to four hours,” said Rumbley, adding that neither Questar nor the department could have prevented the outage that day.

Lawmakers will get an expanded look at the Education Department and its testing program in November when Wilson’s office presents the results of a year-long performance audit, along with findings from a massive survey of Tennessee educators about TNReady.

The two-hour hearing gave lawmakers a platform to take jabs at McQueen and her department for their handling of testing.

Rep. Bo Mitchell admonished the Education Department for tweeting on the second day of testing that Questar “may have experienced a deliberate attack” that morning.

“This gets into the public trust and throwing out information to the public from the Department of Education that the failure was a hack … Whose decision was that to put that out into the public domain without any proof?” asked Mitchell, a Democrat from Nashville.

McQueen clarified that the department never used the word “hack,” but reported that the testing system was experiencing a “pattern of data that was consistent with a cyber attack.” The description was based on what was known as the time, she said.

Sen. Janice Bowling, a Republican from Tullahoma, said Questar’s $2.5 million penalty “seems like a smack on the wrist” given the disruption caused by the company’s mistakes.

McQueen responded that the state is withholding almost $11 million invoiced by Questar for online testing as it continues negotiations. She added that the state’s biggest testing expenses stem from printing and transit costs for paper materials used by about half of its students this year. The state is transitioning to computerized testing and has decided to slow the switch for a second time in the wake of this year’s challenges.

Justin P. Wilson

Questar officials told Chalkbeat last week that the company plans to pursue the state’s new contract next year, but Rep. Craig Fitzhugh told McQueen that he doesn’t want the Minnesota-based company involved after it completes its current contract.

“I don’t think we can let Questar get in the ballgame again,” said the Ripley Democrat.

The proposal will be competitively bid, said Wilson, adding that Questar’s past performance will be taken into account.

For more on how Tennessee got here, read why state lawmakers share blame, too, for TNReady testing headaches.

help wanted

Will third time be a charm? Tennessee searches again for online testing company

PHOTO: Marta W. Aldrich
Education Commissioner Candice McQueen answers questions Thursday at a news conference about changes to Tennessee's testing program. The changes were supported by Dale Lynch (right), executive director of the Tennessee Organization of School Superintendents.

After firing one testing company and hiring another in a pinch, Tennessee plans to launch a fresh search this fall for vendors — forging ahead with its switch to computerized exams, albeit more slowly than initially planned.

Education Commissioner Candice McQueen announced Thursday that the state will seek proposals from one or more companies to take over its troubled standardized testing program beginning in the 2019-20 school year. A track record of successful online testing is a must, she said.

Questar, which has handled the job the last two school years, will continue to oversee the state tests known as TNReady this year under an amended contract. Chief Operating Officer Brad Baumgartner said the company plans to pursue the new contract, too.

“We anticipate successful fall and spring administrations and hope to be afforded the opportunity to continue the momentum,” he told Chalkbeat.

McQueen said the state is ordering numerous changes next school year under Questar, including a modified timeline for transitioning from paper to computerized exams.

Instead of following the state’s initial game plan to have most students testing online next year, only high schoolers will stick with computers for their exams in 2018-19. All students in grades 3-8 — some of whom tested online this spring — will take their TNReady tests on paper.

The exception will be Tennessee’s new science test. Because that assessment is based on new academic standards and won’t count toward student grades or teacher evaluations in its first year, students in grades 5-8 will take it online, while grades 3-4 will test on paper. The idea is that the “field test” provides an opportunity for fifth-graders and up to gain online testing experience in a low-risk environment.

Even with technical problems hampering online testing two of the last three years, McQueen made it clear that computerized exams are the future for all Tennessee students if they want to keep pace with their peers nationally.

“Tennessee is one of less than 10 states who still have a paper test in our lower grade levels,” McQueen said during a news conference.

Local school leaders are equally committed to computerized testing, according to Dale Lynch, executive director of the Tennessee Organization of School Superintendents.   

“We do not want to go back to paper and pencil,” Lynch said. “Online testing is the way to go, but we need to get it right in Tennessee.”

"Online testing is the way to go, but we need to get it right in Tennessee."Dale Lynch, Tennessee Organization of School Superintendents

All of the changes are in response to the series of technical issues that frequently interrupted testing this year, exasperating students and teachers and prompting an emergency state law that rendered the scores mostly inconsequential for one year.

“Teachers, students and families deserve a testing process they can have confidence in, and we are doing everything possible to meet that responsibility,” McQueen said. “We are always committed to listening and improving, and we’ll continue to do just that.”

Questar is Tennessee’s second testing company since 2016, when the state entered the era of TNReady, a new assessment aligned to new academic standards and billed as harder to game. The switch to computerized testing was part of that package.

McQueen fired North Carolina-based Measurement Inc. after its online rollout failed on the first day of testing and led to the cancellation of most state exams that year. Questar, which had come in second for the contract, was brought on as an emergency vendor for $30 million a year. Questar’s two-year contract ends in November, but McQueen wants an extension in order to complete testing for the 2018-19 school year.

The search for a new vendor — or combination of vendors — could be tricky. Only about a half dozen companies can provide online testing for a state the size of Tennessee. That’s why the state Department of Education’s invitation for companies to submit proposals will be structured so that different vendors can bid on different pieces of the work.

“What we’ve learned over time is that there are few vendors who do all of those components well, but some vendors do some pieces of it much better than others,” McQueen said. “We’re going to look for those who have a track record of success online and who we think can manage our program well.”

The state already has taken a step toward that approach. Last month, McQueen announced that Educational Testing Service, also known as ETS, will take over this year’s TNReady design work, such as devising questions and exam instructions. The change will allow Questar to focus on giving and scoring the test and verifying and reporting the results. (ETS also owns Questar. Read more here.)

The legislature’s fiscal review committee recently approved that change, including $12.5 million to pay for ETS’ services, although state officials expect the extra money will be offset by re-negotiating down the cost of Questar’s current contract.