behind the scenes

What a day is like inside Pearson’s test scoring facility

Facing widespread backlash after years of controversies and testing glitches, one of the world’s largest testing companies is taking an unusual approach to quieting critics: It’s opening its doors.

Pearson, a British-based testing conglomerate that recently signed on for a two-year contract to aid in writing and administering Indiana’s ISTEP test, today invited Indianapolis reporters to its north side scoring facility in an effort to reveal how the company hand-scores questions on hundreds of thousands of student exams.

It’s part of a charm offensive from a company that depends on public contracts but is often at the center of public debate over testing in schools.

“We don’t see any reason why we shouldn’t pull back the curtain,” said Scott Overland, Pearson’s new director of media.

With parents and educators often skeptical about how exams are made and scored at a time when test scores can influence everything from teacher bonuses to whether schools are allowed to continue to operate, Pearson is hoping to allay fears by better explaining its work.

“We completely understand how this whole assessment process, the knowledge that parents and educators and the public has about that is limited,” Overland said as he led reporters through Pearson’s scoring facility on the third floor of a mid-sized office building.

The tour featured a short walk through the floor, which consisted of three large rooms and several small offices and conference rooms. Pearson executives and officials in charge of the scoring process explained how scorers, who must have a four-year college degrees, are recruited by reaching out to retired teachers, tutors and other educators. They receive about seven hours of in-person or online training and most learn to score just one open-ended question — including essays and math problems that require students to show their work to demonstrate that they understand the concept.

Multiple choice questions are graded by machine at Pearson’s main scoring facility in Iowa.

Pearson execs on the tour showed reporters how scorers log onto a computer platform where they see scanned images of student papers they must assess and grade. Their work is tested by “validity” questions that supervisors use to test their scorers for accuracy.

Scoring supervisors sit quietly at rows of tables in front of boxy computers in the scoring center. They’re in regular communication with the scorers themselves, who typically work from home across Indiana. Because scorers and supervisors don’t necessarily work regular business hours, many tables were sparsely filled Thursday morning.

Allison Tucker, a scoring supervisor for fourth-grade reading who’s been working with Pearson for more than 10 years, said one of her graders might do 70 questions in an hour. If a scorer gets off track and starts grading incorrectly, Tucker said that’s where the supervisors can step in.

“That’s one of the things that we really take seriously,” Tucker said. “So far it hasn’t been a problem for us.”

Few businesses in the education world are under as much day-to-day scrutiny as testing giants like Pearson, since just a handful of companies compete for lucrative state testing contracts and the chance to sell associated test prep materials for those exams.

Pearson is the largest education company in the world and a leader in standardized test development, having nabbed a contract for the multistate, Common Core-linked PARCC exam and one to handle the scoring for the National Assessment of Educational Progress (NAEP).

Yet it’s an industry frequently under fire when errors are discovered among millions of test questions or when problems arise with scoring or computer testing platforms. Every few weeks during standardized testing season, critics can seize on headlines reporting computer malfunctions or other testing disruptions.

Just yesterday, an employee error caused widespread test cancellation of New Jersey’s PARCC exam.

The problems aren’t limited to Pearson. Indiana’s 2015 ISTEP test, which was haunted by glitches and scoring delays was administered by California-based CTB, a Pearson competitor. CTB also ran into problems in 2013 when about 78,000 Indiana students taking the test on computers were interrupted over the course of several days — an error that forced CTB to pay $13 million in damages to the state.

Indiana then dumped CTB and hired Pearson last year with a more than $30 million contract to administer the 2016 and 2017 ISTEP exams, but the state is now looking to create yet another new exam for 2018.

The new exam will surely generate another a sought-after testing contract. So Pearson could be treating the ISTEP as something of an audition, trying to make a good impression in hopes of ongoing work.

“We recognize very much that this is critically important work we are doing,” said Melodie Jurgens, who oversees the general scoring process. “Our scorers are quite passionate, and they care a lot about how students do. They want to get it right because they know it’s important.”

Indiana is one of the first states where Pearson has invited reporters to tour its facilities, though earlier this week Overland said some national news outlets were given tours of the Iowa facility. The company hasn’t used such strategies in the past, he said, but plans to open up tours in other states going forward.

Granting this level of access to reporters isn’t a common move for testing companies, said Bob Schaeffer, spokesman for The National Center for Fair and Open Testing, an organization that acts as a testing watchdog. He said he’d been contacted by another reporter about a similar tour this past week but had never heard of this approach before.

But given the challenges Pearson has faced recently — including the loss of three major testing contracts in Florida, Texas and New York — it’s not necessarily a surprise.

“All the major testing companies have had computer testing failures,” Schaeffer said. “It shows an incredible pattern of technological failure that is more than the isolated glitch that they like to make it seem.”

Since Indiana switched to Pearson this year, things have gone relatively smoothly. The state officially started its second round of 2016 ISTEP tests this week, and few problems have been reported.

But Schaeffer said Indiana has “jumped from the frying pan into the incinerator” by making its test vendor switch.

“It’s a perverse game of musical chairs in which a state might reject a contract with a vendor for doing a bad job and hires a new vendor who has time available because they just got fired from another job,” Schaeffer said.

year two

Tennessee high schoolers post higher test scores, but some subjects remain a struggle

PHOTO: Marta W. Aldrich
Tennessee Education Commissioner Candice McQueen presents 2017 high school test scores to the State Board of Education.

High school students in Tennessee saw their state test scores rise in 2017, the second year that a new test aligned to the Common Core standards were given in the state.

The increases were modest on average, but sharp for some of the students who have historically struggled most. Just one in five poor students scored at the lowest-level on the ninth-grade English exam, for example, compared to one in three last year.

But in most courses, especially in math, students continued to fall far below the state’s expectations. Even as the state estimates that 11,000 more students met the English proficiency bar this year, two thirds of students still fell below it. And in two advanced math courses, scores actually declined slightly.

The upward trajectory across most subjects puts Tennessee in line with other states that have seen their scores plummet in the first year of new exams, but then rise incrementally afterwards as students and teachers adjust to tougher standards.

Education Commissioner Candice McQueen touted the results Thursday during a brief presentation to the State Board of Education in Nashville.

“These results are encouraging because they show that we’re on the right track,” McQueen said. “As we have moved our standards forward, our teachers and students are meeting those expectations.”

She singled out improvements with historically underserved groups, particularly students with disabilities, and a reduction in the percentage of students performing at the lowest achievement level.

“This positive movement is showing we are taking seriously the work we’re doing with all of our student groups,” McQueen said.

High schoolers scored best on their science exams, which was expected since Tennessee has not yet switched to more rigorous science standards. Those standards will reach classrooms in the fall of 2018.

The statewide scores are the first batch to be released. District- and school-level high school scores come next in August, while results for students in grades 3-8 are due out this fall. Grades 3-8 took TNReady for the first time last school year after their 2016 exams were scuttled amid technical failures.

 

previewing TNReady

Why Tennessee’s high school test scores, out this week, matter more — and less — than usual

PHOTO: Nic Garcia

When scores dropped last year for most Tennessee high school students under a new state test, leaders spoke of “setting a new baseline” under a harder assessment aligned to more rigorous standards.

This week, Tennesseans will see if last year’s scores — in which nearly three-quarters of high schoolers performed below grade level — was in fact just a reset moment.

Education Commissioner Candice McQueen has scheduled a press conference for Thursday morning to release the highly anticipated second year of high school scores under TNReady, which replaced the state’s TCAP tests in 2015-16. (Students in grades 3-8 will get TNReady scores for the first time this fall; last year, their tests were canceled because of a series of testing failures.)

Here’s what you need to know about this week’s data dump, which will focus on statewide scores.

1. Last year’s low scores weren’t a big surprise.

Not only was it the first time Tennessee students took TNReady, it also was the first time that they were being tested on new academic standards in math and language arts known as the Common Core, which reached Tennessee classrooms in 2012.

Other states that switched to Common Core-aligned exams also saw their scores plummet. In New York, for example, the proportion of students who scored proficient or higher in reading dropped precipitously in 2013 during the first year of a new test for grades 3-8.

McQueen sought last year to prepare Tennessee for the same experience. After all, she said, the state was moving away from a multiple-choice test to one that challenges students’ higher-order thinking skills. Plus, while Tennessee students had been posting strong scores on the state’s own exam, they had struggled on national tests such as the ACT, raising questions about whether the previous state test was a good measure of students’ skills.

“We expected scores to be lower in the first year of a more rigorous assessment,” McQueen said after only 21 percent of high school students scored on or above grade level in math, while 30 percent tested ready in English and reading.

2. It’s expected that this year’s scores will rise … and it will be a bad sign if they don’t.

Over and over, state officials assured Tennesseans that 2016 was just the start.

“[We] expect that scores will rebound over time as all students grow to meet these higher expectations — just as we have seen in the past,” McQueen said.

She was referring to the state’s shift to Diploma Standards in 2009, when passing rates on end-of-course tests dropped by almost half. But in subsequent years, those scores rose steadily in a “sawtooth pattern” that has been documented over and over when states adopt new assessments and students and teachers grow accustomed to them.

That includes New York, where after the worrisome results in 2013, the percentage of students passing started inching up the following year, especially in math.

In Tennessee, this year’s high school scores will provide the first significant data point in establishing whether the state is on the same track. Higher scores would put the state on an upward trajectory, and suggest that students are increasingly proficient in the skills that the test is measuring. Scores that remain flat or go down would raise questions about whether teachers and students are adjusting to more rigorous standards.

3. There’s lots more scores to come.

This week’s statewide high school scores will kick off a cascade of other TNReady results that will be released in the weeks and months ahead.

Next comes district- and school-level high school scores, which will be shared first with school systems before being released to the public. That’s likely to happen in August.

In the fall, Tennessee will release its scores for students in grades 3-8, who took TNReady for the first time this year after the 2016 testing debacle. While testing went better this year, the state’s new testing company needed extra time to score the exams, because additional work goes into setting “cut scores” each time a new test is given.

A group of educators just concluded the process of reviewing the test data to recommend what scores should fall into the state’s four new categories for measuring performance: below grade level, approaching grade level, on grade level, or mastered. The State Board of Education will review and vote on those recommendations next month.

4. This year’s scores are lower stakes than usual, but that probably won’t last.

For years, Tennessee has been a leader in using test scores to judge students, teachers, and schools. Like most states, it uses the data to determine which schools are so low-performing that they should be closed or otherwise overhauled. It also crunches scores through a complicated “value-added” algorithm designed to assess how much learning that teachers contribute to their students — an approach that it has mostly stuck with as value-added measures have fallen out of favor across the nation. And unusually, the state exam scores are also supposed to factor into final student grades, this year counting for 10 percent.

But the rocky road to the new tests has temporarily diminished how much the scores count. Because preliminary scores arrived late this spring, most districts opted to grade students on the basis of their schoolwork alone.

And because of the testing transition, the scores won’t be given as much weight in this year’s teacher evaluations — an adjustment that lawmakers made to alleviate anxiety about the changes. Test scores will contribute only 10 percent to teachers’ ratings. Depending on the subject, that proportion is supposed to rise to between 15 and 25 percent by 2018-19.