State's erasure analysis pilot included all tests taken last year

An effort to root out possible cheating that the State Education Department billed as stopgap actually included every single test that elementary and middle school students in New York took last year.

But the state is not yet saying how many red flags turned up when three million students’ answer sheets were scrutinized using a test security procedure known as “erasure analysis.”

For the last two years, under increasing pressure to show that the state’s test scores are meaningful, state education officials have asked the legislature for funding to carry out erasure analysis on students’ answer sheets. The process detects how often answers have been switched from wrong to right, a key barometer of cheating. Erasure analysis helped uncover cheating in other districts, including Atlanta, where 35 educators were recently indicted for their roles in a sweeping cheating scandal.

But in both years, legislators turned down the education officials’ requests. Last year, after legislators rejected a $1 million request to perform erasure analysis on 10 percent of tests, the officials said they would free up funds from their budget for limited testing.

“We’re still going through our process to determine how many and how much,” an education department spokesman said in June.

In the end, state officials said, the department was able to conduct erasure analysis on a far larger scale — and for far lower a price — than they had estimated. Instead of testing a fraction of exams, the department had every single math and reading test scanned for wrong-to-right erasures. Yet instead of spending $10 million, the price tag the department originally gave legislators for comprehensive erasure analysis, the costs were “minimal,” a department spokesman said.

“We were able to devise a more cost-effective process and, as a result, the actual pilot in the 2011-12 school year was appreciably [larger] than was originally proposed,” the spokesman, Jonathan Burman, said in an email.

This year, the department will have every high school Regents exam analyzed, Burman said. That’s on top of the three million elementary and middle school reading and math tests that students will take starting next week.

The analyses are done by 17 regional scanning centers around the state when they process and collect test score data sent to them by schools. The state instructed the centers to install erasure analysis software of their choice onto their scanning computers.

Burman said the state pays the scanning centers to crunch test results, which are used to assess students, teachers, schools, and districts. He said he could not say how much the state spent across the 17 centers, but the most expensive contract, with the Central New York Regional Information Center, is for $400,000. According to the state comptroller, that contract is to “create [an] early detection system to quickly and accurately detect test fraud incidents” and was approved last month.

The scanning centers are only able to detect erasures, and officials said they still want to hire outside vendors who can examine the tests more thoroughly. Additional analysis can show other kinds of suspicious data, including irregular spikes from one year to the next, and unlikely clusters of high scores in one class or one school.

The state has conducted erasure analysis in the past. From 2008 to 2011, the state pulled a sample of high school Regents exams and had them checked for erasures. Sixty-two schools were flagged for possible cheating — 48 in New York City — but just six schools were investigated. The renewed focus on the procedure is part of a broader expansion of test security efforts at the state level.

Burman said the state would ultimately release results from the 2012 erasure analysis to the public. Those results are “still being confirmed and analyzed” now, he said.

“We obviously don’t want to unfairly suggest impropriety on the part of any school, so we have to make certain that the results are valid and reliable before anything is released,” Burman said.