number crunching

Some analysis left undone in data-driven education department

P.S. 199 in the South Bronx, one of the city's top-rated elementary schools in recent years. A high rate of its former students' test scores plummeted once they moved onto middle school.

The Department of Education crunches state test scores in dozens of ways to measure the performance of schools, principals, teachers, and students. But it does not perform a statistical analysis that can reveal whether an elementary school’s graduates have received test scores that far outstrip their actual skills.

Researchers say it would be relatively easy for the department to calculate “swing rates” to find the proportion of students from each school whose scores rise or fall by a statistically unlikely margin when they move to another school. Such an analysis could take some of the burden off of individual educators to report suspicions of cheating.

The city used to conduct swing rate analysis prior to the Bloomberg administration, according to a former testing official, and the state is poised to launch the measure as part of an overhaul of its own approach to test security.

But department officials say the analysis would be too onerous. They also say that they never launch investigations into cheating based on data anomalies alone. Instead, they say they will dispatch investigators only when they receive formal allegations of test improprieties.

The policy means that some top-rated schools whose students’ scores plummet at far higher than the average rate never have their testing practices scrutinized.

For all of the criticism of state tests as being arbitrary and imperfect measures of student performance, they are remarkably stable. In 2011, students who saw their scores fall by more than two standard deviations from the previous year made up just 0.6 percent of the sixth grade test-taking population in English, and 0.4 percent in math. That degree of decline is highly improbable under normal circumstances and is more likely to reflect externalities than real changes in academic proficiency.If one student’s test score plummets to that degree, it might be reasonable to conclude that he had a bad day when the second test was administered. But if an entire cohort of students see their scores plummet, it could be that testing conditions were especially favorable in the first year, maybe illicitly.

Some city schools have posted swing rates many times the 0.6 percent average, according to the New York Times. At the two elementary schools with the highest scores on the city’s 2011 progress reports, P.S. 257 and P.S. 31 in Brooklyn, the swing rates that year were 9 percent and 1.8 percent, respectively.

Neither school raised eyebrows at the department until a whistleblower at a middle school that received their students registered an official complaint about wide discrepancies between the students’ test scores and their actual skills. The allegation triggered investigations at both schools.

But the Brooklyn schools’ swing rates were not even the highest in the city that year. About 30 percent of the 34 students who graduated from P.S. 199 in the South Bronx and went on to a middle school up the road, I.S. 303, saw their test scores drop by more than two standard deviations. P.S. 199 had the 16th highest progress report score that year.

At I.S. 232, a nearby middle school that absorbed 29 students from P.S. 199, the swing rate was more in line with the city average.

But teachers at I.S. 303 said that the high test scores did not correlate to the basic English and math skills that many of the incoming P.S. 199 students demonstrated early on in sixth grade.

“You don’t just forget everything,” said one math teacher. “It just baffled me that they somehow got 3’s and 4’s” in fifth grade.

Multiple teachers at the school said students’ behavior during the sixth-grade state tests suggested that the students had received help during tests before.

Some of the students from P.S. 199 grew frustrated during the tests when I.S. 303 teachers did not tell them the answers to questions that stumped them, the teachers said. The teachers agreed to speak on the condition of anonymity because their principal had not given them permission to speak to reporters.

But even after noticing the score discrepancy, no one at I.S. 303 went to the department that year. It’s a common decision, according to educators from across the city who say allegations usually bring scrutiny first to the people who filed them, sometimes exposing them as whistleblowers to their colleagues.

An investigation at P.S. 199 is now open, a department spokeswoman said. Education officials contacted investigators after GothamSchools asked repeatedly about the school’s anomalous scores.

The department’s Office of Special Investigations is handling the case now after it was referred by the city’s Office of the Special Commissioner of Investigation, a SCI spokeswoman said.

Department officials said their concerns were not based solely on the swing rate. Data points such as swing rates alone are not enough to trigger investigations.

Many of the 37 schools to which the department dispatched testing monitors last year had seen their scores increase in unusual ways. But all but four of them had also been the subject of formal allegations.

In fact, the department does not even calculate schools’ swing rates as part of its regular analysis of schools’ performance. When the accountability division runs the numbers that feed into schools’ annual progress reports, which are based largely on students’ year-to-year growth, it does not aggregate results by sending schools.

Department officials say generating schools’ swing rates is a complicated endeavor. After learning about the score discrepancies at P.S. 199, GothamSchools requested swing data for other top-rated schools. But department officials said for months that running those numbers would be too difficult.

“We have provided you with a detailed analysis of P.S. 199,” spokeswoman Marge Feinberg wrote in an email this summer. “For the other schools, it would take a great deal of time.”

Researchers who have worked with Department of Education data before disputed that claim.

“It would be quite easy to do. Just about anyone with a computer and a basic knowledge of statistics could run these checks,” said Sean Corcoran, a New York University professor who has studied the city’s test score data. “This is something the DOE should do on a regular basis.”

A former testing director in New York City said that under his watch the city used data anomalies to trigger investigations.

“We routinely would look at the change in scores in schools from one year to the next,” said Bob Tobias, the Board of Education’s longtime testing chief who retired in 2002. Schools that showed erratic spikes, Tobias said, “would get a little more scrutiny.”

There are signs that the state might start conducting this type of analysis on its own. One charge given to the brand-new test security chief Tina Sciocchetti, at the State Education Department is to set guidelines for pursuing investigations using data methods that look suspicious test score patterns.

“I think those sorts of statistical analyses are simply a red flag and it’s absolutely true that additional investigation is necessary,” said Sciocchetti.

a 'meaningful' education?

How a Colorado court case could change how public schools everywhere serve students with special needs

Dougco headquarters in Castle Rock (John Leyba/The Denver Post).

The U.S. Supreme Court on Wednesday grappled with the question of what kind of education public schools must provide students with disabilities, hearing arguments in a case that originated with a complaint against a suburban Denver school district and that could have profound implications nationwide.

The case involves a student diagnosed with autism and attention deficit/hyperactive disorder. His parents pulled him out of his Douglas County elementary school, saying he wasn’t making enough progress and the district’s response was lacking.

They enrolled the boy in a private school for children with autism and asked the district to reimburse them for the tuition, arguing their son was due a “free appropriate public education” as required by the 1975 Individuals with Disabilities Education Act.

The law spells out the requirements states must meet to receive federal money to educate special-needs students. The district declined, saying it had met the standard of the law.

The family eventually filed a lawsuit against the district. Lower courts all sided with the district, reasoning that it had provided the child “some” educational benefit — the standard cited in the federal statute at issue.

Lower courts across the nation have varied in their definition of the proper standard. The high court arguments Wednesday centered on whether “some” benefit was good enough, or whether special-needs students deserve a more “meaningful” benefit.

Jeffrey Fisher, an attorney for the boy’s family, told the justices that as a general rule, individualized education plans for special education students should include “a level of educational services designed to allow the child to progress from grade to grade in the general curriculum.”

Throughout the arguments, the justices expressed frustration with what Justice Samuel Alito described as “a blizzard of words” that the law and courts have used to define what’s appropriate for special needs students.

Chief Justice John Roberts said regardless of the term used, “the whole package has got to be helpful enough to allow the student to keep up with his peers.”

Neal Katyal, an attorney for the school district, argued that providing children “some benefit” is a reasonable standard.

“That’s the way court after court has interpreted it,” he said. “It’s worked well. This court shouldn’t renege on that.”

Ron Hager, senior staff attorney for special education at the National Disability Rights Network, attended the oral arguments Wednesday and said he was optimistic the lower court’s ruling would be overturned.

He said if the Supreme Court does overturn the federal Tenth Circuit Court’s ruling and requires a higher standard, it won’t necessarily come with major financial costs for school districts. Instead, he said, it will nudge them to be proactive and provide teacher training and intervention services early on instead of waiting until problems — and the expenses associated with them — snowball later.

Marijo Rymer, executive director of the Arc of Colorado, which advocates on behalf of people with intellectual and developmental disabilities, said she was heartened to see the case advance to the Supreme Court. Establishing a clearer standard on what constitutes a fair and appropriate education for students with disabilities is a civil rights issue, she said.

“It’s critical that federal law, which is what this is based on, be reinforced and supported, and the court is in the position to deliver that message to the nation’s schools and the taxpayers that fund them,” Rymer said.

Both Hager and Rymer acknowledged that even if the Supreme Court establishes a new, higher standard, it could be open to interpretation. Still, they said it would send a strong message to school districts about their responsibilities to students with disabilities.

Summer remix

Ten stories you may have missed this summer (and should read now as the new school year kicks in)

PHOTO: Caroline Bauman
Gabrielle Colburn, 7, adds her artistic flair to a mural in downtown Memphis in conjunction with the XQ Super Schools bus tour in June.

Labor Day used to signal the end of summer break and the return to school. That’s no longer the case in Tennessee, but the long holiday is a good time to catch up on all that happened over the summer. Here are 10 stories to get you up to speed on K-12 education in Tennessee and its largest school district.

TNReady is back — with a new test maker.

Last school year ended on a cliffhanger, with the State Department of Education canceling its end-of-year tests for grades 3-8 in the spring and firing testmaker Measurement Inc. after a series of missteps. In July, Commissioner Candice McQueen announced that Minnesota-based Questar will pick up where Measurement Inc. left off. She also outlined the state’s game plan for standardized tests in the coming year.

But fallout over the state’s failed TNReady test in 2015-16 will be felt for years.

The one-year void in standardized test scores has hit Tennessee at the heart of its accountability system, leaving the state digging for other ways to assess whether all of its students are improving.

Speaking of accountability, Tennessee also is updating that plan under a new federal education law.

The state Department of Education has been working with educators, policymakers and community members on new ways to evaluate schools in answer to the federal Every Student Succeeds Act, or ESSA, which requires states to judge schools by non-academic measures as well as test scores.

Meanwhile, issues of race and policing have educators talking about how to foster conversations about social justice in school.

In the wake of police-related killings that rocked the nation, five Memphis teachers talked about how they tackle difficult conversations about race all year long.

School closures made headlines again in Memphis — with more closings likely.

Closing schools has become an annual event as Tennessee’s largest district loses students and funding, and this year was no exception. The shuttering of Carver and Northside high schools brought the total number of district-run school closures to at least 21 since 2012. And more are likely. This month, Shelby County Schools is scheduled to release a facilities analysis that should set the stage for future closures. Superintendent Dorsey Hopson has said the district needs to shed as many as two dozen schools — and 27,000 seats — over the next four years. A Chalkbeat analysis identifies 25 schools at risk.

Exacerbating the challenges of shifting enrollment, families in Foote Homes scrambled to register their children for school as Memphis’ last public housing project prepared to close this month amid a delay in delivering housing vouchers to move elsewhere.

The new school year has officially begun, with the budget approved not a moment too soon for Shelby County Schools.

District leaders that began the budget season facing an $86 million shortfall eventually convinced county commissioners to significantly increase local funding, while also pulling some money from the school system’s reserve funds. The result is a $959 million budget that gives most of the district’s teachers a 3 percent raise and restores funding for positions deemed critical for continued academic progress.

The district also unveiled its first annual report on its growing sector of charter schools.

With charter schools now firmly entrenched in Memphis’ educational landscape, a Shelby County Schools analysis shows a mixed bag of performance, while calling on traditional and charter schools to learn from each other and promising better ways to track quality.