behind the metrics

IBO: City's school progress reports are flawed but an advance

The system the city uses to award letter grades to schools is complicated and in some ways flawed — but it’s the best system we have.

That’s the conclusion of a report by the Independent Budget Office, the city’s budget watchdog that since 2009 has been charged with scrutinizing Department of Education data. The office examined the city’s progress reports, released annually since 2007, to see whether their underlying metrics produce meaningful results.

The progress reports were meant to radically reorient the way that New Yorkers thought about school performance. Instead of assessing schools simply by the proportion of students passing state tests, the progress reports focus on students’ improvement from year to year. In a precursor to the “value-added” measurements now being used to assess teachers, the reports use a complex and evolving algorithm that controls for student demographics to calculate just how much students have progressed.

The city then assigns each school a letter grade based on its score. The letter grades inform both the city’s decisions about which principals should receive bonuses and which schools should be considered for closure and families’ choices are where to enroll.

The IBO concludes that the progress reports offer a more sophisticated analysis of school performance than ever before — but that there is room for improvement. “The methodology used by the education department is a significant improvement over simply basing measures on comparisons of standardized test scores,” the report concludes. “Still, the School Progress Reports have to be interpreted with caution.”

The IBO looked at three issues: whether the city’s algorithm has successfully controlled for factors outside of schools’ control; whether the reports have reflected long-term shifts as well as short-term changes; and whether minor methodology changes produced outsized score swings.

On the first question, the budget office concluded that overall, progress report scores in a small set of schools, those serving both elementary and middle school students, can be considered “demographically neutral” — or unaffected by student characteristics. But in most cases that was not true, according to the IBO’s analysis.

“All other things equal, elementary, middle, and high schools with a higher percentage of black and Hispanic students were consistently likely to have lower overall scores than other schools,” the report notes. Progress report scores were also lower in high schools with more poor students and more students with disabilities, the IBO concluded.

Confirming previous findings, the IBO also concludes that elementary and middle school scores have been highly volatile, with the majority of those schools receiving three or more different progress report grades since 2007. (High school progress report grades are based on a wider range of variables and have always been more stable.) But the IBO says that changes to the reports’ methodology, particularly around how students’ year-to-year growth is assessed, have made them more stable.

Finally, the IBO’s analysis found that most of the changes made to the progress report methodology in 2010 and 2011 did not affect their overall grade. In general, the office concludes, the city’s reports successfully identified very high- and very low-performing schools under multiple methodologies, but they were less successful at distinguishing among middle performers.

“The distinction between a C and D rating for a school may be the result of the particular methodology that the DOE has chosen, among the many that are possible, rather than the result of school practices or effectiveness,” the report concludes. “Unfortunately, this weakness occurs at precisely the point where high stakes decisions about schools are made.”

Department of Education officials say that while the peer group comparisons do not “completely control” for student characteristics, they do reduce the impact of race and other demographics compared to other measures of school performance. They also point out that the IBO found strong correlations between demographics and school grades in only a handful of the relationships it analyzed. And they note that the IBO’s analysis shows that between half and three quarters of C and D grades issued in 2011 would have been the same using the IBO’s methodology for analyzing score stability.

“As the IBO has recognized, New York City’s progress reports are a huge improvement over other state and district systems for measuring student learning in schools — and for this reason they have become a national model,” said spokesman Matthew Mittenthal in a statement. “Closing the achievement gap in New York City is a core goal of our reform strategy, but as long as it exists we should expect it to show up in school progress reports, which are designed to be an accurate reflection of our schools’ strengths and challenges.”

The complete IBO report is below.

a 'meaningful' education?

How a Colorado court case could change how public schools everywhere serve students with special needs

Dougco headquarters in Castle Rock (John Leyba/The Denver Post).

The U.S. Supreme Court on Wednesday grappled with the question of what kind of education public schools must provide students with disabilities, hearing arguments in a case that originated with a complaint against a suburban Denver school district and that could have profound implications nationwide.

The case involves a student diagnosed with autism and attention deficit/hyperactive disorder. His parents pulled him out of his Douglas County elementary school, saying he wasn’t making enough progress and the district’s response was lacking.

They enrolled the boy in a private school for children with autism and asked the district to reimburse them for the tuition, arguing their son was due a “free appropriate public education” as required by the 1975 Individuals with Disabilities Education Act.

The law spells out the requirements states must meet to receive federal money to educate special-needs students. The district declined, saying it had met the standard of the law.

The family eventually filed a lawsuit against the district. Lower courts all sided with the district, reasoning that it had provided the child “some” educational benefit — the standard cited in the federal statute at issue.

Lower courts across the nation have varied in their definition of the proper standard. The high court arguments Wednesday centered on whether “some” benefit was good enough, or whether special-needs students deserve a more “meaningful” benefit.

Jeffrey Fisher, an attorney for the boy’s family, told the justices that as a general rule, individualized education plans for special education students should include “a level of educational services designed to allow the child to progress from grade to grade in the general curriculum.”

Throughout the arguments, the justices expressed frustration with what Justice Samuel Alito described as “a blizzard of words” that the law and courts have used to define what’s appropriate for special needs students.

Chief Justice John Roberts said regardless of the term used, “the whole package has got to be helpful enough to allow the student to keep up with his peers.”

Neal Katyal, an attorney for the school district, argued that providing children “some benefit” is a reasonable standard.

“That’s the way court after court has interpreted it,” he said. “It’s worked well. This court shouldn’t renege on that.”

Ron Hager, senior staff attorney for special education at the National Disability Rights Network, attended the oral arguments Wednesday and said he was optimistic the lower court’s ruling would be overturned.

He said if the Supreme Court does overturn the federal Tenth Circuit Court’s ruling and requires a higher standard, it won’t necessarily come with major financial costs for school districts. Instead, he said, it will nudge them to be proactive and provide teacher training and intervention services early on instead of waiting until problems — and the expenses associated with them — snowball later.

Marijo Rymer, executive director of the Arc of Colorado, which advocates on behalf of people with intellectual and developmental disabilities, said she was heartened to see the case advance to the Supreme Court. Establishing a clearer standard on what constitutes a fair and appropriate education for students with disabilities is a civil rights issue, she said.

“It’s critical that federal law, which is what this is based on, be reinforced and supported, and the court is in the position to deliver that message to the nation’s schools and the taxpayers that fund them,” Rymer said.

Both Hager and Rymer acknowledged that even if the Supreme Court establishes a new, higher standard, it could be open to interpretation. Still, they said it would send a strong message to school districts about their responsibilities to students with disabilities.

Summer remix

Ten stories you may have missed this summer (and should read now as the new school year kicks in)

PHOTO: Caroline Bauman
Gabrielle Colburn, 7, adds her artistic flair to a mural in downtown Memphis in conjunction with the XQ Super Schools bus tour in June.

Labor Day used to signal the end of summer break and the return to school. That’s no longer the case in Tennessee, but the long holiday is a good time to catch up on all that happened over the summer. Here are 10 stories to get you up to speed on K-12 education in Tennessee and its largest school district.

TNReady is back — with a new test maker.

Last school year ended on a cliffhanger, with the State Department of Education canceling its end-of-year tests for grades 3-8 in the spring and firing testmaker Measurement Inc. after a series of missteps. In July, Commissioner Candice McQueen announced that Minnesota-based Questar will pick up where Measurement Inc. left off. She also outlined the state’s game plan for standardized tests in the coming year.

But fallout over the state’s failed TNReady test in 2015-16 will be felt for years.

The one-year void in standardized test scores has hit Tennessee at the heart of its accountability system, leaving the state digging for other ways to assess whether all of its students are improving.

Speaking of accountability, Tennessee also is updating that plan under a new federal education law.

The state Department of Education has been working with educators, policymakers and community members on new ways to evaluate schools in answer to the federal Every Student Succeeds Act, or ESSA, which requires states to judge schools by non-academic measures as well as test scores.

Meanwhile, issues of race and policing have educators talking about how to foster conversations about social justice in school.

In the wake of police-related killings that rocked the nation, five Memphis teachers talked about how they tackle difficult conversations about race all year long.

School closures made headlines again in Memphis — with more closings likely.

Closing schools has become an annual event as Tennessee’s largest district loses students and funding, and this year was no exception. The shuttering of Carver and Northside high schools brought the total number of district-run school closures to at least 21 since 2012. And more are likely. This month, Shelby County Schools is scheduled to release a facilities analysis that should set the stage for future closures. Superintendent Dorsey Hopson has said the district needs to shed as many as two dozen schools — and 27,000 seats — over the next four years. A Chalkbeat analysis identifies 25 schools at risk.

Exacerbating the challenges of shifting enrollment, families in Foote Homes scrambled to register their children for school as Memphis’ last public housing project prepared to close this month amid a delay in delivering housing vouchers to move elsewhere.

The new school year has officially begun, with the budget approved not a moment too soon for Shelby County Schools.

District leaders that began the budget season facing an $86 million shortfall eventually convinced county commissioners to significantly increase local funding, while also pulling some money from the school system’s reserve funds. The result is a $959 million budget that gives most of the district’s teachers a 3 percent raise and restores funding for positions deemed critical for continued academic progress.

The district also unveiled its first annual report on its growing sector of charter schools.

With charter schools now firmly entrenched in Memphis’ educational landscape, a Shelby County Schools analysis shows a mixed bag of performance, while calling on traditional and charter schools to learn from each other and promising better ways to track quality.