In reports, validation for city's high school gains, but not its data

An independent research group with access to a trove of the city’s education data concluded that most of the Bloomberg administration’s claims of high school progress are credible.

But in a different report commissioned by a nonprofit group that manages some city high schools, researchers found that the city’s tools for evaluating schools do not treat schools with higher-need students fairly.

The two reports come as the Bloomberg administration concludes a three-term spree of policy changes meant to spur improvement in the city’s high schools. The spree included dozens of school closures and the creation of hundreds of new high schools, along with accountability metrics such as the annual “progress report” to make school performance transparent. Whether to continue the policies and accountability measures will be a major choice facing the next mayor. Mayor Bloomberg has said that those efforts have moved more students toward graduation, but critics have pointed to low “college readiness” statistics to argue that students are leaving high school without the skills needed for success afterwards.

According to New York University’s Research Alliance for New York City Schools, which produced the first report, both claims are true. The group, which has unfiltered access to city schools data through a deal with the city Department of Education, examined the performance of 900,000 high school students between 1999 and 2011 and found that students of all backgrounds had posted achievement gains, including an 18-point increase in the four-year graduation rate  — but that few graduated college-ready and large demographic performance gaps remained.

Compared to other analyses of high school student achievement, the report looked at a longer time period and examined results by more subgroups of students.

The report dug into students’ performance data from their middle school years to see how they progressed in high school. Often, eighth-graders with the lowest attendance and test scores saw the largest gains in graduation. For instance, students who scored in the lowest 20 percent on the state’s eighth-grade math and English tests graduated 4 percent of the time in 2001. In 2007, that rate was 22 percent, a 400 percent increase. Chronically absent eighth-graders graduated 9 percent of the time in 2001; they graduated 25 percent of the time in 2007.

That progress stalled when compared to the higher standard of the state’s college readiness metric. Students who struggled with attendance and on their test scores in eighth grade had almost no chance of being “college ready.” Overall, readiness rates rose by just eight percentage points — from 13 percent to 21 percent — between 2001 and 2007, even as graduation rates increased by more than twice that margin.

At a panel about the Research Alliance’s report on Wednesday, Department of Education Chief Academic Officer Shael Polakow-Suransky pushed back against the formula used to measure college readiness. The report defined readiness as graduating with a Regents diploma and scores of 75 and 80 in the English and math Regents exams, the same standard that the state uses. But Polakow-Suransky said a stronger gauge of success is what percentage of students are still enrolled after four semesters of college.

“You actually see scores closer to 34 percent of students,” Polakow-Suransky said. He added, “But even with 35 percent of the kids that’s way too low.”

One tool that the city created to measure high school performance, the progress report, was explicitly designed to give schools credit for helping low-performing students move forward, even if they did not hit proficiency standards. But in the second analysis out this week, commissioned by the nonprofit New Visions, NYU researchers found that the controls the Department of Education employs to make fair comparisons across schools are not adequate.

More specifically, the researchers found that schools are more likely to post low grades if they have many poor students, students of color, or students with disabilities. The finding confirms recent research by the Independent Budget Office, a city office charged with studying Department of Education data.

On the flip side, the report also finds that some schools with many high-achieving students might also have been penalized for not hitting “mathematically impossible growth targets.”

Like the IBO report, the New Visions report argues that the progress report methodology is an improvement over simply comparing schools’ student performance without looking at how quickly students are advancing. Still, the researchers recommended several changes to the progress reports’ methodology, should the next mayor continue using the system to generate scores for each school each year.

“We hope … that this report and the above recommendations will spark a productive conversation about how the high school Progress Report can be improved to better meet its objectives,” wrote the researchers, led by Sean Corcoran, a professor who has also studied the city’s principal training institute and the “value-added” ratings that the city briefly assigned to teachers.

The purpose of the Research Alliance’s high school analysis, according to Executive Director James Kemple, was to lay the groundwork for series of papers on high schools. One paper will examine how high schools have been affected by changes in enrollment patterns, which critics believe led to the downfall to many schools selected for closure. Another will study high schools that “beat the odds” at narrowing the racial and socioeconomic achievement gap and identify what they’re doing differently.

The Alliance — whose board includes teachers union president Michael Mulgrew — received a $3 million grant in 2008 from the Gates Foundation, which also helped former schools chancellor Joel Klein expand the city’s portfolio of small schools before abandoning the strategy several years ago. Kemple said the group’s research was nonpartisan and applied the most rigorous methods for data management, organization, and analysis.

“Our agreements with the Department of Education within NYU is to publish things as we see them, not as we’re dictated to, and nobody can restrict our publications,” he said.