Another round in the fight over charter school research methods

Thanks to commenter Tim, who alerted us to the latest entry in the ongoing conversation on how researchers should properly measure the performance of charter schools.

Researchers from Stanford’s Center for Research on Education Outcomes (CREDO) are responding to their colleague Caroline Hoxby’s criticism of the methodology they used in their national study of charter school performance. That study found that the majority of charter school students in 15 states and the District of Columbia performed as well as or worse than their peers in traditional public schools.

Hoxby, a fellow Stanford researcher, last month released a study of New York City’s charter school students that concluded that charter school students were significantly outperforming students in the city’s district schools. Along with that study, she released a memo arguing that the CREDO study’s more mixed conclusions about the performance of charters were based on a “serious statistical mistake.”

The CREDO researchers, led by director Margaret E. Raymond, responded earlier this week with a long technical analysis refuting Hoxby’s claim. They argue that her criticism is irrelevant to their overall conclusions on charter school performance.

What struck me about the CREDO response, though, was the researchers’ note explaining that the conclusions of the two studies are not necessarily in competition with one another. The two studies have been frequently characterized as presenting conflicting outcomes, but the CREDO researchers write that this might not be the case:

It should be noted that this peer‐reviewed CREDO study found that charter school performance varied considerably.  Some communities and states have gotten the policy right and are able to demonstrate positive charter school results.  The recent results for New York City (NYC) indicate that it, too, has a focused and effective charter policy.  Along with states like Arkansas, Louisiana and Missouri, these results offer a glimpse into the realm of what is possible when the barriers to entry are removed, the authorizers are high functioning and poor quality schools are removed from the field. On the other hand, the CREDO study shows that nationally there are more poorly performing charter schools than good ones, leading to obvious and important questions about why some charters in some communities do better than schools in other environments.

Education Next’s Eric Hanushek has an interesting blog post on what this means for how we interpret the results of the Hoxby study:

[T]he NYC study can be thought of as proof that the best charter schools, as judged by parents, can dramatically outperform the alternative traditional school.  That is important information, but it is impossible to know how to generalize it to other environments with different state laws, different union contracts, different district governance, different financing arrangements, and the like.   Just on the surface, nobody would think that it was possible to generalize from NYC (one million students) to LA (700,000 students), let alone to Kansas City (20,000 students).

Of course, both of these comments accept that the results of the Hoxby study are conclusive, a subject that has been vigorously debated in New York.

One other interesting item of note: neither the CREDO nor the Hoxby study has been published in a peer-reviewed journal, though both have had elements of peer review. The CREDO study was reviewed by four outside experts, Education Week reported. The Hoxby study is currently posted as a working paper by the National Bureau for Economic Research, which opens the research up for review by other scholars, though it’s unclear if any feedback was incorporated into the study before it was widely released. Hoxby also told Education Week that the paper is under consideration for publication by two journals.