New York

Bungled by Design

There's a well-known education research textbook by three distinguished scholars at Harvard entitled By Design.  Judy Singer, one of the authors, once told me that the working title for the book, rejected by Harvard University Press, was Bungled by Design.  That title conveyed the key message of the book, which is that, when it comes to education research, you can't fix by analysis what you bungled by design.  The design of a research study dictates what a researcher can plausibly ask, and the credibility of the claims about what is being studied. The recently-released NYU study of the New York City Principal Leadership Academy comparing graduates of the Aspiring Principals Program to other new NYC principals is, in my view, bungled by design.  This is not a knock on the authors, each of whom I know and respect a great deal.  Rather, it reflects the fact that the NYU researchers were brought in to study the Aspiring Principals Program of the Leadership Academy long after critical design decisions about how to evaluate the impact of the program were made—either by omission or commission. The three key limitations I raise here pertain to selection mechanisms that ideally would have been observed by the researchers.  The inability to understand and model these selection processes undermines the objective of isolating the effect of the Aspiring Principals Program on student outcomes.  (See the comments of Sean Corcoran, lead author of the report, on selection issues here.)
New York

SAT Scores in New York City: A Large and Unrelenting Gap

Yesterday, the College Board released its annual report on the SAT, and New York City was quick to follow suit with data on the performance of NYC high school students on the SAT.  Citywide average scores fell a few points, at the same time that the numbers of Black and Hispanic students taking the SAT increased.  Writing in the Daily News, Rachel Monahan summarized the DOE spin, courtesy of DOE spokesman Andy Jacob:  (a) More Black and Hispanic students took the SAT, and fewer white students did;  (b) the increasing numbers of SAT-takers are less likely to be high performers than SAT-takers in the past;  (c) therefore, let's focus on the increased representativeness of the test-taking group, and ignore the fact that scores fell among Blacks and Hispanics, and that the achievement gap is still huge. I don't think that we should pay too much attention to single-year changes in test scores of any kind, and especially the SAT, which commenter CarolineSF points out are taken by a self-selected group of high school students.  But this year's snapshot nevertheless reveals some hard truths about the performance of New York City's high school students. Let's address the representativeness issue first.  Is there evidence that the rising numbers of Black and Hispanic students taking the SAT reflects a dramatic change in the kinds of students who are taking the SAT?  Can we explain the falling average Black and Hispanic SAT scores as reflecting a new group of low-performing NYC high school students striving to get into college?