On Friday, I began talking about what counts as a big effect.  Turns out I’m reinventing the wheel, as there is an excellent paper by Carolyn Hill and her colleagues at Manpower Development Research Corporation on this topic, entitled “Empirical Benchmarks for Interpreting Effect Sizes in Research.”  But I’ll press onward nevertheless.

Last month, the federal Institute for Education Sciences released the third-year report on the evaluation of the DC Opportunity Scholarship Program, which provides vouchers for K-12 children and youth in the DC Public Schools who win a lottery to attend a private school.  The key outcomes in the study were scale scores on the Stanford Achievement Test (SAT-9) in reading and mathematics.  (Scale scores are converted from “raw” scores based on the number of correct responses to the test.)  The evaluators found that, after three years, students who were offered a voucher scored 4.46 points higher on the SAT-9 reading test, which represented an effect size of .13.  This effect was statistically different from zero.  Interestingly, the impact of being offered a voucher on reading scores was not reliably different from zero for male students.  In mathematics, there was no evidence of a positive effect of being offered a voucher:  after three years, students offered vouchers scored .81 points higher on the SAT-9 math test, an effect that was not statistically different from zero, and which corresponded to an effect size of .03.

Based on how these effect sizes equate with percentile changes, these are pretty small effects, and the presence of an asterisk denoting statistical significance for the effect of being offered a voucher on reading scores for girls alone, and no effects on math scores for either boys or girls, doesn’t justify the political spectacle that surrounds the program.  After three years, the net movement in reading for voucher students starting at around the 34th percentile nationally is about five percentiles;  in math, it’s about one percentile.  Anyone who thinks that effects of this size are altering the life trajectories of DC children is kidding himself.

Part of the hoopla stems from another way in which the size of the voucher effect is being reported:  months and years of additional learning.  The overall effect of 4.5 scale score points in reading is reported as equivalent to 3.1 months of additional learning for members of the treatment group, and the 5.3 point scale score gain for those who actually used the voucher is reported as 3.7 additional months of learning.  The Wall Street Journal’s op-ed page, always good with math, rounded this up to “Children attending private schools with the aid of the scholarships are reading nearly a half-grade ahead of their peers who did not receive vouchers.”