First Person

New York City Charter Lotteries: Hey, You Never Know

A few years ago, the New York State lottery’s slogan was “Hey, you never know.”  In its original formulation, the slogan sought to motivate New Yorkers to play the lottery, a game of chance, on the grounds that you never know unless you play if you are a winner.  But the slogan is a double entendre when applied to Caroline Hoxby’s highly-publicized study of the effects of attending a charter school in New York City.  Propelled by Hoxby’s forceful claims about the superiority of lottery-based research on charter schools, much of the mainstream media has concluded that we now know definitively that New York City charter schools outperform their traditional counterparts—in spite of the fact that her study has not undergone a rigorous peer review process that might identify problems in the study and ways of addressing them.  Today, however, an equally forceful critique prepared by Sean Reardon of Stanford University argues that Hoxby’s research is anything but definitive.  Citing flaws in the statistical analysis of the report, Reardon writes that it “likely overstates the effects of New York City charter schools on students’ cumulative achievement … It may be that New York City’s charter schools do indeed have positive effects on student achievement, but those effects are likely smaller than the report claims.”

Reardon is careful to point out that it’s not possible, based on the information provided in Hoxby’s report and associated documents, to judge the extent of the bias in Hoxby’s estimates of charter school effects on student achievement.  More than anything, he calls for reserving judgment until more information about the study, its data and methods are available, and until the study has undergone rigorous peer review.  Until then, he maintains, it would be unwise to rely on the statistics reported in the study, and the inferences Hoxby and her colleagues draw about charter school effects in New York City.     

Here I’ll mention two of the features of Reardon’s critique that I find particularly persuasive.  The first is that Hoxby used an inappropriate set of statistical models to analyze the data, which likely distorts the charter school effects.  You might be surprised to learn that Hoxby used statistical models at all.  If her results are based on comparing students who won a charter school lottery with students who lost the lottery, and the lottery was fair, balanced and random, why would a model be needed?  It seems like the charter school effect would simply be the difference in the outcomes observed for the lottery winners and the lottery losers.  But comparing lottery winners and losers isn’t really estimating an individual causal effect, because an individual student can’t simultaneously be enrolled in a charter school and a traditional public school.  Even in the context of a lottery, or any other kind of study that can capitalize on a randomization process, such as a clinical drug trial, statistical models come into play to allow for inferences about cause-and-effect relationships.  These inferences are always made in relation to a particular statistical model, and all such models have assumptions.

One of the assumptions that is widely recognized is that a statistical model for causal inference should take account of factors that precede selection into the “treatment”—in this case, enrollment in a charter school versus a traditional public school.  If, hypothetically, charter school attendees were wealthier than traditional public school attendees, we’d want to control for wealth to make the charter and traditional school attendees as comparable as possible.  But it’s just as widely recognized that such a statistical model should not take account of factors that are measured after, and hence potentially influenced by, the treatment.  If attending a charter school increased a student’s motivation, and heightened motivation yields better test scores, then we wouldn’t want to control for motivation in a statistical model for the causal effect of going to charter school on test scores.  That kind of control means that the charter and traditional school attendees are no longer comparable at the time that they began attending a charter versus traditional school, which is the critical time.

Reardon demonstrates that this is precisely what Hoxby and her colleagues do in most of their statistical analyses.  For the analyses of charter school effects on test scores in grades four through eight, she controls for achievement in the prior year—achievement that was observed after the lotteries that determined whether a student enrolled in a charter or traditional public school.  The effects of charter school attendance on test scores in grades four through eight are therefore distorted, but to an unknown degree.  This is not a problem for estimates of the cumulative effect of charter school attendance in grades K-3 on third grade test performance, because the statistical models don’t include prior test scores (as there aren’t any before grade three.)  Reardon therefore finds Hoxby’s estimates of the effect of going to charter school in grades K-3 to be more credible than those for grades four through to eight.  However, the K-3 effect is only one-half to one-third as large as the estimated annual effect of charter school attendance in grades four through eight.    

The second issue is estimation of the cumulative effects of charter school attendance.  Hoxby’s report gets a lot of mileage out of the claim that the effects of attending a charter school from kindergarten to grade eight are large enough to close the performance gap between (predominantly white, upper-class) children in Scarsdale and (predominantly minority, low-income) children in Harlem by 66% in English and 86% in math.  Reardon points out that these figures are based on unrealistic extrapolations.  You can’t simply add up the annual effects of attending a charter school from year to year because the gains decay over time.  Moreover, most of the students in the Hoxby study have been in charter schools for only three or four years;  virtually none have been enrolled in charter schools for as many as nine years, and those would only have been enrolled in the very small number of charter schools that have been open that long, and cannot tell us about the long-term effects of attending the much larger number of newer charter schools.  Reardon’s analysis suggests that Hoxby’s estimate of the cumulative effect of attending a charter school from grades four through eight could be exaggerated by as much as 50%.

Are Caroline Hoxby’s estimates of the effects of attending a charter school rather than a traditional public school in New York City accurate?  Maybe.  But based on Sean Reardon’s critique, probably not.  Hey, you never know.

First Person

Two fewer testing days in New York? Thank goodness. Here’s what else our students need

PHOTO: Christina Veiga

Every April, I feel the tension in my fifth-grade classroom rise. Students are concerned that all of their hard work throughout the year will boil down to six intense days of testing — three for math and three for English language arts.

Students know they need to be prepared to sit in a room for anywhere from 90 minutes to three hours with no opportunity to leave, barring an emergency. Many of them are sick to their stomachs, feeling more stress than a 10-year-old ever should, and yet they are expected to perform their best.

Meanwhile, teachers are frustrated that so many hours of valuable instruction have been replaced by testing, and that the results won’t be available until students are moving on to other classrooms.

This is what testing looks like in New York state. Or, at least it did. Last month, state officials voted to reduce testing from three days for each subject to two, to the elation of students, parents, and teachers across New York. It’s an example of our voices being heard — but there is still more to be done to make the testing process truly useful, and less stressful, for all of us.

As a fifth-grade teacher in the Bronx, I was thrilled by the news that testing time would be reduced. Though it doesn’t seem like much on paper, having two fewer days of gut-wrenching stress for students as young as eight means so much for their well-being and education. It gives students two more days of classroom instruction, interactive lessons, and engagement in thought-provoking discussions. Any reduction in testing also means more time with my students, since administrators can pull teachers out of their classrooms for up to a week to score each test.

Still, I know these tests provide us with critical data about how students are doing across our state and where we need to concentrate our resources. The changes address my worries about over-testing, while still ensuring that we have an objective measure of what students have learned across the state.

For those who fear that cutting one-third of the required state testing hours will not provide teachers with enough data to help our students, understand that we assess them before, during, and after each unit of study, along with mid-year tests and quizzes. It is unlikely that one extra day of testing will offer any significant additional insights into our students’ skills.

Also, the fact that we receive students’ state test results months later, at the end of June, means that we are more likely to have a snapshot of where are students were, rather than where they currently are — when it’s too late for us to use the information to help them.

That’s where New York can still do better. Teachers need timely data to tailor their teaching to meet student needs. As New York develops its next generation of tests and academic standards, we must ensure that they are developmentally appropriate. And officials need to continue to emphasize that state tests alone cannot fully assess a student’s knowledge and skills.

For this, parents and teachers must continue to demand that their voices are heard. Until then, thank you, New York Regents, for hearing us and reducing the number of testing days.

In my classroom, I’ll have two extra days to help my special needs students work towards the goals laid out in their individualized education plans. I’ll take it.

Rich Johnson teaches fifth grade at P.S. 105 in the Bronx.

First Person

My high school told me to apply to 100 colleges — and I almost lost myself in the process

PHOTO: Stephanie Snyder

High schools have become obsessed with “million-dollar scholars,” and it’s hurting students.

Across Memphis, students often are pushed by counselors to apply to as many colleges as possible — as many as 100 — all to push students to reach that million-dollar scholarship mark. The more dollars and college acceptance, the better!

I graduated in 2016, and my experience offers a case study.

I’m a pretty well-rounded individual: In high school, I was a finalist in the Let’s Innovate Through Education program and was able to launch SousChef-Memphis, a culinary nonprofit organization. I was a dual-enrollment student and took honors courses. I was committed to community service. I was vice president of my high school organization, Modern Distinctive Ladies. I was on the bowling team, managed the basketball team, and participated in debate forensics and drama.

I was also told by counselors to apply to 100 colleges. I was never told why that number was chosen, but my peers were told the same. We were often pulled out of class to complete these applications, which took away from instructional time — about an hour per day. My high school also ran on an infraction system, and not turning in college applications and other documents led to disciplinary actions.

The quality of those applications only shed a dim light on the student and person that I am. A hundred applications was never my goal. A hundred applications doesn’t measure the capability, intelligence or worth of me as a student. A hundred applications is just ridiculous!

Schools with similar approaches, though, get glowing media coverage. Meanwhile, a lot of that scholarship money is irrelevant, since a single student obviously can only attend one school.

I think that if I had been counseled properly, I would have had a better grasp on my high school-to-college transition. I ultimately chose to leave Memphis to attend another state university on a full scholarship. Looking back, that school was not the best fit for me. I returned to Memphis to attend our local public university.

A counselor never tried to determine what the absolute best school for me would be. I wasted a lot of time, money and resources trying to figure that out. And I almost lost myself in the process.

I was more than capable of getting back on track, and I did. But not every student can afford to go through what I went through.

High schools need to realize that, while students amassing millions of dollars in scholarships and hundreds of college acceptance letters seems like an accomplishment, the outcome for many students is the total opposite.

Too many students end up not going to a school that is the best fit for them, taking on piles of debt, and dropping out with no workforce experience.

The goal should be that each high school student will graduate having a grasp on their career path (and experience in that field), scholarships to the school of their choice (full rides or little to no debt), and be confident in where they will be spending the next four to six years of their life. Being thorough in the college search and submitting quality applications is what leads to a college that is the best fit for the student, obtaining scholarships, and ultimately graduating.

Here’s what I wish a counselor had told me:

"It is time to start thinking about quality over quantity. Quality counseling. Quality applications. And quality choices about what to do next."Anisah Karim

Try things you like, but don’t overload yourself. Look for summer internships that pay, rather than minimum-wage jobs. Build a network of people who can help you make good decisions about college and work. Research schools with a major you’re interested in, and find out what scholarships they offer. Keep an eye on your GPA and make sure you’re taking the classes you need to graduate. Apply for colleges when applications open and submit the FAFSA form in October.

And most importantly, through all four years of high school, don’t be afraid to ask for help.

It is time to start thinking about quality over quantity. Quality counseling. Quality applications. And quality choices about what to do next.

Anisah Karim is a psychology student at the University of Memphis. She plans to continue her education in speech pathology and otology and eventually start her own private practice. She also plans to launch two new business ventures in the fall and relaunch SousChef in the fall of 2018.