First Person

New York City Charter Lotteries: Hey, You Never Know

A few years ago, the New York State lottery’s slogan was “Hey, you never know.”  In its original formulation, the slogan sought to motivate New Yorkers to play the lottery, a game of chance, on the grounds that you never know unless you play if you are a winner.  But the slogan is a double entendre when applied to Caroline Hoxby’s highly-publicized study of the effects of attending a charter school in New York City.  Propelled by Hoxby’s forceful claims about the superiority of lottery-based research on charter schools, much of the mainstream media has concluded that we now know definitively that New York City charter schools outperform their traditional counterparts—in spite of the fact that her study has not undergone a rigorous peer review process that might identify problems in the study and ways of addressing them.  Today, however, an equally forceful critique prepared by Sean Reardon of Stanford University argues that Hoxby’s research is anything but definitive.  Citing flaws in the statistical analysis of the report, Reardon writes that it “likely overstates the effects of New York City charter schools on students’ cumulative achievement … It may be that New York City’s charter schools do indeed have positive effects on student achievement, but those effects are likely smaller than the report claims.”

Reardon is careful to point out that it’s not possible, based on the information provided in Hoxby’s report and associated documents, to judge the extent of the bias in Hoxby’s estimates of charter school effects on student achievement.  More than anything, he calls for reserving judgment until more information about the study, its data and methods are available, and until the study has undergone rigorous peer review.  Until then, he maintains, it would be unwise to rely on the statistics reported in the study, and the inferences Hoxby and her colleagues draw about charter school effects in New York City.     

Here I’ll mention two of the features of Reardon’s critique that I find particularly persuasive.  The first is that Hoxby used an inappropriate set of statistical models to analyze the data, which likely distorts the charter school effects.  You might be surprised to learn that Hoxby used statistical models at all.  If her results are based on comparing students who won a charter school lottery with students who lost the lottery, and the lottery was fair, balanced and random, why would a model be needed?  It seems like the charter school effect would simply be the difference in the outcomes observed for the lottery winners and the lottery losers.  But comparing lottery winners and losers isn’t really estimating an individual causal effect, because an individual student can’t simultaneously be enrolled in a charter school and a traditional public school.  Even in the context of a lottery, or any other kind of study that can capitalize on a randomization process, such as a clinical drug trial, statistical models come into play to allow for inferences about cause-and-effect relationships.  These inferences are always made in relation to a particular statistical model, and all such models have assumptions.

One of the assumptions that is widely recognized is that a statistical model for causal inference should take account of factors that precede selection into the “treatment”—in this case, enrollment in a charter school versus a traditional public school.  If, hypothetically, charter school attendees were wealthier than traditional public school attendees, we’d want to control for wealth to make the charter and traditional school attendees as comparable as possible.  But it’s just as widely recognized that such a statistical model should not take account of factors that are measured after, and hence potentially influenced by, the treatment.  If attending a charter school increased a student’s motivation, and heightened motivation yields better test scores, then we wouldn’t want to control for motivation in a statistical model for the causal effect of going to charter school on test scores.  That kind of control means that the charter and traditional school attendees are no longer comparable at the time that they began attending a charter versus traditional school, which is the critical time.

Reardon demonstrates that this is precisely what Hoxby and her colleagues do in most of their statistical analyses.  For the analyses of charter school effects on test scores in grades four through eight, she controls for achievement in the prior year—achievement that was observed after the lotteries that determined whether a student enrolled in a charter or traditional public school.  The effects of charter school attendance on test scores in grades four through eight are therefore distorted, but to an unknown degree.  This is not a problem for estimates of the cumulative effect of charter school attendance in grades K-3 on third grade test performance, because the statistical models don’t include prior test scores (as there aren’t any before grade three.)  Reardon therefore finds Hoxby’s estimates of the effect of going to charter school in grades K-3 to be more credible than those for grades four through to eight.  However, the K-3 effect is only one-half to one-third as large as the estimated annual effect of charter school attendance in grades four through eight.    

The second issue is estimation of the cumulative effects of charter school attendance.  Hoxby’s report gets a lot of mileage out of the claim that the effects of attending a charter school from kindergarten to grade eight are large enough to close the performance gap between (predominantly white, upper-class) children in Scarsdale and (predominantly minority, low-income) children in Harlem by 66% in English and 86% in math.  Reardon points out that these figures are based on unrealistic extrapolations.  You can’t simply add up the annual effects of attending a charter school from year to year because the gains decay over time.  Moreover, most of the students in the Hoxby study have been in charter schools for only three or four years;  virtually none have been enrolled in charter schools for as many as nine years, and those would only have been enrolled in the very small number of charter schools that have been open that long, and cannot tell us about the long-term effects of attending the much larger number of newer charter schools.  Reardon’s analysis suggests that Hoxby’s estimate of the cumulative effect of attending a charter school from grades four through eight could be exaggerated by as much as 50%.

Are Caroline Hoxby’s estimates of the effects of attending a charter school rather than a traditional public school in New York City accurate?  Maybe.  But based on Sean Reardon’s critique, probably not.  Hey, you never know.

First Person

I mentor students demoralized about not having a vote. Here’s their plan for getting civically involved before turning 18

Students in the Minds Matter program.

Every Monday night during the school year, I spend time with two wonderful young women. They’re high-achieving high school sophomores from low-income families whose success would be certain if they grew up in a more affluent ZIP code.

Along with a team of other mentors, I help the students improve their writing and communication skills to help them prepare for a successful college career. That’s what I’m prepared to do.

I was less prepared for what they brought to our meeting last week, the first time we met under the tenure of a new president. They talked about feeling the consequences of the national political shift, though at 15, they knew it would be years before they could cast a ballot of their own. “We feel left out of a system that affects us too,” they said.

So our task that night became to expand our ideas about what participation in the American political system really means.

Here are five ideas we came up with, designed to help high schoolers do just that.

1. Meet elected officials. Meeting state senators and representatives during their campaigns is often the easiest way to make contact. Attend a coffee event, a party meeting, or a fundraiser where students can introduce themselves and talk about their concerns. Encourage them to be more than just another face in the crowd.

There are plenty of young, local elected officials to learn from. Dominick Moreno, a prominent Senate Democrat on the state of Colorado’s powerful Joint Budget Committee, got his start running for class president as a high school sophomore. Still only 32, he has already served in the House of Representatives and as mayor pro tem of a Denver suburb.

2. Volunteer on a campaign. This is the best opportunity for students to get an inside look at the political process and can help them establish lasting relationships with real people working in politics.

Some legislators face tough races and are out knocking on doors for months. Others spend their time differently, and in either case, candidates need help reaching out to voters, managing social media accounts, answering emails or organizing events. Plus, this work looks great on student résumés.

I tell students about my own experience. It started small: When I was 10, I passed out stickers for local elected officials at holiday parades. When I was 16, I got the chance to intern at the South Dakota state capitol. At 21, I got my first job in Washington, and at 23 I started lobbying in Colorado, affecting policy that now touches all citizens of the state.

3. Think locally. There are so many small things that students can do that will help their community become a better place on their own timeline. Help students organize a neighborhood clean-up day or tutor at an elementary school. These might feel inadequate to students when they look at the big picture, but it’s important to remind them that these actions help weave a fabric of compassion — and helps them become local leaders in the community.

4. Pre-register to vote. Voting matters, too. It sounds simple, but pre-registering addresses a root cause of low voter turnout — missing deadlines. In Colorado, one must be a U.S. citizen, be at least 16 years old, and reside in the state 22 days prior to the date of the election.

5. Affiliate with a party.
This assures full involvement in the process. Before turning 18, students can still attend party meetings or even start a “Young Democrats/Republicans” group at school. If they don’t feel like they fit with either the Republican or the Democratic parties, that’s OK — unaffiliated voters can now take part in the primary elections and help name either Republican or Democratic leaders.

Talking through these ideas helped the students I work with realize voting isn’t the only way to make a difference. One of my students has started a group that helps other young women know about birth control options, after seeing girls in her high school struggle and drop out after getting pregnant. Other students in the group have asked to learn more about the legislative process and want to testify on legislation.

They’re proving that democracy doesn’t begin and end with casting a ballot — but it does depend on taking interest and taking action.

Zoey DeWolf is a lobbyist with Colorado Legislative Services, based in Denver. She also works with Minds Matter of Denver, a not-for-profit organization whose mission is to help prepare accomplished high school students from low-income families for successful college careers.

First Person

It’s time to retire the myth that any counselor can do the job alone — even at a tiny school

A few of the author's students who graduated last year.

I waited five years to get my dream job as a counselor in a New York City public school. After all of that waiting, I was full of ideas about how I would be able to use my experience to help students navigate what can be an overwhelming few years.

I wanted to make our school counseling more individualized and full of innovative support mechanisms. I wanted our guidance department to be a place that anyone could leave with a grand plan.

A few months into that first year, in fall 2015, it was clear that my vision would be, to put it bluntly, impossible to achieve.

When I received my position at a Harlem high school in District 5, I was assigned to not only take on the responsibilities of a school counselor, but also to act as the college advisor, assign (and then frequently re-shuffle) class schedules for every student, and several other tasks. My school had just under 200 students — enrollment low enough that it was assumed this could all be managed.

This proved to be a very inaccurate assumption. I was working with a group of students with low attendance rates, and many were English language learners or students with disabilities. Many students were overage and under-credited, others were in foster care or homeless, some had returned from incarceration, and a couple were teen parents or pregnant.

The American School Counselor Association recommends a maximum school counselor-to-student ratio of one to 250. I know from experience that extremely high student need makes that ratio meaningless. Almost all of these students needed help in order to be ready to learn. Their needs tripled the feel of our enrollment.

This frequent mismatch between need and numbers puts school counselors like me in the position to do a great disservice to so many students. As the only counselor available, a seemingly small mishap with a task as crucial as graduation certification or credit monitoring could have spelled disaster for a student. I know some seniors missed certain financial aid opportunities and application deadlines, and some ninth, 10th, and 11th graders could have used more academic intervention to help them transition to the next grade level successfully.

My success at keeping our promotion and college admissions rates on the upswing was largely due to my outreach and partnership with community-based organizations that helped support several of our students. Had it not been for their assistance, I wouldn’t have achieved anything near what I did.

I’m still a counselor at my small school, and some aspects of the job have gotten easier with time. I love my job, which I think of as the most rewarding yet intense position in the building. But I still believe that there is almost no case in which only one counselor should be available for students.

Principals and school leaders directly involved with the budget must make sure to effectively analyze the needs of their student population, and advocate for an appropriately sized counseling staff. Small schools face real funding constraints. But ones serving students like mine need more than they’ve gotten.

Students’ social and emotional development and their academic success go hand in hand. Let’s not make the mistake of conflating enrollment numbers with need.

Danisha Baughan is a high school counselor and college advisor. She received her masters in school counseling in May 2010 and has held elementary, middle, and high school counseling positions since then.