First Person

New York City Charter Lotteries: Hey, You Never Know

A few years ago, the New York State lottery’s slogan was “Hey, you never know.”  In its original formulation, the slogan sought to motivate New Yorkers to play the lottery, a game of chance, on the grounds that you never know unless you play if you are a winner.  But the slogan is a double entendre when applied to Caroline Hoxby’s highly-publicized study of the effects of attending a charter school in New York City.  Propelled by Hoxby’s forceful claims about the superiority of lottery-based research on charter schools, much of the mainstream media has concluded that we now know definitively that New York City charter schools outperform their traditional counterparts—in spite of the fact that her study has not undergone a rigorous peer review process that might identify problems in the study and ways of addressing them.  Today, however, an equally forceful critique prepared by Sean Reardon of Stanford University argues that Hoxby’s research is anything but definitive.  Citing flaws in the statistical analysis of the report, Reardon writes that it “likely overstates the effects of New York City charter schools on students’ cumulative achievement … It may be that New York City’s charter schools do indeed have positive effects on student achievement, but those effects are likely smaller than the report claims.”

Reardon is careful to point out that it’s not possible, based on the information provided in Hoxby’s report and associated documents, to judge the extent of the bias in Hoxby’s estimates of charter school effects on student achievement.  More than anything, he calls for reserving judgment until more information about the study, its data and methods are available, and until the study has undergone rigorous peer review.  Until then, he maintains, it would be unwise to rely on the statistics reported in the study, and the inferences Hoxby and her colleagues draw about charter school effects in New York City.     

Here I’ll mention two of the features of Reardon’s critique that I find particularly persuasive.  The first is that Hoxby used an inappropriate set of statistical models to analyze the data, which likely distorts the charter school effects.  You might be surprised to learn that Hoxby used statistical models at all.  If her results are based on comparing students who won a charter school lottery with students who lost the lottery, and the lottery was fair, balanced and random, why would a model be needed?  It seems like the charter school effect would simply be the difference in the outcomes observed for the lottery winners and the lottery losers.  But comparing lottery winners and losers isn’t really estimating an individual causal effect, because an individual student can’t simultaneously be enrolled in a charter school and a traditional public school.  Even in the context of a lottery, or any other kind of study that can capitalize on a randomization process, such as a clinical drug trial, statistical models come into play to allow for inferences about cause-and-effect relationships.  These inferences are always made in relation to a particular statistical model, and all such models have assumptions.

One of the assumptions that is widely recognized is that a statistical model for causal inference should take account of factors that precede selection into the “treatment”—in this case, enrollment in a charter school versus a traditional public school.  If, hypothetically, charter school attendees were wealthier than traditional public school attendees, we’d want to control for wealth to make the charter and traditional school attendees as comparable as possible.  But it’s just as widely recognized that such a statistical model should not take account of factors that are measured after, and hence potentially influenced by, the treatment.  If attending a charter school increased a student’s motivation, and heightened motivation yields better test scores, then we wouldn’t want to control for motivation in a statistical model for the causal effect of going to charter school on test scores.  That kind of control means that the charter and traditional school attendees are no longer comparable at the time that they began attending a charter versus traditional school, which is the critical time.

Reardon demonstrates that this is precisely what Hoxby and her colleagues do in most of their statistical analyses.  For the analyses of charter school effects on test scores in grades four through eight, she controls for achievement in the prior year—achievement that was observed after the lotteries that determined whether a student enrolled in a charter or traditional public school.  The effects of charter school attendance on test scores in grades four through eight are therefore distorted, but to an unknown degree.  This is not a problem for estimates of the cumulative effect of charter school attendance in grades K-3 on third grade test performance, because the statistical models don’t include prior test scores (as there aren’t any before grade three.)  Reardon therefore finds Hoxby’s estimates of the effect of going to charter school in grades K-3 to be more credible than those for grades four through to eight.  However, the K-3 effect is only one-half to one-third as large as the estimated annual effect of charter school attendance in grades four through eight.    

The second issue is estimation of the cumulative effects of charter school attendance.  Hoxby’s report gets a lot of mileage out of the claim that the effects of attending a charter school from kindergarten to grade eight are large enough to close the performance gap between (predominantly white, upper-class) children in Scarsdale and (predominantly minority, low-income) children in Harlem by 66% in English and 86% in math.  Reardon points out that these figures are based on unrealistic extrapolations.  You can’t simply add up the annual effects of attending a charter school from year to year because the gains decay over time.  Moreover, most of the students in the Hoxby study have been in charter schools for only three or four years;  virtually none have been enrolled in charter schools for as many as nine years, and those would only have been enrolled in the very small number of charter schools that have been open that long, and cannot tell us about the long-term effects of attending the much larger number of newer charter schools.  Reardon’s analysis suggests that Hoxby’s estimate of the cumulative effect of attending a charter school from grades four through eight could be exaggerated by as much as 50%.

Are Caroline Hoxby’s estimates of the effects of attending a charter school rather than a traditional public school in New York City accurate?  Maybe.  But based on Sean Reardon’s critique, probably not.  Hey, you never know.

First Person

I’m a Bronx teacher, and I see up close what we all lose when undocumented students live with uncertainty

The author at her school.

It was our high school’s first graduation ceremony. Students were laughing as they lined up in front of the auditorium, their families cheering them on as they entered. We were there to celebrate their accomplishments and their futures.

Next to each student’s name on the back of those 2013 graduation programs was the college the student planned to attend in the fall. Two names, however, had noticeable blanks next to them.

But I was especially proud of these two students, whom I’ll call Sofia and Isabella. These young women started high school as English learners and were diagnosed with learning disabilities. Despite these obstacles, I have never seen two students work so hard.

By the time they graduated, they had two of the highest grade point averages in their class. It would have made sense for them to be college-bound. But neither would go to college. Because of their undocumented status, they did not qualify for financial aid, and, without aid, they could not afford it.

During this year’s State of the Union, I listened to President Trump’s nativist rhetoric and I thought of my students and the thousands of others in New York City who are undocumented. President Trump falsely portrayed them as gang members and killers. The truth is, they came to this country before they even understood politics and borders. They grew up in the U.S. They worked hard in school. In this case, they graduated with honors. They want to be doctors and teachers. Why won’t we let them?

Instead, as Trump works to repeal President Obama’s broader efforts to enfranchise these young people, their futures are plagued by uncertainty and fear. A Supreme Court move just last week means that young people enrolled in the Deferred Action for Childhood Arrivals program remain protected but in limbo.

While Trump and the Congress continue to struggle to find compromise on immigration, we have a unique opportunity here in New York State to help Dreamers. Recently, the Governor Cuomo proposed and the state Assembly passed New York’s DREAM Act, which would allow Sofia, Isabella, and their undocumented peers to access financial aid and pursue higher education on equal footing with their documented peers. Republicans in the New York State Senate, however, have refused to take up this bill, arguing that New York state has to prioritize the needs of American-born middle-class families.

This argument baffles me. In high school, Sofia worked hard to excel in math and science in order to become a radiologist. Isabella was so passionate about becoming a special education teacher that she spent her free periods volunteering with students with severe disabilities at the school co-located in our building.

These young people are Americans. True, they may not have been born here, but they have grown up here and seek to build their futures here. They are integral members of our communities.

By not passing the DREAM Act, it feels like lawmakers have decided that some of the young people that graduate from my school do not deserve the opportunity to achieve their dreams. I applaud the governor’s leadership, in partnership with the New York Assembly, to support Dreamers like Sofia and Isabella and I urge Senate Republicans to reconsider their opposition to the bill.

Today, Sofia and Isabella have been forced to find low-wage jobs, and our community and our state are the poorer for it.

Ilona Nanay is a 10th grade global history teacher and wellness coordinator at Mott Hall V in the Bronx. She is also a member of Educators for Excellence – New York.

First Person

I was an attorney representing school districts in contract talks. Here’s why I hope the Supreme Court doesn’t weaken teachers unions.

PHOTO: Creative Commons / supermac1961

Many so-called education reformers argue that collective bargaining — and unions — are obstacles to real change in education. It’s common to hear assertions about how “restrictive” contracts and “recalcitrant” unions put adult interests over children’s.

The underlying message: if union power were minimized and collective bargaining rights weakened or eliminated, school leaders would be able to enact sweeping changes that could disrupt public education’s status quo.

Those that subscribe to this view are eagerly awaiting the Supreme Court’s decision in the case of Janus v. American Federation of State, County, and Municipal Employees. At issue is the constitutionality of “agency” or “fair share” fees — employee payroll deductions that go to local unions, meant to cover the costs of negotiating and implementing a bargaining agreement.

In states that permit agency fees (there are about 20), a teacher may decline to be part of a union but must still pay those fees. If the Supreme Court rules that those agency fees are unconstitutional, and many teachers do not voluntarily pay, local unions will be deprived of resources needed to negotiate and enforce bargaining agreements.

Based on my experience as an attorney representing school districts in bargaining and contract issues, I have this to say to those hoping the Court will strike down these fees: be careful what you wish for.

Eliminating fair share fees (and trying to weaken unions) represents a misguided assumption about bargaining — that the process weakens school quality. To the contrary, strong relationships with unions, built through negotiations, can help create the conditions for student and school success. Indeed, in my experience, the best superintendents and school boards seized bargaining as an opportunity to advance their agenda, and engaged unions as partners whenever possible.

Why, and how, can this work? For one, the process of negotiations provides a forum for school leaders and teachers to hear one another’s concerns and goals. In my experience, this is most effective in districts that adopt “interest-based bargaining,” which encourages problem-solving as starting point for discussions as opposed to viewing bargaining as a zero-sum game.

Interest-based bargaining begins with both sides listing their major concerns and brainstorming solutions. The touchstone for a solution to be adopted in a bargaining agreement: Is the proposal in the best interests of children? This important question, if embedded in the process, forces both sides to carefully consider their shared mission.

For example, some districts I worked with paid teachers less than comparable neighboring districts did. It would have been unreasonable for unions to insist that their pay be increased enough to even that difference out, because that would mean reducing investments in other items of importance to children, like technology or infrastructure. At the same time, it would have been untenable for management to play “hard ball” and deny the problem, because to do so would likely lead to a disgruntled workforce.

Instead, both sides were forced to “own” the issue and collaboratively craft plausible solutions. That made unions more agreeable to proposals that demonstrated some commitment by the district to addressing the issue of pay, and districts open to other things that they could provide without breaking the budget (like more early release days for professional development).

To be sure, many school administrators could get frustrated with the process of bargaining or having to consult the negotiated agreement when they want to make a change. Some districts would very much like to adopt an extended school day, for example, but they know that they must first consult and negotiate such an idea with the union.

Yet, in districts where school administrators had built a reservoir of goodwill through collective bargaining, disagreement does not come at the cost of operating schools efficiently. Both sides come to recognize that while they inevitably will disagree on some things, they can also seek agreement — and often do on high-stakes matters, like teacher evaluations.

How does this relate to the Supreme Court’s pending decision? Without fees from some teachers, unions may lack the resources to ensure that contract negotiations and enforcement are robust and done well. This could create a vicious cycle: teachers who voluntarily pay fees for bargaining in a post-Janus world, assuming the court rules against the unions, will view such payments as not delivering any return on investment. In turn, they will stop contributing voluntarily, further degrading the quality of the union’s services.

Even more troubling, if fair share fees are prohibited, resentment and internal strife will arise between those who continue to pay the fees and those who refuse. This would undercut a primary benefit of bargaining — labor peace and a sense of shared purpose.

Speaking as a parent, this raises a serious concern: who wants to send their child to a school where there is an undercurrent of bitterness between teachers and administrators that will certainly carry over into the classroom?

It is easy to see the appeal of those opposing agency fees. No one wants to see more money going out of their paycheck. The union-as-bogeyman mentality is pervasive. Moreover, in my experience, some teachers (especially the newer ones) do not recognize the hidden benefits to bargaining contracts.

But, obvious or not, agency fees help promote a stable workplace that allows teachers to concentrate on their primary responsibility: their students. Removing the key ingredient threatens this balance.

Mark Paige is a former school teacher and school law attorney who represented school districts in New England. He is currently an associate professor of public policy at the University of Massachusetts – Dartmouth.