Voucher debate update

First study of Indiana’s voucher program — the country’s largest — finds it hurts kids’ math skills at first, but not over time

PHOTO: Scott Elliott
Cardinal Ritter High School has one of the highest percentages of students paying tuition with state-funded vouchers in Indiana.

A new analysis of Indiana’s school voucher program offers something for both sides in the heated debate about whether public money should be used to fund private school tuition.

The study, obtained by Chalkbeat, shows that students using a voucher saw math achievement fall on average, though students who remained in private school for four years improved to match or outperform public school students in math and English.

The results amount to a Rorschach test for advocates on either side of the issue.

“At the end of four years, English scores are slightly above where students started and math scores are statistically the same — so the trend line is heading the right way,” said Robert Enlow, the president of EdChoice, an Indianapolis-based group that backs voucher and tax credit programs.

“Indiana diverted millions of dollars for years from public schools to private school vouchers, resulting in negative or negligible results for student outcomes,” said American Federation of Teachers president Randi Weingarten in a statement. “This latest study of vouchers should be yet another red flag to [U.S. Education Secretary Betsy] DeVos that she is going down the wrong path.”

The report was provided by researchers Mark Berends and Joseph Waddington after Chalkbeat obtained an earlier version of the study through a public records request to the Indiana Department of Education.

The study has been submitted to a peer-reviewed journal but has not yet been published. (An initial description posted online —  later taken down — has drawn wide media attention, and the researchers also previously presented preliminary findings to separate gatherings of academics and school choice advocates.)

The authors declined to comment on the results but criticized Chalkbeat’s decision to release them.

“It does a disservice to social scientists who want to make sure their research passes peer review before being publicly released,” Berends of the University of Notre Dame and Waddington of the University of Kentucky wrote in an email.

Chalkbeat is publishing the research because it is on a matter of pressing public concern — whether low-income students given public dollars to attend private schools learn more than they would in public schools, as the Trump administration promises to push for more voucher programs like Indiana’s.

“In Indiana, we’ve seen some of the best pro-parent and pro-student legislation enacted in the country,” DeVos recently said, referring to the state’s private and charter school initiatives. Her former advocacy group, American Federation for Children, heavily backed Indiana’s school voucher program while she was the group’s chairperson.

Recent research has found that voucher programs can lead to drops in test scores, but some studies like those in D.C. and Louisiana only examine the first one or two years of the program. The latest analysis, the first statewide study of Indiana’s program, looks at four years of data — and offers evidence that judging programs by short-term results may be unfair.

A spokesperson for the Indiana Department of Education declined to comment on the results, saying the department would do so once the study completes the peer-review process.

Voucher students lose ground in math, but those who stick around see gains in English

The paper examines the first four years — from the 2011-12 school year to 2014-15 — of Indiana’s private school voucher program, the largest in the country.

The initiative was championed by former governor Mitch Daniels and expanded to include middle-class families under Mike Pence, now vice president. In Indiana, participating schools are largely religious, and unlike in some school choice programs, students take state tests and schools can be barred from accepting new voucher students for poor academic performance.

The researchers focus on low-income students in the middle or end of elementary school who switched from public schools to private schools using a voucher, and compare them to similar students who remained in public school.

Relative to low-income students in public schools, those whose family elected to use a voucher were more likely to be female, Latino, and an English-language learner, and less likely to be black or have a disability. The voucher students also had slightly higher initial test scores, though still below the state average.

Student achievement in English over time

Compared to other private school students in the state, voucher recipients were more racially diverse, more likely to be low-income, and had significantly lower test scores.

The study estimates that receiving a voucher led to moderate decreases in math test scores overall. Students who participated in the program for four consecutive years initially saw a drop, but by year four they had caught back up to their public school counterparts.

When looking at English scores, the data suggest that there was no impact, good or bad, of receiving a voucher on average. However, the subset of students who remained in the program for all four years appeared to be doing moderately better in English than those in public schools.

In contrast to students who stuck with the program for several years, those who eventually left private schools saw large decreases in achievement while they were using a voucher.

There were not major differences across students by ethnicity or gender. But students with disabilities saw significant decreases in English test scores, while Catholic schools improved English achievement.

(The study was funded in part by the Walton Foundation, which is a supporter of Chalkbeat. EdChoice is also a Chalkbeat funder. Learn more about our funding here.)

Study validates advocates’ argument not to rush to judgment based on early years

The analysis of the program joins recent research showing that voucher programs can hurt student achievement. But the study’s finding that students who remain in the program improve over time gives new credence to advocates who said it was unreasonable to judge a program based on only one or two years of data.

“The results obviously cast further doubt on proponents’ claims that awarding vouchers to low-income students will immediately boost their math and reading achievement, but they also indicate that the negative initial effects on test scores seen in Louisiana, Ohio, and now Indiana are less concerning than it might appear,” said Marty West, a professor at Harvard, who reviewed the paper at Chalkbeat’s request.

The authors of the study suggest that private schools have gotten better as they have acclimated to new students who were more disadvantaged than those they previously served.

“Over time, voucher students may adjust to their new schools, and private schools may make adjustments that better meet the educational needs of voucher students,” the authors write, though they note that their research can’t confirm either hypothesis.

Research also released Monday on year three of Louisiana’s voucher program showed that negative results in early years of the program dissipated for some students in some subjects, but found continued negative effects for those in younger grades.

“What’s interesting about the Indiana results is that they’re consistent with the Louisiana results … in that they start out pretty negative and get less negative over time,” said Matt Chingos of the Urban Institute who has studied private school scholarships in New York City.

West noted the large achievement drops for those leaving the program may not be a bad sign.

“The fact that the students who switch back were disproportionately those who saw big drops in achievement is encouraging,” he said. “It does suggest that any large negative effects of voucher programs on achievement could be to some extent self-correcting.”

Doug Harris, an economist at Tulane University who has been critical of DeVos, said the the new research in Indiana “still has to give pause to anyone pushing broad federal or statewide [voucher] programs.”

“There are still no examples of statewide programs producing overall positive academic effects,” he said.

Other researchers praise study, but point to limitations

The results come with several important caveats.

First, because vouchers were not assigned through random lottery — unlike in some state programs, like Louisiana’s — the researchers can’t be confident that the results only capture the impact of receiving a voucher, a point the study acknowledges.

“Choosing to apply for and receiving a voucher depends on the active choices of parents and their children,” Berends and Waddington write.

Other researchers said the Indiana study does a good job controlling for that selection bias, though.

“The study is well done,” said Harris. “They try many different methods and the results hold up well.” He did note that there was some evidence that low-income students who took a voucher were more advantaged than poor students in public schools, suggesting the possibility of an “upward bias” in the results.

Second, the Indiana research is only able to look at a subset of the thousands of students who have used a voucher in Indiana to date — late elementary and middle school, low-income students who switched from public to private school. Similarly, the researchers only had data on a small number of students remained in the program for four years. That’s a limitation in using the study to draw conclusions about other students, West said.

Enlow and West both noted that the study only measures academic success with state test scores.

That, West said, means that it “can’t speak to how voucher use may have affected other student outcomes or families’ satisfaction with their child’s school.”

Fact check

To back up claim that schools must change, DeVos cites made-up statistic about the future of work

PHOTO: Gabriel Scarlett/The Denver Post

In a recent interview with the Wall Street Journal, Education Secretary Betsy DeVos made a remarkable claim: “Children starting kindergarten this year face a prospect of having 65 percent of the jobs they will ultimately fill not yet having been created.”

This statistic bolsters DeVos’s view that schools need to radically change to accommodate a rapidly evolving economy.

But there’s a problem: that number appears to have no basis in fact.

A version of the 65 percent claim has been percolating for some time, across the world. After a number of British politicians repeated some iteration of the statistic, the BBC investigated its source.

That report found the claim gained popularity in a 2011 book by Cathy Davidson, a CUNY professor; this in turn was cited by a New York Times article. But attempts to track that claim back to an actual study have failed, which Davidson herself now concedes, saying she no longer uses the figure.

Others making the claim offer an even flimsier citation. For instance, a report released by the World Economic Forum says, “By one popular estimate, 65% of children entering primary school today will ultimately end up working in completely new job types,” and simply cites a series of popular YouTube videos (which doesn’t even appear to make that precise claim).

Some even say the number is higher: A Huffington Post headline said that “85% Of Jobs That Will Exist In 2030 Haven’t Been Invented Yet.” The piece links to a report by Dell, which bases the claim on “experts” at a workshop organized by a group called Institute for the Future.

In short, no one has pointed to any credible research that lands on the 65 percent figure. When asked for a source for DeVos’s statistic, a spokesperson for the Department of Education said the 65 percent figure “might be an underestimation,” pointing to the Dell report, which offers no specific sourcing.

Of course, making predictions about the future of work is inherently tricky. But a recent report by the federal Bureau of Labor Statistics estimated areas where the most new jobs would be created between 2016 and 2026. The positions included software application developers but also personal care aides, nurses, fast food workers, home health aides, waiters, and janitors — and though that’s less than 10 years in the future, these are mostly jobs that have been around for some time.

Sweeping, unsourced claims like this about the future economy are not uncommon — and seem to be a driving force behind some policymakers’ approach to education. The fact that DeVos’s go-to number isn’t backed up by evidence raises questions about the foundation of her view that schools need dramatic overhaul.

After citing the 65 percent figure, DeVos continued, saying, “You have to think differently about what the role of education and preparation is.”

study up

Trump education nominee pleads ignorance about high-profile voucher studies showing negative results

At his confirmation hearing, Mick Zais, the nominee to be second-in-command at the Department of Education, said that he was not aware of high-profile studies showing that school vouchers can hurt student achievement.

It was a remarkable acknowledgement by Zais, who said he supports vouchers and would report to Education Secretary Betsy DeVos, whose signature issue has been expanding publicly funded private school choice programs.

The issue was raised by Minnesota Sen. Al Franken, who asked whether Zais, who was previously the South Carolina schools chief, was “aware of the research on the impact of vouchers on student achievement.”

He replied: “To the best of my knowledge, whenever we give parents an opportunity to choose a school that’s a good fit for their child the result is improved outcomes.”

Franken responded, “No, that’s not true. The academic outcomes for students who used vouchers to attend private school are actually quite abysmal.”

Franken proceeded to mention recent studies from Louisiana, Indiana, Ohio, and Washington, DC that showed declines in test scores after students move to private schools with a voucher.

Zais responded: “Senator, I was unaware of those studies that you cited.”

Franken then asked if Zais’s initial response expressing confidence in school choice was anecdotal, and Zais said that it was.

What’s surprising about Zais’s response is that these studies were not just published in dusty academic journals, but received substantial media attention, including in the New York Times and Washington Post (and Chalkbeat). They’ve also sparked significant debate, including among voucher supporters, who have argued against judging voucher programs based on short-term test scores.

Meanwhile, it’s worth noting that the research confusion was a bipartisan affair at Wednesday’s confirmation hearing.

Although Franken, who referred to a New York Times article on voucher research in his question, was broadly accurate in his description of the recent studies, he said that a DC voucher study showed “significantly lower math and reading scores”; in fact, the results were only statistically significant in math, not reading.

Franken also did not mention evidence that the initial negative effects abated in later years in Indiana and for some students in Louisiana, or discuss recent research linking Florida’s voucher-style tax credit program to higher student graduation rates.

In a separate exchange, Washington Sen. Patty Murray grilled Jim Blew — the administration’s nominee for assistant secretary for planning, evaluation, and policy development — on the performance of Michigan’s charter schools. Murray said that DeVos was “one of the architects of Detroit’s charter school system,” describing the results as “disastrous for children.”

Blew disputed this: “The characterization of the charter school sector in Detroit as being a disaster seems unfair. The most reliable studies are saying, indeed, the charter school students outperform the district students.”

Murray responded: “Actually, Michigan’s achievement rates have plummeted for all kids. In addition, charter schools in Michigan are performing worse than traditional public schools.”

(Murray may be referring to an Education Trust analysis showing that Michigan ranking on NAEP exams have fallen relative to other states. The study can’t show why, or whether school choice policies are the culprit, as some have claimed.)

Blew answered: “The most reliable studies do show that the charter school students in Detroit outperform their peers in the district schools.”

Murray: “I would like to see that because that’s not the data that we have.”

Blew: “I will be happy to get if for you; it’s done by the Stanford CREDO operation.”

Murray: “I’m not aware of that organization.”

CREDO, a Stanford-based research institution, has conducted among the most widely publicized — and sometimes disputed — studies of charter schools. The group’s research on Detroit does show that the city’s charter students were outperforming similar students in district schools, though the city’s students are among the lowest-performing in the country on national tests.