Voucher debate update

First study of Indiana’s voucher program — the country’s largest — finds it hurts kids’ math skills at first, but not over time

PHOTO: Scott Elliott
Cardinal Ritter High School has one of the highest percentages of students paying tuition with state-funded vouchers in Indiana.

A new analysis of Indiana’s school voucher program offers something for both sides in the heated debate about whether public money should be used to fund private school tuition.

The study, obtained by Chalkbeat, shows that students using a voucher saw math achievement fall on average, though students who remained in private school for four years improved to match or outperform public school students in math and English.

The results amount to a Rorschach test for advocates on either side of the issue.

“At the end of four years, English scores are slightly above where students started and math scores are statistically the same — so the trend line is heading the right way,” said Robert Enlow, the president of EdChoice, an Indianapolis-based group that backs voucher and tax credit programs.

“Indiana diverted millions of dollars for years from public schools to private school vouchers, resulting in negative or negligible results for student outcomes,” said American Federation of Teachers president Randi Weingarten in a statement. “This latest study of vouchers should be yet another red flag to [U.S. Education Secretary Betsy] DeVos that she is going down the wrong path.”

The report was provided by researchers Mark Berends and Joseph Waddington after Chalkbeat obtained an earlier version of the study through a public records request to the Indiana Department of Education.

The study has been submitted to a peer-reviewed journal but has not yet been published. (An initial description posted online —  later taken down — has drawn wide media attention, and the researchers also previously presented preliminary findings to separate gatherings of academics and school choice advocates.)

The authors declined to comment on the results but criticized Chalkbeat’s decision to release them.

“It does a disservice to social scientists who want to make sure their research passes peer review before being publicly released,” Berends of the University of Notre Dame and Waddington of the University of Kentucky wrote in an email.

Chalkbeat is publishing the research because it is on a matter of pressing public concern — whether low-income students given public dollars to attend private schools learn more than they would in public schools, as the Trump administration promises to push for more voucher programs like Indiana’s.

“In Indiana, we’ve seen some of the best pro-parent and pro-student legislation enacted in the country,” DeVos recently said, referring to the state’s private and charter school initiatives. Her former advocacy group, American Federation for Children, heavily backed Indiana’s school voucher program while she was the group’s chairperson.

Recent research has found that voucher programs can lead to drops in test scores, but some studies like those in D.C. and Louisiana only examine the first one or two years of the program. The latest analysis, the first statewide study of Indiana’s program, looks at four years of data — and offers evidence that judging programs by short-term results may be unfair.

A spokesperson for the Indiana Department of Education declined to comment on the results, saying the department would do so once the study completes the peer-review process.

Voucher students lose ground in math, but those who stick around see gains in English

The paper examines the first four years — from the 2011-12 school year to 2014-15 — of Indiana’s private school voucher program, the largest in the country.

The initiative was championed by former governor Mitch Daniels and expanded to include middle-class families under Mike Pence, now vice president. In Indiana, participating schools are largely religious, and unlike in some school choice programs, students take state tests and schools can be barred from accepting new voucher students for poor academic performance.

The researchers focus on low-income students in the middle or end of elementary school who switched from public schools to private schools using a voucher, and compare them to similar students who remained in public school.

Relative to low-income students in public schools, those whose family elected to use a voucher were more likely to be female, Latino, and an English-language learner, and less likely to be black or have a disability. The voucher students also had slightly higher initial test scores, though still below the state average.

Student achievement in English over time

Compared to other private school students in the state, voucher recipients were more racially diverse, more likely to be low-income, and had significantly lower test scores.

The study estimates that receiving a voucher led to moderate decreases in math test scores overall. Students who participated in the program for four consecutive years initially saw a drop, but by year four they had caught back up to their public school counterparts.

When looking at English scores, the data suggest that there was no impact, good or bad, of receiving a voucher on average. However, the subset of students who remained in the program for all four years appeared to be doing moderately better in English than those in public schools.

In contrast to students who stuck with the program for several years, those who eventually left private schools saw large decreases in achievement while they were using a voucher.

There were not major differences across students by ethnicity or gender. But students with disabilities saw significant decreases in English test scores, while Catholic schools improved English achievement.

(The study was funded in part by the Walton Foundation, which is a supporter of Chalkbeat. EdChoice is also a Chalkbeat funder. Learn more about our funding here.)

Study validates advocates’ argument not to rush to judgment based on early years

The analysis of the program joins recent research showing that voucher programs can hurt student achievement. But the study’s finding that students who remain in the program improve over time gives new credence to advocates who said it was unreasonable to judge a program based on only one or two years of data.

“The results obviously cast further doubt on proponents’ claims that awarding vouchers to low-income students will immediately boost their math and reading achievement, but they also indicate that the negative initial effects on test scores seen in Louisiana, Ohio, and now Indiana are less concerning than it might appear,” said Marty West, a professor at Harvard, who reviewed the paper at Chalkbeat’s request.

The authors of the study suggest that private schools have gotten better as they have acclimated to new students who were more disadvantaged than those they previously served.

“Over time, voucher students may adjust to their new schools, and private schools may make adjustments that better meet the educational needs of voucher students,” the authors write, though they note that their research can’t confirm either hypothesis.

Research also released Monday on year three of Louisiana’s voucher program showed that negative results in early years of the program dissipated for some students in some subjects, but found continued negative effects for those in younger grades.

“What’s interesting about the Indiana results is that they’re consistent with the Louisiana results … in that they start out pretty negative and get less negative over time,” said Matt Chingos of the Urban Institute who has studied private school scholarships in New York City.

West noted the large achievement drops for those leaving the program may not be a bad sign.

“The fact that the students who switch back were disproportionately those who saw big drops in achievement is encouraging,” he said. “It does suggest that any large negative effects of voucher programs on achievement could be to some extent self-correcting.”

Doug Harris, an economist at Tulane University who has been critical of DeVos, said the the new research in Indiana “still has to give pause to anyone pushing broad federal or statewide [voucher] programs.”

“There are still no examples of statewide programs producing overall positive academic effects,” he said.

Other researchers praise study, but point to limitations

The results come with several important caveats.

First, because vouchers were not assigned through random lottery — unlike in some state programs, like Louisiana’s — the researchers can’t be confident that the results only capture the impact of receiving a voucher, a point the study acknowledges.

“Choosing to apply for and receiving a voucher depends on the active choices of parents and their children,” Berends and Waddington write.

Other researchers said the Indiana study does a good job controlling for that selection bias, though.

“The study is well done,” said Harris. “They try many different methods and the results hold up well.” He did note that there was some evidence that low-income students who took a voucher were more advantaged than poor students in public schools, suggesting the possibility of an “upward bias” in the results.

Second, the Indiana research is only able to look at a subset of the thousands of students who have used a voucher in Indiana to date — late elementary and middle school, low-income students who switched from public to private school. Similarly, the researchers only had data on a small number of students remained in the program for four years. That’s a limitation in using the study to draw conclusions about other students, West said.

Enlow and West both noted that the study only measures academic success with state test scores.

That, West said, means that it “can’t speak to how voucher use may have affected other student outcomes or families’ satisfaction with their child’s school.”

Fact check

Why the school spending graph Betsy DeVos is sharing doesn’t mean what she says it does

PHOTO: U.S. Department of Education

U.S. Secretary of Education Betsy DeVos’s message at an event on Thursday was clear: education leaders have been throwing good money after bad policy.

To make the point, she shared a graph that appears to show that school spending has risen dramatically over the last three decades as student outcomes barely budged.

“Scores [are] continuing to muddle along, unremarkable,” she told one of her predecessors, Bill Bennett, at a Reagan Institute event. “And yet look at the spending. This is not something we’re going to spend our way out of.”

This visualization, which DeVos also shared on Twitter, has become a staple in some education policy circles. Even Bill Gates has offered a version of it. But the graph is misleading, and several recent studies have in fact shown that students gain — in higher test scores, graduation rates, and earnings as adults — when they attend schools that spend more money. And while DeVos highlighted scores in reading, gains in math have been fairly substantial over part of that period.

A spokesperson for DeVos did not immediately respond to a request for additional comment.

Here’s why it doesn’t really mean what DeVos says it does.

The biggest problem with the graphic is that it conflates correlation and causation.

Simply pointing to two trends that coincide doesn’t say anything about how one affects the other. It’s possible that if education spending hadn’t increased, test scores would have actually declined. Maybe the extra money actually stopped a dramatic deterioration in school quality, for example. There’s just no way to know based on this data.

Here’s another example: Over the past couple decades, charter schools and vouchers have expanded dramatically. By DeVos’s own logic, those would be failed reforms because they didn’t coincide with a sufficient increase in NAEP scores. But that would also be an unfair conclusion, since we don’t know what would have happened without the increase in choice programs.

The two trends are also put side-by-side in an arbitrary way.

This graph has two y axes: one based on school spending, and the other based on scale scores on the National Assessment of Educational Progress. But these two variables are very different. Should an extra $1,000 correspond with 1 point on NAEP scores? Five points?

The decision is subjective, but it determines how large the spending changes look and how small the NAEP changes look, too.

Spending is up in education, but it’s up elsewhere too.

It’s true that in inflation-adjusted dollars, school spending has risen. The economy has also grown over that same time. That matters, because if spending were to hold steady in education while increasing in other areas of the economy, it might become increasingly hard to recruit and retain teachers of equal quality.

Indeed, while education spending has risen, teacher pay compared to similar professionals has fallen. (At the same time, the number of teachers and the number of non-teaching staff in schools has increased significantly, which some have argued illustrate bad spending priorities.)

The story would also look different for math scores.

NAEP scores have been relatively flat for the past decade, but in math they increased substantially the decade beforehand. In fact, a graph could be made showing a tight correspondence between math results and spending.

That’s exactly what Northwestern University economist Kirabo Jackson did. His approach was similar to DeVos’s, with a few changes. He compared NAEP scores in math to the cumulative spending on the average student in the years leading up to the test — the idea being that scores reflect not just the last year of schooling, but several years before that.

Source: Education Next

His graphs look quite different than DeVos’s, simply by tightening the range of scores, looking at math rather than reading, and tweaking the spending measure. (Jackson said that a similar pattern exists for reading, but that it’s less pronounced.)

Recent research has generally shown that school spending really does matter.

To be clear, Jackson’s graphs are also correlational, and like DeVos’s cannot prove cause and effect.

That’s why he and others have conducted sophisticated statistical analyses to isolate the impact of resources on schools. In one recent study, Jackson and colleagues compared states that made more spending cuts in the wake of Great Recession, and found that they did worse on NAEP as a result. In other words, there is a decent case that the recent stagnation in NAEP scores is due at least in part to spending cuts.

That jibes with a larger body of research. Other recent national studies have linked more spending to better NAEP scores, higher graduation rates, better economic outcomes, and greater social mobility. Studies in specific states including California, MassachusettsOhio, and New York have also found test score gains from spending increases.

Of course, none of that proves that the additional money spent over the past few decades have been well used. It doesn’t even prove that those extra dollars have been helpful — the above studies are of specific places or time periods, and their individual approaches could be disputed, too.

There is one recent study, which DeVos often cites, showing that a federal turnaround program that involved a substantial infusion of money for struggling schools didn’t make a noticeable difference.

Still, the graph that Devos offers is not, on its own, a strong case that resources for schools don’t make a difference. Available evidence suggests the opposite.

“If reduced spending contributed to the problem, it seems reasonable to conclude that increased spending may alleviate it,” said Jackson, the Northwestern researcher. “This is not to say that increased spending is some silver bullet, but rather that it should probably be part of a comprehensive set of policies to improve U.S schools.”

after douglas

Betsy DeVos avoids questions on discrimination as school safety debates reach Congress

Education Secretary Betsy DeVos prepares to testify at a House Appropriations Labor, Health and Human Services, Education and Related Agencies Subcommittee hearing in Rayburn Building on the department's FY2019 budget on March 20, 2018. (Photo By Tom Williams/CQ Roll Call)

U.S. Education Secretary Betsy DeVos fielded some hostile questions on school safety and racial discrimination as she defended the Trump administration’s budget proposal in a House committee hearing on Tuesday.

The tone for the hearing was set early by ranking Democrat Rep. Rosa DeLauro, who called aspects DeVos’s prepared remarks “misleading and cynical” before the secretary had spoken. Even the Republican subcommittee chair, Rep. Tom Cole, expressed some skepticism, saying he was “concerned about the administration continuing to request cuts that Congress has rejected.”

During nearly two hours of questioning, DeVos stuck to familiar talking points and largely side-stepped the tougher queries from Democrats, even as many interrupted her.

For instance, when Rep. Barbara Lee, a Democrat from California, complained about proposed spending cuts and asked, “Isn’t it your job to ensure that schools aren’t executing harsher punishments for the same behavior because [students] are black or brown?” DeVos responded by saying that students of color would benefit from expanded school choice programs.

Lee responded: “You still haven’t talked about the issue in public schools as it relates to black and brown students and the high disparity rates as it relates to suspensions and expulsions. Is race a factor? Do you believe that or not?” (Recent research in Louisiana found that black students receive longer suspensions than white students involved in the same fights, though the difference was very small.)

Again, DeVos did not reply directly.

“There is no place for discrimination and there is no tolerance for discrimination, and we will continue to uphold that,” she said. “I’m very proud of the record of the Office of Civil Rights in continuing to address issues that arise to that level.”

Lee responded that the administration has proposed cuts to that office; DeVos said the reduction was modest — less than 1 percent — and that “they are able to do more with less.”

The specific policy decision that DeVos faces is the future of a directive issued in 2014 by the Obama administration designed to push school districts to reduce racial disparities in suspensions and expulsions. Conservatives and some teachers have pushed DeVos to rescind this guidance, while civil rights groups have said it is crucial for ensuring black and Hispanic students are not discriminated against.

That was a focus of another hearing in the House on Tuesday precipitated by the shooting last month at Marjory Stoneman Douglas High School in Parkland, Florida.

Rep. Jim Sensenbrenner, a Wisconsin Republican, falsely claimed in his opening statement that Broward County Public Schools rewrote its discipline policy based on the federal guidance — an idea that has percolated through conservative media for weeks and been promoted by other lawmakers, including Florida Sen. Marco Rubio and Utah Sen. Mike Lee. In fact, the Broward County rules were put into place in 2013, before the Obama administration guidance was issued.

The Manhattan Institute’s Max Eden, a leading critic of Obama administration’s guidance, acknowledged in his own testimony that the Broward policy predated these rules. But he suggested that policies like Broward’s and the Obama administration’s guidance have made schools less safe.

“Faced with pressure to get the numbers down, the easiest path is to simply not address, or to not record, troubling, even violent, behavior,” he said.

Kristen Harper, a director with research group Child Trends and a former Obama administration official, disagreed. “To put it simply, neither the purpose nor the letter of the federal school discipline guidance restrict the authority of school personnel to remove a child who is threatening student safety,” she said.

There is little, if any, specific evidence linking Broward County’s policies to how Stoneman Douglas shooter Nicholas Cruz was dealt with. There’s also limited evidence about whether reducing suspensions makes schools less safe.

Eden pointed to a study in Philadelphia showing that the city’s ban on suspensions coincided with a drop in test scores and attendance in some schools. But those results are difficult to interpret because the prohibition was not fully implemented in many schools. He also cited surveys of teachers expressing concerns about safety in the classroom including in Oklahoma CityFresno, California; and Buffalo, New York.

On the other hand, a recent study found that after Chicago modestly reduced suspensions for the most severe behaviors, student test scores and attendance jumped without any decline in how safe students felt.

DeVos is now set to consider the repeal of those policies on the Trump administration’s school safety committee, which she will chair.

On Tuesday, DeVos said the committee’s first meeting would take place “within the next few weeks.” Its members will be four Cabinet secretaries: DeVos herself, Attorney General Jeff Sessions, Secretary of Health and Human Services Alex Azar, and Secretary of Homeland Security Kirstjen Nielsen.

Correction: An earlier version of this story said that Rep. Barbara Lee was from Texas; in fact, she represents a California congressional district.