As Indianapolis Public Schools pursues a radical new approach to saving its most troubled schools, district leaders face an increasingly pressing question: Are their reforms working?

In the first year at the first failing school to be restarted as an “innovation” school, the number of students who passed state tests actually dropped. But whether or not one year of test scores says much about a school depends on who you ask.

It is “absolutely” concerning that student passing rates dropped at School 103 in its first year in the district’s innovation program, said Gary Henry, a professor at Vanderbilt who has evaluated turnaround efforts in Tennessee and North Carolina.

“Most of the recent literature on turnover … showed positive results in the first year,” he said.

When the district converts struggling schools to “innovation” status, new principals are given full control over their budgets, staffing and curriculum. Teachers must reapply for jobs and they are not part of the IPS union. But the restarted schools are still part of the district and they still accept all neighborhood kids.

Since the innovation program began, IPS board has restarted three failing schools as innovation schools. The district’s first test case was School 103, where the charter network Phalen Leadership Academies took over management in 2015.

School supporters were likely hoping to see scores creep up a bit since Phalen took over but when ISTEP scores were released this month, the school saw its passing rate on ISTEP fall from 9.6 to 4.6 percent.

Henry says that’s a bad sign. In a recent review of research, he found that every turnaround that was successful had improvements in test scores and proficiency rates in the first year.

But other experts argue that it can take time for new or restarted schools to ramp up.

Brian Gill, a fellow with Mathematica Policy Research, lead a study of takeover schools that found that scores fell in the first year but rebounded in later years.

“I would not expect turnaround efforts that involve major changes to show improvements in student achievement in the first year,” Gill wrote in an email.

The leader of the charter network that runs School 103, Earl Phalen, takes ISTEP passing rates seriously, and he said that in the longterm, the school aims to exceed the state average. But for the first year, test scores weren’t their top priority. Instead, school officials were focused on creating a warm culture where students feel supported and encouraged to be leaders.

“You’ve got to get culture right first before you can really focus on the academics,” Phalen said. “Next year, we expect the state results to go up significantly.”

Some experts agree that the district shouldn’t expect immediate improvement in test scores. In part that’s because it could take a few years for their work to translate into test scores, said Steven Glazerman, who is also at Mathematica. For example, if a school is doing a good job in the early grades, it would be years before those students take the state standardized test.

But other researchers are more concerned by declining passing rates. Henry said that most of the research that shows it takes a few years for a turnaround school to improve was focused on less intense interventions when the teachers were retrained, rather than replaced.

At schools like the IPS innovation schools, where the existing teachers must reapply for their jobs, he said the district should expect to see immediate improvements in student test scores.

Despite his misgivings, Henry said that turnaround schools should be given at least a few years to make improvements before the district decides whether to renew their contracts.

Both Glazerman and Henry agree that there are other indicators the district can look to when it is evaluating restarted schools in the early years.

The turnaround schools Henry has studied where leaders recruit high-quality, experienced teachers and principals have improved significantly, he said. But turnaround schools with novice teachers and principals have actually had drops in performance.

“The disruption (to teaching staff) is probably needed, but the results will depend on what happens after the disruption,” Henry said. “Who is recruited? Are they experienced, high-quality, dedicated teachers and do they get compensation for the extra work that is required in these turnaround schools?”