Heaps of conflicting studies exist on whether teacher incentive pay improves student performance. Now after wading through decades of findings, researchers at Vanderbilt University have come to a conclusion.

It does.

Student test scores have a modest but statistically significant improvement when an incentive pay plan is in place for their teachers, say researchers who analyzed findings from 44 primary studies between 1997 and 2016.

“Approximately 74 percent of the effect sizes recorded in our review were positive. The influence was relatively similar across the two subject areas, mathematics and English language arts,” said Matthew Springer, assistant professor of public policy and education at Vanderbilt’s Peabody College of Education and Human Development.

The academic increase is roughly equivalent to adding three weeks of learning to the school year, based on studies conducted in U.S. schools, and four weeks based on studies across the globe.

But the 16-month analysis, released on Wednesday, also notes some important caveats that may help explain why different studies seem to waffle on the effect of pay-for-performance plans.

“We found that effect sizes are highly sensitive to program design and study context, so there are many issues that policymakers should look at when they’re considering or implementing a merit pay plan,” said researcher Lam D. Pham.

For instance, the evidence suggests that rewarding teachers as a group is almost twice as effective as rewarding them individually based on rank-order.

The rollout is important, too. In one study that found no effect on student test scores in Chicago, researchers noted that teachers who could have been impacted didn’t appear to understand that a merit pay plan existed.

“I don’t think there’s one design that will work across the board,” Pham added. “You have to consider the local teacher labor market. For instance, designing an incentive plan for a rural area where there aren’t many teachers to hire or retain is going to look different than in somewhere like Washington, D.C., where there are tons of teachers available.”

Merit pay in education is a longstanding idea dating to the early 1700s in Great Britain, and numerous U.S. cities used some form in the early 1900s. The approach has garnered considerable attention in the last decade in America as the federal government has awarded more than $2 billion across some 30 states to design and launch teacher performance pay systems.

Teacher pay is significant because salaries account for nearly 60 percent of school expenses nationwide, and research is clear that teachers matter more to student achievement than any other aspect of schooling (although out-of-school factors matter more). About 95 percent of public school districts set teacher pay based on years of experience and highest degree earned, but merit pay advocates argue that the approach needs to change.

In Tennessee, dozens of school systems have tied salary increases to effectiveness in the classroom, including improved student test scores. The shift was enabled under a 2010 law passed as part of the state’s winning bid for a federal Race to the Top grant. But many teachers have said they don’t trust Tennessee’s evaluation system and are frustrated by the elevated focus on testing.

Pham emphasizes the importance of engagement, teacher support and working through questions about merit pay, which he says takes time.

“You can’t just do it in one or two years and expect to see huge gains,” he said.

“And if we don’t build capacity and support (for teachers), it won’t work,” he added, citing variables such as coaching, better resources, stronger school leadership and smaller classes. “I think where we should look next is pairing pay incentives with teacher supports.”

Emerging studies also suggest that merit pay can improve teacher recruitment and retention, which has been found to contribute to improving student performance, particularly in low-income areas.

You can find the full report here.