a high-stakes evaluation

The Gates Foundation bet big on teacher evaluation. The report it commissioned explains how those efforts fell short.

PHOTO: Brandon Dill/The Commercial Appeal
Sixth-grade teacher James Johnson leads his students in a gameshow-style lesson on energy at Chickasaw Middle School in 2014 in Shelby County. The district was one of three that received a grant from the Gates Foundation to overhaul teacher evaluation.

Barack Obama’s 2012 State of the Union address reflected the heady moment in education. “We know a good teacher can increase the lifetime income of a classroom by over $250,000,” he said. “A great teacher can offer an escape from poverty to the child who dreams beyond his circumstance.”

Bad teachers were the problem; good teachers were the solution. It was a simplified binary, but the idea and the research it drew on had spurred policy changes across the country, including a spate of laws establishing new evaluation systems designed to reward top teachers and help weed out low performers.

Behind that effort was the Bill and Melinda Gates Foundation, which backed research and advocacy that ultimately shaped these changes.

It also funded the efforts themselves, specifically in several large school districts and charter networks open to changing how teachers were hired, trained, evaluated, and paid. Now, new research commissioned by the Gates Foundation finds scant evidence that those changes accomplished what they were meant to: improve teacher quality or boost student learning.  

The 500-plus page report by the Rand Corporation, released Thursday, details the political and technical challenges of putting complex new systems in place and the steep cost — $575 million — of doing so.

The post-mortem will likely serve as validation to the foundation’s critics, who have long complained about Gates’ heavy influence on education policy and what they call its top-down approach.

The report also comes as the foundation has shifted its priorities away from teacher evaluation and toward other issues, including improving curriculum.

“We have taken these lessons to heart, and they are reflected in the work that we’re doing moving forward,” the Gates Foundation’s Allan Golston said in a statement.

The initiative did not lead to clear gains in student learning.

At the three districts and four California-based charter school networks that took part of the Gates initiative — Pittsburgh; Shelby County (Memphis), Tennessee; Hillsborough County, Florida; and the Alliance-College Ready, Aspire, Green Dot, and Partnerships to Uplift Communities networks — results were spotty. The trends over time didn’t look much better than similar schools in the same state.

Several years into the initiative, there was evidence that it was helping high school reading in Pittsburgh and at the charter networks, but hurting elementary and middle school math in Memphis and among the charters. In most cases there were no clear effects, good or bad. There was also no consistent pattern of results over time.

A complicating factor here is that the comparison schools may also have been changing their teacher evaluations, as the study spanned from 2010 to 2015, when many states passed laws putting in place tougher evaluations and weakening tenure.

There were also lots of other changes going on in the districts and states — like the adoption of Common Core standards, changes in state tests, the expansion of school choice — making it hard to isolate cause and effect. Studies in Chicago, Cincinnati, and Washington D.C. have found that evaluation changes had more positive effects.

Matt Kraft, a professor at Brown who has extensively studied teacher evaluation efforts, said the disappointing results in the latest research couldn’t simply be chalked up to a messy rollout.

These “districts were very well poised to have high-quality implementation,” he said. “That speaks to the actual package of reforms being limited in its potential.”

Principals were generally positive about the changes, but teachers had more complicated views.

From Pittsburgh to Tampa, Florida, the vast majority of principals agreed at least somewhat that “in the long run, students will benefit from the teacher-evaluation system.”

Source: RAND Corporation

Teachers in district schools were far less confident.

When the initiative started, a majority of teachers in all three districts tended to agree with the sentiment. But several years later, support had dipped substantially. This may have reflected dissatisfaction with the previous system — the researchers note that “many veteran [Pittsburgh] teachers we interviewed reported that their principals had never observed them” — and growing disillusionment with the new one.

Majorities of teachers in all locations reported that they had received useful feedback from their classroom observations and changed their habits as a result.

At the same time, teachers in the three districts were highly skeptical that the evaluation system was fair — or that it made sense to attach high-stakes consequences to the results.

The initiative didn’t help ensure that poor students of color had more access to effective teachers.

Part of the impetus for evaluation reform was the idea, backed by some research, that black and Hispanic students from low-income families were more likely to have lower-quality teachers.  

But the initiative didn’t seem to make a difference. In Hillsborough County, inequity expanded. (Surprisingly, before the changes began, the study found that low-income kids of color actually had similar or slightly more effective teachers than other students in Pittsburgh, Hillsborough County, and Shelby County.)

Districts put in place modest bonuses to get top teachers to switch schools, but the evaluation system itself may have been a deterrent.

“Central-office staff in [Hillsborough County] reported that teachers were reluctant to transfer to high-need schools despite the cash incentive and extra support because they believed that obtaining a good VAM score would be difficult at a high-need school,” the report says.

Evaluation was costly — both in terms of time and money.

The total direct cost of all aspects of the program, across several years in the three districts and four charter networks, was $575 million.

That amounts to between 1.5 and 6.5 percent of district or network budgets, or a few hundred dollars per student per year. Over a third of that money came from the Gates Foundation.

The study also quantifies the strain of the new evaluations on school leaders’ and teachers’ time as costing upwards of $200 per student, nearly doubling the the price tag in some districts.

Teachers tended to get high marks on the evaluation system.

Before the new evaluation systems were put in place, the vast majority of teachers got high ratings. That hasn’t changed much, according to this study, which is consistent with national research.

In Pittsburgh, in the initial two years, when evaluations had low stakes, a substantial number of teachers got low marks. That drew objections from the union.

“According to central-office staff, the district adjusted the proposed performance ranges (i.e., lowered the ranges so fewer teachers would be at risk of receiving a low rating) at least once during the negotiations to accommodate union concerns,” the report says.

Morgaen Donaldson, a professor at the University of Connecticut, said the initial buy-in followed by pushback isn’t surprising, pointing to her own research in New Haven.

To some, aspects of the initiative “might be worth endorsing at an abstract level,” she said. “But then when the rubber hit the road … people started to resist.”

More effective teachers weren’t more likely to stay teaching, but less effective teachers were more likely to leave.

The basic theory of action of evaluation changes is to get more effective teachers into the classroom and then stay there, while getting less effective ones out or helping them improve.

The Gates research found that the new initiatives didn’t get top teachers to stick around any longer. But there was some evidence that the changes made lower-rated teachers more likely to leave. Less than 1 percent of teachers were formally dismissed from the places where data was available.

After the grants ran out, districts scrapped some of the changes but kept a few others.

One key test of success for any foundation initiative is whether it is politically and financially sustainable after the external funds run out. Here, the results are mixed.

Both Pittsburgh and Hillsborough have ended high-profile aspects of their program: the merit pay system and bringing in peer evaluators, respectively.

But other aspects of the initiative have been maintained, according to the study, including the use of classroom observation rubrics, evaluations that use multiple metrics, and certain career-ladder opportunities.

Donaldson said she was surprised that the peer evaluators didn’t go over well in Hillsborough. Teachers unions have long promoted peer-based evaluation, but district officials said that a few evaluators who were rude or hostile soured many teachers on the concept.

“It just underscores that any reform relies on people — no matter how well it’s structured, no matter how well it’s designed,” she said.

Correction: A previous version of this story stated that about half of the money for the initiative came from the Gates Foundation; in fact, the foundation’s share was 37 percent or about a third of the total.

Future of Teaching

Tentative contract includes big raises for IPS teachers

PHOTO: Dylan Peers McCoy
Teachers would receive significant raises under a tentative new contract with IPS.

A month after voters approved a vast funding increase for Indianapolis Public Schools, Superintendent Lewis Ferebee’s administration and the district teachers union have reached a tentative deal for a new contract that would boost teacher pay by an average of 6.3 percent.

The agreement was ratified by union members Wednesday, according to a statement from teachers union president Ronald Swann. It must be approved by the Indianapolis Public Schools board, which is likely to consider the contract next week, before it is final.

Swann did not provide details of the agreement, but it was outlined in union presentations to teachers on Wednesday ahead of the ratification vote. The deal would cover the 2018-19 school year, and teachers would receive retroactive pay back to July 2018. The prior contract ended in June.

Raising teacher pay was a key part of the sales pitch district leaders used to win support for a referendum to raise $220 million over eight years from taxpayers for operating expenses. The referendum passed with wide support from voters last month, and although the district will not get that money until next year, the administration can now bank on an influx of cash in June 2019. Teachers could receive another raise next year, once the money from the referendum begins flowing.

The proposed deal would bring pay raises for new and experienced teachers. First year teachers in the district would see their salaries jump to $42,587, about $2,600 above the current base salary, according to the presentation to teachers. Returning teachers would move up the pay scale, with most receiving raises of about $2,600.

The deal also brings a reward for teachers who are at the top of the current scale. The top of the scale would rise to $74,920 by adding several stops above the current maximum of $59,400. That means teachers who are currently at the top of the scale would be able to move up and continue getting raises.

Many longtime teachers in the district also earn additional pay for advanced education, but teachers who joined the district more recently are not eligible for that extra money.

Teachers who received evaluations of ineffective or needs improvement in 2017-18 are not eligible for raises.

The new contract is the second time in recent years that teachers have won substantial raises in Indianapolis Public Schools. After four years of painful pay freezes, Ferebee negotiated a contract in 2015 that included a large pay increase. Teacher pay is especially important for the district because it is competing with several surrounding communities to staff schools.

Health care costs would go up this year, a policy shift that was advocated by the Indy Chamber, which urged the district to reduce health insurance spending as part of a plan to shift more money to teacher salaries.

The contract includes a provision that was piloted last year allowing the district to place newly hired teachers at anywhere on the salary schedule. It’s designed to allow the district to pay more for especially hard-to-fill positions.

Teachers at some troubled schools, known as the transformation zone, would also be eligible for extra pay on top of their regular salaries at the discretion of the administration. That money would come from state grants specifically targeted at transformation zone schools.

The idea of allowing superintendents to pay some teachers in their districts more than others is controversial.

Teacher Pay

‘Our teachers have waited long enough’: Educators say Indiana needs to act now on teacher pay

PHOTO: Shaina Cavazos
Students in Decatur Township work on physics problems with their teacher.

Educators and advocates are pushing state leaders to take action this year to raise teacher compensation — not to wait for additional research, as Gov. Eric Holcomb proposed last week.

“Our teachers have waited long enough,” said Teresa Meredith, president of the Indiana State Teachers Association, the state’s largest teachers union. “It doesn’t take a two-year study to discover what we already know: teachers need to be valued, respected, and paid as professionals.”

Holcomb’s proposal last week to study raises in the upcoming budget-writing session and make bigger steps in 2021 didn’t sit well with some, since lawmakers and advocates spent the fall talking up the need to make teacher salaries competitive with other states. But given the state’s tight budget situation, Holcomb suggested studying the impact of raises for at least a year, as well as looking at how much money would be needed and how districts would be expected to get the money to teachers.

Read: Raising teacher pay likely to be at the forefront for Indiana lawmakers and advocates in 2019

The proposal drew quick criticism. Education leaders and advocacy groups took to Twitter to express their hopes that Holcomb and lawmakers would find ways to address teacher salaries this year as well as into the future.

“IN must respond now,” State Superintendent Jennifer McCormick tweeted Friday morning, remarking that too many teachers across the state are leaving the profession because pay is too low. “Kids deserve & depend upon excellent teachers.”

“We can’t wait to act because Hoosier children are counting on all us to come together to ensure our schools can attract and retain the best teachers,” Justin Ohlemiller, executive director of Stand for Children Indiana, said in a blog post titled “The time to act on teacher pay is now.

ISTA’s 2019 legislative agenda, released Monday, will continue pushing for lawmakers and state leaders to find creative solutions to raise teacher pay and make Indiana competitive with other states.

And ISTA says they might have voters on their side. A recent ISTA poll of more than 600 Hoosiers, conducted by Emma White Research, shows that funding for education is a priority across the state, with more than 86 percent of those sampled supporting sending more money to public schools. About 72 percent of people polled believe educators are underpaid.

But it’s unclear if there would be enough money in the budget to spend on across-the-board raises after other funding obligations are met, such as funding needed by the Department of Child Services to deal with effects of the state’s opioid crisis. Senate Democrats have called for $81 million a year to ensure 5 percent raises for teachers and counselors over the next two years. Republicans have strong majorities in both chambers.

Neither ISTA, lawmakers, Holcomb nor other education groups have released specific plans for either how much they’d like to see set aside for teachers or strategies for how a pay increase could feasibly be carried out. However, the effort has brought together some unlikely allies — the union, a vocal advocate for traditional public schools, rarely aligns its education policy with groups like Stand and Teach Plus Indiana that have favored increased school-choice options, such as charter schools.

With limited dollars to go around, the focus will have to also be on how to make existing education dollars go farther, Meredith said. She, along with Republican House Speaker Brian Bosma last month, pointed to the need to curtail spending on administration, which, they argue, could free up money for other expenses such as teacher compensation.

Some have also pointed to the state’s recent budget surplus and reserves as evidence that Indiana could spend more on education if there was political will to do so.

“The surplus has come on the backs of educators and their students,” Meredith said. “Elected leaders must do more. They must do more to declare teacher pay a priority in this session, and they must take action.”

ISTA is also hoping lawmakers will act to:

  • Restore collective bargaining rights so educators can negotiate work hours and class size, as well as salaries and benefits.
  • Remove teacher evaluation results from decisions about salary until the state’s new ILEARN test has been in place for a few years.
  • Invest in school counselors, psychologists, and social workers
  • Strengthen regulations for charter and virtual charter schools, including putting a moratorium on new virtual schools until those safeguards can be enacted.
  • Study districts that have focused on how to best teach students who have experienced trauma.

Indiana’s next legislative session begins in January.

Correction: Dec. 11, 2018: This story has been updated to reflect that Stand for Children Indiana doesn’t take a position in regards to private school vouchers.