value added?

Eight years ago, the L.A. Times published teachers’ ratings. New research tells us what happened next.

In 2010, the Los Angeles Times did something that hadn’t been done before.

The newspaper published test-score data for thousands of the city’s public school teachers, assigning them a rating based on how they influenced students’ results.

It caused a firestorm. Critics, including many teachers, railed against the measures as misleading and poorly constructed, warning that the data would demoralize teachers. The L.A. Times itself defended the release as necessary transparency.

In an accompanying story, one local teacher suggested it might also help children by empowering parents “to demand a good teacher.”

New research suggests that’s what happened next — but only for certain families.

Publishing the scores meant already high-achieving students were assigned to the classrooms of higher-rated teachers the next year, the study found. That could be because affluent or well-connected parents were able to pull strings to get their kids assigned to those top teachers, or because those teachers pushed to teach the highest-scoring students.

In other words, the academically rich got even richer — an unintended consequence of what could be considered a journalistic experiment in school reform.

“You shine a light on people who are underperforming and the hope is they improve,” said Jonah Rockoff, a professor at Columbia University who has studied these “value-added” measures. “But when you increase transparency, you may actually exacerbate inequality.”

That analysis is one of a number of studies to examine the lasting effects of the L.A. Times’ decision to publish those ratings eight years ago. Together, the results offer a new way of understanding a significant moment in the national debate over how to improve education, when bad teachers were seen as a central problem and more rigorous evaluations as a key solution.

The latest study, by Peter Bergman and Matthew Hill and published last month in the peer-reviewed journal Economics of Education Review, found that the publication of the ratings caused a one-year spike in teacher turnover. That’s not entirely surprising, considering many teachers felt attacked by the public airing of their ratings.

“Guilty as charged,” wrote one teacher with a low rating. “I am proud to be ‘less effective’ than some of my peers because I chose to teach to the emotional and academic needs of my students. In the future it seems I am being asked to put my public image first.”

But a separate study, by Nolan Pope at the University of Maryland, finds the publication of the ratings may have had some positive effects on students, perhaps by encouraging schools to better support struggling teachers.

Pope’s research showed that Los Angeles teachers’ performance, as measured by their value-added scores, improved after their scores were published. The effects were biggest for the teachers whose initial scores were lowest, and there was no evidence that the improvement was due to “teaching to the test.”

“These results suggest the public release of teacher ratings could raise the performance of low-rated teachers,” Pope concluded.

The two studies offer divergent pictures of the consequences of L.A. Times’ move. Pope did not find that higher-scoring students moved into the classrooms of higher-scoring teachers, while Bergman and Hill didn’t find clear evidence that teachers improved.

Those varying results are not entirely surprising, since the researchers used different methods. Pope’s research compared the same teachers before and after their value-added scores were published. Bergman and Hill took advantage of the fact that the L.A. Times only published scores for teachers who taught 60 or more students between 2003 and 2009, creating a natural experiment. The researchers then compared teachers who had taught just more than 60 kids to those teachers who had taught just under 60.

Rockoff of Columbia said he found both studies credible.

A third study, published in 2016, looks at an entirely different question: Did housing prices in Los Angeles increase near schools with more highly rated teachers?

Not really, according to the paper. That’s somewhat surprising, because past research has shown that housing zoned for schools with higher overall test scores and ratings is more expensive.

The researchers suggest that that might be because families had a hard time understanding what the ratings represented, and that some may have tuned out because of the surrounding controversy.

The results come in a very different political climate than around the time of the public release of the scores, when conversations about teacher performance had reached a fever pitch.

In 2010, then-Secretary of Education Arne Duncan praised the publication of teacher ratings. He used federal carrots and sticks to encourage states to use student test scores as part of how teachers are judged, a policy most states adopted.

But since then, states like New York and Virginia have barred the public release of this performance data, while media organizations have increasingly shied away from publicizing them. The new teacher evaluation systems have run into political challenges, and in some cases not had the hoped-for effects on student performance. And the federal education law passed in 2016 specifically banned the secretary of education from pushing teacher evaluation rules.

resentment and hurt

‘We are all educators:’ How the teachers strike opened at a rift at one Denver middle school network that will take time to mend

PHOTO: Melanie Asmar/Chalkbeat
Students at Kepner Beacon Middle School work on an assignment.

For the first time since the Denver teacher strike exposed divisions in their ranks, the 100 adults who make the Beacon middle school network run gathered in the same room.

Teachers, some still wearing red for the union cause, brought breakfast burritos to share. Upbeat soul music pumped through the speakers, an attempt to set a positive tone.  

Speaking to the group assembled Friday for a long-scheduled planning day in the cafeteria of Grant Beacon Middle School, Executive Principal Alex Magaña opened by acknowledging the awkwardness that had taken a toll on a school community that prides itself on a strong culture.

Some teachers who had gone on strike — exhausted by the experience and exhilarated by the outcome — felt snubbed. Where was the celebration of what they had just fought for?

School administrators were smarting for another reason: A large number of teachers did not return to work on Thursday after the tentative pact was signed, making for another hard day.

Just as it was starting, the effort to heal the Beacon school community was stumbling a bit.

One day after the end of the three-day strike over teacher pay, Denver students had a day off Friday, giving leaders at the 147 district-managed schools affected by the strike the opportunity to begin repairing any damage done. The district administration shared resources with schools, too, including a “lessons learned” tipsheet from the recent Los Angeles teachers strike.

The challenge is proving unexpectedly daunting at the city’s two Beacon schools — Grant Beacon in east Denver and Kepner Beacon in southwest Denver — which share a common central administrative staff, approach, and mission to serve the city’s neediest students.

“It’s never been administration-versus-teachers, district-versus-teachers, in the culture we have created here,” said Magaña, who oversees the two schools. “We have a lot of good leadership, a lot of input from teachers. But this caught everyone kind of by surprise.”

By “this,” Magaña means the tension on the two campuses before, during, and after the strike that put Denver in an unfamiliar national glare. The 93,000-student district is better known for its unique brand of at times controversial education reform — of which the Beacon network is part — than it is labor strife and division in the educator ranks.

As it became evident that the teachers union was intent on striking, Magaña said he sent a message to his teachers, staff, and administrators.

“I called it out two weeks ago: Be careful with what you say, because it’s going to cause harm and impact our culture,” he said. “Everyone has their own right to make their own individual decision. Respect it. And people were trying to respect it.”

From Magaña’s perspective, it didn’t always happen. At Kepner Beacon, where 96 percent of students qualify for subsidized lunches, the young corps of teachers “grouped together and suddenly had this camaraderie, which is something that is part of our culture and that makes us successful,” Magaña said.

All but a few Kepner Beacon teachers went out on strike. Magaña harbored concerns, though, saying some striking teachers “were guilting teachers into joining for solidarity.” Teachers who crossed picket lines told him they felt alienated, he said.

Linsey Cobb, a special education teacher and special education team leader at Kepner Beacon, disputes that. She said every teacher wholeheartedly supported each teacher’s decision.

Cobb herself was torn about striking. She said she stood with teachers fighting for a system they believed would pay them a better, fairer wage. But the third-year teacher ended up reporting to work as usual Monday morning, feeling too strong of a pull to fulfill her responsibilities supporting students with individualized education plans — the complex and sometimes confounding binding documents for students with special needs.

Cobb said she was not fully prepared by what she experienced that morning.

“Even though I am very close with my students, I felt incredibly isolated,” she said. “I got the weirdest feeling. I got a lot of, ‘Miss, why aren’t you striking? Don’t you believe what teachers are fighting for?’ I was like, ‘I do!’ I had a little bit of an internal struggle.”

After attending the big teachers union rally Monday at the Capitol, Cobb said she woke up Tuesday and decided to join her colleagues picketing, which she did for the strike’s duration.

The strike brought to the forefront just how different the two Beacon campuses are. At Grant Beacon, 80 percent of students qualify for subsidized lunch — slightly above the district average. That part of the city, like much of Denver, is gentrifying. The southwest Denver neighborhood around Kepner is not. The school is a safe harbor from violence and trauma.

About half of Grant Beacon students showed up for school during the strike, and six in 10 teachers joined the strike. Four miles and a world away at Kepner Beacon, 90 percent of students showed up for school — and all but a few teachers were out on strike.

Against the backdrop of the strike, Magaña said he emphasized that words matter. Everyone in the buildings, he thought, not just teachers, ought to be considered educators. That was the role everyone was thrust into — administrators, deans, and district central office staff who through no choice of their own had to cover for absent teachers. Magaña, too. He taught math.

“We maintained a positive culture through a really weird and complicated time,” said Tristan Connett, who as Kepner’s dean of students was pressed into service to teach eighth-grade reading and language arts. “Not just for students, but all the adults, everyone included.”

Outside Kepner Beacon each morning of the strike, teachers huddled over donuts and coffee. Parents brought them hand-warmers in the 20-degree chill. One teacher sat in her car with the engine running recording a video message to her students, telling them where she was and spelling out the day’s lesson plan before she joined the picket line, Cobb said.

The Beacon schools promote character-building and use personalized learning, using data and technology to tailor instruction to individual students. As “innovation schools,” the schools are exempt from some state laws and aspects of the teachers union contract. Both schools were “green,” the second-highest ranking, in the district’s most recent school ratings.

Cracks in school culture did show during the strike. Magaña said one teacher at Grant Beacon was hurt by the negative reaction he received from striking colleagues.

The strike’s sudden end just after 6 a.m. Thursday led to mixed messages and confusion about what was expected of teachers that day, deepening rifts at the Beacon schools.

Cobb, the Kepner special education teacher, said teachers somehow got what turned out to be incorrect information from the union that they couldn’t be late for the start of school if they wanted to return.

Many striking teachers did not come back to school Thursday. That was out of step with the district as a whole, which saw the vast majority of teachers back in classrooms.

Some Beacon teachers, Magaña said, “said they were mentally and physically exhausted.” What, he asked, does that tell everyone who took on unfamiliar roles to keep the schools open?

When teachers, administrators, and staff arrived for Friday morning’s meeting, they congregated at tables with colored pencils and “reflection forms.” Everyone was asked to write down answers to two questions: What did you learn about yourself? What did you learn about your colleagues?

“I also brought out the obvious — the elephant in the room,” Magaña said. “There are hurt feelings. There is resentment from teachers to staff to students to parents. That is something we can’t pretend isn’t there, and we put it out there and acknowledge it to move forward.”

The message from the network administration left a number of teachers disappointed.

“Every teacher who went out on strike believed in it, we got this victory, and it wasn’t celebrated as a whole,” Cobb said. “It was more like there was an acknowledgement of what we want to repair. OK, but we felt like we deserved a little celebration for what we accomplished.”

Several teachers took up administrators’ offers to speak in private throughout the day, and when everyone gathered to wrap things up, Cobb said there was acknowledgement of what teachers had accomplished. Magaña said Saturday he doesn’t regret starting off the day like he did.

“We had to acknowledge all of the feelings of the group,” he said. “It was about all of us working together for a common ground.”

Under the tentative deal union members are expected to vote on next week, all of the teachers in the Beacon network will see their base pay increase. The incentives Kepner Beacon teachers receive for teaching in a “highest priority” school will be slightly smaller but will continue.

After the Presidents’ Day holiday Monday, teachers and students will return to school on Tuesday and try to the maintain culture that has contributed to promising academic progress.

“It’s about trust,” Magaña said. “Some of it was cracked a little it. There was no contention in the room (Friday). It was really coming in with openness and willingness by everyone to say, ‘It’s done, and we did the right thing for ourselves. Now it’s time to come closer together.’”

“Normalcy will happen,” added Cobb, the teacher. “But it might take a bit.”

bonus

Aurora school district numbers shows some positive results from hard-to-staff bonus

Students work on algebra problems in a college-level course at Hinkley High School in Aurora.

When the Aurora school district offered some teachers and service providers a bonus for accepting or returning to hard-to-staff positions, the district saw less turnover in those jobs and had more of them filled by the start of the school year.

But the results weren’t consistent across schools, and there were differences in how teachers and other support staff responded to the bonus. Some schools still saw big increases in turnover. And the district still couldn’t fill all positions by the start of the school year.

In a report that district staff will present to the Aurora school board Tuesday, survey responses show the bonus was most influential for new special service providers, such as nurses, occupational therapists, or speech language pathologists. But only 33 percent of new teachers coming into the district said the bonus made an impact on their decision.

Aurora administrators refused to talk about the findings ahead of the board meeting. When the district first announced the bonuses, Superintendent Rico Munn said he had hoped the pilot bonus system would help the district attract more candidates, fill more vacancies, and retain more employees. The union objected to the bonuses. The union and the district begin negotiations next month on how to spend $10 million that voters approved to raise teacher pay.

An arbitrator ruled that the district should have negotiated the terms of the bonuses with the union first, but the school board refused to uphold the finding. District officials had indicated that the results of the pilot incentives would play a role in what changes they propose going forward, and it’s not clear where the school board, a majority of whom were elected with union support, will come down.

On a state and national level, incentives for teachers are being questioned after Denver teachers went on strike, in part over a disagreement about how effective incentives can be and whether that money is better spent on base pay. Ultimately, the tentative agreement that ended the strike on Thursday maintained a number of bonuses, including $2,000 for educators in hard-to-staff positions.

In the Aurora pilot program, the district offered a bonus for special education, secondary math and secondary science teachers at 20 targeted schools. If staff in those positions committed to returning to their job for this year, they could get $3,000. If they returned, but did not give an early commitment, the bonus would be $2,500.

The same rules applied for other positions such as psychologists, nurses, occupational therapists, and speech pathologists, but those employees were eligible at all district schools. New employees in those positions could get $2,500.

To pay for the bonuses, the district had set aside $1.8 million from an unexpected increase in revenue due in part to rising property values. The district only ended up spending about $1.1 million.

Among 229 eligible teachers, 133 returned to their jobs, committing early, and another 29 returned without making an early commitment, meaning about 70 percent of teachers were retained and received the bonus.

Of the 20 schools at which teachers of math, science, and special education received incentives, turnover went down at 13 schools, up at another five, and stayed the same at two.

Among 184 staff members in the other hard-to-staff positions districtwide, 141 returned to their jobs, or 77 percent, all of them committing early and receiving the higher bonus.

The report doesn’t compare those numbers with previous years’.

Ramie Randles, a math teacher, was at Aurora West Collegiate Prep last year and received the bonus. But, she says, she had already decided to return to the same job this school year even before she learned about the bonus.

“To be honest with you it’s nice to get a little extra, but it’s a very small amount that’s not going to sway me one way or another,” Randles said.

In the second quarter of the school year, she left her job at Aurora West and is now teaching math at North Middle School.

The bonus is offered at both schools, but it wasn’t a factor, she said.

“I just feel like I want to feel valued in a job,” Randles said. “If I’m feeling like I’m happy that affects not just me, it affects my students. It affects my coworkers.”

According to the district, 98.26 percent of those who received a bonus remain in the same position as of this week.

Fill rates, which represent how many of the district’s positions are filled by the start of the school year, show an increase, although often small, among all positions except for school psychologists.

Fill rates over time: Did Aurora have more positions filled at the start of this school year than in the past?

Position 16-17 17-18 18-19
Secondary math teachers at 20 schools 91.5% 92.6% 93.4%
Secondary science teachers at 20 schools 93.5% 93.8% 94.8%
Special education teachers at 20 schools 92.6% 89.4% 90.24%
Nurses, district-wide 87.3% 94.6% 98%
Occupational therapists, district-wide 95.4% 80% 96.1%
Psychologists, district-wide 94.4% 96% 95.4%
Speech language pathologists, district-wide 75% 81.4% 85.4%

Another goal of the pilot was to help the district save money by decreasing the use of contract agencies to fill important positions.

The report found that compared with last year, fewer positions were filled through contract agencies.

The Aurora district “was one of the few districts in the metro area that did not provide some form of differentiated pay or incentive for hard-to-fill subject areas,” according to the district. As examples, the report cites Cherry Creek, Denver, and Douglas school districts.

Bruce Wilcox, president of Aurora’s teachers union, said the union has “no interest in pay like Denver does.”

He is against the bonus because he disagrees with setting up different pay for people doing the same jobs in different schools, and because he doubts it will have a long- term effect.

“For some, maybe money was enough to lure them in, but will it be enough to lure them in over a period of time?” Wilcox asked. “Money’s nice and every teacher needs it, let’s be honest, but is it enough to make you continue to work if the leadership and culture aren’t there?”

Tuesday, Aurora staff will also present the school board with an update on overall strategies to improve teacher recruitment and retention. Among those strategies: the development of new training for principals, including on how to motivate and retain high-performing employees.

Another report on the pilot incentives will be prepared this fall with final numbers of how many teachers stayed.

Find turnover rates for the pilot, by school, in the district’s report below. Note: The colors in the second column represent a comparison over the prior year with green showing that it is a lower rate than in the past.