research shows

Gates Foundation study paints bleak picture of teaching quality

The study measured teachers against the criteria in Charlotte Danielson's Framework for Effective Teaching rubric, which is used in New York as a tool for observing teachers. Teachers scored better at classroom management than they did on measures of higher-order instructional challenges, such as asking productive questions.

A historic look inside the nation’s classrooms, including some in New York City, painted a bleak picture, according to a report released by the Bill and Melinda Gates Foundation today.

The second installment of the foundation’s ambitious Measures of Effective Teaching study, the report focuses on the picture of teaching yielded by five different classroom observation tools. It also scrutinizes those tools themselves, concluding that they are valuable as a way to help teachers improve but only useful as evaluation tools when combined with measures of student learning known as value-added scores.

The conclusion is a strong endorsement of the Obama administration’s approach to improving teaching by implementing new evaluations of teachers that draw on both observations and value-added measures. New York State took this approach to overhauling its evaluation system when it applied for federal Race to the Top funding.

Among the group of five observation tools the foundation studied is the rubric now being piloted in New York City classrooms as part of stalled efforts to implement the changes to teacher evaluation, Charlotte Danielson’s Framework for Effective Teaching.

Through all five lenses, instruction looked mediocre in an overwhelming majority of more than 1,000 classrooms studied, the report concludes. There were some bright spots. Many teachers were scored relatively well for the aspect of teaching known as “classroom management” — keeping students well-behaved, making sure they are engaged.

But teachers often fell short when it came to other elements of teaching, such as facilitating discussions, speaking precisely about concepts, and carefully modeling skills that students need to master. These higher-order skill sets, the report notes, are crucial in order for students to meet the raised standards outlined in the Common Core.

The study is the most expansive known examination of instruction in the U.S., reviewing more than 1,000 teachers for this report and nearly 3,000 for the study. Its lead authors are the economists Thomas Kane, of Harvard, and Douglas Staiger of Dartmouth, although more than a dozens researchers contributed to the study.

The evaluations were conducted by trained evaluators, who watched clips from videotape of more than 1,000 teachers around the country and then judged whether the teaching exhibited certain traits outlined in the observation tools.

One complicated aspect of the study is that it doesn’t just ask what the observation tools have to say about teaching; it also asks whether those observation tools are good ways to measure teaching at all. The question is crucial to the contentious teacher quality debate.

Motivated by the Obama administration’s focus on improving teaching by improving the way teachers are evaluated, the teacher quality debate has been dominated by a search for a better evaluation tool. The idea is that if school districts could have a better way to sort teachers, then they could increase quality by rewarding those who are most effective and improving or removing those who are less effective.

The study offers a qualified endorsement of the five observation tools it studied, saying that they should be one of multiple evaluation measures but that no one observation tool should be a sole measure. While the study found that all five observation tools had a positive association with student achievement as measured by value-added scores, the associations were not perfect.

And the tools’ reliability was relatively low — lower, in some cases, than the famously volatile judgments of value-added measures. When different observers used the same tool to evaluate the same teacher, they sometimes gave very different scores.

But the report does endorse using the observation tools in combination with value-added measures, as New York’s new evaluation system does. When researchers combined multiple observation tools’ judgments of teachers together — and then combined those with the teachers’ value-added scores, the result was a view of a teacher that was more able to predict future student achievement, the report says.

A final complication worth noting is that the study’s ultimate arbiter of what makes a good evaluation tool is itself under heavy scrutiny. That arbiter is a teacher’s value-added score, an estimate that attempts to extrapolate the amount of student learning for which a teacher can be held responsible, excluding other factors such as a student’s family income level.

A study that was the subject of a story in today’s New York Times found that value-added scores indeed are useful predictors not only of student achievement, but other measures of life success. Researchers have cast doubts on value-added measures’ validity, citing a host of concerns from the measures’ volatility to whether a high value-added score reflects true student learning or simply effective test prep.

Though an overhaul of teacher evaluation in New York has been stalled by the failure of teachers unions and school districts to agree on how to conduct it, both the New York City teachers union and the Department of Education agreed to participate in the Gates Foundation study when it launched in 2009. The union helped recruit teachers to join, and ultimately, teachers from about 100 schools signed up to have their lessons videotaped and analyzed.

“It takes the politics out of what’s being measured,” UFT president Michael Mulgrew said when the union first agreed to participate. “Teachers are very frustrated with the political debate. They are always saying, ‘why don’t you just come into the classroom?’ That’s what this is doing.”

Since then, the politics over teacher quality has grown even more heated.

Last summer, a GothamSchools reader who had worked in a school piloting the Danielson evaluation said it was very hard for teachers to be rated “effective.”

Betsy DeVos

‘Underperformer,’ ‘bully,’ and a ‘mermaid with legs’: NYMag story slams Betsy DeVos

PHOTO: New York Magazine
A drawing of DeVos commissioned by an 8-year-old starts the New York Magazine article.

A new article detailing Betsy DeVos’s first six months as U.S. education secretary concludes that she’s “a mermaid with legs: clumsy, conspicuous, and unable to move forward.”

That’s just one of several brutal critiques of DeVos’s leadership and effectiveness in the New York Magazine story, by Lisa Miller, who has previously covered efforts to overhaul high schools, New York City’s pre-kindergarten push, and the apocalypse. Here are some highlights:

  • Bipartisan befuddlement: The story summarizes the left’s well known opposition to DeVos’s school choice agenda. But her political allies also say she’s making unnecessary mistakes: “Most mystifying to those invested in her success is why DeVos hasn’t found herself some better help.”
  • A friend’s defense: DeVos is “muzzled” by the Trump administration, said her friend and frequent defender Kevin Chavous, a school choice activist.
  • The department reacts: “More often than not press statements are being written by career staff,” a spokesperson told Miller, rejecting claims that politics are trumping policy concerns.
  • D.C. colleagues speak: “When you talk to her, it’s a blank stare,” said Charles Doolittle, who quit the Department of Education in June. A current education department employee says: “It’s not clear that the secretary is making decisions or really capable of understanding the elements of a good decision.”
  • Kids critique: The magazine commissioned six portraits of DeVos drawn by grade-schoolers.
  • Special Olympics flip-flop: DeVos started out saying she was proud to partner with the athletics competition for people with disabilities — and quickly turned to defending a budget that cuts the program’s funding.
  • In conclusion: DeVos is an underperformer,” a “bully” and “ineffective,” Miller found based on her reporting.

We’ve reached out for reaction from DeVos’s team and will update when we hear back.

teachers with borders

Schools near state lines perform worse — and rules discouraging teachers from moving may be to blame

PHOTO: Alan Petersime

Want a leg up in school? Don’t attend one near a state border.

That’s the surprising finding of a new study published in the Economics of Education Review. The likely culprit: certification and pension rules that discourage teachers from moving between states, limiting the labor pool on each side of the border.

The peer-reviewed paper focuses on test scores at public middle schools near a state boundary. Eighth-graders attending those schools, the researchers find, perform consistently worse in math than students at non-boundary schools. (The results are negative in reading, too, but smaller and not always statistically significant.)

One reason the findings ought to catch the attention of policymakers across the country: the data comes from 33 states, including big ones like Florida, New York, and Texas.

“We estimate that roughly 670,000 students are enrolled in middle schools nationally that are [considered] ‘intensely affected’ by a state boundary in our study,” the researchers write.

Of course, schools and students are not randomly assigned to be near state boundaries, so the study can’t definitively conclude that boundaries are the cause of lower performance. But the researchers — Dongwoo Kim, Cory Koedel, Shawn Ni, and Michael Podgursky, all of the University of Missouri — control for a number of student characteristics that might affect performance.

And while the study can’t pinpoint why a boundary seems to hurt test scores, the researchers have a theory: “state-specific pension and licensing policies” that discourage teachers from moving between states, likely forcing border schools to draw from a more limited pool of potential teachers.

In some places, those pension rules mean a substantial loss of retirement wealth if teachers move states mid-career. Complicated licensure rules that in some cases require experienced teachers to take certification exams or obtain additional degrees can also make that kind of switch practically difficult. Other research has found that teachers rarely move across state lines, even if they live near a boundary.

Why might that harm performance of schools near state lines?

Say a school in New York City has two science teachers and no math teachers, while a school right across the river in New Jersey has two math teachers and no science teachers. If each school needs exactly one teacher per subject, the solution is easy in theory: the New York City school gets a math teacher and loses a science one, and vice versa for the New Jersey school. But if certification or pension rules prevent that from happening, both schools lose out — and student achievement might suffer.

States aren’t typically eager to change those policies, though, for several reasons.

For one, states that require prospective teachers to clear a high bar to become certified may worry that making it too easy for an out-of-state teacher to receive a license could reduce teacher quality. A study from North Carolina provides some evidence for this argument, showing that teachers trained elsewhere were less effective than teachers trained in-state, though the difference was very small.

Another argument is that limiting teachers’ ability to bring pension money along with them when they move helps states hold on to their educators — even if they are in turn harmed when they can’t recruit teachers from elsewhere.

The latest study suggests that the net impact of those restrictions are negative. Still, the effects on students are quite small, implying that changes to pension and certification policies are unlikely to lead to large improvements in student performance.

But, the study points out, policies that eliminate the harm from attending school near a state line could help hundreds of thousands of students.

“Although the boundary effects are small on a per-student basis, they are spread across a very large population,” the researchers write.