human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

weekend update

How the education world is reacting to racist violence in Charlottesville — and to Trump’s muted response

PHOTO: Andrew Dallos/Flickr
A rally against hate in Tarrytown, New York, responds to the violence in Charlottesville.

For educators across the country, this weekend’s eruption of racism and violence in Charlottesville, Virginia, offered yet another painful opportunity to communicate their values to families, colleagues, and community members.

Many decried the white supremacists who convened in the college town and clashed with protesters who had come to oppose their message. Some used social media to outline ideas about how to turn the distressing news into a teaching moment.

And others took issue with President Donald Trump’s statement criticizing violence “on many sides,” largely interpreted as an unwillingness to condemn white supremacists.

One leading education official, U.S. Education Secretary Betsy DeVos, followed Trump’s approach, criticizing what happened but not placing blame on anyone in particular:

DeVos’s two most recent predecessors were unequivocal, both about what unfolded in Charlottesville and whom to blame:

Leaders of the nation’s two largest teachers unions responded directly to Trump:

The American Federation of Teachers, Weingarten’s union, is supporting vigils across the country Sunday night organized by chapters of Indivisible, a coalition that emerged to resist the Trump administration. The union also promoted resources from Share My Lesson, its lesson-plan site, that deal with civil rights and related issues.

“As educators, we will continue to fulfill our responsibility to make sure our students feel safe and protected and valued for who they are,” Weingarten said in a statement with other AFT officials.

Local education officials took stands as well, often emotionally. Here’s what the superintendent in Memphis, which is engaged in the same debate about whether Confederate memorials should continue to stand that drew white supremacists to Charlottesville, said on Twitter:

Teachers in Hopson’s district return for the second week of classes on Monday. They’ve helped students process difficult moments before, such as a spate of police killings of black men in 2016; here’s advice they shared then and advice that teachers across the country offered up.

We want to hear from educators who are tackling this tough moment in their classrooms. Share your experiences and ideas here or in the form below. 

Betsy DeVos

‘Underperformer,’ ‘bully,’ and a ‘mermaid with legs’: NYMag story slams Betsy DeVos

PHOTO: New York Magazine
A drawing of DeVos commissioned by an 8-year-old starts the New York Magazine article.

A new article detailing Betsy DeVos’s first six months as U.S. education secretary concludes that she’s “a mermaid with legs: clumsy, conspicuous, and unable to move forward.”

That’s just one of several brutal critiques of DeVos’s leadership and effectiveness in the New York Magazine story, by Lisa Miller, who has previously covered efforts to overhaul high schools, New York City’s pre-kindergarten push, and the apocalypse. Here are some highlights:

  • Bipartisan befuddlement: The story summarizes the left’s well known opposition to DeVos’s school choice agenda. But her political allies also say she’s making unnecessary mistakes: “Most mystifying to those invested in her success is why DeVos hasn’t found herself some better help.”
  • A friend’s defense: DeVos is “muzzled” by the Trump administration, said her friend and frequent defender Kevin Chavous, a school choice activist.
  • The department reacts: “More often than not press statements are being written by career staff,” a spokesperson told Miller, rejecting claims that politics are trumping policy concerns.
  • D.C. colleagues speak: “When you talk to her, it’s a blank stare,” said Charles Doolittle, who quit the Department of Education in June. A current education department employee says: “It’s not clear that the secretary is making decisions or really capable of understanding the elements of a good decision.”
  • Kids critique: The magazine commissioned six portraits of DeVos drawn by grade-schoolers.
  • Special Olympics flip-flop: DeVos started out saying she was proud to partner with the athletics competition for people with disabilities — and quickly turned to defending a budget that cuts the program’s funding.
  • In conclusion: DeVos is an underperformer,” a “bully” and “ineffective,” Miller found based on her reporting.

Updated (July 31, 2017): A U.S. Education Department spokesperson responded to our request for comment, calling the New York Magazine story “nothing more than a hit piece.” Said Liz Hill: “The magazine clearly displayed its agenda by writing a story based on largely disputed claims and then leaving out of the article the many voices of those who are excited by the Secretary’s leadership and determination to improve education in America.”