human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

money matters

Report: Trump education budget would create a Race to the Top for school choice

PHOTO: Official White House Photo by Shealah Craighead
President Donald Trump and U.S. Secretary of Education Betsy DeVos participate in a tour of Saint Andrews Catholic in Orlando, Florida.

The Trump administration appears to be going ahead with a $1 billion effort to push districts to allow school choice, according to a report in the Washington Post.

The newspaper obtained what appears to be an advance version of the administration’s education budget, set for release May 23. The budget documents reflect more than $10 billion in cuts, many of which were included in the budget proposal that came out in March, according to the Post’s report. They include cuts to after-school programs for poor students, teacher training, and more:

… a $15 million program that provides child care for low-income parents in college; a $27 million arts education program; two programs targeting Alaska Native and Native Hawaiian students, totaling $65 million; two international education and foreign language programs, $72 million; a $12 million program for gifted students; and $12 million for Special Olympics education programs.

Other programs would not be eliminated entirely, but would be cut significantly. Those include grants to states for career and technical education, which would lose $168 million, down 15 percent compared to current funding; adult basic literacy instruction, which would lose $96 million (down 16 percent); and Promise Neighborhoods, an Obama-era initiative meant to build networks of support for children in needy communities, which would lose $13 million (down 18 percent).

The documents also shed some light on how the administration plans to encourage school choice. The March proposal said the administration would spend $1 billion to encourage districts to switch to “student-based budgeting,” or letting funds flow to students rather than schools.

The approach is considered essential for school choice to thrive. Yet the mechanics of the Trump administration making it happen are far from obvious, as we reported in March:

There’s a hitch in the budget proposal: Federal law spells out exactly how Title I funds must be distributed, through funding formulas that sends money to schools with many poor students.

“I do not see a legal way to spend a billion dollars on an incentive for weighted student funding through Title I,” said Nora Gordon, an associate professor of public policy at Georgetown University. “I think that would have to be a new competitive program.”

There are good reasons for the Trump administration not to rush into creating a program in which states compete for new federal funds, though. … Creating a new program would open the administration to criticism of overreach — which the Obama administration faced when it used the Race to the Top competition to get states to adopt its priorities.

It’s unclear from the Post’s report how the Trump administration is handling Gordon’s concerns. But the Post reports that the administration wants to use a competitive grant program — which it’s calling Furthering Options for Children to Unlock Success, or FOCUS — to redistribute $1 billion in Title I funds for poor students. That means the administration decided that an Obama-style incentive program is worth the potential risks.

The administration’s budget request would have to be fulfilled by Congress, so whether any of the cuts or new programs come to pass is anyone’s guess. Things are not proceeding normally in Washington, D.C., right now.

By the numbers

After reshaping itself to combat declining interest, Teach For America reports a rise in applications

PHOTO: Kayleigh Skinner
Memphis corps members of Teach For America participate in a leadership summit in last August.

Teach for America says its application numbers jumped by a significant number this year, reversing a three-year trend of declining interest in the program.

The organization’s CEO said in a blog post this week that nearly 49,000 people applied for the 2017 program, which places college graduates in low-income schools across the country after summer training — up from just 37,000 applicants last year.

“After three years of declining recruitment, our application numbers spiked this year, and we’re in a good position to meet our goals for corps size, maintaining the same high bar for admission that we always have,” Elisa Villanueva Beard wrote. The post was reported by Politico on Wednesday.

The news comes after significant shake-ups at the organization. One of TFA’s leaders left in late 2015, and the organization slashed its national staff by 15 percent last year. As applications fell over the last several years, it downsized in places like New York City and Memphis, decentralized its operations, and shifted its focus to attracting a more diverse corps with deeper ties to the locations where the program places new teachers. 

This year’s application numbers are still down from 2013, when 57,000 people applied for a position. But Villanueva Beard said the changes were working, and that “slightly more than half of 2017 applicants identify as a person of color.”