human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

the end

A 60-year-old group that places volunteers in New York City schools is shutting down

PHOTO: August Young

Citing a lack of support from the city education department, a 60-year-old nonprofit that places volunteers in New York City schools is closing its doors next month.

Learning Leaders will cease operations on March 15, its executive director, Jane Heaphy, announced in a letter to volunteers and parents last week.

In the message, she said the group had slashed its budget by more than a third, started charging “partnership fees” to participating schools, and explored merging with another nonprofit. But the city pitched in with less and less every year, with no guarantee of consistency, she said.

“This funding volatility has created insurmountable challenges to the long-term viability of our organization,” Heaphy wrote. “We regret the vacuum that will be created by our closure.”

The group — which began as part of the city school system but became its own nonprofit in the 1970s — says its volunteers work with more than 100,000 students in more than 300 schools every year, many of them faithfully. When then-84-year-old Carolyn Breidenbach became the group’s 2013 volunteer of the year, she had been helping at P.S. 198 on the Upper East Side daily for 12 years.

Heaphy’s full message to volunteers is below:

Dear [volunteer],

It is with a heavy heart that I write to inform you Learning Leaders will cease operations on March 15 of this year. This organization has worked diligently over the last few years to sustain our work of engaging families as Learning Leaders, but the funding landscape has become too challenging to keep our programs going. While we have been able to increase our revenues from a generous community of funders, we have ultimately come to the conclusion that without a consistent and significant base of funding from the NYC Department of Education, we cannot leverage foundation grants, individual donors, or school fees sufficiently to cover program costs.

In the face of growing financial challenges, Learning Leaders reduced its costs as thoughtfully as possible — and in ways that did not affect our program quality. Rather, we sought to deepen and continually improve our service to schools and families while eliminating all but the most necessary costs. These efforts reduced our budget by more than 35 percent.

At the same time, we sought greater public support for our work with schools and families across the city. We are grateful to the foundations and individual donors that have believed in our work and provided financial support to keep it going. We were gratified when schools stepped up to support our efforts through partnership fees. While these fees only covered a portion of our costs, the willingness of principals to find these funds within their extremely tight school budgets was a testament to the value of our work.

Throughout an extended period of financial restructuring Learning Leaders advocated strongly with the Mayor’s Office and the DOE [Department of Education] for a return to historical levels of NYC DOE support for parent volunteer training and capacity building workshops. While we received some NYC DOE funding this year, it was less than what we needed and was not part of an ongoing budget initiative that would allow us to count on regular funding in the coming years. Several efforts to negotiate a merger with another nonprofit stalled due to the lack of firm financial commitment from the DOE. Over time, this funding volatility has created insurmountable challenges to the long-term viability of our organization.

We regret the vacuum that will be created by our closure. If you have questions or concerns about opportunities and support for family engagement and parent volunteer training, you can contact the NYC DOE’s Division of Family and Community Engagement at (212) 374-4118 or [email protected].

On behalf of the board of directors and all of us at Learning Leaders, I offer heartfelt thanks for your partnership. We are deeply grateful for your work to support public school students’ success. It is only with your dedication and commitment that we accomplished all that we did over the last 60 years. We take some solace in knowing that we’ve helped improve the chances of success for more than 100,000 students every year. The Learning Leaders board and staff have been honored to serve you and your school communities.
Sincerely,

Jane Heaphy
Executive Director

Rise & Shine

While you were waking up, the U.S. Senate took a big step toward confirming Betsy DeVos as education secretary

Betsy DeVos’s confirmation as education secretary is all but assured after an unusual and contentious early-morning vote by the U.S. Senate.

The Senate convened at 6:30 a.m. Friday to “invoke cloture” on DeVos’s embattled nomination, a move meant to end a debate that has grown unusually pitched both within the lawmaking body and in the wider public.

They voted 52-48 to advance her nomination, teeing up a final confirmation vote by the end of the day Monday.

Two Republican senators who said earlier this week that they would not vote to confirm DeVos joined their colleagues in voting to allow a final vote on Monday. Susan Collins of Maine and Lisa Murkowski of Alaska cited DeVos’s lack of experience in public education and the knowledge gaps she displayed during her confirmation hearing last month when announcing their decisions and each said feedback from constituents had informed their decisions.

Americans across the country have been flooding their senators with phone calls, faxes, and in-person visits to share opposition to DeVos, a Michigan philanthropist who has been a leading advocate for school vouchers but who has never worked in public education.

They are likely to keep up the pressure over the weekend and through the final vote, which could be decided by a tie-breaking vote by Vice President Mike Pence.

Two senators commented on the debate after the vote. Republican Lamar Alexander of Tennessee, who has been a leading cheerleader for DeVos, said he “couldn’t understand” criticism of programs that let families choose their schools.

But Democrat Patty Murray of Washington repeated the many critiques of DeVos that she has heard from constituents. She also said she was “extremely disappointed” in the confirmation process, including the early-morning debate-ending vote.

“Right from the start it was very clear that Republicans intended to jam this nomination through … Corners were cut, precedents were ignored, debate was cut off, and reasonable requests and questions were blocked,” she said. “I’ve never seen anything like it.”