data-driven decisionmaking

Why we won't publish individual teachers' value-added scores

Tomorrow’s planned release of 12,000 New York City teacher ratings raises questions for the courts, parents, principals, bureaucrats, teachers — and one other party: news organizations. The journalists who requested the release of the data in the first place now must decide what to do with it all.

At GothamSchools, we joined other reporters in requesting to see the Teacher Data Reports back in 2010. But you will not see the database here, tomorrow or ever, as long as it is attached to individual teachers’ names.

The fact is that we feel a strong responsibility to report on the quality of the work the 80,000 New York City public school teachers do every day. This is a core part of our job and our mission.

But before we publish any piece of information, we always have to ask a question. Does the information we have do a fair job of describing the subject we want to write about? If it doesn’t, is there any additional information — context, anecdotes, quantitative data — that we can provide to paint a fuller picture?

In the case of the Teacher Data Reports, “value-added” assessments of teachers’ effectiveness that were produced in 2009 and 2010 for reading and math teachers in grades 3 to 8, the answer to both those questions was no.

We determined that the data were flawed, that the public might easily be misled by the ratings, and that no amount of context could justify attaching teachers’ names to the statistics. When the city released the reports, we decided, we would write about them, and maybe even release Excel files with names wiped out. But we would not enable our readers to generate lists of the city’s “best” and “worst” teachers or to search for individual teachers at all.

It’s true that the ratings the city is releasing might turn out to be powerful measures of a teacher’s success at helping students learn. The problem lies in that word: might.

Value-added measures do, by many readings, appear to do the job that no measure of a teacher’s quality has done before: They estimate the amount of learning by students for which a teacher, and no one else, is responsible, and they do this with impressive reliability. That is, a teacher judged to be more effective one year by value-added is likely to continue to be judged effective the next year, and the year after that.

But this is not true for every teacher — hardly. Many teachers will be mislabeled; no one disputes this. Value-added scores may be more reliable than existing alternatives, but they are still far from perfectly reliable. It’s completely possible, for instance, that a teacher judged as less effective one year will be judged as very effective the next, and vice versa.

As we reported two years ago, when the NYU economist Sean Corcoran looked at New York City’s value-added data, he found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

The fluctuation is acknowledged by even the strongest supporters of using value-added measures to evaluate teachers. One of the creators of the city’s original value-added model, the Columbia economist Jonah Rockoff, compares value-added scores to baseball players’ batting averages. One of his reasons: In each case, the year-to-year fluctuations of an individual’s score are about the same.

“If someone hit, you know, .280 last year, that doesn’t guarantee they’re going to hit .280 next year,” Rockoff said today. “However, if you hit .210 last year and I hit .300, there’s a very high likelhood I’m going to hit more than you next year, too. Whereas if you hit .280 and I hit .278, we’re basically the same.”

Another challenge is that many researchers still aren’t convinced that value-added scores are measuring the right sort of teacher impact. The challenge lies in the flaws of the measures on which value-added scores depend — standardized state test scores.

Tests are supposed to measure what a student has learned about a subject, but they can also reflect other things, like how well her teacher prepared her for the test, or how well she mastered the narrow band of the subject the test assessed.

The test-prep concern is magnified by findings that a single teacher can generate two different value-added scores if evaluators use two different student tests to determine them. The Gates Foundation’s Measures of Effective Teaching study calculated value-added scores for teachers based on both state tests and more conceptual tests. They found substantial differences between the two, according to an analysis by the economist Jesse Rothstein of the University of California at Berkeley.

“If it’s right that some teachers are good at raising the state test scores and other teachers are good at raising other test scores, then we have to decide which tests we care about,” Rothstein said today. “If we’re not sure that this is the test that captures what good teaching is, then we might be getting our estimates of teaching quality very wrong.”

Flags about exactly what high value-added ratings reward are also raised by studies that ask how the ratings match up with measures of what teachers actually say and do in the classroom. Heather Hill,  professor at Harvard’s Graduate School of Education, rated math teachers’ teaching quality based on an observation rubric called the Mathematical Quality of Instruction, which looks at factors like whether the teacher made mathematical errors and the quality of her explanations. Then Hill compared the math teaching rating to value-added measures.

Two individual cases stood out: One teacher had made a slew of math errors in her teaching, and the other had failed to connect a class activity to math concepts. But teachers’ value-added scores put them at the top of their cohort.

There is some reason to think that value-added measures reflect more than test prep. Rockoff points out that while different tests can produce different value-added scores for the same teacher, the two measures are still correlated. Using different tests, he said, is akin to looking at slugging percentage rather than batting average. “I’m sure those two things are positively correlated, but probably not one for one,” he said.

More persuasively, a recent study by Rockoff and two other colleagues concluded that value-added measures can actually predict long-term life success outcomes, including higher cumulative lifelong income, reduced chance of teen pregnancy, and living in a high-quality neighborhood as an adult. The study examined an anonymous very large urban school district that bears several similarities to New York City.

That study targeted another concern about value-added measures: that teachers score consistently well year after year not because of something they are doing, but because they consistently teach students with certain advantages.

Rothstein has used value-added models to conclude that fifth-grade teachers have strong effects on their students’ performances in third-grade — something they could not possibly influence, unless value-added scores reflect not just teachers’ influence but also advantages brought by students.

Rockoff and his colleagues evaluated the possibility by testing a question. If high-value added teachers do well because they get the “better” students of those in their grade, then their students’ high test score growth would be linked with mediocre performance in other classrooms. That would mean that, when researchers looked at growth for the entire grade, the “better” students’ growth would be canceled out by their less lucky peers. But the scores were not canceled out, suggesting that effective teachers did more than just have unusually good students.

None of this means that we won’t write about what the data dump includes or that we might not publish an adapted database that strips out information linking the city’s data to individual teachers. With more than 90 columns in the Excel sheet the city has developed — and more than 17,000 rows, representing the number of reports issued over their two-year lifespan — the release might well enable us to examine the city’s value-added experiment in new ways.

Value-added measures certainly aren’t going away. City officials only stopped producing Teacher Data Reports because they knew the State Education Department is preparing its own. The measures, which are expected to come out in 2013, will make up 25% of the evaluation for teachers of math and English in tested grades.

first steps

Superintendent León secures leadership team, navigates evolving relationship with board

PHOTO: Patrick Wall
Superintendent Roger León at Tuesday's school board meeting.

As Newark’s new superintendent prepares for the coming academic year, the school board approved the final members of his leadership team Tuesday and began piecing together a roadmap to guide his work.

The board confirmed three assistant superintendents chosen by Superintendent Roger León: Jose Fuentes, the principal of First Avenue School in the North Ward; Sandra Rodriguez, a Hoboken principal who previously oversaw Newark Public Schools’ early childhood office; and Mario Santos, principal of East Side High School in the East Ward. They join three other assistant superintendents León selected for his team, along with a deputy superintendent, chief of staff, and several other officials.

The three assistant superintendents confirmed Tuesday had first come before the board in June, but at that time none of them secured enough votes to be approved. During last month’s meeting, the board assented to several of León’s leadership picks and to his decision to remove many people from the district’s central office, but it also blocked him from ousting several people.

This week, Board Chair Josephine Garcia declined to comment on the board’s reversal, and León did not respond to a request for comment.

What is clear is that the board and León are still navigating their relationship.

In February, the board regained local control of the district 22 years after the state seized control of the district due to poor performance and mismanagement. The return to local control put the board back in charge of setting district policy and hiring the superintendent, who previously answered only to the state. Still, the superintendent, not the board, is responsible for overseeing the district’s day-to-day operations.

During a board discussion Tuesday, Garcia hinted at that delicate balance of power.

“Now that we’re board members, we want to make sure that, of course, yes, we’re going to have input and implementation,” but that they don’t overstep their authority, she said.

Under state rules, the board is expected to develop district goals and policies, which the superintendent is responsible for acting on. But León — a former principal who spent the past decade serving as an assistant superintendent — has his own vision for the district, which he hopes to convince the board to support, he said in a recent interview on NJTV.

“It’s my responsibility as the new superintendent of schools to compel them to assist the district moving in the direction that I see as appropriate,” he said.

Another matter still being ironed out by the board and superintendent is communication.

León did not notify the full board before moving to force out 31 district officials and administrators, which upset some members. And he told charter school leaders in a closed-door meeting that he plans to keep intact the single enrollment system for district and charter schools — a controversial policy the board is still reviewing.

The district has yet to make a formal announcement about the staff shake-up, including the appointment of León’s new leadership team. And when the board voted on the new assistant superintendents Tuesday, it used only the appointed officials’ initials — not their full names. However, board member Leah Owens stated the officials’ full names when casting her vote.

The full names, titles and salaries of public employees are a matter of public record under state law.

Earlier, board member Yambeli Gomez had proposed improved communication as a goal for the board.

“Not only communication within the board and with the superintendent,” she said, “but also communication with the public in a way that’s more organized.”

The board spent much of Tuesday’s meeting brainstorming priorities for the district.

Members offered a grab bag of ideas, which were written on poster paper. Under the heading “student achievement,” they listed literacy, absenteeism, civics courses, vocational programs, and teacher quality, among other topics. Under other “focus areas,” members suggested classroom materials, parent involvement, and the arts.

Before the school year begins in September, León is tasked with shaping the ideas on that poster paper into specific goals and an action plan.

After the meeting, education activist Wilhelmina Holder said she hopes the board will focus its attention on a few key priorities.

“There was too much of a laundry list,” she said.

early dismissals

Top Newark school officials ousted in leadership shake-up as new superintendent prepares to take over

PHOTO: Patrick Wall
Incoming Newark Public Schools Superintendent Roger León

Several top Newark school officials were given the option Friday to resign or face termination, in what appeared to be an early move by incoming Superintendent Roger León to overhaul the district’s leadership.

The shake-up includes top officials such as the chief academic officer and the head of the district’s controversial enrollment system, as well as lower-level administrators — 31 people in total, according to documents and district employees briefed on the overhaul. Most of the officials were hired or promoted by the previous two state-appointed superintendents, Cami Anderson and Christopher Cerf, a sign that León wants to steer the district in a new direction now that it has returned to local control.

The officials were given the option to resign by Tuesday and accept buyouts or face the prospect of being fired by the school board at its meeting that evening. The buyouts offer a financial incentive to those who resign voluntarily on top of any severance included in their contracts. In exchange for accepting the buyouts, the officials must sign confidentiality agreements and waive their right to sue the district.

Earlier this week, León submitted a list of his choices to replace the ousted cabinet-level officials, which the board must approve at its Tuesday meeting. It’s not clear whether he has people lined up to fill the less-senior positions.

It’s customary for incoming superintendents to appoint new cabinet members and reorganize the district’s leadership structure, which usually entails replacing some personnel. However, many staffers were caught off guard by Friday’s dismissals since León has given little indication of how he plans to restructure the central office — and he does not officially take the reins of the district until July 1.

A district spokeswoman and the school board chair did not immediately respond to emails on Friday about the shake-up.

Some staffers speculated Friday that the buyout offers were a way for León to replace the district’s leadership without securing the school board’s approval because, unlike with terminations, the board does not need to sign off on resignations. However, it’s possible the board may have to okay any buyout payments. And it could also be the case that the buyouts were primarily intended to help shield the district from legal challenges to the dismissals.

León was not present when the staffers learned Friday afternoon that they were being let go, the employees said. Instead, the interim superintendent, Robert Gregory, and other top officials broke the news, which left some stunned personnel crying and packing their belongings into boxes. They received official separation letters by email later that day.

The people being ousted include Chief Academic Officer Brad Haggerty and Gabrielle Ramos-Solomon, who oversees enrollment. Also included are top officials in the curriculum, early childhood, and finance divisions, among others, according to a list obtained by Chalkbeat.

In addition to the 31 being pushed out, several assistant superintendents are being demoted but will remain in the district, according to the district employees.

There was concern among some officials Friday about whether the turnover would disrupt planning for the coming school year.

“I don’t know how we’re going to open smoothly with cuts this deep,” one of the employees said. “Little to no communication was provided to the teams about what these cuts mean for the many employees who remain in their roles and need leadership guidance and direction Monday morning.”