human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

Newark Enrolls

After changes and challenges, Friday’s deadline to enroll in Newark schools finally arrives

PHOTO: Patrick Wall/Chalkbeat
A student fills out an information sheet at Central High School's booth at the citywide school fair in December.

Newark families have just a few hours left to apply to more than 70 public schools for next fall.

At noon on Friday, the online portal that allows families to apply to most traditional and charter school will close. After that, they will have to visit the district’s enrollment center. Last year, nearly 13,000 applications were submitted.

The stakes — and stress — are greatest for students entering high school. Each year, hundreds of eighth-graders compete for spots at the city’s selective “magnet” high schools, which many students consider their best options.

This year, those eighth-graders have to jump through an extra hoop — a new admissions test the magnets will use as they rank applicants. District students will sit for the test Friday, while students in charter and private schools will take it Saturday.

That’s news to many parents, including Marie Rosario, whose son, Tamir, is an eighth-grader at Park Elementary School in the North Ward.

“I don’t know nothing about it,” she said. District officials have been tight-lipped about what’s on the new test, how it will factor into admissions decisions, or even why introducing it was deemed necessary.

Students can apply to as many as eight schools. Tamir’s top choice was Science Park, one of the most sought-after magnet schools. Last year, just 29 percent of eighth-graders who ranked it first on their applications got seats.

“I’m going to cross my fingers,” Rosario said.

Students will find out in April where they were matched. Last year, 84 percent of families applying to kindergarten got their first choice. Applicants for ninth grade were less fortunate: Only 41 percent of them got their top choice, the result of so many students vying for magnet schools.

This is the sixth year that families have used the online application system, called Newark Enrolls, to pick schools. Newark is one of the few cities in the country to use a single application for most charter and district schools. Still, several charter schools do not participate in the system, nor do the vocational high schools run by Essex County.

Today, surveys show that most families who use the enrollment system like it. However, its rollout was marred by technical glitches and suspicions that it was designed to funnel students into charter schools, which educate about one in three Newark students. Some charter critics hoped the district’s newly empowered school board would abolish the system. Instead, Superintendent Roger León convinced the board to keep it for now, arguing it simplifies the application process for families.

Managing that process has posed challenges for León, who began as schools chief in July.

First, he ousted but did not replace the district’s enrollment chief. Then, he clashed with charter school leaders over changes to Newark Enrolls, leading them to accelerate planning for an alternative system, although that never materialized. Next, the district fell behind schedule in printing an enrollment guidebook for families.

Later, the district announced the new magnet-school admissions test but then had to delay its rollout as León’s team worked to create the test from scratch with help from principals, raising questions from testing experts about its validity. Magnet school leaders, like families, have said they are in the dark about how heavily the new test will be weighted compared to the other criteria, including grades and state test scores, that magnet schools already use to rank applicants.

Meanwhile, León has repeatedly dropped hints about new “academies” opening inside the district’s traditional high schools in the fall to help those schools compete with the magnets. However, the district has yet to hold any formal informational sessions for families about the academies or provide details about them on the district website or in the enrollment guidebook. As a result, any such academies are unlikely to give the traditional schools much of an enrollment boost this year.

District spokeswoman Tracy Munford did not respond to a request Thursday to speak with an official about this year’s enrollment process.

Beyond those hiccups, the enrollment process has mostly gone according to plan. After activating the application website in December, the district held a well-attended school fair where families picked up school pamphlets and chatted with representatives. Individual elementary schools, such as Oliver Street School in the East Ward, have also invited high school principals to come and tell students about their offerings.

American History High School Principal Jason Denard said he made several outings to pitch his magnet school to prospective students. He also invited middle-school groups to tour his school, and ordered glossy school postcards. Now, along with students and families across the city, all he can do is wait.

“I’m excited to see the results of our recruitment efforts,” he said. “Not much else is in my control — but recruitment is.”

reunion

Jubilation, and some confusion, as Denver schools begin their post-strike recovery

PHOTO: Eric Gorski/Chalkbeat
Lupe Lopez-Montoya greets students at Columbian Elementary School on Thursday after Denver's three-day teacher strike.

Lupe Lopez-Montoya got the text message at 6:15 a.m. The strike was over, the colleague who’d been keeping her up to speed on news from her union told her.

And so as district and teachers union officials celebrated their deal at the Denver Public Library, Lopez-Montoya on Thursday resurrected her regular commute to Columbia Elementary School, the northwest Denver school where she’s the longest-serving tenured teacher.

For the past three days, Lopez-Montoya had stayed out of school as Denver Public Schools and the Denver Classroom Teachers Association sparred over how teachers in Denver are paid. She still had unanswered questions about the future. But first, it was time to greet students.

One girl ran across the sidewalk to give a hug. “Ms. Montoya’s back,” she shouted, burying her face in the teacher’s side.

Lopez-Montoya began to welcome students into the building. “Line up and you can go get breakfast,” she told one boy. “I’m happy you’re here today.”

The moment kicked off what was a not-quite-normal day in Denver schools. With a deal coming just an hour before some schools were due to start for the day, teachers and families had little time to adjust their plans. The district also had too little time to reopen early childhood classes that have been closed all week; those will reopen on Friday, officials said.

Building principals got an email around 7 a.m. telling them that the strike was over and that many teachers would be coming back to work. In some buildings, central office staff and substitutes who had been filling in for teachers were already on site when teachers returned to work. And some students who commute long distances didn’t get the word in time to go to schools.

By early afternoon, about 81 percent of teachers in district-run schools had shown up to work, and about 83 percent of students had, according to Denver school district spokesman Will Jones.

In the parking lot outside Mathematics and Science Leadership Academy, a west Denver elementary school, teachers still clad in the red of their cause gathered in the parking lot to walk in together. Asked what the mood was, reading intervention teacher Denise Saiz said, “Relief!” and her colleagues responded with an enthusiastic “Yes!”

“We’re really excited to get back to the kids,” said fifth grade teacher Emily Mimovich.

While strike participation was high at the school — it was founded with support from the state and local union as a teacher-led school — the teachers said they did not believe there would be hard feelings between those who walked and those who didn’t.

“We have such a good relationship,” said Reyes Navarro, who teaches Spanish to kindergarten and first-grade students. “We understand that you have to do whatever is right for you.”

Parents also said they were eager to have their children get back to school.

Sarah Murphy, who has a third-grader and a 4-year-old preschool student, said this morning was a “bit of a scramble, albeit a welcome one.” She had both children signed up for a day camp at the University of Denver and was unsure if she should send them to school or to camp because she wasn’t sure teachers would be back at school. Then she got word that camp was canceled for both children — but Denver Public Schools preschools are still closed.

“We were left trying to figure out what to do with our ECE4 since they were still closed,” she wrote to Chalkbeat. “We know how hard this has been on everyone, but this morning proved that the ECE program was the hardest hit, short notice every step of the way. We very much look forward to tomorrow when both are back in their normal school routines.”

Not everyone returned to school. Some teachers said they needed a day to collect themselves after an emotional experience. Judy Kelley, a visual arts teacher at Martin Luther King Jr. Early College, said she and many of her colleagues live too far away from the school to get there on short notice.

But the majority of teachers who went back to work Thursday describe positive experiences returning to their schools. Ryan Marini, a social studies teacher and football coach at South High School, called it the “best day” in 17 years of teaching.

Suzanne Hernandez, a first-grade teacher at Westerly Creek Elementary in Stapleton, said she and other teachers gathered to walk into school together at 7:45 a.m.

Teachers thanked the subs from central office who were in the building. They headed back to their roles in central offices.

Early in the morning, Hernandez sent a message to her students’ parents to let them know teachers would be back in school. The reaction from students as she greeted them was “overwhelming,” she said.

With the help of paraprofessionals who had prepared lessons, Hernandez said teachers were able to jump right back to where they left off last week.

As far as Valentine’s Day, that celebration will be pushed back until Friday afternoon, she said. For now, teachers are having their own celebration, happy to be back in the classroom.

“I think if there’s any sort of effect from the strike, it’s been a positive one,” she said. “It’s been a very unifying experience.”

Allison Hicks, a teacher at Colfax Elementary, said returning teachers were greeted with hugs, music, smiles and tears.

“After finding out an hour ago we were going back to work, I scrambled home to get ready,” she wrote to Chalkbeat. “I am exhausted, but so excited to be back with my students. This day won’t be normal, but knowing I did my part to put my students first is the best feeling to have.”

Kade Orlandini, who teaches at John F. Kennedy High School, said most of the teachers are back in her building, despite the short notice.

“Teachers were ready, and we are jumping right back into instruction,” she wrote. “Attendance seems lower than normal, but students who are here are ready to learn. The atmosphere in the school is very positive all around.”

What was your experience going back to school? What other questions do you have? Take our survey

Chalkbeat’s Ann Schimke, Erica Meltzer, and Yesenia Robles contributed reporting to this article.