For years, the city touted its improved test scores, saying that higher and higher percentages of its students were proficient on the standardized exams.
A new report by the Independent Budget Office, which tracked the change in year-to-year test scores of individual students for the same period, disputes the gains the city claimed.
That’s because more than 60 percent of students from a single cohort who were tested from third to sixth grade between 2006 to 2009 on the English language arts exam didn’t improve their proficiency levels, the IBO analysis found. Thirty percent ended up at a higher level and eight percent ended up at a lower level.
“At a time when the city was saying things were getting better in the school system, it looks different when you look at performances of the individual student,” said Ray Domanico, director of education research at the IBO.
Domanico acknowledged that there were some limitations to looking at proficiency levels alone. He said comparing state test scores from one year to the next is less than ideal, because a proficiency level in third grade doesn’t necessarily mean a student learned nothing if they earn the same level in subsequent grades.
But given the data, Domanico said proficiency levels — rather than raw scores — was the only possible metric to measure individual student progress, in an era when education officials are increasingly evaluating schools and teachers based on students growth on test scores. He added that proficiency level is the most relevant metric to the public, because the city uses that data to make decisions about student promotion and admission, and shares it with parents.
“We feel that the proficiency levels are real,” Domanico said.
But Department of Education officials say the IBO report contains a “fundamental flaw” that represents exactly the wrong way to go about tracking students. Because student performance levels vary from grade to grade, officials said, the city must produce and track growth percentiles rather than individual performance to get accurate results.
“As we explained to the IBO, their research is invalid,” DOE spokeswoman Jessica Scaperotti said in a statement. “Testing experts know that performance levels on New York State tests cannot be compared from grade to grade without additional analysis, which this study failed to complete or consider.”
The IBO report tracked test scores for 46,000 students who were in third grade in 2006 and followed them as they advanced from third grade to sixth grade. It stopped short in 2010 when the students would have been in seventh grade. That was the year when thousands of students saw their proficiency level drop in response to the state raised standards required of a student to be deemed proficient.
The students who saw the highest rate of improvement had the most room for growth, according to IBO’s analysis. Eighty percent of students who received a proficiency level score one or two in third grade achieved higher proficiency levels by sixth grade. Those findings confirm much about what is already known of the inflated test score bubble that existed prior to 2010. For years, the bar for proficiency on the reading and math tests lowered and proficiency levels steadily improved as a result.
IBO officials nodded toward these data limitations in their report, but added that it was still an important first step toward monitoring the department’s use of performance data.
“Increasingly, schools and educators are being held responsible for the year-to-year change in the achievement scores of individual students, yet there has been no public report of the school system’s overall performance on this account,” the report reads.
Next year, New York State will begin factoring individual student growth on standardized tests from one year to the next on teacher evaluations. Teachers whose students improve their scores on the tests compared to the year earlier will be rewarded with higher ratings when their annual evaluations come out.
The IBO analysis also disaggregated the data by race and ethnicity to determine if achievement gaps had narrowed or widened. IBO found that the gap between Hispanic and white students declined three percent, but found “no progress in reducing the white-black achievement gap” up to 2009. In recent years, the achievement gap has narrowed slightly, according to the National Assessment of Educational Progress, which is considered a more reliable indicator.
The report also found that the gaps between Asian students and every other ethnic group widened significantly.
The IBO consulted with Fred Smith, a former Department of Education employee who recently wrote critically about New York State tests.
The full report is below.
IBO Report on Grade-to-Grade Achievement