behind the metrics

IBO: City's school progress reports are flawed but an advance

The system the city uses to award letter grades to schools is complicated and in some ways flawed — but it’s the best system we have.

That’s the conclusion of a report by the Independent Budget Office, the city’s budget watchdog that since 2009 has been charged with scrutinizing Department of Education data. The office examined the city’s progress reports, released annually since 2007, to see whether their underlying metrics produce meaningful results.

The progress reports were meant to radically reorient the way that New Yorkers thought about school performance. Instead of assessing schools simply by the proportion of students passing state tests, the progress reports focus on students’ improvement from year to year. In a precursor to the “value-added” measurements now being used to assess teachers, the reports use a complex and evolving algorithm that controls for student demographics to calculate just how much students have progressed.

The city then assigns each school a letter grade based on its score. The letter grades inform both the city’s decisions about which principals should receive bonuses and which schools should be considered for closure and families’ choices are where to enroll.

The IBO concludes that the progress reports offer a more sophisticated analysis of school performance than ever before — but that there is room for improvement. “The methodology used by the education department is a significant improvement over simply basing measures on comparisons of standardized test scores,” the report concludes. “Still, the School Progress Reports have to be interpreted with caution.”

The IBO looked at three issues: whether the city’s algorithm has successfully controlled for factors outside of schools’ control; whether the reports have reflected long-term shifts as well as short-term changes; and whether minor methodology changes produced outsized score swings.

On the first question, the budget office concluded that overall, progress report scores in a small set of schools, those serving both elementary and middle school students, can be considered “demographically neutral” — or unaffected by student characteristics. But in most cases that was not true, according to the IBO’s analysis.

“All other things equal, elementary, middle, and high schools with a higher percentage of black and Hispanic students were consistently likely to have lower overall scores than other schools,” the report notes. Progress report scores were also lower in high schools with more poor students and more students with disabilities, the IBO concluded.

Confirming previous findings, the IBO also concludes that elementary and middle school scores have been highly volatile, with the majority of those schools receiving three or more different progress report grades since 2007. (High school progress report grades are based on a wider range of variables and have always been more stable.) But the IBO says that changes to the reports’ methodology, particularly around how students’ year-to-year growth is assessed, have made them more stable.

Finally, the IBO’s analysis found that most of the changes made to the progress report methodology in 2010 and 2011 did not affect their overall grade. In general, the office concludes, the city’s reports successfully identified very high- and very low-performing schools under multiple methodologies, but they were less successful at distinguishing among middle performers.

“The distinction between a C and D rating for a school may be the result of the particular methodology that the DOE has chosen, among the many that are possible, rather than the result of school practices or effectiveness,” the report concludes. “Unfortunately, this weakness occurs at precisely the point where high stakes decisions about schools are made.”

Department of Education officials say that while the peer group comparisons do not “completely control” for student characteristics, they do reduce the impact of race and other demographics compared to other measures of school performance. They also point out that the IBO found strong correlations between demographics and school grades in only a handful of the relationships it analyzed. And they note that the IBO’s analysis shows that between half and three quarters of C and D grades issued in 2011 would have been the same using the IBO’s methodology for analyzing score stability.

“As the IBO has recognized, New York City’s progress reports are a huge improvement over other state and district systems for measuring student learning in schools — and for this reason they have become a national model,” said spokesman Matthew Mittenthal in a statement. “Closing the achievement gap in New York City is a core goal of our reform strategy, but as long as it exists we should expect it to show up in school progress reports, which are designed to be an accurate reflection of our schools’ strengths and challenges.”

The complete IBO report is below.

Behind the brawl

Three things to know about the Tennessee school behind this week’s graduation brawl

PHOTO: Arlington Community Schools
Arlington High School is a 2,000-plus-student school in suburban Shelby County in southwest Tennessee.

Arlington High School is considered the crown jewel of a 3-year-old district in suburban Shelby County, even as its school community deals with the unwelcome attention of several viral videos showing a fight that broke out among adults attending its graduation ceremony.

The brawl, which reportedly began with a dispute over saved seats, detracted from Tuesday’s pomp and circumstance and the more than $30 million in scholarships earned by the school’s Class of 2017. No students were involved.

“It was unfortunate that a couple of adults in the audience exhibited the behavior they did prior to the ceremony beginning and thus has caused a distraction from the celebration of our students’ accomplishments,” Arlington Community Schools Superintendent Tammy Mason said in a statement.

Here are three things to know about the 13-year-old school in northwest Shelby County.

With more than 2,000 students, Arlington is one of the largest high schools in Shelby County and is part of a relatively new district.

It’s the pride of a suburban municipality that is one of six that seceded from Shelby County Schools in 2014 following the merger of the city and county districts the year before. (School district secessions are a national trend, usually of predominantly white communities leaving predominantly black urban school systems.) More than 70 percent of Arlington’s students are white, and 6 percent are considered economically disadvantaged — in stark contrast to the Memphis district where less than 8 percent are white, and almost 60 percent are considered economically disadvantaged.

The school’s graduation rate is high … and climbing.

Last year, after adding interventions for struggling students, the school’s graduation rate jumped a full point to more than 96 percent. Its students taking the ACT college entrance exam scored an average composite of 22.5 out of a possible 36, higher than the state average of 19.9. But only a fifth scored proficient or advanced in math and a third in English language arts during 2015-16, the last school year for which scores are available and a transition year for Tennessee under a new test.

PHOTO: Caroline Bauman
Education Commissioner Candice McQueen visits with students at Arlington High School during a 2016 tour.

The school was in the news last August when Tennessee Education Commissioner Candice McQueen visited its campus.

The commissioner spoke with students there to kick off her statewide listening tour that’s focused on ways to get students ready for college and career. McQueen highlighted the school’s extracurricular activities and students’  opportunities to intern for or shadow local professionals. She also complimented Arlington for having an engaged education community. 

poster campaign

How one Memphis student is elevating the conversation about school discipline

PHOTO: Laura Faith Kebede
Posters created by junior Janiya Douglas have amplified student voices about the culture of White Station High School in Memphis.

Now in her third year of attending a premier public high school in Memphis, Janiya Douglas says she’s observed discipline being handed out unevenly to her classmates, depending on whether they are on the college preparatory track.

PHOTO: Laura Faith Kebede
From left: Janiya Douglas and Michal Mckay are student leaders in Bridge Builders CHANGE program.

“We’re heavily divided in an academic hierarchy,” said Janiya, a junior in the optional program for high-achieving students at White Station High School. “It’s obvious students are treated differently if they are in traditional classes.”

Janiya also has observed racial disparities in how students are disciplined, and the state’s data backs that up. White Station students who are black or Hispanic are suspended at significantly higher rates than students who are white.

Frustrated by what she’s seen, Janiya took her concerns last Friday to the hallways of White Station and hung 14 posters to declare that “our school doesn’t treat everybody equally.”

By Monday morning, the posters were gone — removed by school administrators because Janiya did not get prior approval — but not before other students shared images of some of the messages on social media.

Now, Janiya is seeing some fruits of her activism, spawned by her participation in Bridge Builders CHANGE, a student leadership program offered by a local nonprofit organization.

In the last week, she’s met with Principal David Mansfield, a school counselor and a district discipline specialist to discuss her concerns. She’s encouraged that someone is listening, and hopes wider conversations will follow.

The discussions also are bringing attention to an online petition by the education justice arm of Bridge Builders calling for suspension alternatives across schools in Memphis.

White Station often is cited as one of the jewels of Shelby County Schools, a district wrought with academic challenges. The East Memphis school is partially optional, meaning some students test into the college prep program from across the county.

But Janiya and some of her classmates say they also see an academically and racially segregated school where students zoned to the traditional program are looked down upon by teachers. Those students often get harsher punishments, they say, than their optional program counterparts for the same actions.

“Our school doesn’t treat everybody equally. A lot of groups aren’t treated equally in our school system,” junior Tyra Akoto said in a quote featured on one poster.

“If we get wrong with a teacher, they’ll probably write us up. But if a white student was to do it, they’ll just play it off or something like that,” said Kelsey Brown, another junior, also quoted in the poster campaign.

A district spokeswoman did not respond to questions about disciplinary issues raised by the posters, but offered a statement about their removal from the school’s walls.

White Station is known for “enabling student voice and allowing students to express their opinions in various ways,” the statement reads. “However, there are protocols in place that must be followed before placing signs, posters, or other messages on school property. Schools administrators will always work with students to ensure they feel their voices are heard.”

PHOTO: @edj.youth/Instagram
Members of the education justice arm of the Bridge Builders CHANGE program

To create the posters, Janiya interviewed about two dozen students and had been learning about about school discipline disparities as part of the Bridge Builders CHANGE program.

State discipline data does not differentiate academic subgroups in optional schools. But white students in Shelby County Schools are more likely to be in an optional school program and less likely to be suspended. And statewide in 2014-15, black students were more than five times as likely as white students to be suspended.

White Station reflects those same disparities. About 28 percent of black boys and 19 percent of black girls were suspended that same year — significantly higher than the school’s overall suspension rate of 14 percent. About 17 percent of Hispanic boys and 7 percent of Hispanic girls were suspended. By comparison, 9 percent of white boys and 2 percent of white girls were suspended.

Shelby County Schools has been working to overhaul its disciplinary practices to move from punitive practices to a “restorative justice” approach — a transition that is not as widespread as officials would like, according to Gina True, one of four specialists implementing a behavior system called Positive Behavior Interventions and Supports, or PBIS.

“The whole goal is to not get them suspended, because we want to educate them,” said True, who met this week with Janiya and several other students from Bridge Builders. “When students are cared for emotionally, they perform better academically. As counselors, that’s what we’ve been saying for years.”

Janiya acknowledges that she didn’t follow her school’s policy last week when hanging posters without permission at White Station. But she thinks her action has been a catalyst for hard conversations that need to happen. And she hopes the discussions will include more student input from her school — and across the district.

“Those most affected by the issues should always be a part of the solution,” she said.

Correction: April 10, 2017: A previous version of this story said Janiya put up 50 posters at her school. She designed 50 but actually posted only 14.