moving the goalposts?

New York City is now focusing on academic progress at its lowest-performing schools. Here’s why some experts say that’s a problem.

PHOTO: Stephanie Snyder
The Richard R. Green campus in the Williamsbridge neighborhood of the Bronx houses multiple schools in the Renewal program.

Ever since Orchard Collegiate Academy was assigned to the city’s Renewal program for low-performing schools, leaders of the Manhattan high school have worked to boost attendance: meeting weekly to hone their strategy, and teaming up with a community organization to contact families to find out why certain students aren’t showing up.

The effort seems to be paying off. The school’s attendance rate increased from 76.6 percent two years ago to 81.5 percent last year, just barely meeting its assigned attendance goal (though still well below the city high school average of nearly 88 percent).

But while Orchard Collegiate — and the 85 other schools in Mayor Bill de Blasio’s signature turnaround effort — are expected to maintain progress, they were not issued new attendance goals through the Renewal program. Nor did they receive official goals on the quality of the school’s leadership, classroom instruction or school climate, benchmarks that have been issued for the past two years.

Education officials say this is intentional: As the program nears the end of its third year, they argue, it’s time to shift focus toward academic outcomes like state test scores and graduation rates.

Yet even staunch supporters of the city’s approach to school turnaround expressed concerns with that model. They worry that expecting academic gains too soon will set the program up for disappointing results on a national stage, that it sends mixed signals to long-struggling schools over what the city wants them to focus on, and that the program’s success will be tied too closely to student test scores, which may not accurately reflect student learning.

“What they’re really saying is that schools should be showing outcomes in the second year of implementation, which is not realistic,” said Megan Hester, a senior associate at the Annenberg Institute for School Reform, who works closely with community organizations that partner with Renewal schools. “It’s a political timeline and not a school research-based timeline.”

The city’s shift toward evaluating Renewal on academic outcomes is part of a key de Blasio promise: that the program would yield “fast and intense improvement” within three years, spurred by social services like mental health clinics, social workers and vision screenings combined with extra academic support.

That overarching strategy — slated to cost $850 million through 2019 — won over many advocates who criticized his predecessor’s approach, which involved dozens of contentious school closures.

But those same supporters have long worried that expecting speedy transformation is unrealistic, given that schools in the program started out among the city’s lowest performers. Research has shown school turnarounds can take five to ten years, well beyond the city’s initial three-year timeframe, if they happen at all.

Figuring out how much progress is realistic to expect — and when — is a difficult task, experts say, complicating the city’s desire to establish incremental goals struggling schools can realistically achieve.

To thread that needle, the city assigned schools two different types of benchmarks. The first — called “leading indicators” in education jargon — looked at data like attendance and whether the school has strong leadership or rigorous instruction, as measured by surveys and observations.

Those goals were meant to offer initial signs about whether the turnaround efforts were taking hold. Progress on those standards has been uneven: 20 percent of the city’s 86 Renewal schools met none of their leading indicators last year, while 15 percent met all of them.

The second kind of indicators are called “student achievement benchmarks,” and focus on proficiency in reading and math as measured by state test scores for elementary and middle school students, and metrics like graduation rates and state exit-exam completion for high schools students. Some of those benchmarks require only tiny improvements, but many schools have still struggled to meet them.

“Certainly by the end of year three we expect to see noticeable improvements in student achievement,” Eric Ashton, the education department’s executive director for school performance, said in an interview. He and other officials stressed that the city would still pay close attention to attendance and school climate, and Renewal schools that did not meet last year’s goals are still expected to meet them.

Michelle Renée Valladares, associate director of the National Education Policy Center at the University of Colorado at Boulder, said she applauded the city’s initial focus on both school climate and academic goals, but worries about the shift toward academic outcomes measured by test scores.

She noted that boosts in attendance might actually have a negative effect on scores, and standardized tests are not ideal short-term measures of student learning.

“Now all these kids are showing up, who weren’t showing up and are probably behind, and now we want you to get their test scores up instantly,” said Valladares, who has advised the city’s education department on how to measure the program. “It doesn’t happen instantly.”

For now, multiple Renewal school leaders said the lack of new leading indicators was unlikely to change their work — at least in the short term.

“Right now, I feel like we’re doing all we can to raise attendance,” said one Renewal school administrator, who said his school was unaware that the city did not plan to issue new leading indicators this year, and spoke on condition of anonymity.

But, he added, the city’s shift might inspire a change of his own. “If they’re telling us we’re no longer measuring it,” he said, “does that mean we might move resources? Maybe.”

The benchmarks aren’t just a way of incentivizing schools to make certain changes, experts said. They have implications for the approach itself.

“Other people are looking to [New York City] about how do sustainable community schools work … and know they’re working,” said Valladares. “It’s really important for them to get it right.”

Follow the money

New York City’s finance watchdog demands answers on $600 million school turnaround program

PHOTO: Alex Zimmerman
City Comptroller Scott Stringer.

The city’s top financial watchdog didn’t wait even a week before pressing Chancellor Richard Carranza on whether the “Renewal” school turnaround program is living up to its nearly $600 million price tag.

“While some Renewal schools have shown improvements,” Comptroller Scott Stringer wrote in a letter to Chancellor Richard Carranza, “inconsistent progress across all Renewal schools suggests the need for a more thorough review of the program’s components and their overall impact.”

The letter, sent just three days after Carranza officially took office, asks for a detailed accounting of how Renewal schools spent money on core elements of the program, including teacher training and extending the school day for an hour — as well as any evidence that those efforts are paying off or being monitored. Two independent evaluations by outside researchers suggest the program has produced only mixed results.

Stringer’s letter appears to be motivated at least in part by a recent round of hotly contested school closures. Since the program’s launch in 2014, 16 of 94 original Renewal schools have been merged or closed. (Another 21 schools are slowly easing out of the program after city officials said they made enough progress.)

“With the decision to now close schools that have not made sufficient progress,” Stringer wrote, “I question whether there have been adequate direction and accountability measures in place to ensure that all school received allocations with sufficient time to show progress, and were directing new resources to high impact programs and interventions.”

Stringer’s letter came just weeks before Carranza began raising his own questions about the Renewal program, which gives long-struggling schools extra academic support and social services. In an interview with Chalkbeat, the new schools chief said the Renewal program did not appear to have a single clear “theory of action.”

The comptroller’s probe also comes at a precarious moment for the program: It is without a permanent leader and it’s also unclear whether the city will phase out or reconfigure it. (Carranza told Chalkbeat he is committed to running a turnaround program of some kind.)

Stringer also touched on a number of other aspects of the program that have drawn criticism from school communities, including how the city identifies which schools should be closed and how the education department helps families find new schools.

According to the comptroller’s letter, multiple schools that met the exact same number of city benchmarks received different decisions about whether they should be closed.

While Stringer acknowledged that the city conducts a holistic review in making closure decisions, “the lack of transparency about these additional factors and how school closure decisions are made is breeding needless distrust in communities.”

An education department spokeswoman, Toya Holness, said the department is “reviewing the comptroller’s letter and will provide a formal response.”


closures ahead

As New York City prepares to close more struggling ‘Renewal’ schools, here’s what we know about ones they’ve shuttered before

PHOTO: Alex Zimmerman
Monroe Academy for Visual Arts and Design, a Renewal school, was closed last year.

In the coming days, struggling schools in the city’s “Renewal” improvement program will learn whether they get more time to mount a comeback — or will be shut down for good.

New York City education officials are expected to announce soon which of the low-performing schools will close at the end of this academic year. The decision will have enormous consequences for students and teachers who will have to find new schools — and will likely rekindle debate about the effectiveness of Mayor Bill de Blasio’s $582 million effort to turn around troubled schools by infusing them with social services and academic support.

As the Renewal program passed its third birthday in November — a date by which the mayor promised to decide which schools aren’t measuring up — officials have been tight-lipped about which schools are on the chopping block.

Chalkbeat analyzed the previous rounds of closures — nine schools out of the original 94 — to understand which schools might be targeted this time. Perhaps the clearest finding is that it’s difficult to predict which schools the city will shutter.

While all the closed schools had very low graduation rates and test scores, so do other Renewal schools that were spared. The analysis shows there are no strict rules about which schools are shut down and which are given more time to turn around.

That said, here are some takeaways from previous Renewal school closures:

Almost all the closed schools struggled to retain students.

Seven of the nine closed schools enrolled fewer students in the year they were shuttered than when they entered the Renewal program in 2014 — and six shed more than a fifth of their students.

Many of the schools had struggled to recruit and retain students even before the program started — once it did, the schools struggled to staunch the flow. The city considers such shrinkage an existential problem; officials have suggested that schools with fewer than 250 students can become unsustainable since school funding is based partly on enrollment.

Six of the closed schools enrolled fewer than 200 students their final year — including Brooklyn’s M.S. 584, which lost about 25 percent of its population since the Renewal program started, leaving it with just 78 students.

Meeting the city’s goals doesn’t guarantee survival.

The city assigned each Renewal school annual goals around attendance, graduation rates, test scores, and other measures. (The goals have been criticized as overly modest.)

In the past, officials have said “all options are on the table” — including closure — for schools that fail to meet their goals. But those the city has actually shuttered have been all over the map.

For instance, the city closed the Bronx’s Leadership Institute the year after it hit 71 percent of its goals — more than most schools in the program. At the other end of the spectrum, it shuttered Monroe Academy for Visual Arts and Design, which met just 14 percent of its goals.

In the past, education department officials have said they consider a range of factors when deciding which schools to close, including their academic performance, feedback from families, staff turnover, and previous improvement efforts.

“When making decisions about school closures we carefully assess each school based on multiple measures,” department spokesman Michael Aciman said in an email. “In every case where we’ve proposed a closure, we’ve prioritized family engagement and guaranteed that every student has higher-performing school options.”

However, the mix of factors means it isn’t clear which schools are most at risk of closure. The fact that some were shuttered after meeting most of their city-issued goals only adds to the mystery.

Small gains in graduation rates and test scores aren’t enough.

A handful of shuttered Renewal high schools had boosted their graduation rates while they were in the program, while some middle schools got more students to pass the state exams.

However, the gains were usually small and the majority of students were still struggling.

At the Essence School, the share of students who passed the reading tests more than doubled since it became a Renewal school. But even with that bump, still only 5 percent passed. Meanwhile, math proficiency barely ticked up to 3 percent.

At the high-school level, every shuttered Renewal school saw an uptick in graduation rates.

The increases ranged from 5 to 17 percentage points. However, because most of the schools enrolled were relatively small, they could boost their graduation rates by several points simply by helping a few additional students earn diplomas.

And the schools with the biggest gains — a 17 point jump at Foundations Academy and a 13 point spike at Foreign Language Academy of Global Studies — came at schools with serious enrollment challenges. In their final year, both served fewer than 100 students.