research report

Three years in, some signs of (slight) academic growth at struggling ‘Renewal’ schools

PHOTO: Patrick Wall
Mayor Bill de Blasio at Brooklyn Generation School — part of the Renewal program

When Mayor Bill de Blasio launched an aggressive and expensive campaign to turn around the city’s lowest performing schools, he made a big promise: Schools would see “fast and intense” improvements within three years.

Almost exactly three years later, and after flooding 78 schools with more than $386 million in new social services and academic support, there are signs that the Renewal program has generated gains in student learning. The evidence is based on two newly updated analyses of test score data — one from Marcus Winters, a fellow at the conservative-learning Manhattan Institute, and the other from Aaron Pallas, a professor at Teachers College.

But the researchers caution that those improvements are modest — when they exist at all — and don’t yet match the mayor’s lofty promises.

The results may have implications far beyond New York City, as a national and political test case of whether injecting struggling schools with resources is more effective than closing them.

The two researchers previously reviewed the first two years of test score data in elementary and middle schools in the Renewal program: Winters found a positive effect on test scores, while Pallas generally found little to no effect.

Now, as the program reaches its third birthday, the pair of researchers have updated their findings with new test score data from last school year, and largely reaffirmed their earlier conclusions.

“We’re not seeing large increases” in student achievement, Pallas said. “And the reality is it’s hard to get large increases in struggling schools.”

Some advocates have argued that it is too early to expect big shifts in test scores, and that infusing schools with extra social services like mental health counseling and vision screenings are valuable in themselves. But de Blasio’s promise of quick academic turnaround has invited questions about Renewal’s effectiveness and whether resources can be more effective in improving low-performing schools than shuttering them.

To assess the program’s academic effect, Pallas compared changes in Renewal school test scores to other schools that had similar test results and student demographics when the program started, but did not receive extra support.

The biggest gains Pallas found were concentrated at the elementary level.

Over the past three school years, 20 elementary schools in the Renewal program have made larger gains on average in math and reading than 23 similar schools that didn’t get extra resources. The proportion of elementary school students considered proficient in reading at Renewal schools increased from 7 percent in 2014 to 18 percent last year — an 11-point jump. Meanwhile, the comparison schools also saw gains, but only by seven percentage points, giving Renewal schools a four percentage point advantage.

At the middle school level, the results are less encouraging. The 45 Renewal middle schools did not collectively outperform a group of 50 similar schools outside the program in reading or math.

In math, for instance, Renewal school students improved from 5 percent proficient to 7 percent. However, the comparison schools outside the program improved by roughly the same margin — increasing proficiency from 6 to 9 percent (and still far below city average). In reading, Renewal middle schools showed slightly less growth than the comparison group.

City officials have argued that Pallas’ findings are misleading partly because Renewal schools and the comparison schools are not actually comparable. Renewal schools, they say, were designated based on a range of factors like school climate or teacher effectiveness, not just student demographics and test scores.

“The schools included in the study are neither similar nor comparable in quality and a comparison of the two dissimilar groups is unreliable at best,” Michael Aciman, an education department spokesman, said in a statement. Aciman added that Renewal schools have made larger gains in reading and math than similar schools across the state, and have made progress in reducing chronic absenteeism and improving instruction.

Pallas notes that there are some limitations to his approach, and acknowledges that he could not account for some differences between the two groups, such as the quality of a school’s principal. He also does not use student-level data, for instance, which would allow a more fine-grained analysis of whether the Renewal program is boosting student achievement. But Pallas, and other researchers who have previously reviewed his data, have said his model is rigorous.

The Manhattan Institute’s Winters found more positive trends than Pallas, consistent with his earlier findings. Using an approach that evaluates whether Renewal schools are outperforming historical trends compared with schools outside the program, Winters found that the Renewal program appeared to have a statistically significant effect on both reading and math scores — roughly equivalent to the difference in student achievement between charter schools and traditional district schools in New York City.

Asked about how to interpret the fact that his results tended to be more positive, Winters said either interpretation is plausible.

“It’s hard to tell which of these is exactly right,” he said. But “neither of us are finding results that are consistent with what we would expect if the program is having a large positive effect.”

Highs and lows

Some Colorado schools brace for state intervention, while others cheer their progress

PHOTO: Nic Garcia

Two Colorado school districts and six individual schools failed to show enough improvement to raise their ratings under the state’s accountability system. Unless they successfully appeal their preliminary ratings this fall, they’ll remain on state-mandated improvement plans. Those include the 7,500-student Adams 14 district based in Commerce City, which is likely to face additional state intervention this school year, and Aurora Central High School.

And two new schools also face the potential for state intervention, Central Elementary School in Adams 14 and Minnequa Elementary in the Pueblo 60 district.

One school district and six individual schools that had previously faced state intervention improved enough to get off Colorado’s “accountability clock,” according to preliminary school ratings released Monday.

“Our students are working harder than they ever have, and it’s making a big difference,” said Superintendent Deirdre Pilch of the Greeley-Evans district north of Denver. In that district, two schools on state improvement plans and four others that had been placed on warning did well enough to get out of state scrutiny. “By every measure we use, we are seeing students move in the right direction. I am so proud of people locking arms and coming together to do that work.”

Colorado’s school accountability system rates districts based on achievement on state literacy, math, and science tests, on annual academic growth, and on postsecondary readiness as measured by graduation rates, dropout rates, scores on college entrance exams, and enrollment in college.

Schools go on performance watch or “on the clock” if their rating places them in one of the lowest two tiers – turnaround or priority improvement – and they face state intervention if they don’t move into a higher tier after five years.

The ratings released Monday are considered preliminary. Districts can request that the state reconsider districtwide or school ratings based on, for example, progress in literacy and math in the early grades or measures of high school achievement that don’t show up on state tests.

The deadline to file a request to reconsider is Oct. 15. The ratings will be finalized in December.

The State Board of Education has four options when deciding the future of schools whose performance remains in the two lowest tiers of the five-point scale for years on end. The board can close the school, hand it over to a charter management organization, contract with a third party to help run the school or create an innovation plan that spells out strategies and exemptions from district and state policy to improve student learning.

The options for districts are similar but include drastic and politically challenging steps like pushing for merger with a higher-performing district.

So far, the State Board has taken a collaborative approach and largely approved the plans that schools and districts brought forward. At the same time, state lawmakers, at the request of the Colorado Department of Education, approved changes to require schools to show more sustained improvement to get off performance watch and encourage schools to take action earlier in the process.

Colorado doesn’t have the option of state takeover that’s been exercised in places like New Jersey and Tennessee, with mixed results.

The next few years will serve as an ongoing test of how well this system of carrots and sticks works to help long-struggling districts.

“We are excited to see the progress made in some schools around the state to improve student performance, especially for some of those that have persistently struggled,” said Alyssa Pearson, the Colorado Department of Education’s deputy commissioner for accountability and performance, in an email. “These schools and their districts have had a laser-like focus on the needs of students. They have done this through two to three high-leverage priorities around data-driven instruction, leadership development, and culture to better meet the needs of all students. The specific actions vary, depending on the local context and need.”

Perhaps one of the best examples of the process working is the Greeley-Evans district north of Denver. When Pilch arrived in 2015, 10 schools were on the accountability clock, and three of them ultimately required state-approved improvement plans: Prairie Heights Middle Schools, Franklin Middle School, and Martinez Elementary.

Franklin Middle School got off the clock last year, and with this round of ratings, Prairie Heights and Martinez Elementary were also freed from state supervision. Four additional Greeley-Evans schools came off performance watch this year without outside intervention.

Three Greeley-Evans schools, though, will be on the state’s watch list if their ratings don’t change in appeals.

Pilch said the district poured resources and support into the schools that needed to improve and took advantage of state grants for leadership training and other professional development. The district received extensive reviews of what was and wasn’t working within its schools from state evaluators and took those findings seriously in crafting improvement plans.

“The schools that are on priority improvement and turnaround, they’ve been our priority in terms of support and in terms of ensuring the right resources and the right leaders are in place,” Pilch said. “We’ve been very intentional about maintaining and training quality leaders at these schools. We’ve seen fewer leaders turn over.”

Also showing improvement was the Westminster district, which had one more year to make progress on its plan and is now off the clock. Officials there claimed vindication for the district’s competency-based learning model, in which students are grouped by their understanding of a certain subject and can progress to another level as soon as they show that they’ve mastered that class content.

Results for Aurora schools, where district officials have been using new reforms to intervene in low-performing schools, have been mixed. The district as a whole improved enough to dodge state action last year. This year’s rating stayed at “improvement.”

The school with the most years of bad ratings, Aurora Central High School, failed to improve. The school is already on a state plan for improvement and has a year left to earn a higher rating before it would have to return to face the state again. Previously, state officials essentially blessed a district plan to continue rolling out interventions it was already trying, with extra help from an outside partner. If the school has to return to the state, officials could take more drastic action.

Three schools that had earned the lowest rating of turnaround last year — including Lyn Knoll Elementary — improved. Paris Elementary, which was facing state intervention this year if it didn’t raise it’s Priority Improvement rating, also managed a higher score, to avoid state sanctions.

If final school ratings remain the same, no other Aurora schools this year would be as close to state action. Three schools — Gateway High School, North Middle School and Virginia Court Elementary — would be entering year four, meaning they would have one year to show improvement before being at risk for state intervention.

“Aurora Public Schools continues to see gains and movement in the right direction,” district officials said in an emailed statement. “We have some great momentum that we will continue to build upon. While we recognize that we need to make more improvements at faster rates, we will dig into our data to plan how we will best leverage our strengths and address our challenges.”

District officials said the results from schools in what the district calls its Action Zone – schools that have individual plans for some flexibility from district rules – show the district has the right structure in place to keep improving. However, two schools in that zone, Aurora Central and Boston K-8, did not improve within the state framework and Boston actually earned a lower state rating this year, though it is not on the clock.

The tiny Sheridan district south of Denver, which got off the clock in 2014 after years of effort, was rated in the second-lowest tier, putting the district back on performance watch. Pat Sandos, the new superintendent there, said the news was a hard way to start his tenure, but it also brings a sense of urgency to improving student performance.

“We’re looking at the data really hard and breaking out the content areas,” he said. “We see opportunity for growth with [English-language learners]. It seems like that’s something that’s not just us, but for a lot of at-risk districts.

“Some of the work that we’re already doing is realigning and looking at the curriculum. We’re really focusing hard on that, on establishing a framework that gets teachers focused on what kids need to know at each grade level. And we’re paying a lot of attention to professional development.”

Other schools that got off the clock after facing state intervention include Hope Online Academy Middle School, authorized through the Douglas County school district, Manaugh Elementary School in the Cortez-Montezuma district in southwest Colorado, and Bessemer Elementary School in the Pueblo 60 district.

Two other Pueblo 60 schools still face state supervision: Risley International Academy of Innovation and Heroes Middle School. So does the little Aguilar district in southern Colorado, where the both the district and its joint junior-senior high school have struggled with years of low performance, and Hope Online Learning Academy Elementary School.

The ratings will be finalized by December, and schools that have spent six years or more in turnaround or priority improvement status will face new or ongoing state intervention.

Editor’s note: This story has been updated to include comment from Colorado Department of Education Deputy Commissioner Alyssa Pearson. 

Follow the money

New York City’s finance watchdog demands answers on $600 million school turnaround program

PHOTO: Alex Zimmerman
City Comptroller Scott Stringer.

The city’s top financial watchdog didn’t wait even a week before pressing Chancellor Richard Carranza on whether the “Renewal” school turnaround program is living up to its nearly $600 million price tag.

“While some Renewal schools have shown improvements,” Comptroller Scott Stringer wrote in a letter to Chancellor Richard Carranza, “inconsistent progress across all Renewal schools suggests the need for a more thorough review of the program’s components and their overall impact.”

The letter, sent just three days after Carranza officially took office, asks for a detailed accounting of how Renewal schools spent money on core elements of the program, including teacher training and extending the school day for an hour — as well as any evidence that those efforts are paying off or being monitored. Two independent evaluations by outside researchers suggest the program has produced only mixed results.

Stringer’s letter appears to be motivated at least in part by a recent round of hotly contested school closures. Since the program’s launch in 2014, 16 of 94 original Renewal schools have been merged or closed. (Another 21 schools are slowly easing out of the program after city officials said they made enough progress.)

“With the decision to now close schools that have not made sufficient progress,” Stringer wrote, “I question whether there have been adequate direction and accountability measures in place to ensure that all school received allocations with sufficient time to show progress, and were directing new resources to high impact programs and interventions.”

Stringer’s letter came just weeks before Carranza began raising his own questions about the Renewal program, which gives long-struggling schools extra academic support and social services. In an interview with Chalkbeat, the new schools chief said the Renewal program did not appear to have a single clear “theory of action.”

The comptroller’s probe also comes at a precarious moment for the program: It is without a permanent leader and it’s also unclear whether the city will phase out or reconfigure it. (Carranza told Chalkbeat he is committed to running a turnaround program of some kind.)

Stringer also touched on a number of other aspects of the program that have drawn criticism from school communities, including how the city identifies which schools should be closed and how the education department helps families find new schools.

According to the comptroller’s letter, multiple schools that met the exact same number of city benchmarks received different decisions about whether they should be closed.

While Stringer acknowledged that the city conducts a holistic review in making closure decisions, “the lack of transparency about these additional factors and how school closure decisions are made is breeding needless distrust in communities.”

An education department spokeswoman, Toya Holness, said the department is “reviewing the comptroller’s letter and will provide a formal response.”