eyes on NYC

New York City’s community schools guru on the program’s massive expansion and why the schools are ‘here to stay’

PHOTO: Stephanie Snyder
Chancellor Carmen Fariña (middle) and Chris Caruso (right) visited East Flatbush Community Research School in 2016.

Chris Caruso is running one of the biggest education experiments in New York City.

The executive director of New York City’s community schools program, Caruso is responsible for delivering on one of Mayor Bill de Blasio’s core education promises: rapidly transforming hundreds of schools into community hubs with extra social services, additional learning time — and even washing machines.

It’s a model that has quickly gained steam. By September, 215 schools serving just over 100,000 students will be part of the city’s community schools initiative, which also encompasses de Blasio’s Renewal turnaround program. New York’s community schools program is considered a key national test case of whether the approach will pay off.

The responsibility to make sure it does rests partly with Caruso, who began working as a program director in a community school in Washington Heights nearly two decades ago. Chalkbeat caught up with him recently to talk about how he measures community schools’ success, what the program’s future looks like, and the challenges of quickly scaling up social services for thousands of students.

This transcript has been edited for length and clarity.

Chalkbeat: This is the largest community schools program of its kind. Do you feel a lot of pressure to make sure it is seen by the public as a success?

Caruso: I think anytime you work on public policy change in New York City, eyes are on you. So we’ve done pre-K at a scale that no one else has done, we’ve done after-school on a scale that no one else has done, and that’s part of the territory.

So yes, you feel that. But for me, and my team, and this administration, I think it brings us energy and drives us to show the potential of the success.

The program has obviously scaled up really quickly. Do you feel like there are tradeoffs in expanding that fast?

I think doing something at a large scale really positions you to have a number of people going through a similar experience at the same time, and we’ve seen real value in schools and partners in learning from one another.

If you were to pilot something in a handful of schools, you have the ability to really be involved and direct day-to-day operations, but you lose the ability to create this learning community, a diverse learning community, and so that’s something that we’ve really seen gain traction.

So now that this program has been off the ground for three years, what are the big problems you’re trying to address as the program moves into its next phase?

I think [one] of the things that we’re looking at for next year is how do we ensure a high level of service and quality across all the schools?

We’ve had principal transitions, we’ve had some [community-based organization] transitions, we’ve had community school directors change, and so we have a diverse pool in terms of where people are at in the stages of development of being a community school.

That’s something that as a system we need to be able to adapt and meet the needs of those schools. A specific thing that we’re doing around that is in our first two years of operation, we held a monthly conference where all the community school directors would come together and we’d do group learning, we’d do individual things, we’d have seminars.

Next year, we’re going to be changing that model and doing more cohorts based on where those schools are at, based more on geography. Getting 215 schools together is a lot harder than 150, but this will allow us to really differentiate our support in a more meaningful way.

You’ve said before that community schools shouldn’t be thought of as a turnaround strategy — something [former U.S. Education Secretary] John King agrees with. Does that mean that providing these extra social services, partnerships and programs is worthwhile, regardless of whether it produces academic gains?

This is an equity strategy. There are neighborhoods in this city where kids have access to far fewer resources, whether those are healthcare resources, learning experiences, relationship resources. And so community schools are a strategy to level that playing field. There’s evidence to back that up. A long-term investment in [services] leads to higher rates of attendance, lower rates of chronic absenteeism, greater connectedness to school — and all those things lead to better academic performance.

It sounds like you’re saying that these supports help create the conditions necessary for a long-term academic boost. Does that mean you’re not paying a lot of attention to [whether] test scores go up this year or next year?

I think it’s impossible not to pay attention to that. That’s the reality and we have that data. We’re looking extremely closely at chronic absenteeism and average daily attendance. And you know we’ve seen a decrease in chronic absenteeism of almost 7.5 percent since the program started. Citywide, [the decrease in schools is] less than 2 percent, so we’ve been really pleased with that progress.

We’re looking at school culture and climate and so we’re looking at the number of suspensions and incidents and seeing decreases there. And we’re very much looking at graduation rates and how students are doing. We’re seeing positive movement there and we expect that we will continue and that will deepen as the school culture changes, as kids feel more connected to adults and to their peers, and as they can see better and they’re healthier and they’re ready to learn.

Many of those [measures] are getting better citywide so it’s hard to know to what extent that is caused by community schools versus some of these broader trends. How do you try to separate that out?

One way we do that is you look at the schools in our portfolio and these are schools that are disproportionately serving children living in poverty, serving English language learners and students with disabilities, and you kind of look at growth among a cohort compared to citywide growth. And that’s one way that you can measure the differences between a particular intervention and the general progress that a district or a system is making.

You’ve said before that strong instructional practices are a key element of community schools. What percentage of your time is spent on thinking about that part of what schools do?

Strong instruction is what schools need to be doing, and so we have an infrastructure in the Department of Education through our superintendents and our Division of Teaching and Learning to support that. That’s not part of my core responsibility.

My role in my team is to help schools integrate partnership resources, and many times schools are looking to partner to support instruction. So that might be: How do you take a momentary break in the middle of a literacy lesson to get kids to be able to focus again? That’s an instructional practice, but that’s not about how you help kids get phonemic awareness. There are elements of kind of being present on managing emotions, on the social emotional skills, that we spend more of our time thinking about. I’m not writing math or ELA curriculum.

I’m curious how much has this model permeated the city more broadly? There are a lot of schools that have partnerships with community-based organizations but aren’t in the city’s official program.

There’s a cohort of schools out there that were implementing this model [before the official program launched]. One of the things we’ve tried to do in our scaling is to bring more of those schools into the fold so that they have access to the same types of supports as the other schools.

The number of schools that might not consider themselves community schools but that are looking at partnerships and that are looking at the whole child in a different way — I think that’s grown exponentially.

And so when Chancellor [Carmen] Fariña visits schools now and is asking a principal — regardless of whether they’re a community school or not — tell me about your [community based] partner, tell me about your after-school program, how are they helping meet the needs of your students, and how are you aligning your supports? That’s huge.

Does the city see this as similar to pre-K, where once you do it, it just becomes part of the system — a feature of New York City public schools?

Yeah, I think so. This mayor ran on that and we have a deputy mayor and a chancellor who have championed that. This is something, again, the fact that it’s not a solely a top-down approach, this is something that communities have been organizing around and advocating for for a while.

I think the depth of the roots of support are deep, and I think that we as a department now are organized around this. It intuitively makes better sense on how we align resources and support schools. So yeah, I think community schools are here to stay.

approaching deadline

In year three of New York City’s massive school turnaround program, the big question is: What’s next?

PHOTO: Stephanie Snyder
Chancellor Carmen Fariña speaks with East Flatbush Community Research School Principal Daveida Daniel during a visit of the Renewal middle school.

On a recent Tuesday morning, Schools Chancellor Carmen Fariña walked into a classroom at Longwood Preparatory Academy in the Bronx, where students were absorbed with gleaming iMacs and recording equipment.

She paused for a moment, watching the teacher shuttle between students experimenting with audio-editing software.

“Look at the attention these kids are getting,” Fariña said, praising the school’s new vocational program in digital media. “It’s a feeling of renewed vigor and energy.”

With the smell of fresh paint still hanging in the air, Fariña’s visit was meant to highlight the enormous investments the city has made in dozens of schools that have floundered for years — including this one.

Under its former name, Banana Kelly, the school suffered from one of the highest dropout rates in the city, churned through four principals in five years, and struggled with serious safety incidents. (A previous principal was doused with pepper spray, and in 2012 was shot with a BB pellet outside the building.)

Now — as one of several back-to-school check-ins at some of the 78 schools the city is currently trying to revamp — Fariña was eager to praise the school’s energetic principal and its recent gains. Its attendance and graduation rates have improved in recent years, though its 2016 graduation rate (which is the latest figure publicly available) still lagged behind schools with similar student populations.

The visit comes as Mayor Bill de Blasio’s three-year effort to revive long-struggling schools, including the former Banana Kelly, is rapidly approaching its third birthday this November. The “Renewal” program — which has cost at least $383 million so far — is arguably the mayor’s most ambitious education reform, an effort to nurse some of the city’s lowest-performing schools back to health with extra social services and academic support rather than shut them down.

And while some experts say it’s too soon to expect big payoffs, de Blasio’s three-year timeline for “fast and intense improvement” has invited scrutiny into whether the program is translating into better outcomes for students.

PHOTO: Alex Zimmerman
Schools Chancellor Carmen Fariña tours Longwood Prep with Principal Asya Johnson (right) and student Heaven Molina.

“Enough time has elapsed that there is an appetite for looking at results,” said Aaron Pallas, a Teachers College professor who has studied the program.

Meanwhile, the exit strategy for schools in the Renewal program remains unclear. Despite promises that schools released from the program won’t lose the extra support they’ve come to rely upon, some school personnel are nervous that extra funding, counselors, and social services could be scaled back.

“There are some principals whose reaction is: ‘We really need to get it together because next year these [partner organization] resources might not be here,’” said Derek Anello, a program director with the nonprofit Partnership with Children who oversees staff in four Renewal schools. “Lots of people are experiencing it as the last of the three years.”

Planning for life after Renewal

Felicia Guerrier has helped usher in a new wave of social services at  P.S. 298 Dr. Betty Shabazz in Brownsville, Brooklyn, a Renewal school where 96 percent of students come from low-income families, and which used to suffer from “a love drought and a resource drought,” as she recently put it.

She supervises two new full-time social workers, oversees vision and dental services now offered out of the school’s auditorium, and coordinates with a nursing service that helps keep student health issues like asthma and diabetes in check — resources she says have led to a big boost in attendance and family involvement at the school.

But she’s already started handing off some of her responsibilities, out of a concern that the nonprofit she works for, Partnership with Children, might eventually be forced to reduce its role at the school. (Some nonprofits in Renewal schools are unsure whether their contracts will be extended past this school year.)

Guerrier explained that she’s now taking a back seat in meetings, making sure the assistant principal is in the loop to coordinate with health providers, and positioning the school’s parent coordinator to help run a food pantry for students and their families.

“I’m feeling the pressure to make sure there is some type of lasting power with what’s happening even when I’m gone,” she said.

While the mayor vowed in 2014 that the original cohort of 94 Renewal schools would be revamped or shuttered within three years (16 schools have already been closed or folded into other schools), the city has indicated the program will continue beyond this year. And officials stressed that the work of nonprofit organizations like Partnership with Children won’t end even if schools are taken out of the program.

“Steady improvement is key, and of course we will evaluate each school that is ready to transition from the program and provide them with the right supports to maintain their improvement,” Aimee Horowitz, the executive superintendent for Renewal schools, said in a statement.

An unclear exit strategy

Even as the Renewal schools move forward with their reforms, a big question hangs over them: How exactly do they exit the program? So far, no schools have left it without being combined with another school or closed.

This August, Mayor de Blasio said that would change. In addition to more closures, he said some schools could graduate out of the program, and others might stay past the three-year deadline.

“We have to work out the details,” de Blasio said, “but we’re not going to leave a school in the lurch.”

In making decisions about which schools to shutter or merge in the past, the city has looked at test scores, enrollment changes, principal effectiveness, and attendance rates — though officials have said there aren’t strict cutoffs, making it difficult to predict which schools could depart the program this fall.

If schools are released from Renewal, it remains to be seen whether they’ll continue to enjoy the same level of support and extra resources.

Brian Bradley, principal of Renaissance School of the Arts in East Harlem, said he is preparing for the possibility that this will be his school’s last year in the program. Renewal, he said, has made a real difference: Additional training for teachers has improved classroom instruction, aggressive outreach is boosting attendance, and the community school director has taken over once-overlooked administrative responsibilities.

“We have a great partnership and that has been the number one thing,” he said.

The school only banked on three years of support, but Bradley noted he has come to rely on extra funding the school receives to lengthen the school day. It’s a feature of the Renewal program that has a dual payoff, he said: more time for student learning and a pay bump that helps reduce teacher turnover.

While he’s already looking for sponsorships or other ways to fund his new programs, he’s aware that his school might have improved its way out of extra money and help.

“I have definitely used the phrase ‘victim of our own success,’” he said, “and that could be the reality.”

Looking for results

The program’s three-year anniversary doesn’t just create uncertainty for Renewal schools, it also raises questions about whether de Blasio’s signature turnaround program is working.

Some advocates of the mayor’s strategy have expressed concerns that his promise of rapid improvement was too aggressive. They say school turnarounds usually require well over three years, especially when they hinge on cultivating partnerships with social service organizations — a new task for many school leaders.

City officials have pointed to better attendance, test scores and graduation rates in some schools, but many others have not yet made significant academic gains. And researchers who have tried to sort out whether the program has led to academic improvements have reached mixed conclusions.

Using three years of test score data (including the results released last month), the Manhattan Institute’s Marcus Winters found that the program is generally boosting math and English scores in elementary and middle schools.

But using a different statistical method that compares Renewal schools to similar ones that didn’t receive extra resources, Teachers College’s Pallas found the program appeared to have no effect on test scores or graduation rates. (He has not yet updated his analysis to include the latest test scores.)

Renewal schools remain under pressure to raise their scores. To aid in that process, education staffers who work with the schools have pushed them to target increasingly specific groups of underperforming students, according to Partnership with Children’s Derek Anello.

“In the prior two years we were more general in what the goals were,” Anello said. Now, “There’s really a microscope on every number and how we move the needle.”

Yet even if schools don’t make huge strides this year, some observers say the mayor is unlikely to change course. Many argue that adding social services to high-need schools enhances students’ health and wellness, even if it doesn’t result in swift academic improvements. The city has invested heavily in creating social service-rich “community schools,” which include more than 130 schools outside the Renewal program.

Even if Renewal’s academic results are mixed, Professor Pallas of Teachers College predicts that de Blasio won’t face strong political pressure to scale back his resource-intensive approach to school improvement, which has generally earned support from local politicians and the teachers union.

The program “resonates with progressives’ desire to support community-based schools,” he said. But, he added, “at some point somebody’s got to make the difficult decision about whether the benefits are worth the investments they’re making.”

fact-check check

The NY Times tried to fact-check the mayor’s claims about Renewal test scores. Researchers say its analysis fell short.

PHOTO: Creative Commons / DMC Wilcox
The New York Times building

Reading the coverage of New York City’s Renewal program, it would be easy to conclude that the program isn’t working for most schools.

And on Thursday night, the New York Times continued in that vein, with a story about the turnaround program headlined: “For $582 Million Spent on Troubled Schools, Some Gains, More Disappointments.”

The story is framed as a fact-check on Mayor Bill de Blasio, who said at a press conference that Renewal schools are showing signs of progress since they had outpaced the city’s average growth in English and math scores.

In evaluating that claim, the Times points out that despite gains at some schools, most Renewal schools have actually not made progress closing the gap between their original scores three years ago and the city average. Some of the program’s fiercest critics seized on the analysis.

But according to three academics who study school performance, two of whom have studied the Renewal program’s impact, the Times’ characterization of the program as producing spotty results is problematic for the same reason de Blasio’s original claim of success doesn’t hold water. That’s because comparing Renewal test score data to city averages is poor evidence of whether the program is working.

The Times’ analysis can’t actually establish a causal effect.

The Times frames its analysis this way:

“To track the effects of the program, which gives schools a longer day and access to special services like vision care for students or mental health supports, The New York Times analyzed Renewal school performance on the 2016 and 2017 tests, as compared with the 2015 scores.”

The phrasing suggests that it’s reasonable to infer “the effects of the program” from test score changes, which is simply not possible, according to Thomas Dee, director of Stanford’s Center for Education Policy Analysis.

That’s because establishing a program’s effect depends on a model that can sort out what would happen to test scores without the program at all. One way to isolate that effect could be to compare low-performing schools that didn’t make it into the Renewal program with those that did, and study the difference in scores between the two groups of schools. But neither the education department nor the Times analysis attempted to do that, making those claims about the program’s impact misleading, experts said.

“There’s often this tacit assumption that we’ve learned about the true effect of the program from comparisons like this, and any researcher worth their salt will tell you that’s not the case,” Dee said.

Amy Virshup, a Times metro editor, defended the story’s analysis. “The piece never presumes to judge the success or failure of the Renewal program based on the ELA and math test results,” Virshup wrote in an email. “Judging whether Renewal is working or not would require many more data points and much more analysis.”

Renewal schools may be serving different students than when the program started.

Another reason the test scores could be misleading is that it has been well-reported that Renewal schools have lost a significant share of their students, continuing an enrollment drop-off that has persisted for years.

And since higher-performing students may be more likely to find a new school, it’s plausible that Renewal schools are serving a more challenging student body than when the program started.

“In general, the students that are most [likely to leave] are those who are higher performing,” said Marcus Winters, who wrote a report about the Renewal program for the conservative-leaning Manhattan Institute.

If that’s true, year-over-year test score comparisons wouldn’t be completely fair, since they would simply pick up changes in which students are served by Renewal schools — instead of the program’s real effect.

“The Times really didn’t do anything to ensure that their comparison to other schools was really comparable,” said Teachers College professor Aaron Pallas.

There’s no mention of rigorous research that has attempted to show causal effects.

Two researchers have tried to suss out whether the Renewal program is creating positive academic changes, and have reached different conclusions.

In an analysis that compares Renewal schools to similar ones that didn’t enter the program, Pallas found the program had essentially no effect on graduation rates or test scores. Meanwhile, using a different statistical model, the Manhattan Institute’s Winters found that the program is actually creating meaningful academic benefits.

The Times doesn’t cite either of those analyses, which would complicate the picture. The results of those research efforts suggest the Times’ description of mixed results is certainly plausible, but it isn’t directly supported by the data analyzed in the story.

Still, Winters said, the Times analysis is worth doing, as long as there are caveats, missing in this case, about what it can and can’t explain. “I don’t think it’s the definitive analysis of what’s going on,” Winters said. “But it’s not nothing.”