An education U-turn

Do struggling schools in New York City’s Renewal turnaround program outperform those left out? A new analysis suggests no

PHOTO: Twitter/NYC Mayor's Office
When Mayor Bill de Blasio unveiled the Renewal program in November 2014, he said the city would "move heaven and earth" to help the struggling schools improve. (Photo: Twitter/NYC Mayor's Office)

Just months after taking office, Mayor Bill de Blasio announced a big bet: He would flood schools that were among the city’s worst with extra social services and academic support and give them three years to improve.

Nearly three years — and $386 million — later, city officials say the program is showing academic benefits, citing increases in test scores and graduation rates among its 86 “Renewal” schools.

But those improvements look very different when the schools are compared against others that are most like them, according to a first-of-its-kind analysis. Schools that had similarly low graduation rates and test scores when the program started — but did not receive an infusion of new resources — posted similar growth on those two metrics.

That finding doesn’t mean the Renewal program isn’t making a difference at those schools. But it does illustrate the pitfalls of the mayor’s promise of “fast and intense” improvement — and how far the city still has to go to prove its school turnaround strategy has been worth the investment.

“With an initiative this ambitious, there’s a lot of political pressure for the administration to demand — and claim — quick academic gains,” said Megan Hester, an organizer at the Annenberg Institute for School Reform. But “deep, sustainable institutional change takes time.”

The new findings come from Aaron Pallas, a Teachers College professor who conducted a review of test score and graduation rate data comparing Renewal schools with others that were similarly low-performing when the program started, but did not get the extra resources. (Chalkbeat interviewed several principals at these under-the-radar schools and found that in many cases, they are charting their own approaches to serving some of the city’s highest-need students.)

Pallas matched the city’s 78 current Renewal schools with 90 others with similar graduation rates, test scores and student demographics — a comparable proportion of black and Hispanic students, for instance, and similar percentages of students with disabilities or coming from low-income families. Each Renewal school was matched with three different configurations of comparison schools based on how similar they were along those categories to avoid the possibility that any single match would skew the results.

(City officials disputed this method of comparing schools because they said it does not account for differences in school performance beyond test scores and graduation rates.)

First, the graduation rates: City officials have often noted that the 31 current Renewal high schools have boosted graduation rates from an average of 52 to an average of 59 percent since the program started more than two years ago.

Graduation rates: How Renewal schools compare to their “nearest neighbors”

But over the same time frame, 23 high schools that are demographically similar to Renewal schools and started with slightly lower graduation rates (averaging 50 percent) made bigger gains. When each Renewal high school is matched to its three “nearest neighbors,” the comparison schools improved to graduate 63 percent of their students — nearly six percentage points more growth than the Renewal schools.

That number fluctuates slightly depending on which combination of comparison schools are matched with Renewal schools. But the comparison schools outperformed Renewal schools using three different combinations of matches. After eliminating two comparison schools that made unusually large gains, the comparison schools still outperformed Renewal schools, though only by a small, statistically unreliable margin.

Using the same approach for English and math proficiency in grades 3-8, as measured by state tests, Pallas found there were no significant differences between Renewal schools and the comparison group, which started out with similar scores.

Twenty Renewal elementary schools, for instance, increased the percentage of their students proficient in reading from 7 to 15 percent over the two previous school years — an eight percentage point jump. But when matched to the Renewal schools, the 23 comparison schools jumped seven points, almost the same margin.

Elementary school math proficiency among Renewal schools grew to 10 percent, a three point bump — but the comparison schools jumped two points, winding up at an identical level of student proficiency as the Renewal schools. (Citywide, reading proficiency has jumped almost 10 points over the same period, and math proficiency has increased just over 2 points.)

At the middle school level, there was virtually no measurable difference in growth in Renewal school math and reading scores measured against the comparison group.

“Based on the best evidence I see,” Pallas said, “the program is not having a meaningful impact on academic outcomes.”

"I think the administration made itself very vulnerable by postulating big expectations for academic growth in renewal schools."Norm Fruchter,a researcher at NYU

The education department disputed that conclusion, arguing the findings are “grossly misleading” because the Renewal schools and the comparison schools aren’t comparable to begin with. City officials said all Renewal schools met a series of criteria — historically low performance, being labeled as “focus” or “priority” schools by the state because of their performance, and posting low scores on the school’s quality review — while none of the comparison schools met all three.

Eric Ashton, the education department’s executive director for school performance, noted that Renewal schools were chosen partly based on qualitative judgements of principal or teacher effectiveness — factors that weren’t directly considered in Pallas’s model.

“Renewal schools are the ones we expected to do worse to begin with,” Ashton said. “Just because you have two schools that start with the same graduation rates and demographics, that doesn’t mean they’re the same quality.”

Pallas acknowledged that there are some limitations to his approach. Unlike an experiment where schools are randomly assigned to the Renewal program, he used a statistical model to match schools with similar starting test scores, graduation rates, and student demographics in an attempt to tease out the effect that Renewal is having compared with schools outside the program.

That means differences in school quality that Pallas didn’t measure — principal effectiveness, for instance — could be important variables in explaining why Renewal schools did not improve more than the comparison schools. Still, Pallas argues his model is strong enough to show that the Renewal program has not yet produced convincing academic gains, a conclusion outside researchers said is reasonable.

“This research shows that positive gains by Renewal schools were not better than schools with similar demographics,” said Jonah Rockoff, an education researcher at Columbia University who has studied New York City schools in the past and reviewed Pallas’s data.

Rockoff echoed the city’s argument that Renewal schools were chosen because they were struggling, making comparisons difficult. But, he said, the findings are still “probably an indication that the Renewal program didn’t have dramatic short-run effects.”

Some supporters of the Renewal approach also point out that many of the city’s efforts, including partnerships with social service providers, took months to implement — leaving Renewal schools with just one full year of extra help so far.

And despite de Blasio’s claim that the schools would show quick gains, the city has also been clear that it sees improvements in attendance, student engagement, and school culture as measures of the program’s success. (City officials pointed out that attendance at Renewal schools has ticked up, and that graduation rates have increased faster since the program started.)

“I think the administration made itself very vulnerable by postulating big expectations for academic growth in Renewal schools,” said Norm Fruchter, a researcher at New York University, and a former de Blasio appointee to the Panel for Educational Policy. “The scale of resources for the system to intervene in poverty is a very hard thing to do, and it’s likely you won’t soon see the results you want.”

Clarification: This story has been updated to clarify that the elimination of two comparison schools that made unusually large gains in graduation rates resulted in a difference that was not statistically reliable.

Cultural Relevancy

Some teachers are spending their summer creating culturally relevant content

Participants at the Reimagining Education summer institute share thoughts in a workshop.

Growing up as a student of color in New York, Abigail Salas Maguire says she struggled to connect with the material she read.

“When I was in school and I was reading ‘Little House On The Prairie,’ that was hard to relate to–meadows, and rolling hills, what is that? I don’t know what that is,” said Maguire.

Now as a fourth grade teacher at the Emily Dickinson Elementary School in Manhattan, Maguire acknowledges that finding relatable, authentic literature can be a “struggle,” even for her. She wants not only books in Spanish for her dual language students but ones that represent them as well.

Last week, Maguire and more than 400 other educators came together at the Reimagining Education summer institute at Teachers College of Columbia University to learn how to provide students with academic experiences that reflect the diversity of their schools.

Amy Stuart-Wells founded the institute three years ago primarily to provide white educators with hands-on strategies for creating culturally relevant subject matter for their diverse student bodies.  She noticed a disconnect between policymakers and educators that she said was preventing the education system from adapting to the shifts in its student population.

In New York City, as in districts across the U.S., there is a gap between the race or ethnicity of teachers and many of their students. Eighty-three percent of students in New York City are Asian, black or Latino, while only 39 percent of teachers are, according to 2015-16 state data compiled by Education Trust-New York, an advocacy group that works to improve outcomes for students of color. (Education Trust and Chalkbeat both receive funding from the Gates Foundation.)

Some recent studies have linked teachers of color to better outcomes for students of color. They indicate that having a black teacher leads to black students having better test scores and fewer suspensions and expulsions.

The city has some programs in place to bridge this divide.  Earlier this year, the city pledged to spend $23 million on anti-bias training for its public school teachers. This includes $4.8 million for training for implicit bias and culturally responsive practices in 2019. The funding was largely a response to a push by activists, fueled by allegations of the mistreatment of students of color in some classrooms and discriminatory practices by some administrators.

But Stuart-Wells notes that the institute focuses more on teaching white educators, who make up the majority of the teaching force, how to create culturally relevant lesson plans and materials.

“This is a very grassroots kind of change. It’s not going to come through Washington, D.C. It has to come from the educator staff, from the people who have the expertise, in order for the program to be able to be effective,” said Stuart-Wells.

Over the course of four days last week, educators from 24 states and other countries heard from experts whose work was focused on equity pedagogy and had made strides in creating comfortable and inclusive environments for their students. After the presentations, participants broke off into workshops–or “pool parties,” as they were called–to debrief and share some of the ways they would apply what they were learning to their own communities.

Many of the people who lead the presentations and workshops are people of color, notes Detra Price-Dennis, an assistant professor of Elementary and Inclusive Education at the Teachers College of Columbia who has been a part of the institute since its creation.

We had faculty of color, who are former teachers and who have gone through professional development leading, and other people of color lead the workshops and give keynotes,” said Price-Dennis. “Part of it is also understanding that as a white educator you don’t have to be a person of color to appreciate the genius of the students in front of you- just appreciate.”

The institute touched upon the importance of providing students of color with spaces to learn specifically about their culture and shared some examples of this in practice. This year at the Harry S. Truman High School in the Bronx, students with families from African countries like Ethiopia, South Africa, and the Ivory Coast came together every Thursday afternoon to learn about their nation’s history and politics. Known as the “Africa Club,” the group was created by assistant principal Jordana Bell, who saw an opportunity to make students feel more comfortable and connected to their community.

Before the Africa club, there was no connection between teachers and students– it was just get your work, get out,” said Abel Zewde, a student at Harry S. Truman who spoke with a group of his peers about his experience in the Africa club. But in our differences we found unity. We learned what it truly means to be in a social group that you’re appreciated in.”

For recent Truman graduate Mannixs Paul Jr., being able to learn about his country at school meant finally being proud of his heritage.

“I never liked to wear native clothes from Nigeria,” said Paul. “But last week I looked at [Nigerian clothes] and I almost cried because I was like, it’s what I am missing! For a long time I’ve been trying to push away the culture. Now, I’m trying to make it a big part of my life.”

Training is just the starting point and may not be enough on its own, said Nicholas Donohue, president of the Nellie Mae Education Foundation, who attended the institute. Ensuring that students are given equal access and culturally relevant material not only means changing what is taught and how it’s taught, but also changing who is leading education reform and who holds positions of power.

“If you really want to change the system, the practices matter, but they will pale and fail if we don’t address structures,” said Donohue. “Reorganization isn’t enough, it’s about the policy and rules of engagement that drive the effort.”

For Stuart-Wells, being able to connect educators under a shared cause is what makes the institute meaningful.

“We are uniting people who are struggling with these issues, so we are creating and tapping into a movement to reclaim this field and honor the diversity of cultures that are around us, and bring them into the curriculum,” she said. “You come, we learn a lot and we stay connected.

An education U-turn

Carmen Fariña wants to help New York City teachers get better at teaching. But some of her own reforms are getting in the way.

PHOTO: Christina Veiga
The math team at P.S. 294 in the Bronx discuss a recent lesson during the 80 minutes of professional development time carved out by the city's most recent contract with the teachers union.

It was a Monday afternoon and school was out at P.S. 294. But there was plenty of learning happening inside the blue-and-yellow building in the Bronx.

Teams of teachers were gathered in classrooms on almost every floor. One group discussed a recent math lesson on how to identify patterns; another analyzed which questions had stumped students during recent statewide tests. A third was thinking about new ways to encourage discussion in the classroom.

In each huddle, they were learning a valuable lesson from each other: how to become better teachers.

What’s happening at P.S. 294 is what Carmen Fariña envisioned when she became chancellor of the country’s largest school system. Among the veteran educator’s most deeply held beliefs is that school improvement starts in the classroom — by helping teachers get better at teaching.

“To me, everything that happens in the classroom is the most crucial thing in the building,” Fariña told Chalkbeat.

Many of Fariña’s reforms reflect that vision, including the city’s contract with the teachers union, which carves out time for professional development each week. But another set of changes Fariña made — overhauling the education department bureaucracy — has sometimes worked at cross purposes, taking power away from those who know schools best.

Strapped superintendents and staffers sidelined in support centers now oversee much of the training teachers encounter. Fariña herself acknowledges it has sometimes been a struggle to meet the diverse needs of schools under the new system.

One Bronx principal said he sees that struggle firsthand.

“What some people call ‘supporting instruction with professional development,’ other people would call ‘bloated bureaucracy,’” the principal told Chalkbeat. “I have no interest in their professional development, and they don’t know my school.”

***

Like much of what has happened at the education department under Mayor Bill de Blasio, the chancellor’s emphasis on teaching the teachers marks a radical shift from the preceding administration.

Former Mayor Michael Bloomberg and Chancellor Joel Klein wanted great teachers in every classroom, too. But their position was that it was easier to hire top talent than cultivate it. Instead of pouring resources into teacher improvement, they set about measuring teachers to weed out those who were ineffective.

“Joel didn’t believe in professional development at all,” said Eric Nadelstern, who served as deputy chancellor for school support and instruction under Klein. “His question was, ‘Is it easier to change the teacher — or to change the teacher?’” Klein himself did not agree to be interviewed for this story.

When Fariña took the helm, educators took heart that one of them was in charge again. With 50 years of experience in New York City classrooms, she was the first chancellor in more than a decade who didn’t need a waiver, which the state requires when a school leader does not have the experience set by law for the job.

“When de Blasio named Fariña chancellor, it was a message,” said Norm Fruchter, a researcher at New York University who previously served as a de Blasio appointee to the Panel for Educational Policy. “The pendulum was going to shift back towards valuing instruction.”

In one of her first moves as chancellor, Fariña helped hammer out a contract with the United Federation of Teachers, the union that had clashed for years with Bloomberg and Klein. Among its most significant changes: giving teachers 80 minutes after school every Monday to work on improving their craft. The contract also created new leadership positions that gave extra pay to skilled teachers who agreed to take on coaching roles in their schools.

Taken together, those moves helped create a structure for helping teachers improve within their own schools.

“The thing with the most value in schools is time,” said Phil Weinberg, deputy chancellor of the department’s Division of Teaching and Learning. “The biggest thing that we’ve done is to honor the fact that learning has to happen by creating time.”

Share your thoughts on the quality of New York’s professional development for educators in our short survey.  

***

In the education world, there is much debate around whether professional learning really works. Plenty of research suggests that typical models do not. Educators have their own disparaging vocabulary to describe those models: drop-and-go, spray-and-pray, even drive-by professional development. The idea is that one-off lectures and workshops are rarely effective in changing teacher practice, let alone improving how much students are learning.

However, recent research suggests there are ways to get it right. A review of 35 different studies, released in June by the Learning Policy Institute, found common themes in professional learning programs that actually improve student performance. Those programs provide coaching, are collaborative and typically happen on the job — much like what’s happening at P.S. 294.

P.S. 294 The Walton Avenue School serves students who are traditionally among the city’s lowest-performing — those who are homeless, learning English, or have disabilities. Yet it outperforms the city average on standardized tests.

PHOTO: Christina Veiga
Daniel Russo (center) working with the math team at P.S. 294

The school has taken on Algebra for All, a de Blasio initiative that helps schools change the way they teach math. P.S. 294 also has teacher leaders paid to share their knowledge with teams of their colleagues. Those teams then work together in the 80 minutes each week reserved for professional development. All of that comes together under a principal, Daniel Russo, who makes sure teachers get the feedback they need to improve their practice.

“We come back every couple of months and say, ‘How are we doing on this? What fell by the wayside and what are ways that we can do better?’” Russo said. “Everyone is going to contribute to, and benefit from, the greater knowledge that there is in the room.”

For all its ambitions, the 80 minutes don’t always work as planned. In about a dozen interviews with teachers and principals, many school staff said they appreciate that the Monday sessions have provided time and space to think about their practice. But others said that time can feel wasted or forced.

“Everyone is very busy at our school, and that’s just another meeting that has to take place to plan more meetings,” a Bronx high school teacher told Chalkbeat. (The teacher, like many educators interviewed for this story, agreed to speak only on the condition of anonymity.)

“A lot of times we’re not really sure what we’re going to do on a given day,” the teacher said. ”It’s not very focused throughout the year.”

***

Why, then, are some schools making good use of the new training time and at others, teachers feel like it’s being frittered away?

One factor: changes to the way principals are supervised and how schools get support.

Under Bloomberg and Klein, principals who needed help turned to dozens of “networks” scattered throughout the city. Principals opted into networks based on their schools’ needs, regardless of where the school or network were located in the city. The network providers were expected to solve problems for schools, or principals could vote with their feet and join different networks.

"That’s the system-wide idea of support now: taking people away from kids."David Baiz, former principal of Global Technology Preparatory

As chancellor, Fariña took a different approach. She promptly rebuilt the department’s Division of Teaching and Learning, which had been dissolved after she left the DOE in 2007. Once again, there was an office at the Department of Education’s central headquarters dedicated to actively helping schools decide what and how to teach.

She also empowered superintendents, calling them the “instructional leaders” of their districts, and upped the years of experience required to land the job. They evaluate principals but are also responsible for making sure schools get the support they need.

In the place of networks, Fariña opened “field support centers,” which serve hundreds of schools but don’t hold supervisory power. Unlike networks, most centers only work with schools located in the same borough. Superintendents and support centers are expected to work together to help schools improve teaching.

Crucially, that doesn’t always happen. The result can work against the 80 minutes, by distancing decision-making about professional development from schools — and complicating it, too.

Our principal is “held with her hands behind her back,” said Corey Taylor, a music teacher at P.S. 33 in the Bronx. “She has to do what she’s being told by her higher-ups.”

Now, principals are expected to ask their superintendents for help, who then turn to field support centers. Some principals and support centers do work directly together, though Weinberg said that’s not the preferred system.

“The ideal thing is that you’re in constant conversation with your superintendent,” he said. “It would be hard for each borough field support center to hear 145 different requests every day, from each of their schools.”

PHOTO: Christina Veiga
Teachers at a training for Computer Science for All, a citywide initiative

Relationships between superintendents and support centers don’t always run smoothly, and both are tasked with overseeing many schools. Superintendents have staffs of around six people, yet may be responsible for dozens of schools. Support centers work with up to 323 schools, with an average caseload just below 200.

With superintendents acting as a filter between schools and support centers, many principals report a divide between what they’re offered and what they want to learn.

“There’s a disconnect between the reality of what’s going on in classrooms, and the offerings,” one Manhattan principal told Chalkbeat. “It usually comes down to: Teachers need to learn, very specifically, techniques, tips, philosophies that affect their own practice.”

When they work well, support centers might send staff to a school to provide targeted help requested by its principal. But, faced with heavy caseloads, the centers often respond to schools’ needs by creating borough-wide professional development sessions that can vary in quality. In the city’s most recent survey of principals, only 73 percent said they were satisfied with the support they get from the centers.

One Manhattan teacher said she went to sessions offered by the support center last year and was disappointed with what she found. The presenters led a lesson on “guided reading,” a technique that includes introducing vocabulary and breaking students into groups, but they seemed fuzzy on how to execute the practice in the classroom.

“Teachers were actually correcting them,” the teacher said. “They’re removed and they forget what it’s like to be a teacher.”

***

Despite Fariña’s emphasis on classroom-based learning, many of the support centers’ professional development sessions are happening outside schools, while class is in session. At three separate support centers, almost all the trainings for teachers offered during the month of May were held during school hours.

"We’ve set aside the time. We’ve set up the space. Can we just manage it?"Michael Mulgrew, president of the teachers union

That wouldn’t have happened under Bloomberg, according to Nadelstern, the former Klein deputy. He said his policy was that teachers and principals should not be pulled away from schools while students are in the building.

“That’s the system-wide idea of support now: taking people away from kids,” said David Baiz, the former principal of Global Technology Preparatory in East Harlem. “That’s not really the best way that adults learn: to sit in a meeting away from the context of their work environment and then try to come back and incorporate it.”

In addition to out-of-office professional development, superintendents host monthly meetings, pulling principals out of their schools for the entire day. In some cases, they include meals paid for by vendors who present professional development sessions based on educational products they’re selling.

“There’s just this feeling among almost every principal that I know,” a Bronx principal told Chalkbeat. “Like meeting after meeting after meeting and requirement after requirement are being added, and really drowning out the time needed for real collaboration.”

In an interview with Chalkbeat, Fariña admitted that professional development run by outside vendors is “not that effective.” She also acknowledged there have been growing pains as the superintendents and field support centers try to meet the needs of all the schools they serve.

“It’s been more of a struggle in some places where there was a more diverse need,” she said.

PHOTO: Christina Veiga
Schools chancellor Carmen Fariña looks over a card from students in her office.

To address that, Fariña said the centers have been working on “modules” based on different areas of need. A module may highlight effective strategies for teaching students who are learning English, for instance, and come with a series of professional development courses that can be run over a period of multiple weeks.

“Each principal can adapt it as they see fit,” Fariña said.

***

Weinberg said it is easy, in a system as large as New York City, to point to “random” weak points.

“What our real goal is, is continuous improvement,” he said. “I think that we make mistakes, oftentimes, by looking at one anecdotal example as a way of disproving a larger movement.”

Michael Mulgrew, president of the teachers union, said the department needs to pay closer attention to how schools are using the resources that are now available. While the the 80 minutes of professional development time is a game-changer, he said, it can also vary in usefulness depending on school culture, principal leadership and how well superintendents and the field support centers can provide help.

“We’ve set aside the time. We’ve set up the space. Can we just manage it?” Mulgrew said. “The fact that the chancellor made this a priority when she came in is the reason why you see the school system moving forward. My fear is, have we reached a plateau?”

It may be tricky, however, to balance the kind of oversight that Mulgrew envisions with the personalization that teachers and principals say is necessary for effective professional development. But the city is evaluating its own work to make sure it’s hitting the mark for teachers and schools.

“Teaching is really fascinating and difficult work,” Weinberg said. “We need to approach this hard job with the humility that says we have the ability to learn more — and we want to learn more.”

Chalkbeat reporters Monica Disare and Alex Zimmerman contributed to this report.