study shows

The $100 million question: Did Newark’s school reforms work? New study finds big declines, then progress

PHOTO: TechCrunch/Creative Commons
Mark Zuckerberg

It was announced with much fanfare on Oprah in 2010: dramatic changes were coming to Newark’s schools, financed with $100 million from Facebook founder Mark Zuckerberg.

Those changes — including a new teachers’ contract and the expansion of charter schools — proved controversial and challenging to implement. But there hasn’t been a clear answer to the key question: Are students learning more now than they were then, thanks to the reform effort?

A new study, released Monday through the National Bureau of Economic Research, is among the first to try to answer.

It finds that by 2016, Newark students were making greater gains on English tests than they were in 2011. But the results are not uniformly positive. It finds no impact in math. And in both subjects, the reforms seem to have come with a cost: student achievement declined substantially in the first three years of the changes.

The study was funded by the Chan-Zuckerberg Initiative and conducted by a number of Harvard researchers, including Tom Kane, who said that the study’s results were independent of its funding source.

“This study confirms the progress that is being made in Newark schools,” Newark schools superintendent Chris Cerf said in a statement, “and shows that reforms undertaken — particularly in areas like citywide enrollment and expansion of high quality schools — are making a real difference for Newark students.”

Their findings have both educational and political import.

The 2010 announcement was heralded as a rare and powerful bipartisan alliance, with Democratic Mayor Cory Booker and Republican Governor Chris Christie united in their plan to make sweeping changes to Newark’s struggling schools and backed by Zuckerberg’s millions.

(The money — $100 million with another $100 million match — seemed eye-popping, though it only accounted for 4 percent of school spending in the city over the five years of the grant.)

The changes would include not only charter schools and performance-based pay for teachers, but also the closure of a number of district schools, and new enrollment system encompassing district and charter schools.

The initiatives soon came under scrutiny. Residents and students pushed back forcefully against some of the changes implemented by schools chief Cami Anderson. Ras Baraka, a school principal and sharp critic of Anderson, was elected mayor in 2014, beating Shavar Jeffries, who was more amenable to the reforms. Anderson was replaced by Cerf in June 2015.

Journalist Dale Russakoff wrote a largely critical account of changes that focused on how a large share of the Zuckerberg money went to high-paid consultants. Since, media reports have largely suggested that the approach failed and that the money was wasted.

Now, the central characters have largely moved on. But Zuckerberg and Booker, now a U.S. senator, are rumored to have national political ambitions, including potential runs for president in 2020.

Here are the key takeaways:

The overall effect of the reforms on student learning was mixed.

“By the fifth year of reform, Newark saw statistically significant gains in English and no significant change in math achievement growth,” the researchers conclude. “Perhaps due to the disruptive nature of the reforms, growth declined initially before rebounding in recent years.”

Source: “Assessing the Impact of the Newark Education Reforms”

The research, also released as a non-technical report, looks at two ways the reforms may have affected students: by making existing schools better and by moving students to more effective schools, including charters. Moving students to better schools did help, but existing schools didn’t consistently improve – and in the first three years got substantially worse in both subjects.

Something we still don’t know is what the cumulative impact of the reforms was on a student who attended Newark schools for five straight years. The study doesn’t answer that.

Another perhaps surprising finding, considering the common description of Newark schools as failing: The district had a growth rate before the changes that was about average for similar districts in New Jersey.

Students seemed to benefit from school closures.

The study finds students whose school was closed subsequently saw higher test score growth, particularly those moving into better schools. That’s consistent with other research. But Newark did not seem to close its worst schools, and even shut down a few schools that were average or above average. That might have limited the positive effects of closures.

Charter schools continued to outperform the district, but have grown less effective.

Part of the reform strategy was to expand Newark’s charter sector, since charter schools had been shown to substantially raise student test scores, relative to the district. The latest study found that charters continued to do better than the district, but the gap has essentially been cut in half.

That’s because charters’ effectiveness has decreased since 2011. It’s not clear why, but three times as many students attend charter schools in Newark now compared to 2010. That influx of new students and accompanying growing pains may be part of the explanation. 

The study also shows that charter schools serve different students than the district. Newark’s charter students are more likely to be African-American and female, and less likely to have a disability or limited proficiency in English.

The study comes with a few important caveats.

The spike in test-score growth toward the end of the five-year grant coincided with the introduction of a new test aligned with the Common Core, the PARCC. It also coincided with an increase in students opting out of state tests, both in Newark and statewide. The researchers try to account for this, but it’s not entirely clear if those changes skewed the findings.

Also, the researchers came to their conclusions by comparing test score growth of Newark’s students to students with similar backgrounds and in similar schools across New Jersey. That doesn’t guarantee that the study is able to isolate the effects of the reforms, but does allow for comparisons to places without the Zuckerberg money or attention.

The results don’t show whether the reforms “worked” — because that’s a complicated question.

The study is focused on standardized test scores, a significant limitation that means it doesn’t speak to other effects of the reforms on students. A separate analysis, funded by the Community Foundation of New Jersey and also released Monday, points out that high school graduation rates in Newark rose substantially in 2016 and 2017, after remaining flat between 2012 and 2015. Enrollment in the city schools has also trended upward in recent years. 

Source: “Moving Up: Progress in Newark’s Schools from 2010 to 2017”

The results also don’t account for political turmoil or the sense that the reforms were done to — rather than with — the community in Newark, whose schools had been under state control for a over two decades. An agreement was finalized in September to return them to community control.

“Ultimately we’re giving the parents the opportunity to have their democratic rights back,” Baraka told NPR, who argued in the same interview that the Zuckerberg dollars had not improved the school district. “There is no real kind of causal relationship between that money and the development of the traditional public schools in Newark.”

outside the box

How one Chicago principal is leaning on data to help black boys

PHOTO: Adeshina Emmanuel

Test scores were rising at Fuller Elementary School when Marilyn McCottrell took over in 2016. Yet troubling trends loomed behind the numbers.

“A lot of growth has been made,” said McCottrell, Fuller’s third principal in six years. “But that growth is not equal among students.”

She’s talking about black boys.

Black girls had driven most of Fuller’s academic improvement since the 2012-13 school year, when Chicago Public Schools handed management of the Bronzeville school over to the non-profit Academy for Urban School Leadership, which replaced the staff and principal in a turnaround effort. Black boys had improved much slower. They got most of the school’s Ds and Fs, and were much less likely than girls to meet or approach expectations for college readiness on state tests.

PARCC Scores

Last school year, McCottrell and her staff crunched the data and made changes at Fuller to shorten the gaps between boys and girls. The stakes are high. Black boys, especially those from low-income households, are more prone than their sisters to falling behind in school and running into the juvenile criminal justice system. As adults, they are more likely to be arrested, imprisoned, or chronically unemployed. McCottrell believes what Fuller did, starting with painstakingly crunching data at the school, classroom and individual levels, could help other schools do better for black boys.

But she wants to be clear about something: Black boys don’t need to be “saved.”

“They need to be respected and appreciated for the differences and the unique gifts that they bring to the educational experience,” she said.

Black boys
PHOTO: Adeshina Emmanuel
Fuller Elementary School students (from left)Tyrese Robinson-Guy, Terrell Johnson, and Jasean Waters at a community garden in Bronzeville.

 

Fuller, a Level 1 school in good standing, occupies the corner of St. Lawrence Avenue and 42nd Street in Bronzeville. Nearly all of its 370 students are black and come from economically disadvantaged households. About half of the teachers are white, and about half are African-American. When CPS turned over management of Fuller, it was seeking to lift up a school that had been on academic probation five consecutive years. Fuller still has far to go. In 2017, only 10 percent of Fuller students were ready for the next level compared to 26 percent across the school district and 34 percent across the state. Growth has been above average, but, as McCottrell said, that growth hasn’t been equal.

PARCC Scores

Last August, McCottrell arrived at Fuller for a training session for teachers bearing handouts packed with data on black boys’ grades and test scores. Middle school reading teacher Arlicia McClain was shocked to see the stark disparities.

“It made me buck up and say I need to talk to these students,” she said. “I need to know what is going on that is preventing them from improving. Is it me? Is it something going on with them individually? Is it something they are missing?”

Girls’ math scores had increased by 193 percent compared with 90 percent for boys since the turnaround effort began in the 2012-13 school year. The gender performance gap was even more striking in reading, where black girls’ scores jumped 140 percent compared with 31 percent for boys. 

As McClain and other teachers reflected on the numbers, they recounted their  own experiences in the classroom. For example, they could all name which students were removed from class the most for disciplinary reasons, and nearly all were black boys.

Arlicia McClain
PHOTO: Courtesy of Arlicia McClain
Fuller Elementary School teacher Arlicia McClain.

McClain realized she tended to call on black girls more in class.

McClain, African-American herself, wondered if she was favoring girls or failing to challenge boys enough, and how that could affect their learning. She resolved to push black boys more during her second year at Fuller. 

She also left the session with another big take-away: A lot of boys who wouldn’t participate in classroom-wide sessions engaged more in small groups. Wedding the data to her realizations has helped the young teacher come up with tailored approaches for struggling students.

“Look at them as individuals who want to learn, but who sometimes need the individualized attention to do that,” McClain said. “If you really are about the progression of black youth, you’re going to need to be individual-focused, and you’re going to need the data to do it.”

In the 2016-17 school year, for the subjects of English language arts and math, about 70 percent of all Ds and Fs at Fuller went to black boys.

In the first quarter of last school year, McCottrell and her staff revised Fuller’s grading policies in hopes of addressing the disparity.

They switched to what McCottrell called “a more equitable grading scale,” where the lowest a student could score is a 50, adopted a “no-opt out policy” for homework, so children who failed to turn in their homework by deadline wouldn’t automatically get a zero and had to make up assignments, and allowed students to redo certain parts of failed tests and quizzes after reteaching.

By the end of the first quarter, the numbers of Ds and Fs had decreased by nearly half.

But black boys were still getting about the same percent of them as before.

So McCottrell decided to go in for a closer look.

“The numbers only tell part of the story,” she said.

McCottrell ate with boys in the lunchroom. She played flag football with them at recess. She sat with them in class, assisted their teachers, and taught her own lessons across grades and subjects.

She talked to the boys — and listened.

Jasean Waters, a black boy

Jasean Waters, 13, said he found it hard to focus on his school work.

Some distractions come from inside the classroom, like the bullies Jasean’s run into. Other distractions live in the world outside Fuller, like the gun violence whose victims are overwhelmingly black males.

“It’s a big struggle for us,” he said. “There’s a lot of people dying around here, so we gotta watch our backs, and when we’re walking home we feel like we’re unsafe, so we just focus on us being safe. It’s hard to focus on school.”

Boredom is another issue. Jasean said that he does well in math, but struggles sometimes with reading, and that his interest wanes with the lack of characters and authors he can relate to in school texts. That sounded familiar to McCottrell.  When she spoke with boys, she heard that school amounted to a seven-hour suppression of their personalities, interests, and voices — especially in reading and English classes, where black voices and black writers were missing.

“When kids have to pick a book for independent reading, they don’t relate to the characters in those classroom libraries,” she said. “It’s really hard coming to a class everyday when nothing relates to you.”

Parcc Scores

McCottrell decided to teach an optional African-American literature class every Friday during a weekly “intervention time” for students needing help in reading and math  About 17 boys showed up on the first day and read excerpts from Ralph Ellison’s “Invisible Man,” whose protagonist proclaims, “I am invisible, understand, simply because people refuse to see me.”

McCottrell said many of the boys could expertly analyze the Harlem Renaissance classic, because they related to the idea of not being heard, seen, or understood for who they really are. The students offered examples like the portrayal of black men in the media.

“Many of them were saying things like, ‘I’m not a gangbanger, but this is what people think I am, because I’m dark or because I’m tall,’” she said. “They talked about it in the context of their teachers not knowing who they are.”

The class soon doubled as word of mouth drew others in. Jasean, a C student at the start of the class, joined them. He said he learned things he hadn’t been introduced to before. He read about segregation, speeches by Martin Luther King, and books like “Bud, Not Buddy,” about a 10-year-old black orphan during the Great Depression.

He said he rededicated himself to doing 100 minutes of reading a night and by the end of last school year earned an A in reading. He said he raises his hand to ask and answer questions in class more.

“It feels good,” he said.

Jasean’s grandmother, local school council member Regina Waters, praised McCottrell’s hands-on approach with students and her efforts to build one-on-one relationships with the boys.

“She’s upfront with the kids, and she knows all the kids by name which is unusual in the short time she’s been there,” Waters said.

McCottrell
PHOTO: Adeshina Emmanuel
Marilyn McCottrell

Fuller’s boys closed the gap with girls in several ways over last school year.

They went from getting 70 percent of the Ds and Fs in English and math to 60 percent. In 2016-17, 46 percent of boys compared with 55 percent of girls were on track, meaning they earned a C or higher in reading and math and had an attendance rate of at least 95 percent. In 2017-18, the percentage of boys on track increased by 23 percentage points compared to 19 points for girls. But sitting in her office at Fuller one day earlier this summer, McCottrell admitted something about her efforts for black boys.

“Nothing is solved,” she said.

Despite some progress last school year, when the 2018-19 school year starts, black boys at Fuller will still lag behind black girls. Forces outside of education like poverty, mass incarceration, and racial discrimination will continue to disadvantage black youth in ways that manifest in classrooms, where they land heaviest on black boys.

The odds aren’t yet even for black boys at McCottrell’s school, or at most schools across America. However, McCottrell believes that educators learned a lot that they can build on down the line.

Next year, McCottrell said she’s urging teachers to incorporate more of the black experience and black voices into lesson plans and to increase small-group instruction.

Teachers are having more data conferences with McCottrell and with each other to guide instruction and target specific students’ needs. McCottrell is also promoting more social-emotional learning techniques and restorative practices rather than punitive approaches to discipline, and incorporating cultural awareness and bias training into teachers’ professional development.

Marlene Aponte, the Academy for Urban School Leadership’s director of coaching,  said that in some ways Fuller’s story resembles other schools’ in the years after turnarounds. After focusing on rigorous instruction and ambitious growth targets,“we’re starting to really hone in on some of the pieces that we may have overlooked, such as gender bias, gender equity, access in equity,” she said.

McCottrell wants her boys to have the tools to succeed. She knows there are some issues that her school won’t be able to solve.

But it’s a start.

testing accountability

Pressuring schools to raise test scores got diminishing returns, new study of No Child Left Behind finds

PHOTO: Ann Schimke

Does tightening the screws on schools and teachers lead to benefits for students?

For the past couple of decades, school reform efforts have assumed that the answer is yes. Setting ambitious goals, and putting pressure on schools to reach them, would push students ahead. And past research has shown that math scores rose as more states began threatening and sanctioning schools with low test scores in the 2000s.

But a new study shows that continuing to “raise the bar” during the No Child Left Behind era only had a modest effect at best. That raises questions about whether the small gains were worth the political controversy, and what critics claim were the educational costs, of putting a greater focus on test scores.

“These results suggest that the ratcheting [up] of test-based accountability pressures alone is not enough to sustain improvements in student achievement,” conclude researchers Vivian Wong, Coady Wing, David Martin, and Anandita Krishnamachari.

Their paper, which has not been formally peer-reviewed, focuses on the several years after the federal No Child Left Behind law was signed in 2002. The law — which passed with bipartisan support but would eventually draw bipartisan ire — required states to test students annually and set goals for schools. Schools that didn’t meet them faced sanctions.

States each set their own targets using different tests. But the researchers attempted to ask the same questions of each state: How hard was it for each school to hit its goals, and how did that change between 2003 and 2011? Then, they looked at how students did on the National Assessment for Educational Progress. Did states see larger gains on the federal low-stakes test after making life tougher on schools?

In many states, it really did become harder and harder for schools to measure up. In 2008, Education Week noted that California’s school failure rate jumped from 34 to 48 percent between 2007 and 2008. In Vermont, the climb was even steeper: from 12 percent of schools failing to 37 percent.

This added pressure, the authors conclude, seemed to lead to national gains in eighth grade math and reading. But the effect was tiny: about half a point in both subjects. (For comparison’s sake, the difference in performance between white and black students in eighth grade math was 32 points on the latest test.)

“Though they find positive effects, like everyone in this literature, they are small [effects],” Tom Dee, a Stanford education professor.

That said, the gains were largest for certain disadvantaged groups: English language learners, Hispanic students, students with disabilities, and students who started at the lowest levels of performance.

There was no evidence of higher standards causing any improvements in fourth grade math or reading.

If this shows that raising the bar doesn’t do much, though, past research has shown that just having a bar can make a big difference.

In states that didn’t have accountability systems at all before No Child Left Behind, creating them led to big gains on national low-stakes math tests: 8 points in fourth grade and 5 points in eighth grade, according to a study from Dee.

Together, this research bolsters a theory known as the “accountability plateau” — that creating tougher rules boost performance, but ratcheting up the pressure leads to diminishing returns.

“It seems like when you implement an accountability system there’s an initial bump, but after that continued gains are hard to come by,” said Morgan Polikoff, a professor at the University of Southern California who has studied standards and accountability systems.

Dee was more skeptical of this idea. Schools’ goals were getting harder and harder to reach just as criticism of the law was cresting and politicians were considering changes.  

“Districts may have understood it was a nudge and a wink and it didn’t really have teeth,” he said of the law.

No Child Left Behind’s replacement, the Every Student Succeeds Act, takes a different tack. Instead of giving each state discretion in how many schools are identified as failing and requiring them to ramp up the consequences over time, the law requires each state to identify 5 percent of schools as low-performing.

The latest study suggests that might be a preferable approach if states are able to figure out better ways to help a small group of struggling schools improve. Turnaround efforts — including a prominent federal program backed by a lot of money — have often produced disappointing results.

“It remains unclear how states will implement ESSA,” write the researchers. “But the federal law will likely not succeed if performance requirements are not accompanied by additional support for educators.”