Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

School choice

Secret CPS report spotlights big vacancies, lopsided options for students

The school district says the report will help inform how it invests in and engages with communities. Communities groups worry the document will be used to justify more school closings, turnarounds and charters.

An unreleased report by a school choice group backed by the business community paints in stark detail what many Chicagoans have known for years: that top academic schools are clustered in wealthier neighborhoods, and that fewer black and Latino students have access to those schools.

The report highlights startling figures: About 27 percent of black students are in the district’s lowest-rated schools, compared with 8 percent of Latino students and 3 percent of whites. It also says that while Chicago Public Schools has more than 150,000 unfilled seats, 40 percent, or 60,000 of them, are at top-ranked schools. That surplus will grow as enrollment, which has been plummeting for years, is projected to decline further by 5.1 percent over the next three years. What that means is the cash-strapped district is moving toward having nearly one extra seat for every two of its students.

The document effectively shows that, in many areas of the city, students are skipping out on nearby options, with less than half of district students attending their designated neighborhood schools.

In a city still reeling from the largest mass school closure in U.S. history, this report could lay groundwork for another round of  difficult decisions.

The “Annual Regional Analysis” report, compiled by the group Kids First Chicago on CPS’ behalf, has been circulating among select community groups but has not been made public. It comes on the heels of a report showing students’ high school preferences vary with family income level. Students from low-income neighborhoods submit more applications than students from wealthier ones and apply in greater numbers for the district’s charter high schools.

The group behind the latest report has had many iterations: Kids First is a new name, but its origins date back to 2004, when it started as the charter fundraising group Renaissance Schools Fund. That was during the Renaissance 2010 effort, which seeded 100 new schools across the city, including many charters. The group changed its name to New Schools Chicago in 2011 and again rebranded this year as Kids First, with a greater focus on parent engagement and policy advocacy.

The report has caused a stir among some community groups who’ve seen it. Because the school district has used enrollment figures to justify closing schools, some people are worried it could be used to propose more closings, turnarounds, and charter schools.

“To me this is the new reason [for school closings],” said Carolina Gaeta, co-director of community group Blocks Together, which supports neighborhood schools. “Before it was academics, then it was utilization, now it’s going to be access and equity. Numbers can be used any way.”

In a statement on the report, Chicago Teachers Union Spokeswoman Christine Geovanis blasted Mayor Rahm Emanuel’s administration for policies that she alleged “undermine enrollment at neighborhood schools,” such as the proliferation of charter schools, school budget cuts, and building new schools over the objection of community members.

Reached by phone Thursday, Kids First CEO Daniel Anello confirmed that his organization helped put the report together, but declined to comment on its contents, deferring to the district. CPS Spokeswoman Emily Bolton acknowledged the report’s existence in a statement emailed to Chalkbeat Chicago that said the school district “is having conversations with communities to get input and inform decisions” about where to place particular academic programs. The statement said CPS is still in the process of drafting a final version of the document, but gave no timetable. Mayor Rahm Emanuel’s office didn’t grant requests for interviews about the Annual Regional Analysis.

Below is a preview of the report provided to Chalkbeat Chicago.

Gaps in access to arts and IB programs

Data released this week from the district’s GoCPS universal high school application clearly shows what academic programs are most in demand: selective enrollment programs that require children to test in;  arts programs; and career and technical education offerings, or CTE.

The Kids First’s analysis puts those findings into context, however, by detailing how supply is geographically uneven, especially when it comes to arts. Maps in the report divide the city into regions defined by the city’s planning department and show how highly-desirable arts programs are not spread equally throughout the city, and are most concentrated along the northern lakefront and downtown.

PHOTO: Sam Park
This map shows the number of fine & performing arts program seats available per 100 elementary school students in each planning area.

Worse, four regions offer 10 or fewer arts seats per 100 students, including the Bronzeville/South Lakefront region that includes neighborhoods such as South Shore, Woodlawn, Kenwood and Hyde Park. They are also scarce in the West Side region, which includes Austin, North Lawndale, and Humboldt Park and in the Northwest neighborhoods of Belmont Cragin, Dunning, and Portage Park.

The report also shows an imbalance in the number of rigorous International Baccalaureate programs.

This map shows the number of IB program seats per 100 students available to elementary and high school students in each planning area.

The highest number of IB seats are in the wealthy, predominately white Lincoln Park area. In contrast, there are far fewer IB seats in predominantly black communities such as  Englewood and Auburn Gresham, Ashburn and in the predominantly Latino Back of the Yards.

When it comes to selective-enrollment elementary school programs such as gifted centers and classical schools, which require students to pass entrance exams, options tend to be concentrated, too, with fewer choices on the South and West sides of the city. This map shows where selective enrollment high school options are most prevalent:

PHOTO: Sam Park
This map shows the number of selective enrollment high school seats available per 100 students in the city’s planning regions.

STEM programs are more evenly distributed across Chicago than both IB and selective enrollment schools, yet whole swaths of the city lack them, especially on the South Side, including the Greater Stony Island. As the other maps show, that region lacks most of the high-demand academic programs the district has to offer.

PHOTO: Sam Park
This map shows the number of STEM program seats available per 100 elementary school students.

Racial disparities in school quality

The analysis also shows disparities in quality of schools, not just variety.

At CPS, 65 percent of students districtwide are enrolled at Level 1-plus or Level 1-rated schools. But only 45 percent of black students and 72 percent of Latino students are in those top-rated seats, compared with 91 percent of white students.

The disparities are even more severe given that the school district is mostly Latino and black, with fewer than one in 10 students identified as white. 

A page from a presentation of the Annual Regional Analysis showed to select community groups.

In the Greater Lincoln Park region, 100 percent of elementary schools have one of the top two ratings — the highest concentration of them in the city.  The highest concentration of top-rated high school seats, 91 percent, is in the Central Area, which includes Downtown and the South Loop.

The lowest concentration of top-rated elementary seats, 35 percent, is in the Near West Side region, and the lowest concentration of high school seats, 14 percent, is in the West Side region.

Long commutes from some neighborhoods

The number of students choosing schools outside their neighborhood boundaries has increased in recent years.

But the report shows that school choice varies by race: 44 percent of black students attend their neighborhood elementary school, compared with 67 percent of Latino students, 69 percent of white students, and 66 percent of Asian students. For high schoolers, only 14 percent of black students attend their neighborhood school, compared with 28 percent of Asians, 30 percent of Latinos, and 32 percent of whites.

More students enrolling outside their neighborhood attendance boundaries means more and more students have longer commutes, but how far they travel depends on their address. 

Again, this is an area where the Greater Stony Island area stands out.

A graphic from the Annual Regional Analysis executive report that shows how far elementary school students in each of the city’s 16 planning regions travel from their homes to school. The data shows that students on the South and West Sides tend to have longer commutes.

The average distance traveled for elementary school students is 1.5 miles — but K-8 students in Greater Stony Island travel an average of 2.6 miles. The average distance to class for high schoolers citywide is 2.6 miles, but students in the Greater Stony Island region travel an average of 5 miles, about twice the city average. 

A graphic from the Annual Regional Analysis executive report that shows how far high school students in each of the city’s 16 planning regions travel from their homes to school. The data shows that students on the South and West Sides tend to have longer commutes.

Looking forward

The introduction to the Annual Regional Analysis describes it as “a common fact base” to understand the school landscape. It clearly states the intent of the report is to assist with district planning, not to provide recommendations.

It still bothers Wendy Katten, founder of Raise Your Hand, who has seen the report and said it tells little about how kids are actually learning at schools.

“It sounds like some data a company would use to reduce inventory at a manufacturing plant,” she said.

Gaete with Blocks Together said the numbers in the report are also missing important context about how the proliferation of charter schools, a lack of transparent and equitable planning, and a lack of support for neighborhood schools in recent decades has exacerbated school quality disparities across race and neighborhoods in Chicago, one of the nation’s most diverse but segregated cities.

It’s unclear when the final study will be published, or how exactly the school district will use its contents to inform its decisions and conversations with communities.

But an event posting on the website for Forefront, a membership association for “nonprofits, grantmakers, public agencies, advisors, and our allies,” mentions a briefing for the report on Oct. 10.

Kids First Chicago CEO Dan Anello and CPS Director of Strategy Sadie Stockdale Jefferson will share the report there, according to the website.

Q and A

In a wide-ranging interview, Carranza takes issue with admissions to New York City’s gifted programs

PHOTO: Christina Veiga
Since becoming schools chancellor, Richard Carranza has questioned city admissions methods that critics say exacerbate segregation. Here, he speaks to a crowd at a town hall about school diversity.

Ever since the city launched a push to scrap the entrance exam for its vaunted specialized high schools, Chancellor Richard Carranza has made it clear that he doesn’t believe a single test should be used to make school admissions decisions.

In an exclusive back-to-school interview with Chalkbeat on Friday, he said that also goes for the city’s gifted and talented programs.

Just like specialized high schools, gifted programs are deeply segregated. Only 22 percent of students in gifted programs are black or Hispanic, compared with 70 percent citywide. And just like specialized high schools, admission to most of the city’s gifted programs hinges solely on the results of an exam.

“I think that’s not a good idea,” Carranza said. “When you look at the disparities in representation across this system, you have to ask the question, ‘Do we have the right way of assessing and making decisions about students?’”

Most students enter gifted programs when they’re in kindergarten, so they are only 4 years old when they take the test — an approach that Carranza questioned.

“There is no body of knowledge that I know of that has pointed to the fact that you can give a test to a 4-year-old or a 5-year-old and determine if they’re gifted,” he said. “Those tests — and it’s pretty clear — are more a measure of the privilege of a child’s home than true giftedness.”

A full transcript of our interview with the chancellor is coming soon. We’ll have interesting insights about Carranza’s relationship with his predecessor, what he thinks about the city’s Renewal turnaround program now that he’s had time to get to know it better, and the problems he’s trying to solve with a recent bureaucratic overhaul. Here are some highlights to hold you over until then.

Why few schools may get shuttered under Carranza’s leadership — even though he’s ‘not scared’ of closures

In one of his very first moves as chancellor, Carranza spared a storied Harlem school that was slated for closure. Since then, he has shaken up the school’s leadership, initiated new partnerships, and brought in a different support structure.

It’s just one example, but it could be a hint of what’s to come during Carranza’s tenure.

The school that won the reprieve is a part of the mayor’s high profile Renewal program, which aims to boost student learning by offering social services and a longer school day. The program has shown mixed results, at best, and many Renewal schools have been shuttered after failing to make progress. 

Carranza indicated there could be more closures ahead: “Let me be clear: I’m not scared of closing a school if it’s not serving the needs of the students,” he said.

But he added: “My experience — nine times out of 10 — has been that we haven’t done all we can do to give schools that are struggling to improve the right conditions, the right resources and the right support to actually improve.”

Did Carranza push City Hall to do something about segregation at specialized high schools?

City Hall has indicated that its plans to overhaul admissions at the city’s vaunted specialized high schools had been in the works for some time. Indeed, de Blasio promised to do something about the stark underrepresentation of black and Hispanic students at the schools during his first run for mayor.

Carranza wouldn’t reveal much about what happened behind the scenes in the lead-up to the city’s June announcement that officials would lobby to scrap the exam that serves as the sole entrance criteria for specialized high schools. The chancellor said he brought up the issue in his talks with the mayor before coming onboard, and said his boss shared the same vision.

“I can tell you the mayor is passionate about making sure that our schools are just as diverse as our city,” Carranza said.

Asked whether he personally played a role in the decision, Carranza would only say that the mayor “knew what he was getting,” when he was tapped to be chancellor.

He later added: “One of the things that I appreciate is, that what the mayor hired was an educator to be the chancellor, and he lets me do my job.”

Alex Zimmerman contributed reporting.