Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

IPS School Board Race 2018

Indiana teachers union spends big on Indianapolis Public Schools in election

PHOTO: Dylan Peers McCoy/Chalkbeat
IPS board candidate signs

The political arm of Indiana’s largest teachers union is spending big on the Indianapolis Public Schools board. The group donated $68,400 to three candidates vying for seats on the board this November, according to pre-election campaign finance disclosures released Friday.

The three candidates — Susan Collins, Michele Lorbieski, and Taria Slack — have all expressed criticism of the current board and the leadership of Superintendent Lewis Ferebee. Although that criticism touches on many issues, one particular bone of contention is the district’s embrace of innovation schools, independent campuses that are run by charter or nonprofit operators but remain under the district’s umbrella. Teachers at those schools are employed by the school operators, so they cannot join the union.

The trio was also endorsed by the IPS Community Coalition, a local group that has received funding from a national teachers union.

It’s not unusual for teachers unions to spend on school board elections. In 2016, the union contributed $15,000 to an unsuccessful at-large candidate for the Indianapolis Public Schools board. But $68,400 dwarfs that contribution. Those disclosures do not capture the full spending on the election. The three candidates endorsed by Stand for Children Indiana — Mary Ann Sullivan, Dorene Rodríguez Hoops, and Evan Hawkins — are likely getting significant unreported benefits.

Stand for Children, which supports innovation schools, typically sends mailers and hires campaign workers to support the candidates it endorses. But it is not required to disclose all of its political activity because it is an independent expenditure committee, also known as a 501(c)(4), for the tax code section that covers it. The group did not immediately respond to a request for information on how much it is spending on this race.

The candidates’ fundraising varied widely in the reporting period, which covered the period from April 14 to Oct. 12, with Taria Slack bringing in $28,950 and Joanna Krumel raising $200. In recent years, candidates have been raising significantly more money than had been common. But one recent candidate managed to win on a shoestring: Elizabeth Gore won an at-large seat in 2016 after raising about $1,200.

Read more: See candidates’ answers to a Chalkbeat survey

One part of Stand for Children’s spending became visible this year when it gave directly to tax campaigns. The group contributed $188,842 to the campaign for two tax referendums to raise money for Indianapolis Public Schools. That includes a $100,000 donation that was announced in August and about $88,842 worth of in-kind contributions such as mailers. The group has a team of campaign workers who have been going door-to-door for months.

The district is seeking to persuade voters to support two tax increases. One would raise $220 million for operating funds, such as teacher salaries, over eight years. A second measure would raise $52 million for building improvements. Donations from Stand for Children largely power the Vote Yes for IPS campaign, which raised a total of $201,717. The Indiana teachers union also contributed $5,000.

Here are the details on how much each candidate has raised and some of the notable contributions:

At large

Incumbent Mary Ann Sullivan, a former Democrat state lawmaker, raised $7,054. Her largest contribution came from the Indy Chamber Business Advocacy Committee, which donated $4,670. She also received $1,000 from Steel House, a metal warehouse run by businessman Reid Litwack. She also received several donations of $250 or less.

Retired Indianapolis Public Schools teacher Susan Collins, who is one of the candidates supported by the union, raised $16,422. The Indiana Political Action Committee for Education contributed $15,000. She also received several donations of $200 or less.

Ceramics studio owner and Indianapolis Public Schools parent Joanna Krumel raised $200. Her largest contribution, $100, came from James W. Hill.

District 3

Marian University Executive Director of Facilities and Procurement and Indianapolis Public Schools parent Evan Hawkins raised $22,037. His largest contributions from individuals were from businessmen Allan Hubbard, who donated $5,000, and Litwack, who donated $2,500. The Indy Chamber Business Advocacy Committee contributed $4,670 and web design valued at $330. He also received several donations of $1,000 or less. His donors included IPS board member Venita Moore, retiring IPS board member Kelly Bentley’s campaign, and the CEO of The Mind Trust, Brandon Brown.

Frost Brown Todd trial attorney and Indianapolis Public Schools parent Michele Lorbieski, who is one of the candidates supported by the union, raised $27,345. The Indiana Political Action Committee for Education contributed $24,900. She also received several contributions of $250 or less.

Pike Township schools Director of Information Services Sherry Shelton raised $1,763, primarily from money she contributed. David Green contributed $116.

District 5

Incumbent Dorene Rodríguez Hoops, an Indianapolis Public Schools parent, raised $16,006. Her largest contributors include Hubbard, who donated $5,000; the Indy Chamber Business Advocacy Committee, which gave $4,670 and web design valued at $330; and the MIBOR PAC, which contributed $1,000. She also received several contributions of $500 or less, including from Bentley.

Federal employee and Indianapolis Public Schools parent Taria Slack, who is one of the candidates supported by the union, raised $28,950. The Indiana Political Action Committee for Education contributed $28,500.

Innovation zone

Two more Denver schools win additional freedom from district rules

PHOTO: J. Zubrzycki/Chalkbeat
Alex Magaña, then principal at Grant Beacon Middle School, greeted students as they moved between classes in 2015.

Two more Denver schools this week won more flexibility in how they spend their money and time. The schools will create a new “innovation zone,” bringing the district’s number of quasi-autonomous zones to three.

The Denver school board on Thursday unanimously approved the schools’ application to operate more independently from district rules, starting in January.

The new zone will include Grant Beacon Middle School in south Denver and Kepner Beacon Middle School in southwest Denver. The two schools are high-performing by the district’s standards and follow a model that allows students to learn at their own pace.

With just two schools, the zone will be the district’s smallest, though Beacon leaders have signaled their intent to compete to open a third school in the growing Stapleton neighborhood, where the district has said it will need more capacity. The district’s other two innovation zones have four and five schools each.

Schools in zones are still district schools, but they can opt out of paying for certain district services and instead spend that money on things that meet their specific needs, such as additional teachers or aides. Zones can also form nonprofit organizations with their own boards of directors that provide academic and operational oversight, and help raise extra dollars to support the schools.

The new zone, called the Beacon Schools Network Innovation Zone, will have a five-member board of directors that includes one current parent, two former parents, and two community members whose professional work is related to education.

The zone will also have a teacher council and a parent council that will provide feedback to its board but whose members won’t be able to vote on decisions.

Some Denver school board members questioned the makeup of the zone’s board.

“I’m wondering about what kinds of steps you’re going to take to ensure there is a greater representation of people who live and reside in southwest Denver,” where Kepner Beacon is located, asked school board member Angela Cobián, who represents the region. She also asked about a greater representation of current parents on the board.

Alex Magaña, who serves as executive principal over the Beacon schools and will lead the new zone, said he expects the board to expand to seven members within a year. He also said the parent council will play a key role even if its members can’t vote.

“The parent council is a strong influence,” he said. “If the parent council is not happy, that’s going to be impacting both of the schools. I don’t want to undersell that.”

Other Denver school board members questioned the zone’s finances and how dependent it would be on fundraising. A district summary of the zone’s application notes that the zone’s budget relies on $1.68 million in foundation revenue over the next 5½ years.

Magaña said the zone would eventually seek to expand to four schools, which would make it more financially stable. As for philanthropic dollars, he said the zone would work to ensure any loss of revenue doesn’t hurt the schools’ unique programs or enrichment.

“I can’t emphasize enough that it won’t impact the schools,” he said.

Ultimately, Denver school board members said they have confidence in the Beacon model and look forward to seeing what its leaders do with their increased autonomy.