Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

Time crunch

Specialized high schools lawsuit could delay admissions decisions, New York City says in court filings

PHOTO: Christina Veiga/Chalkbeat
Asian-American parents and community leaders gathered in Brooklyn to learn about a lawsuit against part of the city's plan to integrate specialized high schools.

New York City students may have to wait longer than usual to learn where they’ve been accepted to high school as the city prepares for a ruling in a lawsuit challenging integration efforts, according to court records filed Wednesday.

The city asked Judge Edgardo Ramos to rule by Feb. 25 on a preliminary injunction to block admissions changes aimed at enrolling more black and Hispanic students in the city’s prestigious but segregated specialized high schools.

A decision would be needed by then in order to meet the “latest feasible date to mail offer letters,” which the city says would be March 18 given the tight timeline around the admissions process.  The delay would apply to all students, not just those vying for a seat at a specialized high school. 

“DOE is mindful that a delay in the mailing of high school offers will increase anxiety for students and families, cause complications for those students considering private school offers, and require specialized high schools and non-specialized high schools to reschedule and restaff open houses,” the letter said.

A spokesman for the education department said the city will “communicate with families when a final offer date has been determined.” Letters were originally scheduled to be sent by March 4.

Filed in December, the lawsuit against the city seeks to halt an expansion of the Discovery program, which offers admission to students who scored just below the cutoff on the exam that is the sole entrance criteria for specialized high schools. The Discovery expansion, slated to begin this year, is one piece of Mayor Bill de Blasio’s plan to diversify the elite high schools.

Asian-American parents and community organizations say the expansion unfairly excludes their children. Asian students make up 62 percent of enrollment at specialized high schools, but comprise only 16 percent of the student body citywide.

The suit calls for a preliminary injunction, which would put the Discovery expansion on hold while the case winds its way through the courts — and would disrupt the admissions cycle already underway for eighth-graders enrolling in high school next year. The process of matching students to specialized high schools was scheduled to begin this week, the city’s letter states.

The plaintiffs wrote a letter supporting the city’s timeline for a decision on the preliminary injunction, saying their aim is to stop the proposed admissions changes “before they can have the anticipated discriminatory effect.”

Hitting pause on the Discovery program expansion would mean the education department has to recalculate the cutoff score for admission to specialized high schools, consult with principals, work with the test vendor to verify scores, and generate offer letters to send to students, city attorney Marilyn Richter wrote in a letter to the judge.

The education department “has never made such a significant course adjustment midstream in the process before,” she wrote.

Education Inequalities

Is Michigan ready for a ‘grand bargain’ to improve its struggling education system?

PHOTO: Lori Higgins/Chalkbeat
Teresa Weatherall Neal, superintendent of the Grand Rapids school district, speaks during a panel discussion that also featured, from left, Detroit Free Press columnist Rochelle Riley, Detroit Superintendent Nikolai Vitti and former U.S. Education Secretary John King.

A new political dynamic in Lansing has put Michigan in a position to potentially see the kind of education transformation that helped catapult Massachusetts to its status as the top-performing state in the nation, says a former U.S. Secretary of Education.

That bit of optimism from John King, who served as education secretary from 2016 to 2017 and is now the president and CEO of The Education Trust, came after nearly three hours of sobering discussion during an event Wednesday about the need to address inequities in education in Michigan.

Several times, King used the term “grand bargain” to describe what Michigan needs. It’s a term many Detroiters will remember from the Detroit bankruptcy and the deal that was a key part of getting the city out of bankruptcy.

As it relates to education, the term “grand bargain” has been used to refer to the bipartisan agreement struck more than two decades ago in Massachusetts that had broad buy-in from business and education groups, teachers, and parents, to improve academic achievement. The gist: The state invested more money in schools. In return, standards and accountability were increased.

King said Michigan is poised to reach the same consensus and invest more money in education, in particular investing money more equitably so the highest-need students are getting the most funding. He said there needs to be a thoughtful approach to accountability, as well as more investments in teacher preparation and support.

He sees it happening because of new leadership at the state level, including Democratic Gov. Gretchen Whitmer and new Republican leaders of the House and Senate.

“This is a moment where the new leadership in Lansing could come together … and it could be truly transformative.”

The event Wednesday, which took place at the Charles H. Wright Museum of African American History, was organized by the Education Trust-Midwest, a Royal Oak-based education advocacy and research organization. The topic was inequities in education and the need to provide equitable opportunities for children regardless on where they live.

Amber Arellano, executive director of the organization, said Michigan ranks 43rd out of 47 states for the funding gap between poor and wealthy school districts.

“Students and families pay the price for this under-investment,” Arellano said.

She said the experiences of states like Massachusetts that have seen striking improvement provide hope for Michigan because they show transformation can happen over five to 10 years.

Michigan has been falling behind other states in performance on the National Assessment of Educational Progress, a national exam that tests a representative sample of students in each state. The state’s performance, in fact, has shown little to no improvement over the last decade.

“We rise and we fall together. In Michigan’s case, we’re falling together,” Arellano said.

The audience at the event’s two panel discussions also heard from speakers such as Nikolai Vitti, superintendent of the Detroit school district. He said that often, when K-12 educators talk about the need for more resources, “you see the rolling eyes of lawmakers.”

But, Vitti said, “It takes funding to educate children. And it takes more funding to educate children who enter … with more challenges.”

Panelists agreed that Michigan is at a pivotal moment because of its new leadership. And they came up with solutions they think will make a difference.

Melody Arabo, outreach specialist at EdReports.org and former Michigan Teacher of the Year, said the state needs to address a lack of resources for educators. She said that in a classroom of 30 students, a teacher can have some reading at the kindergarten level and others “who can read better than I can.”

The materials that teachers have “are not meeting those needs. Teachers spend an average of 12 hours a week going online looking for resources.”

David Meador, vice chairman and chief administrative officer of DTE Energy, said Michigan should look at what successful states have done and adopt best practices. Just as important, he said:“Stick with it. Don’t change it every year.”

For King, the solution to improving schools is simple and starts at a young age.

“If Michigan is going to improve, it will need a surge in high quality early learning that prepares every child for kindergarten or beyond,” King said.