Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

Future of Schools

Mike Feinberg, KIPP co-founder, fired after misconduct investigation

PHOTO: Photo by Neville Elder/Corbis via Getty Images

Mike Feinberg, the co-founder of the KIPP charter network, has been fired after an investigation into sexual misconduct, its leaders announced Thursday.

KIPP found “credible evidence” connected to allegations that Feinberg abused a student in the late 1990s, according to a letter sent to students and staff. Feinberg denies the allegations.

“We recognize this news will come as a shock to many in the KIPP Team and Family as we struggle to reconcile Mr. Feinberg’s 24 years of significant contributions with the findings of this investigation,” the letter says.

It’s a stunning move at one of the country’s best-known charter school organizations — and one where Feinberg has been in a leadership role for more than two decades. Feinberg started KIPP along with Dave Levin in Houston in 1994, and Levin brought the model to New York City the next year. The network became known for its “no excuses” model of strict discipline and attention to academic performance.

KIPP says it first heard the allegation last spring. The network eventually hired the law firm WilmerHale to conduct an external investigation, which found evidence that Feinberg had sexually harassed two adults, both alums of the school who were then employed by KIPP in Houston, the network said.

“In light of the nature of the allegations and the passage of time, critical facts about these events may never be conclusively determined. What is clear, however, is that, at a minimum, Mr. Feinberg put himself into situations where his conduct could be seriously misconstrued,” KIPP wrote in the letter, signed by CEO Richard Barth and KIPP’s Houston leader, Sehba Ali.

Feinberg’s lawyer, Chris Tritico, told the Houston Chronicle that Feinberg had not been fully informed about the allegations against him.

“The treatment he received today from the board that he put in place is wrong, and it’s not what someone who has made the contributions he’s made deserves,” Tritico said.

Read KIPP’s full letter here.

Knock knock

House call: One struggling Aurora high school has moved parent-teacher conferences to family homes

A social studies teacher gives a class to freshman at Aurora Central High School in April 2017. (Photo by Yesenia Robles, Chalkbeat)

When Aurora Central High School held traditional parent-teacher conference nights, fewer than 75 parents showed up.

This year, by taking the conferences to students’ homes, principal Gerardo De La Garza says the school has already logged more than 400 meetings with parents.

“This is something a lot of our families wanted,” De La Garza said. “We decided we wanted to add home visits as a way to build relationships with our community. The attendance at the traditional conferences was not where we wanted it to be.”

The home visits aren’t meant to reach every single student, though — the school has more than 2,000 enrolled this year. Instead, teams of teachers serving the same grade of students work together to identify students who need additional help or are having some issues. On Fridays, when the school lets out early, teachers are to go out and meet with those families. In some cases, they also schedule visits during other times.

Some parents and students say they weren’t made aware about the change and questioned if it was a good idea, while others welcomed the different approach.

“I felt when we go home that’s kind of our space, so I wasn’t comfortable with it,” said Akolda Redgebol, a senior at Aurora Central. Her family hasn’t had a home visit. “My parents, they thought it was a little odd, too.”

A father of another Aurora Central senior spoke to the school board about the change at a meeting earlier this month.

“There’s been a lot of changes over all these years, but one thing we could always count on was the opportunity to sit down with our child’s teachers during parent teacher conferences,” he said. “I hope this new program works, I really do, but why stop holding parent teacher conference nights at the high school? I haven’t had a single meeting. I haven’t met any of his teachers this year. Also why weren’t the parents told? I got two text messages, an email, and a phone call to let me know about a coffee meeting, but not a single notice about cancelling parent teacher conferences.”

Research examining the value of parent-teacher conferences is limited, but researchers do say that increased parent engagement can help lift student achievement. This year, the struggling Commerce City-based school district of Adams 14 also eliminated traditional parent-teacher conference nights from their calendar as a way to make more use of time. But after significant pushback from parents and teachers, the district announced it will return to the traditional approach next year.

Aurora Central High School is one of five in Aurora Public Schools’ “innovation zone,” one of Superintendent Rico Munn’s signature strategies for turning around struggling schools.

The school reached a limit of low performance ratings from the state and last year was put on a state-ordered improvement plan. That plan allowed the school to press on with its innovation plan, which was approved in 2016 and grants it some autonomy for decisions on its budget, school calendar, and school model.

As part of the school’s engagement with parents, the school in the last few years has hired a family liaison, though there’s been some turnover with that position. The school also hosts monthly parent coffee nights, as has become common across many Aurora schools.

As part of the innovation plan, school and community leaders also included plans to increase home visits.

Home visits have also become popular across many school districts as another way to better connect with families. Often, teachers are taught to use the visit as a time to build relationships, not to discuss academic performance or student behavior issues.

That’s not the case at Aurora Central. Principal De La Garza said it is just about taking the parent-teacher conference to the parent’s home. And teachers have been trained on how to have those conversations, he said.

The innovation plan didn’t mention removing conference nights, however.

De La Garza said that’s because parent-teacher conferences are still an option. If parents want to request a conference, or drop by on Fridays to talk to teachers, they still can.

Those Fridays when students end classes early are also the days teachers are expected to make house calls to contact families.

Teachers are expected to reach a certain number of families each Friday, though school and district staff could not provide that exact number.

Bruce Wilcox, the president of the Aurora teachers union, said that it’s important to better engage families, but that balance is needed so not all of the responsibility is put on teachers who are already busy.

Wilcox said he would also worry about teachers having less access to resources, such as translators, during home meetings.

Maria Chavez, a mother of a freshman at Aurora Central, just had a home visit last week. She learned about the school’s strategy when she was called about setting up the visit.

Another, older daughter, was the interpreter during the home meeting with three teachers.

“For me, it was a nice experience,” Chavez said. “As parents, and even the kids, we feel more trust with the teachers.”

Chavez said she goes to parent-teacher conferences with her elementary-aged daughter, but doesn’t always have time for conferences with her high-school-aged daughter, so the home visit was convenient. Chavez also said she was able to ask questions, and said the teachers were able to answer her concerns.

“Maybe I wouldn’t say this should be how every conference happens,” she said, “but it is a good idea.”