where's the research

Summit Learning declined to be studied, then cited collaboration with Harvard researchers anyway

English teacher Adelaide Giornelli works with ninth grade students on computers at Shasta charter public high school, part of the Summit public school system. (Photo by Melanie Stetson Freeman/The Christian Science Monitor via Getty Images)

Summit Learning, a fast-growing “personalized learning” system, touts a partnership with Harvard researchers even though Summit actually turned down their proposal to study the model.

The online platform is backed by Facebook founder Mark Zuckerberg’s philanthropy and is now being used in 380 schools across the U.S.

The program “is based on collaborations with nationally acclaimed learning scientists, researchers and academics from institutions including the Harvard Center for Education Policy Research,” Summit’s website says. “Summit’s research-backed approach leads to better student outcomes.” Schools have used that seeming endorsement to back up their decision to adopt the model.

In fact, though, there is no academic research on whether Summit’s specific model is effective. And while Summit helped fund a study proposal crafted by Harvard researchers, it ultimately turned them down.

“They didn’t tell us explicitly why,” said Tom Kane, a Harvard education professor and faculty director of the Center for Education Policy Research. “All I can say is that the work that we did for Summit involved planning an evaluation; we have not measured impacts on student outcomes.”

Summit’s founder Diane Tavenner said the organization had a number of reasons for not moving forward with the proposed study, including its potential to burden teachers and to limit the platform’s ability to change or grow. Their general approach is backed by other research, she said, and their track record as a charter network.

As to the mention of the Harvard center on Summit’s website, Tavenner said the organization had learned a lot from the process of developing a potential study. Tavenner said that, after Chalkbeat began reporting this story, she offered to change the website’s language, but said Kane had not asked her to do so.

More broadly, Tavenner says she is skeptical of the usefulness of large-scale research of the sort the Harvard team proposed, saying the conclusions might be of interest to journalists and philanthropists, not schools.

“I’m not willing to give up what’s best for kids for those two audiences,” Tavenner told Chalkbeat last month.

It’s a notable stance for Summit, given its ambitious claims and the platform’s wide reach.

As “personalized learning” becomes a more popular idea among those trying to improve America’s schools, Summit’s platform has been adopted for free by schools across the country. That’s thanks largely to the backing of the Chan Zuckerberg Initiative, the philanthropy poised to receive Zuckerberg’s billions. Summit’s model has drawn praise from parents and teachers in some schools, but proven controversial in others.

Regardless, CZI’s support means Summit could continue to grow rapidly — which has some observers wondering when its backers will show that what it’s offering is particularly effective.

“I do think that there is an obligation to provide credible evidence to schools when you’re trying to convince them to adopt things,” said John Pane, a researcher at the RAND Corporation who has extensively studied personalized learning initiatives.

Summit spreads, but research talks with Harvard team fizzle

Summit’s claims about a Harvard collaboration have their roots in conversations that began in  late 2016.

Zuckerberg’s wife, pediatrician Priscilla Chan, took a fateful tour of a school in the Summit Public Schools charter network two years earlier. The network soon began working with a Facebook engineering team to build out its technology.

Summit’s model has a number of components: a curriculum in core subjects for grades four through 12; weeks scheduled for students to deeply examine a topic of interest; long-term mentors for students; and a technology platform, which serves as the approach’s organizing structure. The goal is to better engage students and to give them more control over what and how they learn, Summit says.

By the 2016-17 school year, Summit had rolled out its program to more than 100 schools outside its own network. That’s also about when Summit started talks with Harvard professors Marty West and Kane.

An ideal study might have randomly assigned schools or students to use the learning platform, creating two groups that could be compared. That was a non-starter for Tavenner, as it would limit schools’ access to the platform. If 250 schools were assigned to use it, and another 250 expressed interest but were not, for example, that would be bad for students, she said last month while discussing the organization’s approach to research.

“Am I really going to say to 250 people, ‘You know what, we’re not going to actually help you, even though we actually could right now?’” she said.

Kane says they came up with a few alternatives: comparing students using Summit to others not using it in the same school or comparing schools that had adopted Summit to similar schools that hadn’t. They suggested tracking test scores as well as suspensions and attendance, measuring the effectiveness of the support offered to teachers, and using surveys to measure concepts important to Summit, like whether students felt in control of their schoolwork.

But Summit passed on an evaluation. “After many conversations with Harvard and the exploration of multiple options, we came to recognize that external research would need to meet certain baseline criteria in order for us to uphold in good faith our partnership with schools, students, and parents,” Tavenner said.

Metrics were a particular concern. “Standardized tests are not good measures of the cognitive skills,” a Summit spokesperson said, saying the organization had developed better alternatives. “Attendance and discipline are not measures of habits of success, full stop.” Tavenner said she feared that a study could stop Summit from being able to make changes to the program or that it might stop participating schools from adding new grades. (Kane and West say their plan wouldn’t have limited growth or changes.)

Tavenner told Chalkbeat that research of the kind the Harvard team was offering isn’t needed to validate their approach. Summit is based on decades of research on ideas like project-based learning, she said, citing the organization’s report titled “The Science of Summit.”

Dan Willingham, a University of Virginia educational psychologist, said that’s useful, but not the same as knowing whether a specific program helps students.

“You take a noticeable step down in confidence when something is not research-based but rather research-inspired,” he said, while noting that many education initiatives lack hard evidence of success. “There’s a hell of a lot going on in education that’s not being evaluated.”

What about Summit’s original charter network, now 11 schools? Summit cites internal data showing its graduates have success being accepted to college. But outside research is limited. A 2017 study by the Stanford-based group CREDO found that attending Summit led to modest declines in students’ reading scores and had no clear effect in math, though it looked at only a small portion of the network’s students.

The Summit charter schools are also part of an ongoing study of economically integrated charter schools, and a few were included in two widely cited RAND studies looking at personalized learning, though they didn’t report any Summit-specific information. California’s notoriously limited education data access has stymied more research, Tavenner said.

What does philanthropy owe the public?

Today, Summit’s learning platform has far outpaced its charter network. About 380 schools, with over 72,000 students, use the platform; the national charter network KIPP, by comparison, runs 224 schools serving around 100,000 students.

Summit now gets its engineering help from the Chan Zuckerberg Initiative, not Facebook. That philanthropic partnership has fueled its growth: While CZI has not disclosed how much it’s given to Summit, the Silicon Valley Community Foundation — through which CZI funnels much of its education giving — lists grants to Summit totalling over $70 million in 2016 and 2017.

Summit has also netted $2.3 million for the platform from the Bill and Melinda Gates Foundation in 2016, and another $10 million in 2017. (CZI, the Gates Foundation, and the Silicon Valley Community Foundation are all funders of Chalkbeat.)

Some major foundations regularly invest in research to better understand whether their gifts are doing good, noted Sarah Reckhow, a Michigan State professor who studies education philanthropy. In a number of instances, that research comes to unfavorable conclusions, like a Gates-funded study on its teacher evaluation initiative or a Walton Family Foundation-backed evaluation of charter schools’ propensity to screen out students with disabilities. (A Gates spokesperson said that part of its $10 million to Summit was set aside for “measurement and evaluation.”)

Reckhow said she hasn’t yet seen that same inclination from CZI. And she worries that school districts might be less likely to carefully examine programs that are offered free of charge, like Summit.

“If you reduce that barrier, you’re making it potentially more likely to adopt something without as much scrutiny as they otherwise might do,” she said. “That increases the obligation of Summit and CZI to evaluate the work.”

CZI spokesperson Dakarai Aarons said the organization is committed to research and to Summit, and pointed to a number of schools and districts that saw academic improvements after introducing Summit’s platform. “As the program grows, we look forward to expanded research to help measure its long-term impact,” he said.

Tavenner said Summit is exploring other options to prove its approach is working, including talking to researchers who study continuous improvement. “We can’t just keep saying no to [randomized studies],” she said. “We’ve got to have another way, but I don’t have another way yet.”

Researchers Kane and West, for their part, say Summit’s concerns about evaluating its evolving model should also raise questions about Summit’s swift spread.

“The evaluation we proposed would have assessed the impact of the model at that point in time, even if the model continued to evolve,” they wrote in an email. “When a model is still changing so radically that a point in time estimate is irrelevant, it is too early to be operating in hundreds of schools.”

“Unfortunately, Summit is closer to the rule than the exception,” they said.

new money

House budget draft sends more money to schools, but not specifically to teacher raises

PHOTO: Laura Faith Kebede/Chalkbeat

Despite months of heated debate, Indiana House Republicans are not setting aside extra dollars for meaningful teacher raises in their version of the state’s $14.5 billion education budget plan released Monday night.

Even though lawmakers are proposing preserving a controversial merit-based bonus pool and adding small amounts for teacher training programs, their budget draft would largely leave it up to school districts to dole out raises through increased overall funding.

The budget draft proposes increasing what Indiana spends on schools overall by $461 million — or 4.3 percent — through 2021, a little more than increases in years past. The basic per-student funding that all districts get would jump from $5,352 per student this year to $5,442 per student in 2020 and $5,549 per student in 2021. House lawmakers are also adding in a one-time payment of $150 million from state reserves that would pay down a pension liability for schools. But while lawmakers and Gov. Eric Holcomb have said that pension payment would free up about $70 million in schools’ budgets each year, the state likely wouldn’t require the cost-savings be passed along to teachers.

Although increasing teacher pay is a top goal for House Republicans, lawmakers have crafted bills that hinge on districts spending less money in areas such as administration or transportation rather than adding more money to school budgets and earmarking it for teacher salaries.

Their criticism of school spending has raised the ire of superintendents and educators who say they have little left to cut after years of increasing costs and state revenue that has barely kept pace with inflation.

But budget draft, which is expected to be presented to and voted on by the House Ways and Means Committee on Tuesday, doesn’t completely omit efforts to incentivize teachers to stick around. Unlike Holcomb’s budget proposal, House lawmakers are keeping in the current appropriation of $30 million per year for teacher bonuses.

The House budget draft would also set aside $1 million over two years for a teacher residency pilot program and $5 million over two years for schools that put in place career ladder programs that allow teachers to gain skills and opportunities without leaving the classroom.

Teacher advocacy groups, such as the Indiana State Teachers Association and Teach Plus, have been supportive of residency and career ladder programs, but the organizations have also called for more action this year to get dollars to teachers. Additionally, the ideas aren’t new — similar programs have been proposed in years past.

Calls for the hundreds of millions of dollars it would take to raise teacher salaries to be more in line with surrounding states will likely go unheeded for now as the state instead prioritizes other high-profile and expensive agencies, such as the Department of Child Services and Medicaid.

But while plans for major teacher pay raises appear to be on hold, House lawmakers are looking to boost funding in other areas of education to support some of the state’s most vulnerable students.

The budget draft would increase what the state must spend on preschool programs for students with disabilities from the current $2,750 per-student to $2,875 in 2020 and $3,000 in 2021 — the first such increase in more than 25 years.

House lawmakers are also proposing the state spend more money on students learning English as a new language, at $325 per student up from $300 per student now. While all schools with English learners would receive more money per student under this plan, the new budget draft removes a provision that had previously allocated extra dollars to schools with higher concentrations of English learners.

A 2017 calculation error and an uptick in interested schools meant state lawmakers did not budget enough money for schools with larger shares of English-learners in the last budget cycle, so they ended up getting far less than what the state had promised. But even the small increases were valuable, educators told Chalkbeat.

House lawmakers also suggested slashing funding for virtual programs run by traditional public school districts. Going forward, funding for both virtual charter schools and virtual schools within school districts would come in at 90 percent of what traditional schools receive from the state — now, only virtual charter schools are at the 90 percent level. It’s a marked change for House lawmakers, who in years past have asked that virtual charter school funding be increased to 100 percent.

The virtual funding proposal comes as lawmakers are considering bills that would add regulations for the troubled schools, where few students pass state exams or graduate.

The budget draft also includes:

  • $5 million per year added to school safety grants, totaling $19 million in 2020 and $24 million in 2021
  • Doubling grants for high-performing charter schools from $500 per student to $1,000 per student, at a cost of about $32 million over two years. The money is a way for charter schools to make up for not receiving local property tax dollars like district schools, lawmakers say.
  • $4 million per year more to expand the state’s private school voucher program to increase funding for certain families above the poverty line. Under the plan, a family of four making between $46,000 and $58,000 annually could receive a voucher for 70 percent of what public schools would have received in state funding for the student. Currently, those families receive a 50 percent voucher.
  • About $33 million over two years (up from about $25 million) for the state’s Tax Credit Scholarship program.

rethinking the reprieve

Indiana lawmakers take step to eliminate generous ‘growth-only’ grades for all schools, not just those in IPS

PHOTO: Anthony Lanzilote

A panel of Indiana lawmakers took a first step Monday to stop giving new and overhauled schools more generous state A-F grades that consider only how much students improve on tests and cut schools slack for low test scores.

The House Education Committee was initially looking to clamp down on Indianapolis Public Schools’ innovation schools, barring them from using student test score improvement as the sole determinant in their first three years of A-F grades. The more generous scale has boosted IPS’ performance as it launches a new strategy of partnering with charter operators, by allowing some innovation network schools to earn high marks despite overall low test scores.

But lawmakers expanded the scope of the bill to stop all schools from receiving what are known as “growth-only grades” after Chalkbeat reported that IPS’ overhauled high schools were granted a fresh start from the state — a move that would allow the high schools to tap into the more lenient grading system.

“I want to be consistent, and I felt like [grading] wasn’t consistent before, it was just hodge-podge,” said committee Chairman Bob Behning, an Indianapolis Republican. “We need to be transparent with parents.”

Read: Why it’s hard to compare Indianapolis schools under the A-F grading system

The committee unanimously approved the bill. If it passes into law, Indianapolis Public Schools stands to be one of the districts most affected. Growth-only grades for innovation schools have given the district’s data a boost, accounting for eight of the district’s 11 A grades in 2018. All of its high schools could also be eligible for growth-only grades this year.

Indianapolis Public Schools officials did not immediately respond to requests for comment. In the past, they have defended the two-tiered grading system, arguing that growth on state tests is an important window into how schools are educating students. Growth-only grades were originally intended to offer new schools time to get up and running before being judged on student test scores.

IPS was also the target of another provision in the updated bill that would add in stricter rules for when and how schools can ask for a “baseline reset” — the fresh start that its four high schools were recently granted.

Read: IPS overhauled high schools. Now, the state is giving them a fresh start on A-F

The resets, which districts can currently request from the state education department if they meet certain criteria that show they’ve undergone dramatic changes, wipe out previous test scores and other student performance data to give schools a fresh start. The reset schools are considered new schools with new state ID numbers.

The state determined a reset was necessary for IPS’ four remaining high schools because of the effects of decisions last year to close three campuses, shuffle staff, and create a new system a new system for students to choose their schools. Each school will start over with state letter grades in 2019.

But Behning and other lawmakers were skeptical that such changes merited starting over with accountability, and they were concerned that the process could occur without state board of education scrutiny. If passed into law, the bill would require the state board to approve future requests for accountability resets.

A state board staff member testified in favor of the change. The state education department did not offer comments to the committee.

Rep. Vernon Smith, a Democrat from Gary, said he didn’t like the fact that a reset could erase a school’s data, adding that he had concerns about “the transparency of a school corporation getting a new number.”

The amended bill wouldn’t remove the reset for IPS high schools, but by eliminating the growth-only grades, it would get rid of some of the incentive for districts to ask for a reset to begin with. Under current law, reset schools are considered new and qualify for growth-only grades. But the bill would require that reset schools be judged on the state’s usual scale, taking into account both test scores and test score improvement — and possibly leading to lower-than-anticipated state grades.

The amended bill would still offer a grading grace period to schools opening for the first time: New charter schools would be able to ask the state to give them no grade — known as a “null” grade — for their first three years, but schools’ test score performance and test score growth data would still be published online. Behning said he didn’t include district schools in the null-grade measure because they haven’t frequently opened new schools, but he said he’d be open to an amendment.

The bill next heads to the full House for a vote.