the school improvement industry

Private groups have long tried to help turn around struggling schools. But it’s not clear if they’re doing any good.

PHOTO: Stephanie Snyder
A meeting of New York City Renewal program for struggling schools in 2014.

School improvement programs have sprouted up around the country to help turn around long-struggling schools. And the money that comes with that has also spurred the growth of external groups offering their services to schools.

But there’s little evidence on whether that school improvement industry, paid for by taxpayers, is actually boosting student learning, according to a study of 151 turnaround providers endorsed by various state agencies.  

It’s a striking finding in light of the significant role third-party groups play in supporting these efforts. But it’s not entirely surprising, considering the disappointing (albeit still debated) track record of turnaround efforts across the country, and anecdotal reports that providers with little experience emerged once the spigot of federal dollars was turned on.

There’s also a less damning explanation: Research is expensive and difficult to undertake. Programs may be successful, even if no study exists to prove it.

Still, the results raise questions about the role of third-party turnaround providers, which may continue as federal education law requires turnaround efforts in each state’s lowest performing 5 percent of schools.

This growth of outside groups has likely come because schools have had limited support to implement turnaround strategies — including from state education departments, many of whom worked with or encouraged use of external providers. In the 2012-13 school year, for instance, 34 states and two-thirds of districts implementing federal turnarounds reported contracting with external consultants to support those efforts.

“Our results suggest that states have not prioritized evidence of impact when endorsing providers to work in their schools and districts,” wrote Coby Meyers and Bryan VanGronigen of the University of Virginia.

The paper, published last month in the peer-reviewed American Journal of Education, examined 13 state websites from 2015 that listed recommended turnaround providers for struggling schools. From there, the researchers created a list of 151 state-backed providers — third-party groups that offer a variety of services to low-performing schools, including professional development, teacher recruitment, extra learning time, and improving assessment, parental involvement, and use of technology.

The researchers combed through those providers’ websites, academic research databases, and even sent emails to each organization, to see if they had research showing that their programs had led to better test scores or graduation rates.

Few did. Only 11 percent of the 151 had research backing their specific program, and then only some of those studies were focused on school turnarounds. That’s especially notable because these were all organizations recommended by state agencies, though the researchers point out, “almost no research exists on how [states] recruit, vet, endorse, or evaluate providers.”

Notably, only half of service providers reviewed claimed on their websites that their services were based in research. This meant only that the group said that its general approach was supported by research, rather than that the particular program was.

In some ways the results are no surprise, partially because of the fairly high bar being set — to be counted as having evidence, a program had to show gains based on a study that attempted to isolate cause and effect. But such studies are expensive, and researchers may be less likely to evaluate one-off programs on their own. Also, certain approaches are simply hard to evaluate. That means many programs have never been studied at all.

One of the providers that did have supportive evidence, based on a study in Ohio, was a University of Virginia program known as Partnership for Leaders in Education. That’s also where Coby Meyers, one of the researchers behind the latest study, works as chief research officer, in addition to being an associate professor at the University Virginia.

Others found to have supporting evidence included Success for All, a schoolwide turnaround program that includes efforts to strengthen professional development and curriculum; City Connects, a Boston-based community schools group, and eMINTS, which focuses on the use of technology in schools.

The researchers say it’s puzzling that so few had strong evidence since they were endorsed by state agencies. “Given how few providers had evidence of impact and how many were not research based, a key question arises: What is the rationale for endorsing providers that lack evidence of impact?” they write.

Word of mouth may play a big role in schools’ decision making. “Schools tend to contract with providers used by other schools in their own districts in the past, regardless of past performance,” according to an older study that looked at schools in Texas in the early 2000s that received federal funds to conduct what was called “comprehensive school reform.”

There’s a still great deal that’s not known about these external providers, including how much money is spent on them.

Meanwhile, turning around struggling schools remains vexing for policymakers. High-profile efforts, including by the federal government and New York City, have proven largely disappointing, though each can point to some bright spots.

The funds designated under federal law to help struggling schools are supposed to go toward “evidence-based interventions.” The standard for what counts as evidence is not especially high, though, focusing on the idea rather than the provider behind it.

Nora Gordon a professor at Georgetown who has researched such third-party groups, said their likelihood of succeeding won’t just be determined by what they plan to do, but also on local context and how they implement it.

“There could be two providers who sell intervention X and one does a good job and one does a bad job,” she said. “If you think about ESSA, they’re both going to say we’re using this research-based strategy.”

Data dive

Hardly any kids passed ISTEP at one of Indiana’s largest schools. Here’s why it’s not getting an F

Indiana Virtual Pathways Academy is one of the state’s largest and fastest-growing schools. But because too few of its students took the state exams — and those who did weren’t enrolled long enough — there is no clear picture of how well the school is educating them.

The virtual charter school, which opened in 2017, more than doubled in size to 6,232 students since last fall, in part because state data shows more than 1,700 students transferred from its troubled sister school, Indiana Virtual School.

But despite Indiana Virtual Pathways Academy’s rapid growth, the school is bypassing a key accountability measure that Indiana thinks is important for transparency: A-F grades, which were approved by the Indiana State Board of Education on Wednesday.

Read: As students signed up, online school hired barely any teachers — but founder’s company charged it millions

Education department officials said the school did not get a grade, despite its high enrollment last year, because it did not test enough students who had been enrolled long enough to have one calculated. State grades are based primarily on student test scores, and virtual schools are known to struggle to get their remote students to sit for exams.

State test participation rate data shows IVPA tested about 19 percent of its 346 10th-graders in 2018 — about 65 students. To use test scores to calculate a school grade, the state requires that at least 40 test-takers must have attended the school for at least 162 days, a majority of the school year. But state officials said that while the school enrolled 48 10th-graders who met the attendance threshold during the testing period, only three of those students took the exams.

Federal requirements say schools must test at least 95 percent of students, and school grades can be affected if a school falls below that percentage. But there is currently no consequence for a school that doesn’t test enough students to get a letter grade.

The students who were tested at IVPA posted poor results: 5.7 percent passed both state English and math exams.

Leaders from Indiana Virtual School and IVPA did not respond to requests for comment on A-F grades or testing participation, but the schools’ superintendent Percy Clark said in an emailed statement that students from varying education backgrounds select IVPA, and that the school was designed to serve students who are far behind their peers academically.

“Our students CHOOSE to come to Indiana Virtual Pathways Academy from many different backgrounds, and we accept everyone regardless of where they are on their academic journey,” Clark said.

Virtual charter school critics say IVPA’s lack of a letter grade is an example of how the schools are able to avoid scrutiny.

“It’s absolutely indefensible,” said Brandon Brown, CEO of The Mind Trust, an organization that advocates for charter schools but has been a vocal critic of virtual charter schools. “When it comes to charter schools, the grand bargain is that the charter school gets increased autonomy, and in exchange, there is greater accountability. It’s hard to see where the accountability is for virtual schools right now.”

In contrast to IVPA, other large virtual schools in the state tested at least 90 percent of their students, and nearly every traditional school in Indiana met the federal threshold for testing students.

Indiana Virtual School, the subject of a Chalkbeat investigation that revealed questionable educational and spending practices, tested about two-thirds of its students in 2018. Students at the school, which received its third F grade from the state this week, did marginally better than at IVPA, but performed far below state averages: 18.6 percent of elementary and middle school students passed both tests, and 4.4 percent of high-schoolers did. State law says schools are up for state board of education intervention when they reach four consecutive F grades.

Brown, who used to work in the Indianapolis mayor’s office overseeing charter schools, said this is where charter school authorizers — the entities charged with monitoring the schools’ operations, finances, and academics — need to be involved. Daleville Public Schools, a small rural district near Muncie, oversees IVS and IVPA. State education leaders have previously questioned whether school districts have the capacity and expertise to oversee statewide charter schools. District leaders did not immediately respond to requests for comment.

“If I was still an authorizer and one of our schools had less than a 20 percent rate of their students taking the ISTEP, we would be mortified, and we would be holding that school accountable with very clear measures,” Brown said. “In light of the tens of millions of dollars used to fund this school, there has to be at least a basic level of accountability, and right now, it’s hard to account for how that money is being spent because we just don’t know.”

With such high enrollment numbers, Indiana Virtual School and Indiana Virtual Pathways Academy could together bring in upward of $35 million from the state for this school year, according to funding estimates from the Legislative Services Agency.

At the state’s other full-time virtual charter schools — including those billed as alternative schools like IVPA — state grades are rising as enrollment grows. Indiana Connections Academy is up to a D this year from an F, and Insight School of Indiana is up to a C from an F. For grades under Indiana’s federal plan, the schools received an F and D, respectively.

Indiana Connections Career Academy enrolled about 70 students last year and received no grade, but education department officials say that is because it had too few students to calculate one, despite testing more than 95 percent of them. It’s not uncommon for small schools — especially high schools that have just one tested grade — to not get a grade. This year, the school’s enrollment is up to about 300 students.

Virtual charter school accountability has become a hot issue in Indiana. Earlier this year, the state board of education convened a committee to study virtual charter schools, which have grown rapidly here in recent years. And last month, the committee released a series of recommendations, including slowing growth of new virtual charter schools to 15 percent per year — after a school hits 250 students — for their first four years.

Getting students who are located remotely to sit for state exams is a challenge for virtual schools. Melissa Brown, head of Indiana Connections Academy, said dogged work contacting and keeping up with students has made some of the difference for her school, both in students taking tests and improving on them.

“Our teachers are relentless in trying to engage with kids,” she said. “We are by no means where we want to be. We still have a lot of work to do. But 8-point growth is something that we’re celebrating today.”

Melissa Brown said the school is also offering students who come in behind grade level more ways to make up their classes and incentives for them to stay at the school. For example, she said the school has a lot of over-age eighth-graders who should be in high school. Instead of just drilling their eighth-grade classes, they also have a chance to try out high school-level work — a taste of what’s to come, Brown said. So far, it’s working.

“We’re just trying to be really creative about helping kids progress,” she said.

At Insight, school director Elizabeth Lamey said she’s excited by how the high school has been able to help students show more growth on state tests. Currently, the school, which opened in 2016, is getting grades calculated only on how much students improve on state tests from one year to the next, not their proficiency or other measures such as graduation rate.

Lamey said improving the school’s curriculum and focusing on remediation and teacher training contributed to their progress and sets them up to continue that work.

“We hope to see even more growth this year,” Lamey said. “We know that it’s a rougher road, the older students get, to remediate. It takes more time, and we are slow and steady — we keep moving forward.”

Accountability issues will continue to be important for virtual charter schools as their enrollments grow.

Indiana’s five full-time virtual charter schools enroll about 13,000 students. Although it appears that total virtual charter school enrollment in Indiana has declined since 2017-18, those figures include the closing of low-performing Hoosier Virtual Academy. The school enrolled 1,170 students when it closed in June, which was far lower than the 3,342 it was recorded as having at the beginning of that school year.

Comparing enrollment totals between fall of 2017 and fall of 2018, every virtual charter school currently open in the state saw enrollment rise, with the exception of Indiana Virtual School. Indiana Connections Academy and its sister school, Indiana Connections Career Academy, gained nearly 400 students between them. Insight is also up 45 students.

Virtual charter schools tend to have volatile enrollment patterns in part because of how easily students can enroll and withdraw — their families don’t have to move, and they can live anywhere in the state. Students moving between schools is not unique to virtual schools, but those schools do tend to see higher instances of mobility than traditional schools.

That means it can be hard to determine just how much virtual school enrollment has changed from one year to the next — enrollment numbers reported in the fall might fluctuate wildly through the rest of year.

Are Children Learning

These are the 7 schools IPS leaders are most worried about

PHOTO: Dylan Peers McCoy
School 48 is one of the campuses identified for a quality review.

Seven schools will be getting a closer look, and possible intervention, after Indianapolis Public Schools administration identified them as some of the lowest-performing schools in the district.

Superintendent Lewis Ferebee’s administration announced this week the schools that were chosen for school quality reviews based on performance on the state ISTEP exam. This is the second year the district has initiated its own assessment of struggling schools, which will include district visits to the schools, and interviews with leaders, staff, and families.

The reviews are designed to help schools improve, district officials said. But campuses could also face the possibility of being restarted as innovation schools. If that happens, they would likely be taken over by outside charter or nonprofit operators, who would overhaul the schools with largely new staff. Schools can also be selected for restart based on repeated failing grades from the state.

One of the seven schools identified by the district last year was ultimately restarted as an innovation school. The other schools received different kinds of help such as working with schools to help teachers collaborate better, officials said.

“This is a clear example of our commitment to helping drive improvement at these schools where we see there’s a lack of improvement,” Ferebee said.

One sign that less drastic efforts helped is that only one campus, School 48, appeared on the list for the second year in a row.

“As a matter of fact, a few of the schools from last year had some of the highest growth that we saw in the district,” said Andrew Strope, the district’s performance and continuous improvement officer.

One wrinkle for the district is that three of the seven schools identified are already innovation schools. That raised concerns for board member Venita Moore.

“I was surprised to see these … innovation schools on the list,” Moore said. “But I think it does provide our community insight that we take seriously the quality of the education that our children are receiving.”

When innovation schools are created, the operators have contracts with Indianapolis Public Schools. Those agreements typically stipulate that the contracts can be ended if the schools receive D or F grades from the state for three or more consecutive years.

Ferebee cautioned, however, that restarting them again would create more upheaval. “Often times that creates instability that is not always helpful,” he said. “The goal, I just want to continue to reiterate, is to ensure we can help these schools improve their performance.”

These are the seven schools identified as having test scores in the bottom quarter and growth scores in the bottom half for the district.

  • Stephen Foster School 67
  • Eleanor Skillen School 34
  • Thomas Gregg Neighborhood School 15
  • Ignite Achievement Academy at Elder Diggs 42
  • Kindezi Academy at Joyce Kilmer 69
  • James Russell Lowell School 51
  • Louis B. Russell Jr. School 48