the evolution of research

Beyond the test score horse race: 5 big questions researchers are asking about charter schools

PHOTO: Dylan Peers McCoy

The latest big charter school study was sweeping in scope, looking at thousands of students in 26 states across three school years.

But the study (and lots of other research on charter schools) uses that data to answer a relatively narrow question: How do students, usually in grades 4-8, perform on math and reading tests compared to students in traditional public schools?

This could be called the “test score horse race.” Some researchers are moving beyond that, to try to understand issues like what specific charter approaches are most effective and how charter schools affect larger communities.

“A number of new research studies are beginning to investigate some more nuanced questions with regard to charters,” University of Michigan professor Brian Jacob wrote recently.

Here are a few of the big questions that some researchers are examining — and other important questions that have received little attention.

1. How do charter schools impact students beyond standardized test scores?

In education research, test scores are the coin of the realm because they are readily available. But experience and research suggest that schools affect a lot more: from attendance and behavior to college attendance, future employment, and earnings. A key question is whether test scores predicts any of those longer-term outcomes.

For answers, researchers have looked at charter schools in Boston, Florida, and Texas, as well as specific networks of schools including the Harlem Children’s Zone and the Noble Network in Chicago. Results have been mixed so far, and none of the studies has been national in scope.

2. Why are some charter schools especially effective?

We know that certain charter schools are better than others at raising test scores — but why?

There have been only a handful of studies on this, and research from New York City and Massachusetts indicate that tenets of “no excuses” charters — like extra instructional time, intensive tutoring, and frequent feedback for teachers — make a difference.

There may be other reasons for high test scores: additional resources, peer effects, harsh discipline, teaching to the test, or pushing out low-achieving students. Existing studies on those explanations have been limited in scope.

“I’d like to see more work done on how charter schools operate differently than traditional public schools — a little bit [of] getting in the black box,” Jacob said. “What are the differences in policies and practices?”

3. Why do charter schools in some places outperform charters elsewhere?

Many point to differences among authorizers, the entities — often state or local school boards — that approve and oversee charter schools. Some argue that tight controls on which schools open and the closure of low-performing schools is essential to the success of charters.

There are a handful of studies on how individual authorizers are related to charter performance, as well as the impact of charter school closures.

State policies, like the extent to which charter schools can grow, how much autonomy they have from regulations, and how much funding they get, also likely make a difference. There is remarkably little research on these questions, and what does exist is correlational.

4. How are charter schools changing over time?

“The Evolution of Charter School Quality” in Texas

Most research on charter schools functions as a snapshot that may be less useful over time. Meanwhile, the number of students attending charter schools is growing in many states, and the sector is evolving in other ways, too.

Studies in Texas and North Carolina find that charter performance improves over time, but the schools serve more advantaged students. Research in New Jersey and Arizona also suggest charters as a sector may get better as low-performing ones close down.

5. How do charter schools affect public education more broadly — including finances and segregation?

Looking only at the relative performance of charters against district schools ignores their overall impact. If charters cause traditional public schools to do better or worse through competition, for example, horse-race studies won’t capture that.

Research has found that charter schools generally have no impact or small positive effects on the test scores of students in surrounding public schools, though it is difficult to nail down cause and effect.

The financial impact of charter expansion has been studied less. Existing research has shown that charters create additional costs for districts, but that places with large charter sectors can adapt over time.

On the issue of segregation, there is evidence that charter schools in North Carolina, Pennsylvania, Indianapolis, Michigan, and Texas have exacerbated racial segregation; other studies of multiple states and using national data have found little or no impact, however.

Studies of New Orleans’ public school system, which is composed of nearly all charters, have shown that expansion of charters (as well as a number of other reforms) led to large gains in student achievement, but also caused modest increases in racial segregation in city high schools.

Jacob said he is currently working on a study of the evolution of Michigan’s charter sector. More work needs to be done, he says, to track how the expansion of choice reshapes how families select schools.

an intervention

Struggling Aurora elementary school gets creative to improve — but bigger changes may be coming

First graders at Paris Elementary in Aurora use toys and light pointers to help focus while reading individually. (Photo by Yesenia Robles, Chalkbeat)

When a fifth-grade boy was having trouble waking up in the morning and getting to school on time, officials at his Aurora school, just a block away, came up with an idea.

They gave the student a buddy — another fifth grader who lives two doors down. The boy agreed to knock on his classmate’s door every morning so the two could walk to school together.

“We had to get creative,” said Shannon Blackard, the interim principal of Paris Elementary. “He became more motivated to get to school on time.”

That buddy system is part of a broader push that has led to better attendance at the school — one emphasis to boost student achievement under a district improvement plan already in place. But bigger changes may soon be coming to Paris Elementary.

The 363-student school has logged five straight years of low performance, triggering the next step in Aurora Public Schools’ system for intervening in low-performing schools. District officials this fall put out a “request for information” from interested parties — which could include charter schools and consultants — seeking ideas for more aggressive steps for improvement.

After this week’s deadline to respond passes, the district will review the responses and ask the school board to vote on recommendations as soon as next month. It will be one of the first significant decisions for the seven-member board since four teachers-union backed candidates won election last month. Those new members have questioned some of the district’s reform efforts.

Superintendent Rico Munn created the district’s framework for intervening in low-performing schools after taking over the role in 2013.

Paris Elementary is not the first school to reach five years of low performance, but it stands apart because it is already under a district-approved innovation plan. That plan gave the school more flexibility in budgeting and setting the school calendar, and in making hiring decisions.

Speaking at a September school board meeting about schools facing turnaround, Munn characterized the recommendation for Paris Elementary as “the most high-profile.”

The district essentially is trying to step in before the state forces its hand.

On its fifth year of priority improvement, one of the lowest ratings the state gives, Paris is one year away from being on the list of schools requiring state action if it doesn’t improve. This year, the state did assign more points to the school in the ratings compared to last year, but not enough for the school to jump up into a higher category of ratings.

The school is a block away from the University of Colorado Anschutz Medical Campus and is surrounded by apartments and multi-family housing. Every one of the students who attends the school qualifies for free or reduced-price lunch. Eight of every 10 are learning English as a second language. The students come from 16 different countries.

The school’s principal left the role at the beginning of the school year, leaving a district official to step in for a while before an assistant principal was named interim principal for the year.

School officials at Paris, including Blackard, the interim principal, are optimistic that current work at the school is already helping.

For instance, school-level data shows that the number of students reading at grade level this December has more than doubled school-wide compared to the same time last year. The improvements are at every grade level except kindergarten. And for the students that aren’t on grade level, school teams are now meeting regularly to come up with plans for how to help each one catch up.

Improvements are showing in math classrooms, too. Using a new math curriculum, third grade students in one classroom were excitedly engaged last week in an activity to see if they could guess how much a bag of crackers weighed and if they could use the scale to test it.

Early indications for attendance are also positive. The number of students who are chronically absent has dropped by 50 percent. This year so far, 11 percent of students are labeled chronically absent, down from 22.1 percent last year.

Blackard said she isn’t aware of the district’s plans to recommend possible changes to Paris, but said that she expects the school is on track to make improvements anyway.

“I’m very confident,” Blackard said. “We are very focused.”

When Munn discussed the timeline for district recommendations with the school board in September, he described a balancing act between giving schools time to make improvements while stepping in early enough to roll out changes in time when necessary.

“The question is how do we respond, so that we both don’t over-act but don’t react too late,” Munn said.

The district’s framework for dealing with low-performing schools prompts the district to intervene in schools that earn the lowest quality ratings by the state, increasing the level of intervention by the number of years on the clock. Here’s how it works:

When Paris had reached three years of low performance, part of the district’s plan called for the school to adopt the innovation plan and join a so-called innovation zone in northwest Aurora. Along with providing the flexibility of innovation status, the zone is meant to give its five schools the chance to work together and learn from each other.

If the school doesn’t improve enough by next year, the state’s options could include suggesting school closure or asking a charter school or outside group to take over.

Aurora officials have said they want to be proactive about improving schools before they are directed to make changes by the state.

Last year only Aurora Central High School was on the state’s watchlist facing state sanctions. In that case, using the same framework for responding to low-performing schools, district officials were already rolling out an innovation plan giving the school flexibility for changes before the state stepped in.

State officials and state board members recognized the district’s initiative and gave the district a chance to continue rolling out the plan to see if it would result in improvements.

In another case when the district was following the same playbook, the district in 2015 recommended converting low-performing Fletcher Community School into a charter school. The district tapped the Denver network Rocky Mountain Prep, which had responded to a request for information.

After the decision, Fletcher showed some improvements in test scores. This year, Fletcher’s quality ratings showed enough improvement to get off the state’s radar, even before the charter has fully taken over the school. Some teachers and union officials point to that as evidence that the district might have acted too soon instead of considering other options and allowing those efforts to show improvement.

“It’s not that you can’t ultimately get to that conclusion, the question is how do you examine it publicly,” said Bruce Wilcox, union president. “We’re not part of the conversation right now.”

model of inclusion

New York City is placing students with disabilities in mainstream classes. But do they actually feel included?

PHOTO: Alan Petersime

Terell Richards languished at the public middle school in Queens for students with severe disabilities that he attended a few years ago.

It wasn’t just that he found the work so easy he sometimes fell asleep in the back of the classroom, his sister, Kya, said. It was also that he felt so out of place he would sometimes dissolve into tears.

“Just crying and saying how much he just felt like he was in the wrong place and completely lost,” said Kya, who helped her brother, now 19 years old, switch to a private school for students with special needs.

Now, New York City — and districts across the country — have started sending more students like Terell into classrooms alongside their non-disabled peers. But while some research has shown students with disabilities can perform better in mixed-ability settings, a crucial concern has been whether the new environment makes students with disabilities actually feel less isolated and out-of-place.

A new first-of-its-kind study based on surveys of more than 250,000 New York City middle-school students between 2007 and 2012 tries to answer that question.

The study comes with some important caveats: The surveys are conducted annually by the education department, meaning they weren’t written by the study’s authors nor did they oversee how they were administered. Also, the survey period ended just as the city was beginning its major push to move most students with disabilities out of separate classrooms.

It finds that middle-school students with disabilities tend to feel welcomed in schools with non-disabled peers, though their experiences vary by their type of disability. But, more surprisingly, special-needs students in separate classes don’t feel more excluded.

The study, which was funded by the Spencer Foundation, is set to appear in the peer-reviewed journal, Educational Researcher. Here are three big takeaways:

Students with disabilities generally feel included in mainstream schools.

The study tracks whether students feel included at over 500 traditional schools by looking at how students with disabilities and their non-disabled peers responded to five questions from the city’s annual survey: whether students feel welcome at school, whether students with disabilities are included in school activities, if teachers know students’ names, whether students are bullied, and whether they see harassment.

Generally, students with disabilities reported slightly higher levels of inclusion in school activities than non-disabled students, and feel only marginally less welcomed — though they also reported slightly higher levels of bullying and harassment.

About 60 percent of students with special needs either agreed or strongly agreed that students with disabilities are included in all school activities, about two percentage points higher than non-disabled students. A slightly smaller share of special needs students felt welcomed at school compared to students without disabilities — though 92 percent said they felt welcomed. (The patterns are relatively consistent even when controlling for differences in student characteristics like gender, race, or socioeconomic status.)

Advocates said they were both surprised and encouraged by those findings.

“We do worry that in the hands of an unskilled teacher that kids will not necessarily feel welcomed and they’ll still be separated out and made to feel different” said Maggie Moroff, a disability policy expert at Advocates for Children. “It’s pretty exciting to me to see that’s not necessarily true.”

Feelings of inclusion vary widely by disability type.

Although students with disabilities generally reported feeling about as included and welcomed as their peers, there are significant differences based on the type of disability a student has.

Those who were classified as having an “emotional disturbance” — often students who have significant behavioral problems — felt among the least included. They were about 4 percentage points less likely to report feeling welcome or included, compared to non-disabled students, and were also more likely to report harassment than students in any other disability category.

“The emotional disturbance kids are the ones who stand out in their classrooms,” said Leanna Stiefel, the study’s lead author and an economics professor at New York University, adding that they may feel less included because their disabilities are more difficult to hide.

But students with “low-incidence” disabilities such as multiple handicaps, autism, or intellectual disabilities reported more positive feelings than any other group. They were about 10 percentage points more likely than non-disabled peers to report that their schools include students with disabilities, and were slightly more likely to report feeling welcomed.

Students who are segregated based on ability don’t necessarily feel excluded.

Surprisingly, it made little difference whether students with disabilities were in “self-contained” classes — essentially classes comprised only of students with disabilities — or were in classrooms that included non-disabled peers: Both groups reported similar feelings of inclusion. (The findings don’t include students in District 75, a separate set of schools that are even less inclusive, since the schools themselves are only for students with disabilities.)

Moroff, the special education advocate, said the finding surprised her and noted it could reflect that students in more segregated settings aren’t necessarily aware of more inclusive models.

“It’s very possible there that there’s a level of interaction they’re not having,” Moroff said, “that they don’t even expect to be taking place.”