backfill background

Charter school principal: Enrollment policies can skew scores

Two schools whose students have identical test scores would seem to perform differently if they have different enrollment practices, according to a chart produced by a city charter school leader.

It’s not only the teachers union that is arguing that charter schools’ enrollment practices can influence their apparent test performance.

Unlike district schools, charter schools can choose whether to replace students who leave. Charter schools that do not practice “backfill” can end up posting scores that make it look like their performance is better — or worse — than it really is, argues the founding principal of Harlem Link Charter School, Steven Evangelista.

In the Community section, Evangelista explains that when schools opt not to fill empty seats, “survivorship bias” skews test scores toward the results of students who remain enrolled.

The bias renders test scores meaningless, even dangerous, if the scores are not presented alongside context about a school’s enrollment practices, he writes:

Strong schools take the time required to plan, assess, and tweak new initiatives until they become standard operating procedures. The lack of information provided alongside scores obscures this type of growth, creating perverse incentives for schools to “push out” students who are low performers and to “quick fix” by whittling down large original cohorts to smaller groups of survivors, uncompromised by new admittees.

Evangelista says Harlem Link replaces students who depart, knowing that test scores could be adversely affected, in order to keep its budget stable and fulfill its mission of serving needy students. Last year, he writes, the school got lucky: The students who left were, on average, lower-performing than the students who left the previous year, so the appearance of large test school gains was easy to come by.

It’s a phenomenon that the teachers union has been particularly eager to put onto the agenda. After the city released elementary and middle school progress reports for last year on Monday, the union distributed a fact sheet noting high student attrition rates at several top-scoring charter schools. At South Bronx Classical Charter School, for example, between 20 and 40 percent of students that originally enrolled left before they were tested, and no new students replaced them, the union pointed out.

The fact sheet was meant to hammer home the same insinuation — that some charter schools’ strong results were the result of enrollment practices, not superior teaching and learning — that the union leveled at the charter sector’s self-assessment report this spring. ”The report fails to quantify just what the impact is on test scores when students leave charter schools and are not replaced,” said Dick Riley, a UFT spokesman, at the time.

In an even more extreme example of student attrition that the union did not highlight, Harlem Village Academy had 62 eighth-graders sit for state tests last year. But 104 students had been in the fifth-grade class in 2008, meaning that the test-takers represented just 60 percent of the original cohort. If the students who left were more often ones who struggled — which founder Deborah Kenny said was the case in her recent book — the aggregate scores would have been skewed upward.

A self-assessment report by the city’s charter sector released in April said school leaders “have mixed opinions” about backfilling. But it acknowledged that charter schools’ performance probably benefits from the flexibility not to backfill, especially if it is low performers who leave most often. Charter middle schools, which backfill seats least often, post the strongest performance.

Yet backfill doesn’t always cut in a single direction. Some charter schools fill spots that open up and still post consistently high scores, such as the schools in the Icahn network, which had two schools rank in the top 15 progress reports citywide. Other charter schools, including the UFT Charter School, do not fill always all seats that open up yet do not see their scores rise. South Bronx Charter School, which does not practice backfill, lost nearly 50 percent of the students in its first kindergarten class before their fifth-grade year, yet it posted only middling scores.

Indeed, Evangelista writes, a different array of students coming and going would have resulted in totally different scores at Harlem Link. That’s why he is not arguing for or against backfill as policy, but instead for more transparency about how who the students are can influence what the test scores say.

“What I want is for the public to have some understanding of the context behind test scores, so alleged miracles can be put in their proper place, and year-to-year statistical swings that have nothing to do with a school community’s actual performance can be put into their proper perspective,” he writes.

The methodology used in the city’s progress reports, released for last year on Monday, both mitigates against this effect and exacerbates it. A significant portion of the grades are based on the straight test scores, but an even larger portion looks at the improvement of individual students from year to year. Students who left before the most recent year aren’t factored into the progress component, so schools don’t get a boost if lower-performing students leave.

But attrition also means that a school’s progress score is based on improvements among a smaller number of students, often ones who are higher-performing. Plus, as classes shrink and “peer effects” — or the influence of having a higher proportion of high-performing students — kick in, gains might be easier to achieve.

Summer remix

Ten stories you may have missed this summer (and should read now as the new school year kicks in)

PHOTO: Caroline Bauman
Gabrielle Colburn, 7, adds her artistic flair to a mural in downtown Memphis in conjunction with the XQ Super Schools bus tour in June.

Labor Day used to signal the end of summer break and the return to school. That’s no longer the case in Tennessee, but the long holiday is a good time to catch up on all that happened over the summer. Here are 10 stories to get you up to speed on K-12 education in Tennessee and its largest school district.

TNReady is back — with a new test maker.

Last school year ended on a cliffhanger, with the State Department of Education canceling its end-of-year tests for grades 3-8 in the spring and firing testmaker Measurement Inc. after a series of missteps. In July, Commissioner Candice McQueen announced that Minnesota-based Questar will pick up where Measurement Inc. left off. She also outlined the state’s game plan for standardized tests in the coming year.

But fallout over the state’s failed TNReady test in 2015-16 will be felt for years.

The one-year void in standardized test scores has hit Tennessee at the heart of its accountability system, leaving the state digging for other ways to assess whether all of its students are improving.

Speaking of accountability, Tennessee also is updating that plan under a new federal education law.

The state Department of Education has been working with educators, policymakers and community members on new ways to evaluate schools in answer to the federal Every Student Succeeds Act, or ESSA, which requires states to judge schools by non-academic measures as well as test scores.

Meanwhile, issues of race and policing have educators talking about how to foster conversations about social justice in school.

In the wake of police-related killings that rocked the nation, five Memphis teachers talked about how they tackle difficult conversations about race all year long.

School closures made headlines again in Memphis — with more closings likely.

Closing schools has become an annual event as Tennessee’s largest district loses students and funding, and this year was no exception. The shuttering of Carver and Northside high schools brought the total number of district-run school closures to at least 21 since 2012. And more are likely. This month, Shelby County Schools is scheduled to release a facilities analysis that should set the stage for future closures. Superintendent Dorsey Hopson has said the district needs to shed as many as two dozen schools — and 27,000 seats — over the next four years. A Chalkbeat analysis identifies 25 schools at risk.

Exacerbating the challenges of shifting enrollment, families in Foote Homes scrambled to register their children for school as Memphis’ last public housing project prepared to close this month amid a delay in delivering housing vouchers to move elsewhere.

The new school year has officially begun, with the budget approved not a moment too soon for Shelby County Schools.

District leaders that began the budget season facing an $86 million shortfall eventually convinced county commissioners to significantly increase local funding, while also pulling some money from the school system’s reserve funds. The result is a $959 million budget that gives most of the district’s teachers a 3 percent raise and restores funding for positions deemed critical for continued academic progress.

The district also unveiled its first annual report on its growing sector of charter schools.

With charter schools now firmly entrenched in Memphis’ educational landscape, a Shelby County Schools analysis shows a mixed bag of performance, while calling on traditional and charter schools to learn from each other and promising better ways to track quality.

summer mix tape

Ten stories you might have missed over the summer (and should read now as a new school year begins)

PHOTO: Nicholas Garcia
Stand for Children received funding to support parent education.

There is no such thing as time off from covering education. While school doors were shuttered, plenty happened this summer on the Colorado education beat. Here, we’ve compiled stories that we hope prove useful as you ease back into your fall routines.

We’ve got your immunization data right here … 

For the second year, Chalkbeat tracked down immunization data for more than 1,200 schools in Colorado’s largest school districts. Our database revealed that Boulder remains a hotspot for the anti-vaccination movement, students in districts with racial and income diversity are more likely to get their shots and nearly half of schools in the database did a better job this year tracking students’ immunization records. Read our news story about the findings, check out these six charts that dig into the numbers and search for school-level data here.

Denver Superintendent Tom Boasberg reflects on his sabbatical (a break not everyone appreciated)

In June, Denver Public Schools’ longtime schools chief returned from a six-month unpaid sabbatical in South America with his family. “It made us appreciate the extraordinary resources we have here,” he said in an interview about his experience.

A milestone for Colorado charter schools on diversity, but not so much on integration

For the first time, Colorado’s charter schools educated a larger proportion of racial and ethnic minorities than district-run schools, a state report showed. We took a closer look and found that does not mean charter schools are more integrated.

Race, policing and education during a summer on edge

This summer sadly provided no shortage of violence and heartache over issues that sometimes feel like they’re tearing America apart at the seams. We sought to bring some local perspective (and wisdom) to the debate by talking to an ambitious Manual High School student who took up a bullhorn at Denver street protests and to Aurora Public Schools Superintendent Rico Munn.

A middle school’s last-ditch effort to save itself 

An Adams County middle school running out of time to improve has placed its bet on more challenging, more personal teaching — and zero test preparation. Watch Chalkbeat later this week for a report on whether these efforts paid off in the form of improved state test scores. (Hopefully … the data are set to be made public Thursday).

Guess which Colorado school district had a high proportion of teachers designated to lose tenure …

Compared with other large Colorado school districts, Denver Public Schools had a higher proportion of teachers set to lose tenure under a sweeping educator effectiveness law passed six years ago. We surveyed big districts about one of the consequences of Senate Bill 191.

Too darn hot to teach and learn 

As part of its big bond request of voters this fall, Denver Public Schools wants to try to cool off some of its hottest schools. We took a look at where the mercury soars the highest and found that in 12 of the 18 hottest buildings — some of which house more than one school — the number of students who qualify for free or reduced price lunch exceeds the district average.

But the University Club has a lovely lunch menu (and squash courts, too)…  

What if the State Board of Education held a not-so-public meeting with the education commissioner at a private club downtown to prioritize goals, but didn’t get much of anything accomplished? That happened.

What we know — and don’t know — about Colorado remediation rates

Colorado’s college remediation rates inched upward after years of steady decline, a disheartening development. On top of that, we’re not getting the full picture, either, because of incomplete school-level numbers and non-existent district-level data.

Diet Coke: Coming soon to a high school vending machine near you? 

Despite opposition from advocacy groups, Colorado appears headed toward lifting a seven-year ban on diet soda in high schools. The rule change would clear the way for diet soda to be sold in high school vending machines and school stores, though districts could decide not to stock the drinks. We covered the issue before and after the State Board of Education’s initial vote.