number crunching

Some analysis left undone in data-driven education department

P.S. 199 in the South Bronx, one of the city's top-rated elementary schools in recent years. A high rate of its former students' test scores plummeted once they moved onto middle school.

The Department of Education crunches state test scores in dozens of ways to measure the performance of schools, principals, teachers, and students. But it does not perform a statistical analysis that can reveal whether an elementary school’s graduates have received test scores that far outstrip their actual skills.

Researchers say it would be relatively easy for the department to calculate “swing rates” to find the proportion of students from each school whose scores rise or fall by a statistically unlikely margin when they move to another school. Such an analysis could take some of the burden off of individual educators to report suspicions of cheating.

The city used to conduct swing rate analysis prior to the Bloomberg administration, according to a former testing official, and the state is poised to launch the measure as part of an overhaul of its own approach to test security.

But department officials say the analysis would be too onerous. They also say that they never launch investigations into cheating based on data anomalies alone. Instead, they say they will dispatch investigators only when they receive formal allegations of test improprieties.

The policy means that some top-rated schools whose students’ scores plummet at far higher than the average rate never have their testing practices scrutinized.

For all of the criticism of state tests as being arbitrary and imperfect measures of student performance, they are remarkably stable. In 2011, students who saw their scores fall by more than two standard deviations from the previous year made up just 0.6 percent of the sixth grade test-taking population in English, and 0.4 percent in math. That degree of decline is highly improbable under normal circumstances and is more likely to reflect externalities than real changes in academic proficiency.If one student’s test score plummets to that degree, it might be reasonable to conclude that he had a bad day when the second test was administered. But if an entire cohort of students see their scores plummet, it could be that testing conditions were especially favorable in the first year, maybe illicitly.

Some city schools have posted swing rates many times the 0.6 percent average, according to the New York Times. At the two elementary schools with the highest scores on the city’s 2011 progress reports, P.S. 257 and P.S. 31 in Brooklyn, the swing rates that year were 9 percent and 1.8 percent, respectively.

Neither school raised eyebrows at the department until a whistleblower at a middle school that received their students registered an official complaint about wide discrepancies between the students’ test scores and their actual skills. The allegation triggered investigations at both schools.

But the Brooklyn schools’ swing rates were not even the highest in the city that year. About 30 percent of the 34 students who graduated from P.S. 199 in the South Bronx and went on to a middle school up the road, I.S. 303, saw their test scores drop by more than two standard deviations. P.S. 199 had the 16th highest progress report score that year.

At I.S. 232, a nearby middle school that absorbed 29 students from P.S. 199, the swing rate was more in line with the city average.

But teachers at I.S. 303 said that the high test scores did not correlate to the basic English and math skills that many of the incoming P.S. 199 students demonstrated early on in sixth grade.

“You don’t just forget everything,” said one math teacher. “It just baffled me that they somehow got 3’s and 4’s” in fifth grade.

Multiple teachers at the school said students’ behavior during the sixth-grade state tests suggested that the students had received help during tests before.

Some of the students from P.S. 199 grew frustrated during the tests when I.S. 303 teachers did not tell them the answers to questions that stumped them, the teachers said. The teachers agreed to speak on the condition of anonymity because their principal had not given them permission to speak to reporters.

But even after noticing the score discrepancy, no one at I.S. 303 went to the department that year. It’s a common decision, according to educators from across the city who say allegations usually bring scrutiny first to the people who filed them, sometimes exposing them as whistleblowers to their colleagues.

An investigation at P.S. 199 is now open, a department spokeswoman said. Education officials contacted investigators after GothamSchools asked repeatedly about the school’s anomalous scores.

The department’s Office of Special Investigations is handling the case now after it was referred by the city’s Office of the Special Commissioner of Investigation, a SCI spokeswoman said.

Department officials said their concerns were not based solely on the swing rate. Data points such as swing rates alone are not enough to trigger investigations.

Many of the 37 schools to which the department dispatched testing monitors last year had seen their scores increase in unusual ways. But all but four of them had also been the subject of formal allegations.

In fact, the department does not even calculate schools’ swing rates as part of its regular analysis of schools’ performance. When the accountability division runs the numbers that feed into schools’ annual progress reports, which are based largely on students’ year-to-year growth, it does not aggregate results by sending schools.

Department officials say generating schools’ swing rates is a complicated endeavor. After learning about the score discrepancies at P.S. 199, GothamSchools requested swing data for other top-rated schools. But department officials said for months that running those numbers would be too difficult.

“We have provided you with a detailed analysis of P.S. 199,” spokeswoman Marge Feinberg wrote in an email this summer. “For the other schools, it would take a great deal of time.”

Researchers who have worked with Department of Education data before disputed that claim.

“It would be quite easy to do. Just about anyone with a computer and a basic knowledge of statistics could run these checks,” said Sean Corcoran, a New York University professor who has studied the city’s test score data. “This is something the DOE should do on a regular basis.”

A former testing director in New York City said that under his watch the city used data anomalies to trigger investigations.

“We routinely would look at the change in scores in schools from one year to the next,” said Bob Tobias, the Board of Education’s longtime testing chief who retired in 2002. Schools that showed erratic spikes, Tobias said, “would get a little more scrutiny.”

There are signs that the state might start conducting this type of analysis on its own. One charge given to the brand-new test security chief Tina Sciocchetti, at the State Education Department is to set guidelines for pursuing investigations using data methods that look suspicious test score patterns.

“I think those sorts of statistical analyses are simply a red flag and it’s absolutely true that additional investigation is necessary,” said Sciocchetti.

defensor escolar

Memphis parent advocacy group trains first Spanish-speaking cohort

PHOTO: Caroline Bauman
Manuela Martinez (center left) and Lidia Sauceda (center right) are among 19 parents in the first Spanish-speaking class of Memphis Lift’s Public Advocate Fellowship.

Manuela Martinez doesn’t want Spanish-speaking families to get lost in the fast-changing education landscape in Memphis as the city’s Hispanic population continues to grow.

The mother of two students is among 19 parents in the first Spanish-speaking class of Memphis Lift’s Public Advocate Fellowship, a program that trains parents on local education issues.

“We want to be more informed,” said Martinez, whose children attend Shelby County Schools. “I didn’t know I had much of voice or could change things at my child’s school. But I’m learning a lot about schools in Memphis, and how I can be a bigger part.”

More than 200 Memphians have gone through the 10-week fellowship program since the parent advocacy group launched two years ago. The vast majority have been African-Americans.

The first Spanish-speaking cohort is completing a five-week program this month and marks a concerted effort to bridge racial barriers, said Sarah Carpenter, the organization’s executive director.

“Our mission is to make the powerless parent powerful …,” she said.

The city’s mostly black public schools have experienced a steady growth in Hispanic students since 1992 when only 286 attended the former Memphis City Schools. In 2015, the consolidated Shelby County Schools had 13,816 Hispanic children and teens, or 12.3 percent of the student population.

Lidia Sauceda came to Memphis from Mexico as a child; now she has two children who attend Shelby County Schools. Through Memphis Lift, she is learning about how to navigate Tennessee’s largest district in behalf of her family.

PHOTO: Caroline Bauman
Hispanic parents attend a training with the Memphis Lift fellowship program.

“Latinos are afraid of talking, of standing up,” Sauceda said. “They’re so afraid they’re not going to be heard because of their legal status. But I will recommend this (fellowship) to parents. How do we want our kids to have a better education if we can’t dedicate time?”

The training includes lessons on local school options, how to speak publicly at a school board meeting, and how to advocate for your children if you believe they are being treated unfairly.

The first fellowship was led by Ian Buchanan, former director of community partnership for the state-run Achievement School District. Now the program is taught in-house, and the Spanish-speaking class is being led this month by Carmelita Hernandez, an alumna.

“No matter what language we speak, we want a high-quality education for our kids just like any other parent,” Hernandez said. “A good education leads to better opportunities.”

Stopping summer slide

On National Summer Learning Day, Memphis takes stock of programs for kids

PHOTO: Helen Carefoot
Torrence Echols, a rising first-grader in Memphis, builds a tower with giant legos at the Benjamin L. Hooks Library on National Summer Learning Day.

When it comes to summer learning, it’s been a better year for Memphis, where a range of new programs have helped to stem learning loss that hits hard in communities with a high number of low-income students.

On Thursday, Mayor Jim Strickland celebrated that work in conjunction with National Summer Learning Day and against the backdrop of the children’s reading room of the city’s main library.

He estimated that 10,000 children and teens are being reached this summer through learning programs spearheaded through Shelby County Schools, Literacy Mid-South, Memphis Public Libraries, churches and nonprofit organizations across the community.

That’s a record-breaking number, Strickland says, in a city with a lot of students struggling to meet state and local reading targets.

Summer learning loss, also known as summer slide, is the tendency for students to lose some of the knowledge and skills they gained during the school year. It’s a large contributor to the achievement gap, since children from low-income families usually don’t get the same summer enrichment opportunities as their more affluent peers. Compounded year after year, the gap widens to the point that, by fifth grade, many students can be up to three years behind in math and reading.

But this summer for the first time, Shelby County Schools offered summer learning academies across the city for students most in need of intervention. And Memphis also received a slice of an $8.5 million state grant to provide summer literacy camps at nine Memphis schools through Tennessee’s Read to be Ready initiative.

Literacy Mid-South used Thursday’s event to encourage Memphians to “drop everything and read!”

The nonprofit, which is providing resources this summer through about 15 organizations in Greater Memphis, is challenging students to log 1,400 minutes of summertime reading, an amount that research shows can mitigate learning loss and even increase test scores.

Reading is a problem for many students in Memphis and across Tennessee. Less than a third of third-graders in Shelby County Schools read on grade level, and the district is working to boost that rate to 90 percent by 2025 under its Destination 2025 plan.

The city of Memphis, which does not fund local schools, has made Memphis Public Libraries the focal point of its education work. This summer, the library is offering programs on everything from STEM and robotics to art and test prep.

Parents are a critical component, helping their kids to take advantage of books, programs and services that counter the doldrums of summer learning.

Soon after the mayor left the Benjamin L. Hooks Library on Thursday, Tammy Echols arrived with her son, Torrence, a rising first-grader at Levi Elementary School. Echols said they visit regularly to read books and do computer and math games.

“We always do a lot of reading and we’re working on learning sight words,” Echols said as she watched her son build a tower out of giant Lego blocks. “Torrence is a learning child and it’s easy to forget what you just learned if you’re not constantly reinforcing.”

You can find summer learning resources for families from the National Summer Learning Association.