number crunching

Some analysis left undone in data-driven education department

P.S. 199 in the South Bronx, one of the city's top-rated elementary schools in recent years. A high rate of its former students' test scores plummeted once they moved onto middle school.

The Department of Education crunches state test scores in dozens of ways to measure the performance of schools, principals, teachers, and students. But it does not perform a statistical analysis that can reveal whether an elementary school’s graduates have received test scores that far outstrip their actual skills.

Researchers say it would be relatively easy for the department to calculate “swing rates” to find the proportion of students from each school whose scores rise or fall by a statistically unlikely margin when they move to another school. Such an analysis could take some of the burden off of individual educators to report suspicions of cheating.

The city used to conduct swing rate analysis prior to the Bloomberg administration, according to a former testing official, and the state is poised to launch the measure as part of an overhaul of its own approach to test security.

But department officials say the analysis would be too onerous. They also say that they never launch investigations into cheating based on data anomalies alone. Instead, they say they will dispatch investigators only when they receive formal allegations of test improprieties.

The policy means that some top-rated schools whose students’ scores plummet at far higher than the average rate never have their testing practices scrutinized.

For all of the criticism of state tests as being arbitrary and imperfect measures of student performance, they are remarkably stable. In 2011, students who saw their scores fall by more than two standard deviations from the previous year made up just 0.6 percent of the sixth grade test-taking population in English, and 0.4 percent in math. That degree of decline is highly improbable under normal circumstances and is more likely to reflect externalities than real changes in academic proficiency.If one student’s test score plummets to that degree, it might be reasonable to conclude that he had a bad day when the second test was administered. But if an entire cohort of students see their scores plummet, it could be that testing conditions were especially favorable in the first year, maybe illicitly.

Some city schools have posted swing rates many times the 0.6 percent average, according to the New York Times. At the two elementary schools with the highest scores on the city’s 2011 progress reports, P.S. 257 and P.S. 31 in Brooklyn, the swing rates that year were 9 percent and 1.8 percent, respectively.

Neither school raised eyebrows at the department until a whistleblower at a middle school that received their students registered an official complaint about wide discrepancies between the students’ test scores and their actual skills. The allegation triggered investigations at both schools.

But the Brooklyn schools’ swing rates were not even the highest in the city that year. About 30 percent of the 34 students who graduated from P.S. 199 in the South Bronx and went on to a middle school up the road, I.S. 303, saw their test scores drop by more than two standard deviations. P.S. 199 had the 16th highest progress report score that year.

At I.S. 232, a nearby middle school that absorbed 29 students from P.S. 199, the swing rate was more in line with the city average.

But teachers at I.S. 303 said that the high test scores did not correlate to the basic English and math skills that many of the incoming P.S. 199 students demonstrated early on in sixth grade.

“You don’t just forget everything,” said one math teacher. “It just baffled me that they somehow got 3’s and 4’s” in fifth grade.

Multiple teachers at the school said students’ behavior during the sixth-grade state tests suggested that the students had received help during tests before.

Some of the students from P.S. 199 grew frustrated during the tests when I.S. 303 teachers did not tell them the answers to questions that stumped them, the teachers said. The teachers agreed to speak on the condition of anonymity because their principal had not given them permission to speak to reporters.

But even after noticing the score discrepancy, no one at I.S. 303 went to the department that year. It’s a common decision, according to educators from across the city who say allegations usually bring scrutiny first to the people who filed them, sometimes exposing them as whistleblowers to their colleagues.

An investigation at P.S. 199 is now open, a department spokeswoman said. Education officials contacted investigators after GothamSchools asked repeatedly about the school’s anomalous scores.

The department’s Office of Special Investigations is handling the case now after it was referred by the city’s Office of the Special Commissioner of Investigation, a SCI spokeswoman said.

Department officials said their concerns were not based solely on the swing rate. Data points such as swing rates alone are not enough to trigger investigations.

Many of the 37 schools to which the department dispatched testing monitors last year had seen their scores increase in unusual ways. But all but four of them had also been the subject of formal allegations.

In fact, the department does not even calculate schools’ swing rates as part of its regular analysis of schools’ performance. When the accountability division runs the numbers that feed into schools’ annual progress reports, which are based largely on students’ year-to-year growth, it does not aggregate results by sending schools.

Department officials say generating schools’ swing rates is a complicated endeavor. After learning about the score discrepancies at P.S. 199, GothamSchools requested swing data for other top-rated schools. But department officials said for months that running those numbers would be too difficult.

“We have provided you with a detailed analysis of P.S. 199,” spokeswoman Marge Feinberg wrote in an email this summer. “For the other schools, it would take a great deal of time.”

Researchers who have worked with Department of Education data before disputed that claim.

“It would be quite easy to do. Just about anyone with a computer and a basic knowledge of statistics could run these checks,” said Sean Corcoran, a New York University professor who has studied the city’s test score data. “This is something the DOE should do on a regular basis.”

A former testing director in New York City said that under his watch the city used data anomalies to trigger investigations.

“We routinely would look at the change in scores in schools from one year to the next,” said Bob Tobias, the Board of Education’s longtime testing chief who retired in 2002. Schools that showed erratic spikes, Tobias said, “would get a little more scrutiny.”

There are signs that the state might start conducting this type of analysis on its own. One charge given to the brand-new test security chief Tina Sciocchetti, at the State Education Department is to set guidelines for pursuing investigations using data methods that look suspicious test score patterns.

“I think those sorts of statistical analyses are simply a red flag and it’s absolutely true that additional investigation is necessary,” said Sciocchetti.

Top 10

From forest preschools to a secret apology, here are Chalkbeat Colorado’s must-read stories of 2018

PHOTO: Ann Schimke/Chalkbeat
Megan Patterson works with children to make a dam in a creek during a recent "forest school" class.

We spend a lot of time at Chalkbeat chasing the news to keep our readers informed about controversial policy changes, fast-moving debates, and late-breaking decisions.

But we also relish the opportunity to dig deep into issues affecting students and families, shine light on innovative ideas, and hear from dedicated educators making a difference. With that in mind, we’ve gathered 10 of our best stories from 2018.

These stories don’t necessarily chronicle the biggest education issues of 2018, from teacher walkouts to unprecedented state interventions. But they are stories we think are important and insightful, and that we enjoyed reporting and writing. We hope you enjoy reading them.

No walls: Forest preschools let kids run free, but can they change to reach diverse families?

One day this past summer, about a dozen children frolicked by a Jefferson County creek — making pretend tea in small metal buckets, and building dams with sticks and mud.

They were students at Worldmind Nature Immersion School, where children spend all their time outside. So-called forest preschools like Worldmind are beloved by many families but face significant regulatory and logistical barriers in expanding their footprint nationwide. Here in Colorado, a pilot program could lead a new kind of child care license designed for them.

And being licensed could help the schools confront another problem: a lack of diversity among their students. Read more.

Colorado was never ranked 46th for teacher pay. Does this change the debate?

It was an oft-cited statistic: that Colorado, despite its booming economy, ranked 46th in the nation for teacher pay. The eye-popping number found its way onto social media posts and signs at massive teacher rallies last spring. News outlets latched on to it, too.

But it was wrong. Colorado was actually ranked 30th in the nation.

Our story breaks down how the mistake happened (hint: a new data system, an unrevised report) — and how groups with different agendas seized on the snafu to score points. Read more.

This is the letter of apology that Adams 14 leaders never sent

The Adams 14 district in Commerce City is arguably the most troubled and low-performing in Colorado. Just last month, state officials directed Adams 14 to hire an external manager to oversee the district’s operations for at least the next four years.

Back in September, Adams 14 officials considered taking a rare step: saying sorry to the community. But an apology letter was never signed nor sent out.

Chalkbeat obtained a copy of the letter, which makes mention of “various and conflicting priorities, coupled with constant turnover and organizational disarray.” Read more.

Rising test scores and dwindling trust: Denver’s Tom Boasberg leaves a complicated legacy

Colorado’s largest school district experienced a big change this year when longtime leader Tom Boasberg announced he would step down after nearly 10 years in Denver Public Schools.

Because of his school improvement strategies — some of which were controversial and heightened tensions with the community — the district that the new superintendent, Susana Cordova, will inherit in January is vastly different than it was a decade ago.

One tangible difference: Schools that once served as anchors of the community but struggled academically have been closed or replaced. That disappearance was on display on one of Boasberg’s last days, when he held his cell phone close to his mouth and enunciated each word so his GPS would understand his direction: “Montbello High School.” Read more.

Parents in one Aurora high school are visiting classrooms and giving teachers feedback

Like many schools in Colorado, Aurora’s Rangeview High School has a test score gap between white and black students. But the assistant principal there came up with a unique way to try to address it: by inviting black parents to visit classrooms and observe how students are — or are not — engaging with the teacher’s lesson, and then provide suggestions for improvement.

“We give true and honest feedback,” said one parent involved, “if they looked or appeared comfortable, how they interacted with the environment, the temperature of the room.”

Although the assistant principal considers the African American Parent Committee an experiment, she said it’s generating uncomfortable but necessary conversations. Read more.

How education reform became a wedge issue among Colorado Democrats this election year

For years, more moderate Democrats, often working in unison with like-minded Republicans, championed education reform efforts ranging from school choice to holding educators accountable for student performance.

But partly because of backlash against President Donald Trump and his education secretary, those strategies no longer fly with many Democrats — especially left-leaning Democrats who see them as undercutting public education and devaluing the work of teachers.

That sentiment was palpable in Colorado’s Democratic gubernatorial primary, and could shape the next legislative session, which starts in January. “Education is the issue that really stands to divide the left in a very substantial way,” one observer said. Read more.

How a Colorado school district turned things around at 10,000 feet above sea level

School improvement efforts look a little different high in the Rocky Mountains. While many of the strategies used by the 1,000-student Lake County school district are familiar to urban settings, they’ve been retrofitted to meet the needs of a district that’s 100 miles west of Denver.

For example, instead of firing teachers and principals who weren’t accelerating student learning fast enough, the district adopted a new curriculum and gave its teachers lots of training.

“The belief that the people are the problem is wrong,” the superintendent said. “Our teachers are professionals, and we believe in them. We’re proving that there is a framework or a pathway for rural schools to improve that’s about building capacity within your own community.” Read more.

7 things to know about how Colorado schools punish their youngest students

After state lawmakers rejected a bill to limit the use of suspensions in the earliest grades, Chalkbeat wanted to know more about the early childhood discipline landscape in Colorado. Data from the Colorado Department of Education revealed several trends.

Among them: Young black boys are suspended at disproportionate rates. Some rural school districts have the highest early childhood suspension rates in the state.

And despite nationwide debate about the impact of harsh discipline on young children and local efforts to bring the numbers down, suspensions in the early grades are actually going up. Read more.

In Denver’s gentrifying neighborhoods, some middle-class parents are avoiding the school down the block

Many neighborhoods in Denver are gentrifying, with middle-class families moving into what have historically been working-class communities. That type of demographic shift could easily lead to neighborhood schools that are more integrated by family income and race.

But that doesn’t always happen in Denver. Instead, data show that wealthier families – more often than low-income families – are using Denver Public Schools’ universal school choice process to send their kids to schools elsewhere in the city.

That’s a problem because research shows integrated schools boost test scores for students from low-income families without lowering the scores of those from wealthier ones. Denver officials want to see those benefits, but allowing parents to choose may be thwarting them. Read more.

Why this Colorado principal hand delivers birthday cards to more than 2,000 students and staff

Northglenn High School Principal Sharee Blunt is Colorado’s 2018 School Principal of the year — but perhaps even more impressive is the enormous number of birthday cards she hand delivers each year. If you’re one of those people who can barely remember your spouse’s birthday, you’ll be floored by Blunt’s annual feat.

In our interview with Blunt, part of Chalkbeat’s “How I Lead” Q&A series with distinguished school leaders, she talks about what she realized after a mother’s emotional reaction, and why she gave a teacher a pass during a lesson that went awry. Read more.

union power

Charter teachers won big in nation’s first strike. What now?

PHOTO: Yana Kunichoff / Chalkbeat
Teachers from Acero charter schools in Chicago protest stalled negotiations Oct. 24, 2018, as they readied to vote on authorizing a strike.

Some 500 unionized teachers joined in the nation’s first charter strike last week, and succeeded in negotiating wage increases, smaller class sizes and a shorter school day. Their gains could foreshadow next year’s citywide contract negotiations — between the Chicago Teachers Union, with its contract expiring in June, and Chicago Public Schools.

“The issue of class size is going to be huge,” said Chris Geovanis, the union’s director of communications. “It is a critically important issue in every school.”

Unlike their counterparts in charters, though, teachers who work at district-run schools can’t technically go on strike to push through a cap on the number of students per class. That’s because the Illinois Education Labor Relations Act defines what issues non-charter public school teachers can bargain over, and what issues can lead to a strike.

An impasse on issues of compensation or those related to working conditions, such as length of the school day or teacher evaluations, could precipitate a strike. But disagreements over class sizes or school closures, among other issues, cannot be the basis for a strike.

The number of students per class has long been a point of contention among both district and charter school teachers.

Educators at Acero had hopes of pushing the network to limit class sizes to 24-28 students, depending on the grade. However, as Acero teachers capped their fourth day on the picket line, they reached an agreement with the charter operator on a cap of 30 students — down from the current cap of 32 students.

Andy Crooks, a special education apprentice, also known as a teacher’s aide, at Acero’s Sor Juana Inés de la Cruz school and a member of the teachers bargaining team, said that even having two fewer students in a classroom would make a huge difference.

“You really do get a lot more time with your students,” Crooks said. “And if you are thinking about kindergarten in particular, two less 5-year-olds really can help set the tone of the classroom.”

In district-run schools, classes are capped at 28 students in kindergarten through third grade, and at 31 students in fourth through sixth grade. But a survey by the advocacy group Parents 4 Teachers, which supports educators taking on inequality, found that during the 2017-2018 school year, 21 percent of K-8 classrooms had more students than district guidelines allowed. In 18 elementary school classrooms, there were 40 or more students.

The issue came up at last week’s Board of Education meeting, at which Ivette Hernandez, a parent of a first-grader at Virgil Grissom Elementary School in the city’s Hegewisch neighborhood, said her son’s classes have had more than 30 students in them. When the children are so young and active — and when they come into classrooms at so many different skill levels — “the teachers can’t handle 30 kids in one class,” she told the board.

Alderman Sue Garza, a former counselor, accompanied Hernandez. She also spoke before the board about classroom overcrowding — worrying aloud that, in some grades at one school in particular, the number of students exceeded the building’s fire codes. (Board chair Frank Clark said a district team would visit the school to ensure compliance fire safety policies.)

While the Chicago Teachers Union aren’t technically allowed to strike over class sizes, the union does have a history of pushing the envelope when it comes to bargaining.

Back in 2012, when the Chicago Teachers Union last went on strike, they ended up being able to secure the first limit on class sizes in 20 years because the district permitted the union to bargain over class size.

They also led a bargaining campaign that included discussion over racial disparities in Chicago education and school closures, arguing that these trends impacted the working conditions of teachers.

“Even if you can’t force an employer to bargain over an issue, you can push them to bargain over the impact of an issue,” Bob Bruno, a labor professor at the University of Illinois at Champaign-Urbana, explained.

The Chicago Teachers Union also emerged from its 2012 negotiations with guarantees of additional “wraparound services,” such as access to onsite social workers and school counselors.