Data dive

In Boulder, students with special needs are outperforming their state peers but are far behind their own classmates

PHOTO: RJ Sangosti, Denver Post

For two years in a row, Colorado education officials have raised concerns about how students with special needs are falling far behind on state English and math tests.

This week, we’re getting a better sense of which school districts are following the state trend and which ones are bucking it.

The state on Monday released data from the 2016 and 2017 PARCC exams broken down by different student groups. Chalkbeat has spent a few days looking at the data, first by race and ethnicity and then by socioeconomic status. Today we’re looking at how students with learning disabilities in the state’s 10 largest school districts performed compared to their peers.

Students with learning disabilities historically underperform the general student population on standardized tests. However, advocates say that with the right help, students with non-cognitive disabilities should be just as likely to score well.

That isn’t happening in many of the state’s largest school districts. Here’s a look at how the results break down:

Some of the widest gaps are in northern Front Range school districts such as Poudre, Boulder Valley and St. Vrain.

Boulder is a unique case. While the achievement gap there is wide, students with learning disabilities are outperforming their state peers.

Boulder officials suggested a longstanding and close partnership between their special education department and curriculum specialist could be a key factor in the higher scores.

“There’s a lot of cross training,” said Samantha Messier, Boulder’s assistant superintendent of instructional services and equity. “That’s something that is paying off for students.”

Meanwhile, school districts such as Denver and Aurora are rethinking some of their special education policies.

In Denver, the district is pushing to have students with learning disabilities spend more time in general education classrooms.

“The main focus is on providing the individualized supports that students with disabilities need,” said Superintendent Tom Boasberg. “And we’re significantly increasing our inclusion practices. Many more of our students with disabilities are included more of the time with their general education peers.”

The charter-friendly school district has also required its charter schools to take on greater responsibility in meeting the needs of students with disabilities.

The Aurora school district has hired consultants to help the district better identify students with learning disabilities, intervene earlier and provide them with the proper learning environment.

“We’ve gotten better with early interventions,” said Superintendent Rico Munn.

tar heel trivia

New education research? A good chance it’s from North Carolina.

PHOTO: Creative Commons/Boston Public Library

Barbeque. Basketball rivalries. The Blue Ridge Mountains.

Education research?

It’s something else North Carolina is known for, at least among a subset of social scientists.

“North Carolina has really done something special,” says Amy Ellen Schwartz, a professor and the editor of Education Finance and Policy, an academic journal.

“If you look over the last 20 years and focus on the highest quality work, it’s disproportionately work that comes from North Carolina data,” says Dan Goldhaber, an education professor at the University of Washington at Bothell.

North Carolina students aren’t more interesting or easier to find. But a disproportionate share of education research — and therefore, a disproportionate amount of what we know about how certain policies work — comes out of the Tar Heel State.

That’s because North Carolina has kept track of things like student test scores, teacher demographics, and school accountability data since the ‘90s, and also made that information more accessible to researchers than anywhere else.

It works well for those looking for data. But it also underscores a troubling reality: We know much less about how policies play out in places where data is hard to access — and in some cases, may be kept under lock and key for political reasons. That leaves the public to take the best lessons it can from a state that’s home to just 3 percent of the country’s public school students.

“The problem is that what you really want to do is look at lots of places,” said Schwartz, a professor at the Maxwell School at Syracuse University. “You want to be able to leverage the natural experiments and understand the variation in a way that’s really hard to do in one place.”

Of course, researchers in many cases do work productively with local officials to obtain data. And although it appears that North Carolina is the most commonly studied state in education policy, it is by no means the subject of the majority of academic papers. For instance, seven studies published in Education Finance and Policy over the last two years were focused on North Carolina — more than any other state or district, though over 30 others focused on K-12 schooling in the U.S used national data or data from elsewhere.

North Carolina’s popularity is tied to the fact that it is one of the few states where researchers can get student data (that has been anonymized) from a third party, in this case a research center established in 2000 that operates out of Duke University. In most states, the state education department or other state agency controls that information. Many states and districts lack the resources, streamlined systems, or staff capacity that North Carolina’s center has to meet researchers’ requests.

That center also separates policymakers and the keepers of the data — which may be crucial for ensuring information is made available.

“Not every place wants to open up their data and say, ‘Study what you want,’” said Schwartz. “The risk is that a researcher investigates something or casts it in a way that’s not positive for the school district.”

Goldhaber echoed this. “If you’re talking to somebody who’s involved with politics … they’re going to see everything through a political lens. And that when it comes to evaluating programs and policies, people often don’t see much upside,” he said.

In North Carolina, local researchers realized the importance of tracking students and schools over time, according to Duke’s Clara Muschkin, the faculty director of the data center.

When Goldhaber was studying schools there in the 1990s, he recalled, “There was a real belief that people ought to study these issues, and that was kind of pervasive under Gov. Jim Hunt.”

That extended to research that Hunt’s administration might not like. For instance, Goldhaber was interested in studying whether teachers who attained National Board certification were more effective in the classroom. Hunt was the founding board chair of the organization that awarded those certifications, and Goldhaber’s research had previously shown that certification types didn’t make much difference. But that didn’t stop the administration from providing that data to Goldhaber, who ultimately found North Carolina’s board certified teachers were particularly effective.

It’s impossible to say how often political concerns play a role in keeping data from researchers. When politics is involved, researchers themselves may not know, and if they do, they may not want to publicize it in hopes of eventually working out an agreement. (This reporter has heard frequent complaints about politics getting in the way of data access — but in most cases those are made off the record.)

A more subtle method of interference is when officials decide not to collect data in the first place that researchers might use to reach unflattering conclusions. California, Goldhaber said, is a particular culprit.

The largest state in the country has weakened, or declined to improve, its data systems since 2010, and the information that exists is not readily available to researchers. Governor Jerry Brown has argued that educational data is of little use to teachers and schools, and feeds into a test-focused mentality of schooling.

“You are not collecting data or devising standards for operating machines or establishing a credit score,” wrote Brown in a critique of the Obama administration’s Race to the Top program, which encouraged more data collection. “I sense a pervasive technocratic bias and an uncritical faith in the power of social science.”

Goldhaber has found it difficult to study the state’s education policies.

“There is just basic data that we could not get out of California,” he said, referring to a study he and colleagues are undertaking there.

Some places are becoming more cognizant of concerns about a lack of quality research about their schools. In Washington, D.C., the city council is considering funding an education research group and may make its data widely available to researchers. In California, some advocates and policymakers have pushed for improving its data systems, an idea the state’s likely next governor has backed.

In the meantime, those interested in key education questions — in California, DC, and elsewhere — can always look to North Carolina for answers. That’s largely a good thing, says Goldhaber.

“The fact that we are learning things in North Carolina is tremendously useful for informing policy and practice in other states,” he said.

right to know

A bastion of student data privacy, Colorado yields a bit to demands for more openness

PHOTO: Getty Images

Colorado education officials are reconsidering the data privacy rules that for three years in a row have hidden large amounts of student achievement data from public view.

With Thursday’s release of state test results, the public has greater ability to see how well certain groups of students perform on state tests compared with their peers than they’ve had since 2015, when the state adopted a much more stringent approach.

The main change this year is that the public can see results separated by race and ethnicity, disability status, English language learner status, and economic status at the school level, rather than only by grade levels within schools. Combining results across grades creates larger sample sizes, so that in some cases they no longer had to be redacted. The practice of obscuring results for small groups of students had led to large amounts of missing information and vexed advocates for school improvement.

And state education officials are considering additional changes for 2019 that should make more information available – though just how much remains to be seen.

“In this latest release, we were very pleased to be able to see how kids are making progress,” said Van Schoales, executive director of A Plus Colorado, an education reform advocacy group that focuses on research. “We can now see if they met or exceeded standards for groups of 16 or more. We thank the Colorado Department of Education for figuring out some methods for showing that.”

A Plus Colorado was part of a coalition that wrote the Colorado Department of Education last November calling for more transparency. That group includes the Colorado Children’s Campaign, the business-oriented education reform group Colorado Succeeds, civil rights groups like Together Colorado and Padres y Jóvenes Unidos, Democrats for Education Reform, the Colorado League of Charter Schools, and the Colorado Freedom of Information Coalition.

“The current lack of transparency in student data and results is a major challenge to our collective efforts to build a better education system,” they wrote. “Results that are not transparent and comprehensive not only undermine our ability to improve students’ education, but call into question the honesty and integrity of the entire accountability system.”

Advocates and state education officials met through the spring and summer to see if any common ground could be found. The state also hired an outside firm, HumRRO, to assess its data practices.

That firm, according to Deputy Commissioner Alyssa Pearson, found that Colorado was right to change its practices in 2015 but that there was room for greater transparency while still protecting student privacy.

This debate doesn’t involve test results of individual students but rather information about how groups of students at particular schools perform. Advocates for school accountability say the ability to know, for example, how well students with disabilities or those learning English perform in comparison with their peers is key to knowing how well schools are serving all their students.

Colorado was previously one of the most open states in providing detailed information about student performance. But starting in 2015, the Colorado Department of Education withheld a lot more information, particularly when results were separated out by categories of students. Instead of hiding results just for groups of 16 students or smaller, the state hides results from additional categories of students to prevent a careful observer from working backward to determine what was in the redacted field. This is known as complementary suppression.

And instead of showing how many students scored in each tier on state tests, Colorado shows what percentage of students met or exceeded expectations. That makes it hard to know if students who didn’t meet expectations were just below the threshold or way at the bottom.

Colorado plans to keep obscuring results from small groups of students and to continue the practice of complementary suppression. There are still missing results in the 2018 data publicly released so far. However, by combining results from different grade levels within schools, a more complete picture emerges of how each subgroup of students is being served.

No single state or federal law dictates the specific practices that Colorado has adopted. In fact, federal law requires that states make disaggregated data about student performance available to the public, so long as it doesn’t end up revealing personally identifiable information. These practices are what state officials decided was necessary to prevent the release of information traceable to individual students. That provides some wiggle room to reassess those practices.

Pearson said state officials are focused on who needs to know what information and for what purposes and then trying to accommodate that within privacy requirements. They’re also revisiting the question of how likely it is a “reasonable person” could trace certain pieces of data back to individual students.

In addition to making detailed school-level data available, Colorado has revamped its own online search tool, Data Lab, to make it easier for the public to search for how different groups of students performed at particular schools. Right now the information is available across multiple spreadsheets available for download. The data from 2018 tests should be incorporated into Data Lab sometime in September.

Another change on the table: Reducing the threshold for suppressing data. State officials will decide by the end of the year about whether to lower the bar for concealing group data to 10 students, from the current 16. That would mean less data would be suppressed – but how much less isn’t clear yet.

Pearson said the state hopes to make a final decision by the end of the year so that changes can be incorporated into the 2019 testing cycle. It hasn’t been decided yet whether administrators would make that decision on their own or whether the State Board of Education would vote on the policy change.

Along with pressing the state to release more information, advocates for more transparency have also launched a Right to Know website and campaign to encourage districts and schools to share more information with families. Districts are required by law to share information from state assessments with parents, but practices vary widely across the state.

At a recent meeting of the State Board of Education, members of the Colorado Youth Congress, part of the Right to Know coalition, urged the state to do more to put this information in the hands of the people most affected.

“As a student, I want to know if my school is performing strongly or not so I can understand the setting I’m in and if I want to change to a different setting that would benefit me more,” said Jasmine Kabiri, a junior at Silver Creek High School in Longmont. “In a different situation, if we’re shown the performance data of my school, I could help motivate my school toward advancing themselves.”