score scrutiny

Could scoring changes explain the rise in New York’s English test results? Experts say they’re not convinced

PHOTO: Stephanie Snyder

Recent reports have flagged yet another possible explanation for this year’s dramatic statewide improvement in English test scores: a drop in the number of questions students had to answer correctly in order to pass.

The New York Post wrote Friday that students needed fewer raw score points to pass the exam this year. In 2015, for example, third-grade students needed 34 out of 55 points to pass the English test, while in 2016 they needed 28 out of 47 points. Leonie Haimson, executive director of Class Size Matters, called the overall changes “very fishy” on her blog.

But does the scoring mean the test was easier this year? Not necessarily, said both state officials and outside researchers we consulted.

A spokesperson for the state education department said that questions are often, on average, slightly easier or more difficult each year. In order to ensure the tests are comparable, the number of questions students have to answer correctly can fluctuate each year.

“This analysis is flawed, irresponsible and misleading,” said Emily DeSantis, spokesperson for the New York State Education Department, about the New York Post story. “This year’s tests were just as rigorous as those in the past and the cut scores are the same that have been used since 2013.”

Some researchers agreed that slight fluctuations in the raw scores used don’t automatically imply the test was easier to pass. Recalibration is part of a normal process the state undergoes each year to determine the number of “raw score” points a student needs to earn to be considered proficient, said Jennifer Jennings, a New York University professor and testing expert.

If the test questions are slightly harder, the number of raw score points a student needs to earn could drop — that’s part of the process for determining scores.

“That’s like gravity. That’s just something that has to happen in this world. It’s just a principle of assessing,” she said. “Just eyeballing those numbers [for third grade English], that just looks like year-to-year variation to me.”

Researchers will be better able to discern the difficulty of this year’s exam after the state releases a technical report, which typically comes out a year after the exam results are released, said Aaron Pallas, a professor of sociology and education at Teachers College at Columbia University.

New York State Allies for Public Education, leaders of the statewide opt-out movement, put out their own analysis Friday, detailing decreases in the raw scores it took to pass the tests this year in 11 out of 12 exams. (Though, according to their analysis, the percentages of raw score points required to pass the math tests dropped even more sharply than the percentages in English, without a rise in proficiency scores as steep as the state saw on English.)

Whether or not this year’s test proves comparable to last year’s, advocates’ fears are grounded in a long history of questionable scoring. In 2010, the state was forced to acknowledge score inflation and make the tests harder.

Haimson said she had “15 years of skepticism” about the tests. “I don’t think we should be so innocent to allow these manipulations,” she said. “There is no reason to trust these new tests.”

Meanwhile, the researchers we spoke with said any comparisons between this year’s test and last year’s are inherently flawed. The state acknowledged Friday that the tests are not directly comparable because this year’s tests were shorter, for instance, and untimed.

“The state said in their release that the tests are not comparable,” said Dan Koretz, a professor of education at Harvard. “That’s sort of the end of the story. They’re not comparable.”

heads up

Tennessee will release TNReady test scores on Thursday. Here are five things to know.

PHOTO: Getty Images/Kali9

When Tennessee unveils its latest standardized test scores on Thursday, the results won’t count for much.

Technical problems marred the return to statewide online testing this spring, prompting the passage of several emergency state laws that rendered this year’s TNReady scores mostly inconsequential. As a result, poor results can’t be used to hold students, educators, or schools accountable — for instance, firing a teacher or taking over a struggling school through the state’s Achievement School District.

But good or bad, the scores still can be useful, say teachers like Josh Rutherford, whose 11th-grade students were among those who experienced frequent online testing interruptions in April.

“There are things we can learn from the data,” said Rutherford, who teaches English at Houston County High School. “I think it would be unprofessional to simply dismiss this year’s scores.”

Heading into this week’s data dump, here are five things to know:

1. This will be the biggest single-day release of state scores since the TNReady era began three years ago.

Anyone with internet access will be able to view state- and district-level scores for math, English, and science for grades 3-12. And more scores will come later. School-by-school data will be released publicly in a few weeks. In addition, Tennessee will unveil the results of its new social studies test this fall after setting the thresholds for what constitutes passing scores at each grade level.

2. Still, this year’s results are anticlimactic.

There are two major reasons. First, many educators and parents question the scores’ reliability due to days of online testing headaches. They also worry that high school students stopped trying after legislators stepped in to say the scores don’t necessarily have to count in final grades. Second, because the scores won’t carry their intended weight, the stakes are lower this year. For instance, teachers have the option of nullifying their evaluation scores. And the state also won’t give each school an A-F grade this fall as originally planned. TNReady scores were supposed to be incorporated into both of those accountability measures.

3. The state is looking into the reliability of the online test scores.

In addition to an internal review by the Education Department, the state commissioned an independent analysis by the Human Resources Research Organization. Researchers for the Virginia-based technical group studied the impact of Tennessee’s online interruptions by looking into testing irregularity reports filed in schools and by scrutinizing variances from year to year and school to school, among other things.

4. The reliability of paper-and-pencil test scores are not in question.

Only about half of Tennessee’s 600,000 students who took TNReady this year tested on computers. The other half — in grades 3-5 and many students in grades 6-8 — took the exams the old-fashioned way. Though there were some complaints related to paper testing too, state officials say they’re confident about those results. Even so, the Legislature made no distinction between the online and paper administrations of TNReady when they ordered that scores only count if they benefit students, teachers, and schools.

5. Ultimately, districts and school communities will decide how to use this year’s data.

Even within the same district, it wasn’t uncommon for one school to experience online problems and another to enjoy a much smoother testing experience. “Every district was impacted differently,” said Dale Lynch, executive director of the state superintendents organization. “It’s up to the local district to look at the data and make decisions based on those local experiences.”

District leaders have been reviewing the embargoed scores for several weeks, and they’ll share them with teachers in the days and weeks ahead. As for families, parents can ask to see their students’ individual score reports so they can learn from this year’s results, too. Districts distribute those reports in different ways, but they’re fair game beginning Thursday. You can learn more here.

Sharing Stories

Tell us your stories about children with special needs in Detroit

PHOTO: Patrick Wall

Parents of students with special needs face difficult challenges when trying to get services for their children. Understanding their children’s rights, getting them evaluated and properly diagnosed, and creating an educational plan are among the many issues families face.

Chalkbeat Detroit wants to hear more about those issues to help inform our coverage. We are kicking off a series of conversations called a “listening tour” to discuss your concerns, and our first meeting will focus on children with special needs and disabilities. We’re partnering with the Detroit Parent Network as they look for solutions and better ways to support parents.

Our listening tour, combined with similar events in other communities Chalkbeat serves, will continue throughout this year on a variety of topics. In these meetings, we’ll look to readers, parents, educators, and students to help us know what questions we should ask, and we’ll publish stories from people who feel comfortable having their stories told. We hope you’ll share your stories and explore solutions to the challenges parents face.

Our special education listening tour discussion will take place from 5:30-7:30 p.m., Tuesday July 24, at the Detroit Parent Network headquarters, 726 Lothrop St., Detroit.

As our series continues, we’ll meet at locations around the city to hear stories and experiences parents have while navigating the complexities of getting children the education and services they deserve.

Next week’s event includes a panel discussion with parents of children with special needs, responses from parent advocates, and an open discussion with audience members.

Those who are uncomfortable sharing stories publicly will have a chance to tell a personal story on an audio recorder in a private room, or will be interviewed by a Chalkbeat Detroit reporter privately.

The event is free and open to anyone who wants to attend, but reservations are required because space is limited. To register, complete this form, call 313-309-8100 or email frontdesk@detroitparentnetwork.org.

If you can’t make our event, but have a story to share, send an email to tips.detroit@chalkbeat.org, or call or send a text message to 313-404-0692.

Stayed tuned for more information about listening tour stops, topics and locations.