failing grade

Why one Harvard professor calls American schools’ focus on testing a ‘charade’

PHOTO: Alan Petersime

Harvard professor Daniel Koretz is on a mission: to convince policymakers that standardized tests have been widely misused.

In his new book, “The Testing Charade,” Koretz argues that federal education policy over the last couple of decades — starting with No Child Left Behind, and continuing with the Obama administration’s push to evaluate teachers in part by test scores — has been a barely mitigated disaster.

The focus on testing in particular has hurt schools and students, Koretz argues. Meanwhile, Koretz says the tests are of little help for accurately identifying which schools are struggling because excessive test prep inflates students’ scores.

“Neither good intentions nor the value of well-used tests justifies continuing to ignore the absurdities and failures of the current system and the real harms it is causing,” Koretz writes in the book’s first chapter.

Daniel Koretz, Harvard Graduate School of Education

His skepticism will be welcome to families of students who have opted out of state tests across the country and others who have led a testing backlash in recent years. That sentiment helped shape the new federal education law, ESSA.

Koretz has another set of allies in some conservative charter and voucher advocates, including — to an extent — Secretary of Education Betsy DeVos, who criticized No Child Left Behind in a recent speech. “As states and districts scrambled to avoid the law’s sanctions and maintain their federal funding, some resorted to focusing specifically on math and reading at the expense of other subjects,” she said. “Others simply inflated scores or lowered standards.”

But national civil rights groups and some Democratic politicians have made a different case: That it’s the government’s responsibility to continue to use test scores to hold schools accountable for serving their students, especially students of color, poor students, and students with disabilities. (ESSA continues to require testing in grades three through eight and for states to identify their lowest performing schools, largely by using test scores.)

We talked to Koretz about his book and asked him to explain how he reached his conclusions and what to make of research that paints a more positive picture of tests and No Child Left Behind.

The interview has been edited for clarity and length.

Do you want to walk me through the central thesis of your book?

The reason I wrote the book is really the subtitle: we’re “pretending to make schools better.”

Most of the bad news that’s in this book is old news. We’ve been collecting evidence of various kinds about the impact of the very heavy handed, high-stakes testing that we use in this country for a long time. I lost patience with people pretending that these facts aren’t present. So I decided it would be worth writing a book that summarizes the evidence both good and bad about the effects of test-based accountability. When you do that, you end up with an awful lot on the bad side and not very much on the good side.

Can you talk about some of the bad effects?

There are a few that are particularly important. One is absolutely rampant bad test prep. It’s just everywhere. One of the consequences of that is that test scores are often very badly inflated.

There aren’t all that many studies of this because it’s not really a welcome suggestion. When you go to the superintendent and say, “Gee, I’d like to see whether your scores are inflated,” they rarely say, “Boy, we’ve been waiting for you to show up.” There aren’t that many studies, but they’re very consistent. The inflation that does show up is sometimes absolutely massive. Worse, there is growing evidence that that problem is more severe for disadvantaged kids, creating the illusion of improved equity.

Another is increasingly widespread cheating. We, of course, will never know just how widespread because there aren’t resources to examine the data from 13,000 school districts. Everyone knows about Atlanta, a few people know about El Paso, but that’s just the tip of the iceberg.

There’s obviously also — and perhaps this should be on the same par — enormous amounts of stress for teachers, for kids, and for parents. That’s the bad side.

I want to ask a little more about test score inflation. What is the strongest evidence for inflation? And let me give you two pieces that to me seem like potentially countervailing evidence. One piece is when I’m looking at research on school turnaround — like the most recent School Improvement Grant program and also turnaround efforts in New York City — these schools have been under intensive pressure to raise test scores. And yet their test score gains on high-stakes tests have been pretty modest at best. The other example is the Smarter Balanced exam. The scores on the Smarter Balanced exam don’t seem to be going up. If anything, they’re going down.

The main issue is that score inflation doesn’t occur in the same amount everywhere. You’ve come up with two examples where there is apparently very little. There are other examples that are much worse than the aggregate data suggest.

In the case of Smarter Balanced, I would wait and see. Score inflation can only occur when people become sufficiently aware of predictable patterns in the test. You can’t game a test when you don’t know what irrelevant things are going to recur, and that just may take some time.

I’m wondering your take on why some of the strongest advocates for test-based accountability have been national civil rights groups.

One of the rationales for some of the most draconian test-based accountability programs we’ve had has been to improve equity. If you got back to the enactment of NCLB, you had [then-Massachusetts Sen.] Teddy Kennedy and [then-California Rep.] George Miller actively lobbying their colleagues in support of a Republican bill. George Miller summed that up in one sentence in a meeting I went to. He said, “It will shed some light in the corners.” He said that schools had been getting away with giving lousy services to disadvantaged kids by showing good performance among advantaged kids, and this would make it in theory impossible to do that.

Even going back before NCLB, I think that’s why there was so much support in the disability community for including disabled kids in test-based accountability in the 1990s — so they couldn’t be hidden away in the basement anymore. I think that’s absolutely laudable. It’s the thing I praise the most strongly about NCLB.

It just didn’t work. That’s really clear from the evidence.

I think the intention was laudable and I think the intention was why high-stakes testing has gotten so much support in the minority community, but it just has failed.

You mention in your book probably the most widely cited study on the achievement effects of No Child Left Behind, showing that there were big gains in fourth grade math and some gains in eighth grade math, but there wasn’t anything good or bad in reading.

Pretty much. There was a little bit of improvement in some years in reading but nothing to write home about.

So the math gains — and that was on the low-stakes federal NAEP test — they’re just not worth it in your view?

I think the gains are real. But there are some reasons not be terribly excited about these. One is that they don’t persist. They decline a little bit by eighth grade, they disappear by the time kids are out of high school. We don’t have good data about kids as they graduate from high school, but what we do have doesn’t show any improvement.

The biggest reason I’m not as excited as some people are about those gains is we’ve had evidence going back to the 1980s that one of the responses that teachers have had to test-based accountability is to take time out of untested subjects and to put it into math and reading. We don’t know how much of that gain in math is because people are teaching math better and how much is because kids aren’t learning about civics.

That’s, in my view, not enough to justify all of the stuff on the other side of the ledger.

When I’ve looked at some studies on the impact of NCLB on students’ social-emotional skills, the impact on teachers’ attitudes in the classrooms, and the impact on voluntary teacher turnover, they haven’t found any negative effects. They also haven’t found positive effects in most cases. But that would seem to at least in one sense undermine the argument that NCLB had big harmful effects on these other outcomes.

I haven’t seen those studies, but I don’t think what you describe does undermine it. What I would like to see is an analysis of long-term trends not just on teacher attrition but on teacher selection. A lot of what I have heard has really been, frankly, anecdotal. I was once a public school teacher and teaching now is utterly unlike what it was when I taught. It seems unlikely that that had no effect on who opts in and who opts out to be a teacher.

I don’t have evidence of this but I suspect that to some extent different types of people are selecting into teaching now than were teaching 30 years ago.

Can you talk about what you see as good versus bad test prep?

Something that Audrey Qualls at the University of Iowa said was, “A student has only mastered something if she can do it when confronted with unfamiliar particulars.”

Think about training pilots — you would never train pilots by putting them in a simulator and then always running exactly the same set of conditions because next time you were in the plane and the conditions were different you’d die. What you want to know is that the pilot has enough understanding and a good enough command of the physical motions and whatnot that he or she can respond to whatever happens to you while you’re up there. That’s not all that distant an analogy from testing.

Bad test prep is test prep that is designed to raise scores on the particular test rather than give kids the underlying knowledge and skills that the test is supposed to capture. It’s absolutely endemic. In fact, districts and states peddle this stuff themselves.

I take it it’s very hard to quantify this test prep phenomenon, though?

It is extremely hard, and there’s a big hole in the research in this area.

Let’s turn from a backward-looking to a forward-looking discussion. What is your take on ESSA? Do you think it’s a step in the right direction?

This may be a little bit simplistic, but I think of ESSA as giving states back a portion of the flexibility they had before No Child Left Behind. It doesn’t give them as much flexibility as they had in 2000.  

It has the potential to substantially reduce pressure, but it doesn’t seem to be changing the basic logic of the system, which is that the thing that will drive school improvement is pushing people to improve test scores. So I’m not optimistic.

One of things that I argue very strongly at the end of the book is that we need to look at a far broader range of, not just outcomes, but aspects of schooling to create an accountability system that will generate more of what we want. ESSA takes one tiny step in that direction: it says you have to have one measure beyond testing and graduation rates. But if you read the statute it almost doesn’t matter what that measure is. The one mandate is that it can’t count as much as test scores — that’s written in the statute. The notion that it means the same thing to monitor the quality of practice or to monitor attendance rates is just absurd

As I’m sure you know, research — including from some of your colleagues at Harvard — has shown that so-called “no-excuses” charter schools in places like Boston, Chicago, and New York City, have led to substantial test score gains and in some cases improvements in four-year college enrollment. Are you skeptical that those gains are the result of genuine learning?

It depends on which test you’re talking about. Some of the no-excuses charter schools drill kids on the state test, so I don’t trust the state test scores for some of those schools. I think it’s entirely plausible that some of those schools are going to affect long-term outcomes because they’re in some cases replacing a very disorderly environment with a very orderly one. In fact, I would say too orderly by quite a margin.

But those reforms are much bigger than just test-based accountability or just the control structure we call charters. It’s a whole host of different things that are going on: different disciplinary policies, different kinds of teacher selection, different kinds of behavioral requirements, all sorts of things.

A lot of the discussion around accountability, including in your book, is about the measures we should be using to identify schools. I’m interested in your take on what happens when a school is identified by whatever system — perhaps by the holistic system you described in the book — as low performing.

The first step is to figure out why is it bad. I would use scores as an opening to a better evaluation of schools. If scores on a good test are low, something is wrong, but we don’t know what. Before we intervene we ought to find out what’s wrong.

This is the Dutch model: school inspections are concentrated on schools that shows signs of having problems, because that’s where the payoff is. I would want to know what’s wrong and then you can design an alternative. In some cases, it may be the teaching staff is too weak. It may be in some cases the teaching staff needs supports they don’t have. It may be like in the case of Baltimore, they need to turn the heat on. Who knows? But I don’t think we can design sensible interventions until we know what the problems are.

appropriate instruction

Special education in tumult as Adams 14 faces resignations, parent complaints

Javier Abrego, superintendent of Adams 14 School District, speaks to parents at a forum April 17, 2018. (Photo by Hyoung Chang/The Denver Post)

Just weeks from the end of the school year, Ed Collins found out that his 7-year-old son was not getting all the instruction he was supposed to get to address his attention deficit hyperactivity disorder.

Collins said the school in Adams 14 owed his son 18 days of instruction. Much of the time lost was due to suspensions that were contrary to his child’s special education plan, he said.

Likewise, Samantha Ochoa recently found out that her son, a freshman at Adams City High School, is failing all of his classes. Under a plan to address his learning disability, Ochoa said, the school owes him 650 hours of instruction.

The two parents’ stories are among many in the Adams 14 school district in Commerce City, which has had a tumultuous year with special education. A mass midyear exodus of top administrators and teachers has led to confusion at schools, a rotating crew of substitutes, gaps in services, and students not getting the education and support they need.

According to federal law, students who are identified with special needs must receive appropriate help so they can learn. Legally, schools must follow those students’ individualized education plans, although complaints don’t often go to court.

Javier Abrego, the superintendent of Adams 14, a district under a state plan for turnaround, and school principals refused to discuss the issues parents raise. To protect student privacy, school officials generally do not comment on the cases of individual students. But in a prepared statement, Abrego wrote that the district is “not aware of any IEP non-compliance concerns” – even though people have raised issues at school board meetings and during a community meeting earlier this month.

The silence of the district administration is a change for a district whose previous special education staff was dubbed “the dream team.”

“They were able to take a situation, whether it was brought to them by the parent or someone else and say, ‘let’s figure this out,’” said Paula Christina, a child and family advocate for ARC of Adams County, a nonprofit that helps families of special needs children. “You don’t always have that collaboration and skill set with all the districts.”

“It really saddens me to see how things have fallen apart,” Christina said. “Something is not working at the top level at the administration.”

Instead of responding to parents’ charges, district officials have highlighted efforts to teach students at some schools social and emotional skills. But one principal said that while the work is beneficial, it doesn’t help students with severe needs.

The staffing problems in the district have, as of April, included administrators. The district’s director of student services – who oversees special education – resigned, leaving for an assistant director job in Denver. On the next tier, three special education coordinators who help schools create and troubleshoot plans for students with special needs also resigned and will leave at the end of the school year.

Many special education teachers also have left, although officials would not provide numbers. Anecdotally, other administrators, teachers, and union officials say it has been a bad year.

According to a roster of special education teachers still working in late April, two elementary schools had just one teacher left, although the schools also have paraprofessionals.

Hanson Elementary School, where the district’s records show there are 50 students on special needs plans, has one teacher certified to teach special education. Rose Hill Elementary, which has 60 students with special needs plans — including some in a program for students with severe needs from across the district — also had just one teacher as of April, down from three at the start of the year.

That has led to a succession of substitutes filling in.

At Rose Hill, Collins said that his 7-year-old’s classroom teacher has helped his son a lot, but his assigned special education staffer is always changing, something that is particularly rough for his son.

“They’ve been using substitutes,” Collins said. “He doesn’t like change. I’m a single parent. His mom’s gone and his older sisters are with their mom. So he thinks that everybody that cares about him leaves him.”

Families say change is difficult for many children with special needs.

Because special education teachers are considered a hard-to-staff position across the country, many teachers and parents worry about filling so many positions by next school year.

Superintendent Abrego’s prepared statement notes the district is “exploring all options, including incentives to attract special education professionals to the district.” A spokesman also said the district will reconsider appropriate staffing levels once a new director of student services has been hired.

One change that has already been planned is to move the special needs program from Rose Hill to Central Elementary school next year in an effort to get more space and to pool resources of both schools special needs programs.

Apart from staffing, parents and teachers claim there are also problems in the district’s process for identifying students who have special needs, and then gaps in serving them.

Ochoa’s son at Adams City High School, for example, went through most of the school year with teachers unaware that he had a plan for his special needs and needed accommodations, she said.

“They kept assuring me everything was good,” Ochoa said. “That they were modifying his work. But they weren’t.”

The district recently offered Ochoa’s son a one-on-one tutor to help make up the instructional time, but it will likely go into the summer. Ochoa said her son is weary of the plan.

“He tells me, ‘Mom, I feel stupid, I just want to drop out,’” Ochoa said.

Other parents and teachers point to gaps in services for students with severe emotional or social needs.

Because of budget limitations, district officials say that they can only afford to have a districtwide program at one school for students with severe social or emotional needs. This year, that program is serving middle school students.

Kim Cini, the district’s assistant director of student services, said there aren’t many students identified with severe emotional or social needs in elementary or high school, but for those that are, the special education teams at schools can determine how to serve that student in a regular classroom, or if absolutely necessary, students can be sent to get services out of the district.

Currently there are 25 Adams 14 students that the district is paying to receive services elsewhere. That’s down from 38 out-of-district placements last year.

But integrating those students into regular classrooms can be challenging. Barb McDowell, president of the teachers union in Adams 14, said that in those cases teachers need help from qualified special education staff. Without it, McDowell said teachers are also seeing more behavior issues among all students.

With the district’s turmoil, parents complain that their children’s needs are being overlooked. Shelly Hebel said her 14-year-old daughter, who was diagnosed with severe anxiety, depression, and a learning disability, could have used more mental health help specific to her social and emotional needs this year.

Instead, Hebel found out recently that Adams City High School wasn’t following her daughter’s special education plan. Among the requirements of the plan, Hebel said, teachers needed to allow the girl to leave classes early so she could walk across the campus before hallways are crowded. The girl’s emotional problems got so bad, she stopped going to classes.

Hebel said her oldest daughter had similar issues years ago and ended up dropping out of school, so she says this time she’s not taking chances. She is pulling her daughter out of school and will send her to the district’s alternative high school next year.

Meanwhile the district’s assistant director of student services, Cini, is trying to keep other projects moving forward, which she said could help improve services in the district.

“I know that intervention and prevention is our ticket out of constant crisis,” Cini said.

Union president McDowell said some teachers blame this year’s problems in part on cuts made in the current school year to the number of mental health professionals such as counselors and psychologists at schools.

“There was a lot of pushback on that, so next year every school will have a full-time mental health worker,” Cini said. “Everybody’s going to have that.”

In Adams 14, district officials say the responsibility of making sure a student’s plan is being followed belongs to each principal.

Annie Fahnestock, a special education teacher who resigned in January from Rose Hill, said the schools need more oversight.

“It was all left up to the coordinators and the director of special education,” Fahnestock said.

Christina said that she has seen changes that concern her, including instances where principals haven’t been as receptive to suggestions or the district’s ideas about inclusion of students in general education classrooms.

In November, for example, parents complained that the new school administration at Rose Hill was discriminating against students in the districtwide program for severe needs and making them feel unwelcome.

Her nonprofit group provided a training at the school, including for the principal and other staff, on how to talk about disability and how to be sensitive to students’ needs.

Teacher Fahnestock said the district’s special education coordinators and director came to her school often to share their concerns about special education programming with her principal.

“But in the end, it was more like it’s her building,” Fahnestock said. “So she gets final say.”

beyond high school

Report: Memphis students from poor families less likely to have access to advanced coursework

PHOTO: By Glenn Asakawa/The Denver Post via Getty Images

While most high school students in Tennessee’s largest district have access to advanced courses to prepare them for college, most of those classes are concentrated in schools with more affluent families.

Of the 14 high schools in Shelby County Schools that offer more than 40 advanced classes, all but one have a lower percentage of students from poor families than the district.

Those schools educate slightly more than half of high school students in the Memphis district. In contrast, about a quarter of high school students are in schools with 20 or fewer advanced courses, according to a new district report.

District officials say those course offerings in the 2017-18 school year are closely correlated with the size of the school: The larger the student population, the more likely the school is to offer advanced courses. The concentration of schools with more affluent students was not examined in the report.

PHOTO: Shelby County Schools

The findings are scheduled to be presented at next week’s school board meeting as part of the district’s monthly check-in on various statistics on teaching and student learning.

Taking advanced classes in high school introduces students to college-level coursework and in many cases allows them to skip some college classes — saving students thousands of dollars. And because students from low-income families, who make up about 59 percent of Shelby County Schools, lag behind their more affluent peers in college enrollment, they are encouraged to take more advanced courses.

Advanced courses include programs such as such as Advanced Placement, dual enrollment, International Baccalaureate, and honors courses.

Jessica Lotz, the district’s director of performance management who will present the report, said this year’s numbers are better than last year. Since her last report on the topic, three schools now offer advanced courses for the first time.

Staffing is the biggest barrier to offering more advanced courses, she said. So, additional teacher trainings are planned for the summer.

And district plans are underway to increase the number of students taking those courses. The district is also pursuing federal funds to help students from low-income families pay for dual enrollment courses, and also encouraging area colleges to lower the number of students needed to take a class so that smaller schools can participate.

The number of students taking advanced courses is part of the state Department of Education measure of a being ready for college, or a “ready graduate,” under its new accountability plan.

Scroll down to the bottom of this story for a full chart on the number of advanced courses by high school.

Here are the 14 schools with 40 or more advanced courses each:

  • White Station High (143 advanced courses)
  • Central High (116)
  • Middle College High (98)
  • Germantown High (95)
  • Cordova High (79)
  • Overton High (75)
  • Ridgeway High (74)
  • Bolton High (56)
  • Southwind High (55)
  • Whitehaven High (52)
  • Hollis F. Price Middle College High (46)
  • Kingsbury High (45)
  • Memphis Virtual School (43)
  • East High (42)

Note: The number of courses offered refers to unique advanced courses that are available at a given school, not the total number of times/sections the same course is offered for different groups of students.

Four high schools did not offer any advanced courses: Legacy Leadership Academy, a charter school; The Excel Center, an adult learning school; Newcomer International Center, a new high school program for immigrant students; and Northwest Prep Academy, an alternative school.

Of the advanced courses, International Baccalaureate, a high-profile certification program for high school students worldwide, was the least common. Just three more affluent high schools — Ridgeway, Germantown, and Bolton — offered those courses, according to the district’s data.

Dual enrollment, another category of advanced courses, are taught in partnership with an area college and count toward a postsecondary degree. Though the share of Shelby County Schools students taking dual enrollment courses has increased from about 5 to 9 percent since 2014, the percentage slightly decreased this year compared to last school year.

Most of the high schools, offer a total of 183 dual enrollment courses. But only four of the 16 charter schools in the report offered those classes.

About half of high schools in the district offer a total of 194 Advanced Placement courses, which culminate in a test at the end of the year that can count toward college credit if students score well enough. Most of those classes are concentrated in seven more affluent schools.

Those schools are:

  • White Station High (39 AP courses)
  • Central High (20)
  • Cordova High (15)
  • Kingsbury High (13)
  • Overton High (13)
  • Whitehaven High (11)
  • Southwind High (10)

Honors courses, which count toward an advanced high school diploma but do not count for college credit, were the most common with just over 1,000 across the district. Only seven schools, which were either charter schools or alternative schools, did not offer any honors courses.

One of Shelby County Schools’ goals is to increase the percentage of students prepared for college by 2025. Currently, about 90 percent of students who graduate from the district would be required to take remedial classes in college because of low ACT scores, according to state data. That’s usually a sign that their high school did not adequately prepare them for college classes.

A state report released last fall examining where students go after high school showed that 56 percent of Shelby County Schools’ graduating class of 2016 went on to enroll in a four-year college or university, community college, or technical college. That’s compared to 63 percent of students statewide.

One of the report’s recommendations to boost that number was to improve partnerships with universities and increase the number of advanced course offerings — a recommendation Lotz emphasized Tuesday.

Shelby County Schools partners with the following universities and colleges for dual enrollment courses: Bethel University, Christian Brothers University, LeMoyne Owen College, Southwest Tennessee Community College, Tennessee College of Applied Technology, University of Memphis, and William Moore College of Technology (Moore Tech)

Below you can find the advanced course offerings at each district-run and charter school in Shelby County Schools. Below that you can view the district’s full report.