failing grade

Why one Harvard professor calls American schools’ focus on testing a ‘charade’

PHOTO: Alan Petersime

Harvard professor Daniel Koretz is on a mission: to convince policymakers that standardized tests have been widely misused.

In his new book, “The Testing Charade,” Koretz argues that federal education policy over the last couple of decades — starting with No Child Left Behind, and continuing with the Obama administration’s push to evaluate teachers in part by test scores — has been a barely mitigated disaster.

The focus on testing in particular has hurt schools and students, Koretz argues. Meanwhile, Koretz says the tests are of little help for accurately identifying which schools are struggling because excessive test prep inflates students’ scores.

“Neither good intentions nor the value of well-used tests justifies continuing to ignore the absurdities and failures of the current system and the real harms it is causing,” Koretz writes in the book’s first chapter.

Daniel Koretz, Harvard Graduate School of Education

His skepticism will be welcome to families of students who have opted out of state tests across the country and others who have led a testing backlash in recent years. That sentiment helped shape the new federal education law, ESSA.

Koretz has another set of allies in some conservative charter and voucher advocates, including — to an extent — Secretary of Education Betsy DeVos, who criticized No Child Left Behind in a recent speech. “As states and districts scrambled to avoid the law’s sanctions and maintain their federal funding, some resorted to focusing specifically on math and reading at the expense of other subjects,” she said. “Others simply inflated scores or lowered standards.”

But national civil rights groups and some Democratic politicians have made a different case: That it’s the government’s responsibility to continue to use test scores to hold schools accountable for serving their students, especially students of color, poor students, and students with disabilities. (ESSA continues to require testing in grades three through eight and for states to identify their lowest performing schools, largely by using test scores.)

We talked to Koretz about his book and asked him to explain how he reached his conclusions and what to make of research that paints a more positive picture of tests and No Child Left Behind.

The interview has been edited for clarity and length.

Do you want to walk me through the central thesis of your book?

The reason I wrote the book is really the subtitle: we’re “pretending to make schools better.”

Most of the bad news that’s in this book is old news. We’ve been collecting evidence of various kinds about the impact of the very heavy handed, high-stakes testing that we use in this country for a long time. I lost patience with people pretending that these facts aren’t present. So I decided it would be worth writing a book that summarizes the evidence both good and bad about the effects of test-based accountability. When you do that, you end up with an awful lot on the bad side and not very much on the good side.

Can you talk about some of the bad effects?

There are a few that are particularly important. One is absolutely rampant bad test prep. It’s just everywhere. One of the consequences of that is that test scores are often very badly inflated.

There aren’t all that many studies of this because it’s not really a welcome suggestion. When you go to the superintendent and say, “Gee, I’d like to see whether your scores are inflated,” they rarely say, “Boy, we’ve been waiting for you to show up.” There aren’t that many studies, but they’re very consistent. The inflation that does show up is sometimes absolutely massive. Worse, there is growing evidence that that problem is more severe for disadvantaged kids, creating the illusion of improved equity.

Another is increasingly widespread cheating. We, of course, will never know just how widespread because there aren’t resources to examine the data from 13,000 school districts. Everyone knows about Atlanta, a few people know about El Paso, but that’s just the tip of the iceberg.

There’s obviously also — and perhaps this should be on the same par — enormous amounts of stress for teachers, for kids, and for parents. That’s the bad side.

I want to ask a little more about test score inflation. What is the strongest evidence for inflation? And let me give you two pieces that to me seem like potentially countervailing evidence. One piece is when I’m looking at research on school turnaround — like the most recent School Improvement Grant program and also turnaround efforts in New York City — these schools have been under intensive pressure to raise test scores. And yet their test score gains on high-stakes tests have been pretty modest at best. The other example is the Smarter Balanced exam. The scores on the Smarter Balanced exam don’t seem to be going up. If anything, they’re going down.

The main issue is that score inflation doesn’t occur in the same amount everywhere. You’ve come up with two examples where there is apparently very little. There are other examples that are much worse than the aggregate data suggest.

In the case of Smarter Balanced, I would wait and see. Score inflation can only occur when people become sufficiently aware of predictable patterns in the test. You can’t game a test when you don’t know what irrelevant things are going to recur, and that just may take some time.

I’m wondering your take on why some of the strongest advocates for test-based accountability have been national civil rights groups.

One of the rationales for some of the most draconian test-based accountability programs we’ve had has been to improve equity. If you got back to the enactment of NCLB, you had [then-Massachusetts Sen.] Teddy Kennedy and [then-California Rep.] George Miller actively lobbying their colleagues in support of a Republican bill. George Miller summed that up in one sentence in a meeting I went to. He said, “It will shed some light in the corners.” He said that schools had been getting away with giving lousy services to disadvantaged kids by showing good performance among advantaged kids, and this would make it in theory impossible to do that.

Even going back before NCLB, I think that’s why there was so much support in the disability community for including disabled kids in test-based accountability in the 1990s — so they couldn’t be hidden away in the basement anymore. I think that’s absolutely laudable. It’s the thing I praise the most strongly about NCLB.

It just didn’t work. That’s really clear from the evidence.

I think the intention was laudable and I think the intention was why high-stakes testing has gotten so much support in the minority community, but it just has failed.

You mention in your book probably the most widely cited study on the achievement effects of No Child Left Behind, showing that there were big gains in fourth grade math and some gains in eighth grade math, but there wasn’t anything good or bad in reading.

Pretty much. There was a little bit of improvement in some years in reading but nothing to write home about.

So the math gains — and that was on the low-stakes federal NAEP test — they’re just not worth it in your view?

I think the gains are real. But there are some reasons not be terribly excited about these. One is that they don’t persist. They decline a little bit by eighth grade, they disappear by the time kids are out of high school. We don’t have good data about kids as they graduate from high school, but what we do have doesn’t show any improvement.

The biggest reason I’m not as excited as some people are about those gains is we’ve had evidence going back to the 1980s that one of the responses that teachers have had to test-based accountability is to take time out of untested subjects and to put it into math and reading. We don’t know how much of that gain in math is because people are teaching math better and how much is because kids aren’t learning about civics.

That’s, in my view, not enough to justify all of the stuff on the other side of the ledger.

When I’ve looked at some studies on the impact of NCLB on students’ social-emotional skills, the impact on teachers’ attitudes in the classrooms, and the impact on voluntary teacher turnover, they haven’t found any negative effects. They also haven’t found positive effects in most cases. But that would seem to at least in one sense undermine the argument that NCLB had big harmful effects on these other outcomes.

I haven’t seen those studies, but I don’t think what you describe does undermine it. What I would like to see is an analysis of long-term trends not just on teacher attrition but on teacher selection. A lot of what I have heard has really been, frankly, anecdotal. I was once a public school teacher and teaching now is utterly unlike what it was when I taught. It seems unlikely that that had no effect on who opts in and who opts out to be a teacher.

I don’t have evidence of this but I suspect that to some extent different types of people are selecting into teaching now than were teaching 30 years ago.

Can you talk about what you see as good versus bad test prep?

Something that Audrey Qualls at the University of Iowa said was, “A student has only mastered something if she can do it when confronted with unfamiliar particulars.”

Think about training pilots — you would never train pilots by putting them in a simulator and then always running exactly the same set of conditions because next time you were in the plane and the conditions were different you’d die. What you want to know is that the pilot has enough understanding and a good enough command of the physical motions and whatnot that he or she can respond to whatever happens to you while you’re up there. That’s not all that distant an analogy from testing.

Bad test prep is test prep that is designed to raise scores on the particular test rather than give kids the underlying knowledge and skills that the test is supposed to capture. It’s absolutely endemic. In fact, districts and states peddle this stuff themselves.

I take it it’s very hard to quantify this test prep phenomenon, though?

It is extremely hard, and there’s a big hole in the research in this area.

Let’s turn from a backward-looking to a forward-looking discussion. What is your take on ESSA? Do you think it’s a step in the right direction?

This may be a little bit simplistic, but I think of ESSA as giving states back a portion of the flexibility they had before No Child Left Behind. It doesn’t give them as much flexibility as they had in 2000.  

It has the potential to substantially reduce pressure, but it doesn’t seem to be changing the basic logic of the system, which is that the thing that will drive school improvement is pushing people to improve test scores. So I’m not optimistic.

One of things that I argue very strongly at the end of the book is that we need to look at a far broader range of, not just outcomes, but aspects of schooling to create an accountability system that will generate more of what we want. ESSA takes one tiny step in that direction: it says you have to have one measure beyond testing and graduation rates. But if you read the statute it almost doesn’t matter what that measure is. The one mandate is that it can’t count as much as test scores — that’s written in the statute. The notion that it means the same thing to monitor the quality of practice or to monitor attendance rates is just absurd

As I’m sure you know, research — including from some of your colleagues at Harvard — has shown that so-called “no-excuses” charter schools in places like Boston, Chicago, and New York City, have led to substantial test score gains and in some cases improvements in four-year college enrollment. Are you skeptical that those gains are the result of genuine learning?

It depends on which test you’re talking about. Some of the no-excuses charter schools drill kids on the state test, so I don’t trust the state test scores for some of those schools. I think it’s entirely plausible that some of those schools are going to affect long-term outcomes because they’re in some cases replacing a very disorderly environment with a very orderly one. In fact, I would say too orderly by quite a margin.

But those reforms are much bigger than just test-based accountability or just the control structure we call charters. It’s a whole host of different things that are going on: different disciplinary policies, different kinds of teacher selection, different kinds of behavioral requirements, all sorts of things.

A lot of the discussion around accountability, including in your book, is about the measures we should be using to identify schools. I’m interested in your take on what happens when a school is identified by whatever system — perhaps by the holistic system you described in the book — as low performing.

The first step is to figure out why is it bad. I would use scores as an opening to a better evaluation of schools. If scores on a good test are low, something is wrong, but we don’t know what. Before we intervene we ought to find out what’s wrong.

This is the Dutch model: school inspections are concentrated on schools that shows signs of having problems, because that’s where the payoff is. I would want to know what’s wrong and then you can design an alternative. In some cases, it may be the teaching staff is too weak. It may be in some cases the teaching staff needs supports they don’t have. It may be like in the case of Baltimore, they need to turn the heat on. Who knows? But I don’t think we can design sensible interventions until we know what the problems are.

Fixing Special Education

How will Chicago repair the harm from special-education neglect?

PHOTO: Adeshina Emmanuel / Chalkbeat
Laurel Henson, at the podium, spoke at a press conference in Chicago on Nov. 12, 2018, about her 2-year struggle to get a school nurse on staff to help her son, who suffers from seizures.

Illinois may be forcing Chicago Public Schools to repair its broken special education program, but the ambitious effort still begs a critical question: What happens to hundreds of Chicago children who were harmed by the district refusing them services that would  help them learn?

Neither the state nor the school district is saying yet, even as advocates for students in special education have pressed for answers.

Those children include an unnamed third-grader trapped by a tactic the district apparently used to avoid offering services required by federal law.

The child can’t read the word cat or dog, health-legal advocate Barbara Cohen said, but his teacher didn’t believe in giving low grades. So the third grader received a B in English. Then, she told the State Board of Education on Friday, when the child’s mother sought an evaluation for special education services, school officials denied the request based on his having a good grade.

Laura Boedeker, the state’s monitor overseeing special-education reforms, acknowledged that schools vary in understanding the laws and best practices. Her job, she said, “is to have those discussions and explain what good practices look like.”

That’s not likely to satisfy parents and advocates pushing for quicker action that would help families like the third-grader’s. On Friday, they pressed authorities like Boedeker, who previously served as the district’s in-house attorney.

But with a staff of just three, including herself, it’s not clear how fast Boedeker can move. In four-and-a-half months on the job, she’s only visited 10 of the district’s 600-plus schools.

“Do you have enough boots on the ground, enough help to do this work at the rate you need to do it?” asked Illinois State Board of Education member Susie Morrison.

“We could have an army and not have enough boots on the ground,” said Stephanie Jones, the board’s general counsel. “What we need more than anything is eyes and ears that tell us what is going on so we can take action. Unless we can put an ISBE employee in every school, which is unrealistic, we need parents and teachers and staff members to tell us what is going on.”

Recognizing the lag in responding to parents, the state board is weighing whether to extend the one-year deadline for filing complaints about denied or improper services.

It’s possible, Jones said, that “we can wave this until we have a system of corrective action in place.”

Neither the state nor district have answered questions like: How many students could be eligible? When exactly will the system go into effect? And what roles should advocates and schools play?

Boedeker said that federal officials have insisted that teams who put together students’ individualized education programs be involved in the remedy, because “they’re the ones on the front lines with these students.” 

But lawyer Matt Cohen said he and other advocates want a process that involves more people than the IEP team.

A child who, for example, went without a one-on-one aide for many months or who didn’t get placed in therapeutic day school when needed “might have had a profound loss,” Cohen said.

How the district will compensate that family is the question.  

“They may need more than just a few hours of tutoring to make up for that, they may need months and months of additional services and a specialized process to help them catch up,” Cohen said. “We’re encouraging families whose kids were hurt to bring their complaints to the state, and to seek action to get their individual child’s needs met.”

Jones said that board officials and the school district, federal government and special education advocates are discussing school guidelines for identifying students harmed, notifying their families, assessing damages and offering remedies.

About half a year has passed since a state probe found the school district violated students’ rights by routinely delaying and denying services — like aides, therapy, outside placement and busing — to students in what the district calls its Diverse Learners Program.

The state board’s Jones and Boedeker tried to placate critics by preaching patience.

“From the outside looking in it looks really slow,” Jones said, “but I think we’ve accomplished a great deal in the time we have had.”

Patience doesn’t sit well with parents desperately worried about their children.

Laurel Henson, whose son suffers from seizures, said she’s been pushing to get a nurse on staff at Smyser Elementary for two years, but has encountered “delays and excuses.” On Nov. 1, the school finally granted a meeting to discuss an IEP, she said.

“In that time, he’s had a significant increase in seizures at his school causing fatigue, aggression and bed wetting during the night,” she said. Despite her hopes for the monitor, “nothing has improved for my son and it now feels like neither CPS nor the state are accountable for ensuring students like my son have a free and appropriate education.”

ethnic studies

50 years in, why the fight for Mexican-American studies in schools is still in its early stages

PHOTO: Annie Wells/Los Angeles Times via Getty Images
Sonia Salazar, a college student, joins over 1,000 people to commemorate the historic East LA student walkouts of 1968 earlier this year. Mexican-American Studies courses are gaining traction now in K-12 schools after years of growth in higher education, a panel concluded during a recent civil rights conference in San Antonio.

Thirteen-year-old Alejandra Del Bosque knows not everyone gets to take a class like hers.

In it, she’s learned about Mexican-American students who staged walkouts in the late 1960s and early 1970s to protest the lack of resources available to their schools. She’s also learned how her state’s school funding system has still been deemed inadequate in recent court rulings.

“There was so much to learn about my heritage that I didn’t know,” Del Bosque said. “But from what I understand, it’s a unique class that’s not everywhere. For me, as a Mexican-American, it’s exciting.”

Her experience remains relatively rare. Fifty years after televised civil rights hearings galvanized the Chicano movement, academics and activists agree that the push for Mexican-American studies still lacks basic resources. And though interest is increasing, in part thanks to President Trump, growth has been slow — especially in K-12 schools, since college-level programs have traditionally gotten more attention.

“That was a big mistake we made,” Juan Tejeda, a professor at Palo Alto College, said last week. “There should have always been a focus on developing culturally relevant curriculum from pre-K through 12.”

He spoke at an event commemorating the 1968 U.S. Commission on Civil Rights Hearings on Mexican-Americans in the Southwest, where he and others took stock of the movement that emerged in the decades since to better engage Latino students. (Of the 58 million Latinos in the U.S., nearly two-thirds are of Mexican descent, and most were born in the U.S.)

That’s long been a challenge for schools, especially as most educators are white. Some research has suggested that when students see themselves reflected in their curriculum, test scores and graduation rates rise. Another study found that taking an ethnic studies course helped reduce dropout rates.

Not many students have access to those courses, though. There’s no solid national data on how many school districts have some form of Mexican-American studies in their schools. California is understood to have taken the lead, while Tejeda estimated that only about 38 of more than 1,000 Texas districts have started a program.

That’s partly due to ongoing political opposition.

Arizona’s ban on teaching Mexican-American studies back in 2010 was a wake-up call for the movement, Tejeda said. (Last year, a federal court ruled that the state’s move was “racist and unconstitutional,” but Tucson hasn’t reinstated its program yet.)

Over the last decade, Mexican-American professors built a network that evolved into a group called Somos MAS. The group began a push for a standard high school elective course in Texas.

After four years of lobbying, the Texas board of education approved the course last year. Battles have also turned toward materials: When the book to be used in schools for Mexican-American studies was released in 2016, it was described by many Chicano scholars as racist for its portrayal of Mexican-Americans as lazy and un-American. That book was later thrown out, as was another the board didn’t like in 2017. Then came a debate over the course’s name, which just ended in September.

Those fights were about more than details – they were about granting the topic legitimacy, and about making it easy for teachers to introduce the material, said Lilliana Patricia Saldaña, a professor at the University of Texas at San Antonio.

“There were already some teachers here or there taking upon themselves to incorporate the studies into the schools, but it was sporadic, and accurate materials weren’t always easy to find,” Saldaña said. “Approving a course that can be aligned with state standards is ideal and would allow for the programs to be more streamlined.”

Another key challenge: in many cases, limited student interest. At the college level, Our Lady of The Lake University — the host of the hearings in 1968 and the conference last week — considered nixing its Mexican-American studies program in 2012 because of the small number of participating students. It was later saved.

“That also reminded us that if we don’t fight to keep these programs, they will be lost,” Tejeda said. “But what we needed to do was focus on getting students interested while they are younger.”

Saldaña says student interest has grown more recently thanks to political rhetoric around immigration, specifically from President Donald Trump. Trump has disparaged Mexican immigrants, questioned the impartiality of a Mexican-American judge, and made wanting to build a wall on the U.S.-Mexico border the center of many political speeches.

“Between what we are seeing with the current administration in office, and the battle here on the ground over the course we have been fighting for, students are getting a real-time lesson,” Saldaña said.

Somos MAS now hosts an annual summit for K-12 educators to come learn about Mexican-American Studies and how to integrate lessons into their classrooms. The University of Texas at San Antonio also offers a summer training institute that has drawn nearly 100 teachers at its most recent gathering.

It’s not nearly enough, the panelists said. “What needs to happen next is a focus on building infrastructure: such as more teacher training opportunities on how to incorporate MAS in their classrooms; a teacher certificate in Mexican-American Studies, and more advanced degrees in ethnic studies so students see a future in this field of work,” Saldaña said.

Students from KIPP Camino Academy. (Photo by Francisco Vara-Orta)

One school that has moved ahead with Mexican-American studies course is KIPP Camino Academy in San Antonio. After a pilot program two years ago, the class is now an elective for seventh- and eighth-graders.

On Friday, 20 of the KIPP students watched the discussion on the 50-year fight to get Mexican-American studies in their schools with their instructor, JoAnn Trujillo.

“Some of these kids have driven by the university here and never have gotten to step foot on its grounds,” Trujillo said. “So us being here — in part because of the program, and seeing how Mexican-American studies is something special that had to be fought for many years — will plant seeds about going to college and feeling more self-worth.”