As a superintendent, no part of my job was more important than visiting schools and seeing students, teachers, and leaders in action. Those visits kept me grounded, reminding me of the real-world importance of every decision we made in the central office.

But that’s not to say that I always liked what I saw. Sometimes I would drop in on random classes and see inspiring examples of high-quality teaching and learning. But other times I saw teachers going through the motions and students waiting around for the bell to ring.

Whenever I visited one of those lifeless classrooms — and I saw far too many of them — I found myself wondering how the kids manage to show up every day instead of lashing out in rebellion against dull lessons and mediocre teaching. What explains their willingness to keep attending a class taught by a teacher who isn’t interested in them? Maybe it’s the price they have to pay to pass the course and move on toward a diploma, but if they chose to play hooky sometimes, who could blame them? Wouldn’t that be a rational thing to do?

I don’t mean to suggest that boredom is the main reason why students skip school. But it is worth asking how much of a role it plays now that many states are considering the use of absentee rates as a factor in determining school quality.

Under the federal Every Student Succeeds Act, state and local leaders are supposed to find new ways, beyond the use of test scores, to gauge their schools’ performance. If absenteeism becomes one of those factors, then how should we account for its many possible causes? To what extent is a school to blame for the problem, and to what extent is it caused by things outside of a school’s control?

If the school is unsafe, for example, or if teachers have bad relationships with students, or if classroom instruction is flat-out boring, then we might argue that a poor attendance rate is a fair indication of the school’s poor performance overall. But what if students often miss school because they can’t get there — say, because the bus system is unreliable — or because of chronic health problems, or because they have to work to support their family, or because they don’t have clean clothes?

The school can certainly help identify those factors through the use of early warning systems, and perhaps it can even help organize community resources to change those conditions. But, in those circumstances, is it fair to treat the absentee rate as a measure of school quality? (Robert Balfanz offers more detail on these issues here.)

Perhaps a better approach would be to consider each school’s rate of absenteeism in light of the resources it has available. For example, district leaders might choose to give some leeway to a school that has a high absenteeism rate but which has only a single counselor for its 750 students. But what if that school’s absenteeism rate is just as high five years later, after the district has given it funding to hire two more counselors? Does it still deserve some leeway, or has it squandered its resources?

Similarly, what should we make of a school where, in order to improve attendance rates, the principal creates a plan for teachers to conduct home visits, but the teachers association refuses to permit it, arguing that the extra assignment would violate the existing contract? How should that school be compared to one that has more funding, opts to pay teachers to conduct home visits, and sees its attendance improve as result?

Once you get into the weeds, it quickly becomes hard to say whether absenteeism can be treated as valid indicator of school quality at all. The hope, of course, is that if states put a spotlight on the issue and define it as an important ingredient in school effectiveness, then they can create positive incentives for schools that, in turn, help more students learn. (Elaine Allensworth and Shayne Evans have documented how Chicago Public Schools saw improvements in their high school graduation rate after taking steps to boost attendance in ninth grade.)

I’d like to believe that this will work, and I know that many states are betting that it will. But I also believe that it’s critical for policymakers to make decisions based on what they know, not what they hope. And what we know, from decades of efforts to improve the public schools, is that education leaders and policymakers are easily drawn into pursuing simplistic solutions to complex problems.

So it’s entirely predictable that teachers and administrators will look for easy ways to improve their attendance by coercing, cajoling, or threatening parents and kids to get to school. Moreover, it wouldn’t be surprising to see some schools respond by clearing certain kids from their rolls as a way to bring down their absenteeism numbers. (Will states control for that in their formulas?)

But the real question is this: What are schools doing to create good reasons for students to attend? If absenteeism is going to be used to judge school quality, let’s be sure that we’re pulling the right levers to improve it.

Joshua Starr is the CEO of PDK International, an association for educators
. He was previously the superintendent of schools in Montgomery County, Maryland and Stamford, Connecticut. He tweets @JoshuaPStarr.