This article was originally published in The Notebook. In August 2020, The Notebook became Chalkbeat Philadelphia.
As a psychiatrist and a public official, Kamilah Jackson has one foot in the field of medicine and the other in public health.
But there is little question that she finds school behavioral health a much tougher field to make judgments about than medicine.
Jackson, associate medical director for children’s services at Community Behavioral Health, puts it this way: “We don’t have a test that says, ‘You have taken this drug, and we know you are better.’”
Jackson’s observation is reflected by other behavioral health practitioners. For them, the issue isn’t just finding the right answers; it’s finding the right questions.
“The field has historically not lent itself well to evaluation — we need a unified approach,” said Joe Pyle, president of the Thomas Scattergood Behavioral Health Foundation, which makes grants to help providers evaluate themselves. “The science is evolving. I want to start talking about data.”
Schools themselves use standard benchmarks, like graduation rates, dropout rates, and standardized test scores. But mental health programs are less easily assessed.
“There is no universal or consistent way of measuring outcomes, and it’s been a real obstacle. It’s been a challenge,” said Sharon Stephan, co-director of the Center for School Mental Health at the University of Maryland School of Medicine.
Criteria used by school-based mental health programs around the country include:
- Standard academic measures such as test score improvements, attendance.
Suspensions, bullying and disciplinary actions.
Family engagement in a student’s progress.
Fewer referrals to special education.
Reductions in destructive behaviors such as substance abuse.
Reduction in symptoms of depression or other mental illness.
In September, the center and the School-Based Health Alliance received a $1.2 million, four-year grant from the federal Department of Health and Human Services to develop the first national performance standards for school mental health.
At the program level, Stephan says, “efforts are often piecemeal …[and] there’s not a lot of funding for measuring long-term impact.”
She praised Positive Behavioral Interventions and Supports (PBIS) as the individual program with the best evaluation system.
Locally, PBIS is offered in 20 District schools and one charter school under a grant from the Philadelphia Foundation to the Devereux Center for Effective Schools.
Barry L. McCurdy, director of the Devereux center, says the program has reduced the numbers of suspensions and times students are sent to the school office for disciplinary purposes, has improved attendance and has shown some improvement in reading and math scores.
The center is also joining with Children’s Hospital of Philadelphia on a project that seeks to blend mental health services with efforts to improve school climate.
Ideally, McCurdy says, he would like to see more school-wide surveys of student behavioral health needs, but a lot of schools, particularly in urban areas, aren’t prepared to do this.
“It would be negligent to identify students as needing services and not give them the services,” he said. “You could legally get into some difficulty. If you’re saying you’re concerned enough to screen, you’d better provide the services.”
Another difficulty in evaluation cited by behavioral health experts is that some of the key questions to be answered are subjective.
“Evidence-based standards are very sexy,” said Jackson, of Community Behavioral Health. “Everyone’s grabbing hold of them.”
But there are things that they can’t measure.
“How are you functioning in your family?” asked Jackson, citing one example. “How are you functioning within the school context? Are you developing a peer network, being a good family member?”
The city’s Community Behavioral Health department asks its school providers to draw out answers to these questions by interviewing teachers, parents, clinicians, and the students themselves.
Although there are widely accepted tools for measuring such specific mental health issues as depression, these go only so far.
“It takes lots of input from people supporting children,” Jackson says. “Kids come in with such different needs. We need to give providers flexibility.”
Jody Greenblatt, the Stoneleigh Fellow who is charged with improving school climate in the District, feels that the ultimate test of school-based behavioral health programs is how well they enable students to function socially and academically in a normal school setting.
But even here, she said, it’s easier to measure success when dealing, for example, with a child’s disruptive classroom behavior than when helping a child who is withdrawn.
“If they’re not drawing attention to themselves, they [sometimes] don’t get the services they need because they’re quiet,” she said. “It’s something you need to pay attention to.”
Clea McNeely, an associate professor of public health at the University of Tennessee in Knoxville, points out that it can be difficult to structure any behavioral health evaluation because it’s hard to find a comparison “control” group that isn’t receiving it.
McNeely, who has evaluated behavioral health programs for the Robert Wood Johnson Foundation, the William T. Grant Foundation, and others, said that one successful trial involved a charter middle school (grades 6-8) in Baltimore that admitted students by lottery.
McNeely and a research partner used wait-listed students as a control group, measured both groups in areas such as academic performance, civic participation, and lower levels of sexual activity and found that the admitted students did better.
However, “you need external funding [to do that]” she said. “And evaluation budgets are too small.”
Jeanne Lehrer, vice president for youth behavioral health at NorthEast Treatment Centers, agreed.
“It’s challenging just having the resources to gather outcomes,” she said. “Gathering the data, inputting it, analyzing it.
“And I haven’t found a whole lot of standardized tools that are super helpful. We do collect the data now, and we don’t organize it.
“It would be ideal to track a kid two years later, three years later.”
This story is part of a continuing series by the Notebook on student behavioral health, with support from the Van Ameringen Foundation. Contact reporter Paul Jablow with your ideas and feedback. He would like to hear about your experiences with the system: where it has succeeded, where it needs improvement, and what you would like to read more about on this topic.