evaluating evaluation

More work, worse relationships, and better feedback: How teacher evaluation has changed the job of the principal

PHOTO: Laura Faith Kebede/Chalkbeat
Principal Melody Smith discusses how students at A.B. Hill Elementary grew significantly in test scores.

Teacher evaluation overhauls were supposed to reshape the teaching profession. New research suggests they may have had an even greater impact on what it means to be a school principal.

As policy makers overhauled teacher rating systems in the last decade, principals began spending much more of their time watching teachers in action and talking to them about how to improve. But the shift also overwhelmed them with work, stopped them from fulfilling other responsibilities in their schools, and weakened their relationships with teachers.

Those are some of the takeaways of a study based on interviews with dozens of principals in six districts that revamped how teachers were evaluated in the last decade. Typically, the new evaluation systems — often put in place at the urging of the federal government and influential philanthropies, particularly the Gates Foundation — incorporated student performance for the first time but were driven mostly by teachers’ scores on rubrics that attempt to spell out good teaching practices.

While the systems’ adoption often precipitated significant pushback and in some cases have been rolled back, they remain in force in many places — and their long-term consequences are only now becoming clear.

The study, published last week in The Elementary School Journal, focuses on six large districts that embarked on dramatic evaluation changes: Baltimore, Denver, Hillsborough County (Florida), Houston, Memphis, and Metro Nashville. The researchers interviewed 10 principals in each district for about an hour during the 2012-13 school years, selecting a group at random while also ensuring a combination in different grade and performance levels.

One finding stood out. “The role of administrator is drastically changing,” one principal said. “You cannot be just a manager of a building. … You have to be an instructional leader first.”

The study examines only a handful of the many districts that reworked evaluations in the last decade, so it’s unclear how widely the findings apply. But the researchers detected themes across the six locations, which all had somewhat different evaluation systems and strategies for rolling them out. Here’s what they heard.

The new evaluations take way more time than principals feel they have.

The most consistent concern shared with researchers centered on the time it takes to complete the new evaluations. The responsibilities didn’t just include observing teachers more frequently, but also conducting pre- and post-observation conferences, completing additional paperwork and data entry, and using final ratings to make human resources decisions.

“I mean, honest to God. I just can’t do it by myself,” one principal told the researchers. “If I’m running my school, something’s going to lose out, either academics or your school because you’re just one person.”

“We need to [have] a preconference, an observation, and a post conference, you’re talking about, even though a preconference and postconference may only take 20 to 30 minutes and then an observation, you’re looking at two hours,” another principal said.

“It took us like an hour and a half to input the evaluations per teacher, and I have 82 people. So it was like a nightmare, literally,” said another.

In New York City, where the recently proposed teachers union contract reduces the number of required observation from four to two for most teachers, principals say the blanket requirements of the new system impeded their ability to make good choices about how they spend their time.

“We’re better off spending more time with the teachers who need our help than with the teachers who are doing a bang-up job,” the president of the union that represents school administrators told Chalkbeat. “To say time is at a premium for our members is an understatement.”

Principals are spending more time with teachers — and many say that’s a good thing.

Although the use of test scores has driven much of recent public debates about evaluations, the principals interviewed in the study often focused on the role of observations. That’s in line with past research suggesting principals trust that measure more than test scores; it’s also not surprising since principals conducted observations themselves, and those scores accounted for the lion’s share of a teacher’s overall rating.

While some frameworks for evaluating instruction have gotten mixed reviews, principals in the study said they found them helpful. Having an “objective” rubric, the principals told the researchers, allowed them to both give specific feedback to teachers who were improving and to build evidence to dismiss teachers who persistently struggled.

“For me, as a leader, it has given me a tool that I can use to help teachers grow, more than I’ve ever been able to do in the past, ever, because it’s a lot more specific than it ever has been in the past,” one principal said.

“The … rubric is the single most solid evaluative tool that I’ve used in my career,” said another.

“The conversations and the reflection … have gone from, ‘Oh, I thought it went great,’ to a really robust conversation.”  said a third principal.

Whether observations are really as “objective” as principals seemed to believe is not clear. Other research has shown that classroom observations are consistently biased against teachers who serve more students of color and students who start the year with lower test scores.

Principals’ relationships with teachers worsened.

Though there has been modest uptick in teachers who received low ratings, the vast majority of teachers across the country were still rated effective under the new systems.

Still, principals said the new evaluations damaged their relationships with teachers, who weren’t always sold that the new approaches were fair. They cited this dynamic as a major downside of new evaluation systems.

Some principals framed the tension as a lack of understanding on teachers’ part. “It’s very difficult for some teachers to understand that you really can quantify what they’re doing in the classroom, and many of them don’t want you quantifying it,” one said.

Others said the culture in the school had changed as teachers became more fearful of high-stakes evaluation, and thus less open with their principal.

“I feel like every time I walk in, a teacher’s like automatically on pins and needles thinking I’m there in an evaluative capacity,” a principal told researchers. “I just want to go in and see what the kids are doing.”

Principals ended up being less visible in their schools.

One surprising finding is that even as the new evaluation systems required principals to spend lots of time in classrooms, they said they ended up being less visible to educators and students.

“I’m in more classrooms, but I’m not walking through lots of classes. I’m mainly only in four classes or three classes a day,” one principal explained.

“I don’t do lunch duty as much,” another said. “I loved lunch duty because it’s a time to interact with the kids where you can just communicate with them.”

Trisha Arnold, a New York City teacher who helped negotiate the contract terms that reduce required observations there, told Chalkbeat earlier this month that she had experienced this phenomenon in her school.

“Kids aren’t sure who the principal is,” she said, “because they’re bogged down with paperwork.”

What to do with the research is unclear.

How do the benefits and costs of new systems weigh out? That remains a frequently debated question among policymakers and researchers. A recent analysis focusing on the Gates Foundation’s evaluation reforms in a number of districts yielded disappointing results, with little if any gains in achievement. (Two of those districts were the same ones featured in the latest study, which was also funded by the Gates Foundation; additionally, Gates is a supporter of Chalkbeat.) Another study found that evaluations systems likely deterred prospective teachers from entering the profession.

Other research is more upbeat, linking evaluation to test score gains and teacher improvement.  Studies of evaluation efforts in Cincinnati, Chicago, Washington D.C., and Minnesota have all found benefits, as well as national research on performance pay and feedback connected to classroom observations.

The latest research suggests that finding ways to lighten principals’ workloads might be worthwhile. That’s already happened to a degree, with some principals saying they placed more responsibility on assistant principals or other administrators. A separate Washington State study found a hiring spree of assistant principals soon after evaluation systems were put into place. That might be a smart move to help principals — but it also costs money, which could have been spent elsewhere, say, increasing teacher pay or reducing class size.

The new research also underscores how significant the choice of observation rubric is for districts and schools that adopt new evaluation systems. “Policy makers may also want to consider the enormous weight that is being placed on instructional rubrics as a means of guiding principals’ understanding of teacher performance,” the authors say.

Perhaps most importantly, the authors say, there should be more attention on principals’ role in enacting the raft of procedures connected to new evaluations.

“Without a careful examination of how to address principals’ concerns about lack of time,” they conclude, “the risk of principal fatigue, frustration, and eventual turnover remains substantial.”

Follow the ratings

Illinois education officials laud their school ratings — but critics say they don’t go far enough

Illinois rolled out its new school accountability system in the Illinois Report Card late last month.

State education officials publicly lauded their new school rating system Friday, even as a new, nationwide analysis of school improvement plans criticized Illinois’ approach as too hands-off.  

While the state developed a clear rating system as the federal government requires, Illinois falls short in follow-through, according to the report from the Collaborative for Student Success, a non-profit advocacy group, and HCM Strategies, a policy analysis group.  

“The state is taking too limited a role in leading or supporting school improvement efforts,” said the report, which examined how 17 states are implementing school improvement plans under the federal Every Student Succeeds Act, which was passed in 2015 and replaced the No Child Left Behind Act.

Both those federal laws task states with identifying and helping improve underperforming schools and with creating criteria to judge which schools are doing well. Illinois rolled out its new school accountability system in the Illinois Report Card late last month.

State officials disagree with the criticism.

“Illinois is being held up as a model for other states to follow,” said Ralph Grimm, chief education officer of Illinois, speaking at the monthly state board of education meeting on Friday. “The entire (state) team has to be commended for providing useful information.”

Illinois’ rating system places every public school in the state into one of four categories based in part on scores on the annual PARCC standardized tests (click here to see how Chicago schools ranked).

Only about a third of Illinois students scored proficient or higher on PARCC tests administered last spring. In reading, 37 percent of students in grades 3 through 8 met that benchmark, while in math 31 percent did. Despite that, the state awarded 80 percent of its schools a “commendable” or “exemplary” rating. 

The state labeled 20 percent of schools “underperforming” or “low performing,” the only designations that could trigger state action. Intervention measures include improvement plans, visits from specialists, and additional funding.

The state released its ratings just days after Chicago released its own batch of school ratings, which take into account a different set of metrics and a different standardized test.

Grimm said the next step will be asking the state’s lowest-performing schools to draft improvement plans and then connecting them with experts to implement their changes.

The state ratings pay particular attention to how schools educate certain groups of students — such as children of color and English language learners. Improvement plans will focus on ways to raise their achievement levels.

Under the latest state rankings, nearly half of Chicago schools failed to meet the state’s threshold for performance, with a disproportionate number of high schools on the low-performance list. Nearly all of under- and low-performing Chicago high schools are on the South Side and sit in or border on the city’s poorest census tracts.

The state could grant underperforming schools $15,000, and  the lowest performers can apply for $100,000 under its IL-Empower program — which helped schools improve by funneling federal funds to them. Advocates have welcomed the change to a carrot to help schools pull themselves up, after years of sticks that overhauled and cut funding for low-performing schools.

Nationally, the Collaborative for Student Success report applauded Colorado for its streamlined application system, and Nevada for asking districts to directly address equity.

The collaborative criticized Illinois for failing to involve parents and community members in its plan. The group also said the state needs to give districts more guidance on putting together school improvement plans. 

carry on

These 16 Denver charter schools won renewal from the school board

PHOTO: AAron Ontiveroz/The Denver Post
Sebastian Cruz waves to Rev. Leon Kelly as he works with children in a classroom during his after-school program at Wyatt Academy in September 2018.

The Denver school board on Thursday night unanimously renewed agreements with 16 of the district’s charter schools. The lengths of those renewals, however, varied from one year to five years — and signaled the board’s confidence in the schools to deliver a quality education.

The board also accepted Roots Elementary’s decision to close and surrender its charter at the end of this school year. The Park Hill school is facing low enrollment and high costs.

Denver Public Schools is a charter-friendly school district that has for years shared tax revenue and school buildings with its 60 publicly funded, independently operated charter schools. The schools are controversial, though, with opponents viewing them as privatizing public education.

Every charter school in Denver has an agreement with the district that spells out how long it’s allowed to operate. To continue running after that time period, the charter school must seek renewal. The arrangement is part of the deal for charters: They get the flexibility to operate independently, but they must periodically prove to the district that they’re doing a good job.

The school board relies on one set of recommendations from Denver Public Schools staff and a second set of recommendations from a districtwide parent committee in deciding how long a leash to give each charter school. The district staff and the parents on the committee consider factors such as test scores, school culture, financial viability, and the strength of a school’s leaders when making their recommendations.

They also consider a school’s rating on Denver Public Schools’ color-coded scale based largely on academic factors. The School Performance Framework, or SPF, labels schools either blue, green, yellow, orange, or red. Blue means a school has a distinguished academic record, while red means a school is not meeting the district’s expectations.

The staff recommended the school board renew the charters of all 16 schools that applied. Two other charter schools — DSST: Cole Middle School and Compass Academy — are also up for renewal this year. But because they earned the district’s lowest rating, they must go through a separate process in which they will present a detailed improvement plan. Their renewals will depend on the strength of their plans, which is why they weren’t included in this batch.

The board approved the 16 renewals Thursday without discussion. All of the new terms begin next school year. Here’s the rundown:

STRIVE Prep Federal, a middle school in southwest Denver
Year opened: 2006
School rating: Green
Renewal: Five years

DSST: Green Valley Ranch High School, a high school in far northeast Denver
Year opened: 2011
School rating: Green
Renewal: Five years

Rocky Mountain Prep Creekside, an elementary school in southeast Denver
Year opened: 2012
School rating: Green
Renewal: Five years

DSST: College View High School, a high school in southwest Denver
Year opened: 2015
School rating: Green
Renewal: Three years, with a possible two-year extension

KIPP Northeast Denver Leadership Academy, a high school in far northeast Denver
Year opened: 2015
School rating: Blue
Renewal: Three years, with a possible two-year extension

KIPP Northeast Elementary School, an elementary school in far northeast Denver
Year opened: 2015
School rating: Yellow
Renewal: Two years, with a possible three-year extension

Rocky Mountain Prep Southwest, an elementary school in southwest Denver
Year opened: 2015
School rating: Yellow
Renewal: Two years, with a possible three-year extension

Wyatt Academy, an elementary school in northeast Denver
Year opened: 2003
School rating: Yellow
Renewal: Two years, with a possible two-year extension

KIPP Northeast Denver Middle School, a middle school in far northeast Denver
Year opened: 2011
School rating: Yellow
Renewal: Two years, with a possible two-year extension

Downtown Denver Expeditionary School, an elementary school in central Denver
Year opened: 2013
School rating: Yellow
Renewal: Two years, with a possible two-year extension

Denver Justice High School, an alternative high school for at-risk students in central Denver
Year opened: 2009
School rating: Yellow
Renewal: Two years, with a possible one-year extension

REACH Charter School, an elementary school in central Denver
Year opened: 2015
School rating: Yellow
Renewal: Two years, with a possible one-year extension

Monarch Montessori, an elementary school in far northeast Denver
Year opened: 2012
School rating: Orange
Renewal: One year, with a possible two-year extension

STRIVE Prep SMART, a high school in southwest Denver
Year opened: 2012
School rating: Orange
Renewal: One year, with a possible two-year extension

Academy of Urban Learning, an alternative high school for at-risk students in northwest Denver
Year opened: 2005
School rating: Red
Renewal: One year, with a possible one-year extension

Rise Up Community School, an alternative high school for at-risk students in northeast Denver
Year opened: 2015
School rating: Red
Renewal: One year