The panel discussion I moderated on Tuesday evening focused on two video clips of classroom teaching that together lasted just over four minutes. Those four minutes fueled an hourlong discussion about how teachers are getting observed, especially now that New York City is implementing a teacher evaluation system that requires principals to spend far more time inside classrooms than ever before.

Our first two guests, a teacher and assistant principal from a middle school in the South Bronx, discussed quirks of the Danielson Framework, the way a subtle classroom command can derail a lesson and when technology trumps person. A policy analyst with New America NYC, Chalkbeat’s co-host for the event, then joined to offer a perspective on efforts to improve teacher quality around the country. An active online chat for the duration of the event added extra insight.

A summary of the event is below. If you missed it, we taped saved a live broadcast, which you can see (if not always hear) in its entirety here.

1. Danielson sometimes sets too high a bar for teachers 

Even a perfect teacher would have a hard time matching up to the hypothetical classroom educators described in Danielson’s rubric. In Danielson’s ideal, students of “highly effective” teachers finish work early and ask for more, offer one another feedback on writing samples, and seamlessly transition from one activity to another.

“Is ‘Mr. Holland’s Opus’ possible every time you walk into the class?” asked I.S. 3o3 Assistant Principal Monica Brady, one of the panelists, referring to the movie about the inspirational music teacher. “Is it going to happen every moment of that amazing class?”

Brady said teachers having their best days on the job would have trouble aspiring to some Danielson exemplars.

“There are things about it that are nutso,” she said. “Kids can’t have choice every time they walk into a room.”

2. “Fist-to-Five” and other strategies for student-led grouping don’t always work

As a result of Danielson’s unreasonably lofty standards, educators say it can sometimes be smarter not to reach for them at all. That’s tough for ambitious teachers like Danielle Lerro, the I.S. 303 English teacher on the panel. She studied the rubric and knew what it took to be rated “highly effective” for each component.

In the first video shown, Lerro asked students to show their level of understanding of text they had just read, using fingers on one hand, with a fist being a low level and five being high. She used “fist-to-five” and left it up to them to partner with each other based on shared comfort with the material, a practice that is a “highly effective” way to assess instruction in real time, according to Danielson.

But leaving the decision up to students can be “deceptive,” Brady said, especially in a classroom with many high-needs students who might throw up more fingers than they should.

“Fist-to-five is great, but you need to be able to select sometimes who you know is getting it and who’s not,” Brady said she told Lerro.

In the next video, taken several months later, Lerro listed students before the class who she knew would need extra help and worked with them on a similar activity.

“In an ideal world, [students grouping themselves] would happen all the time,” Lerro said. “But in the world we live in it can’t happen all the time.”

3. It’s all about the school

Classroom observations are yet another instance where policy can make sense in theory but fail in implementation. That was a takeaway from some who joined the event’s conversation online.

A teacher on Twitter said that what is constructive in some schools can be destructive and punitive in others.

Maisie McAdoo, an official with the city teachers union, agreed. “A lot depends on administrators getting this right and too many use the rubrics against teachers,” she responded.

Brady acknowledged the concern during the panel. She said observations are not a “gotcha” technique at I.S. 303, noting that she’ll sometimes cancel a planned observation if she realizes early on that students are especially unruly or the teacher is having an abnormally rough day. If she wanted to use Danielson against a teacher, she said she probably could.

“In the same way that it’s setting a standard, it’s also a standard that’s really easy to fudge,” she said. “It’s really easy to use to get someone.”

4. Observation by video can sometimes trump the real thing 

The limitations of using video as a proxy for an actual observer are easy to imagine. One is that video can miss a student’s reaction when a lesson starts to click for him. Another, New America’s Laura Bornfreund said, is that administrators who watch the video later on in their office might not be fully focused, or “guilty of multitasking.”

But Lerro said she recently picked up on an unanticipated advantage. Most administrators are respected authority figures in their schools. So when they pop into classrooms for observations, it often alters the behavior of the class, an externality that could mask a teacher’s weakness in classroom management.

Lerro said Brady was a good example. Students are always on their best behavior when they know Brady is near, Lerro said, so Brady gets the most realistic view into the classroom when she’s watching tape.

5. Educators want unlikely changes to their evaluation plans

When asked to name the one thing they want to see changed in the city’s teacher evaluation system, Brady and Lerro offered some revisions that might never come.

Brady said she worried about the pace at which the system was being imposed on schools. “For me it’s the speed,” Brady said. She said she was worried that the evaluations could breed distrust among teachers and administrators.

Lerro said that she had little confidence in calculations of teachers’ impact on students’ growth on state tests, a method known as “value-added.” Value-added scores calculated by the city starting in 2007 rarely matched up to a teacher’s actual quality, she said.

“They were all over the map based on what I know about good teaching,” Lerro said. “There was no correlation from what I saw.”

Her anecdotal experience matches what studies of the scores have verified. The city abandoned the scores two years ago, but state law requires that 20 percent of a teacher’s 2013-2014 rating be based on a value-added score determined by the state. Gov. Andrew Cuomo’s strident defense of the state’s evaluation law this year suggests that the formula is unlikely to be changed anytime soon.