high stakes

There’s always been confusion surrounding Tennessee’s growth model. With a missing year of data, new questions pile on

PHOTO: Laura Faith Kebede

At a time when scores are about to be used for high-stakes decisions in how to improve Tennessee’s schools, gaps in the state’s data and uncertainty about how scores were derived have left Memphis officials wondering how to interpret the torrent of information.

Last year’s chaotic state testing, which led to the cancellation of the state’s test for grades 3 to 8, left a crucial gap in the data meant to help make decisions about schools and teachers.

School leaders have also said they were puzzled by the state’s methodology in reaching the so-called growth scores upon which districts and schools are judged — particularly by how they arrived at the Memphis district’s low score.

Even those who are paid to sift through the data say they are having trouble getting answers to questions about the growth scores, known as TVAAS. Bill White, chief of planning and accountability for Shelby County Schools, conceded to board members last week that he didn’t know the ins and outs of the complex formula and the changes meant to compensate for the missing data.

“I have personally never been shown all the mathematics behind our data and how this works,” he told board members. “I do know that it has been peer-reviewed and vetted and it’s essentially been held up among those statisticians. But there is a lot that goes on behind the scenes that no one has been able to walk us through.”

The confusion has renewed skepticism about the state’s value-added model, which is supposed to help officials identify the impact that schools and teachers have on student performance. The system relies on the state’s data measuring student growth in districts.

Part of the problem is last year’s botched testing, which is having multiple ripple effects throughout the state.

This year, growth scores are comparing 2016-17 test results with the 2014-15 school year, the most recent data available. That throws a wrench in how to assess which school or teacher is responsible for a child’s growth over a two-year period. And for elementary schools, that means there is no data for fourth graders this year since testing in third grade, the first year students take state tests, was canceled.

In addition, one subject was dropped entirely from TVAAS calculations because social studies questions were a trial run for elementary and middle schools students and did not count.

Statisticians for the most part have figured out how to calculate growth even when a state transitions to a new test. But the missing data creates a whole other host of challenges the revisions attempt to account for.

One Memphis charter leader said he still isn’t quite sure how his school even got a score since last year his highest grade level at the school was third grade, the first year of testing.

“It’s such a convoluted formula, it’s hard for us to understand. We’re not sure how we got (our score),” said the charter leader, who declined to be named because he was still seeking answers from the state.

Damian Betebenner, a senior associate at Center for Assessment that regularly consults with state departments, said missing data on top of a testing transition “muddies the water” on results.

“When you look at growth over two years, so how much the student grew from third to fifth grade, then it’s probably going to be a meaningful quantity,” he said. “But to then assert that it isolates the school contribution becomes a pretty tenuous assertion… It adds another thing that’s changing underneath the scene.”

At the same time, TVAAS scores for struggling schools will be a significant factor to determine which improvement tracks they will be be placed on under the state’s new accountability system as outlined in its plan to comply with the federal Every Student Succeeds Act. For some schools, their TVAAS score will be the difference between continuing under a local intervention model or being eligible to enter the state-run Achievement School District. The school growth scores will also determine which charter schools are eligible for a new pot of state money for facilities.

The state has data analysts based across Tennessee to help districts with their questions and provide data simulations for the complex formula that has been replicated in other states.

“Of course, the reason it is complex is because we want it to be fair for educators and therefore capture as much data and nuance as possible – which is discussed at length in the technical documentation,” said a state department spokeswoman.

The state has also published an overview video of how the formula works and details on the recent changes in a 46-page, formula-packed document from SAS, the private company that calculates teacher and school scores for the state.

But as far as knowing how the state gets from A to Z, White said he still has questions.

“I’ve had some questions about getting access to certain data myself,” said White, who routinely interprets data for the district. “We would like a lot more access to what goes into TVAAS.” (He later declined to elaborate.)

He’s not the only one. When the Tennessee Education Association unsuccessfully sued Knox County Schools over its use of TVAAS in awarding teacher bonuses, access to data on how the scores were calculated was central to the association’s argument that the district denied teachers due process, said Rick Colbert, TEA’s general counsel.

When Colbert attempted to subpoena technical documents on the calculations, SAS blocked it partially because the request would divulge “trade secrets.”

“When they’re called upon to defend it you get a lot of general statements but you can’t get a lot of information to see if you can back that up,” Colbert said. “There’s so much about TVAAS that can’t be explained.”

Board member Mike Kernell called it a double standard and asked White last week if the district could request a demonstration of the complicated formula.

“I think the state department of education ought to show its work if they’re asking children to show their work,” he said.

Q&A

At this Perry Township school, progress isn’t just about testing, it’s ‘the work we do every single day in our classrooms’

PHOTO: Shaina Cavazos
Principal Star Hardimon, celebrates math progress with fourth-graders at Douglas MacArthur Elementary School in Perry Township.

Chalkbeat is talking with principals across the city at schools that made some of the biggest ISTEP gains in 2017 to explore what was behind their school’s progress and identify possible lessons for other schools.

As Principal Star Hardimon hurried down the hallway of Douglas MacArthur Elementary School, she had her sights set on Tom Stahlhut’s fourth grade classroom, where in just minutes students would be packing up for an assembly.

She carried a gold trophy, which is awarded to the classroom that saw the most improvement on math or English practice tests for that month, part of a new program called Evaluate. Kids were already lining up to leave, but she stepped quickly into the room. One student was already on to her surprise.

“Oh, I know what we win!” he said as he and his classmates gathered closely around Hardimon.

“I actually came to your room today because I brought something along for math Evaluate,” Hardimon said. “Mr. Stahlhut’s class went from a 35 percent to a 49 percent. You are the fourth grade winners!”

The students erupted in cheers, waving their arms and jumping up and down as she presented their trophy. These kinds of celebrations aren’t unusual at MacArthur, Hardimon said, and they were especially significant this year given the gains from last year.

MacArthur, which has 805 students in preschool through fifth grade, moved from a B grade from the state in 2016 to an A in 2017. The school’s test passing rates jumped 10.8 percentage points to 63.3 percent of students passing both English and math exams, higher than the state average. Both figures — passing rates and growth — factor into a school’s letter grade.

Find your school’s 2017 ISTEP scores here.

Almost three-quarters of MacArthur students qualify for subsidized meals, and a little more than one-quarter are learning English as a new language. Many of those English-learners are also refugees from Burma, a trend across the district.

The district as a whole last year was focused on tracking student progress on English and math skills though Evaluate. Students and teachers both track results from tests together each month, using a stoplight model — red, yellow, green — to indicate in their records when a student has mastered, say, dividing fractions, and when they need more practice.

Of the Marion County township elementary schools with the highest ISTEP gains, four were from Perry Township. Every Perry principal who spoke to Chalkbeat this fall mentioned the new data tracking system as key to their improvement.

Below are excerpts from a recent conversation with Hardimon to talk about her school’s progress. The interview was edited for clarity and brevity.

What was your reaction when you learned how much improvement you had made this year?

We fully celebrated. We made a banner and every person, from the custodian, cafeteria — every person that supports kids in our building in any way signed the banner, and every person got a cookie, and we did a cookie with the letter “A” on it. We cheered and had some fun in the lunchroom.

Us earning this A wasn’t about the days we took the test. It was about the work we did every single day in our classrooms, at home, during homework, reading on the weekends — it was everything.

What do you think made the difference?

Well, when we initially got our scores back from the previous year, we were bummed. So we really tried to think about what do we need to do, how do we need to look at this test compared to what we’re doing everyday. And I know it’s a new test and there are some different things, and I don’t want to make excuses, so we just needed to figure out what to do.

Every month I met with grade levels to just talk about the data, talk about what we’re doing, talk about what we look like. And teachers would fill out their data tracking sheets, and everybody was really in tune.

The other thing that we really did is in January, we did an all-hands-on-deck, and for third, fourth and fifth grade we pulled our special education, our E.L., our intervention, and our master teachers to pull groups of students out of classrooms so we could work on specific skills during that intervention time. And we also looked at some of the content area time to really home in so kids could get a real 20 minutes of direct instruction on a particular skill. And that’s something that we had not done in that way. And we’re pretty pleased with it.

I really honestly feel that that effort by everyone to really focus in on that bottom 25 percent (of students) regardless of E.L., special education — whatever their needs are — and our general education kids fell into that as well. I think that’s where we earned those points, was with that group.

What is your school community and culture like?

Douglas MacArthur is a very a community-driven school. I have teachers in the building right now who were students here. I have grandparents who always come in and say, “Oh, my kids and now my grandkids go here.” That comes with a lot of pluses and minuses, but the good thing is the people, they believe in this school. They want the best for kids and they’re really willing — they stay for after school activities and they get involved in all our programs.

Our demographic has been changing. Free and reduced lunch numbers since I’ve been here have increased significantly, and this is my fifth year. Just under half of our kids are English-learners, some coming from as part of our refugee community. We have a very small population of African-Americans, however we have more than when I first came, and then the rest are Caucasian. We do have a small population of Hispanic students, and we have the most number of Hispanic students than we did even five years ago. So our community is definitely changing. It used to be Caucasian, mostly.

What is your approach to leadership?

I feel like i’m a very instructional leader. I try and model behavior in almost everything because if I’m not doing it, then I certainly don’t expect a staff member to do it, or a student to do it. So really modeling and holding myself accountable at a very high level. I’m pretty hard on myself. I think that reflection piece needs to be transparent.

I feel like I try really hard to model a professionalism, a pride in something, working hard everyday. That work ethic is important — it’s important for students to see, it’s important for parents to see. They’re trusting us with their babies, and that’s a pretty big deal, so they have to trust me. I think about my own children, and the thoughts I’ve had about administrators that have led their schools, and that has helped me.

Movers & shakers

Former Tennessee Teacher of the Year will lead citywide reading program

PHOTO: Courtesy of Karen Vogelsang
Karen Vogelsang, the 2015 Tennessee Teacher of the Year, will become the executive director of ARISE2Read.

Three years after winning the state’s top award for teaching, Karen Vogelsang is leaving the classroom to lead a citywide early literacy program.

Vogelsang, a fourth grade teacher at Winridge Elementary School, will become the executive director of ARISE2Read, a Christian volunteer organization that matches reading tutors and mentors with struggling second grade readers.

“When we’re presented as teachers with the opportunity to broaden our impact beyond our school, we need to take that seriously,” Vogelsang told Chalkbeat, adding she initially turned the job down a few months ago. “It’s not just the 80 second graders here at Winridge, but the thousands of second graders in Shelby County Schools.”


Tennessee’s 2015 Teacher of the Year on teaching economically disadvantaged students in Memphis


Vogelsang spent 15 years as a banker before switching careers to education in 2003. She became Tennessee’s Teacher of the Year in 2015. And earlier this year, she stepped into a hybrid role on Superintendent Dorsey Hopson’s team to interject a teacher’s voice in policy decisions for Shelby County Schools. Since then, the teacher advisory council has grown to 17 teachers across the district, she said.

Though she won’t be with the district anymore, Vogelsang will still be working toward goals set out by Shelby County Schools in her new position. ARISE2Read, which has mentors in 30 Memphis schools, aims to catch up struggling second grade readers by taking them out of the classroom for 30 minutes once a week with a mentor.

Shelby County Schools has a goal of having 90 percent of third graders reading on grade level by 2025. In 2014, it was only 30 percent with a goal of reaching 60 percent by 2020. According to early 2017 results from a nationally standardized test (MAP), about 50 percent of third grade students were proficient.

“We have a lot of work to do and we can’t do it on the manpower of Shelby County Schools alone,” Vogelsang said. “The fact that this was so focused was part of the attraction (to ARISE2Read) and addresses a need we have in the district.”

The organization also has mentors and students in Fayette, Jackson/Madison, Tipton and Gibson counties and has done training in Knoxville and Houston.

Vogelsang’s class will be turned over to a co-teacher who has been in her classroom since taking on the hybrid role, and she will begin at ARISE2Read on Jan. 4.