Does the release of information on 18,000 educators really increase transparency, or does it open the door for unprofessional -- and misleading -- statistical analysis?
Following a court challenge last year by several media outlets, the New York City Department of Education -- which operates the nation's largest school district -- agreed to release individual evaluation rankings for 18,000 of its teachers. The New York Times is publishing its analysis of the data, including the teachers' names and their school assignments. GothamSchools, an independent online publication, has opted out, citing significant concerns about both the fairness and the accuracy of the data.
Indeed, there is potentially a wide margin of error. The final scores for teachers could be off on average by as much as 35 or 53 percentage points for English and math exams, respectively, the New York Times reported.
Teacher Evaluations: Too Much Change Too Fast?
Questions Raised on the Low Bar For Teacher Licensing Exams
The Pressure to Cheat: Greater Than Ever in LA, Teachers Say
Additionally, some teachers' evaluations reflect test data collected for as few as 10 students."The purpose of these reports is not to look at any individual score in isolation ever," the city's Education Department's Chief Academic Officer Shael Polakow-Suransky told the New York Times. "No principal would ever make a decision on this score alone and we would never invite anyone, parents, reporters, principals, teachers, to draw a conclusion based on this score alone."
That might be easier said than done. In many cases, it's easier to gather data than it is to comprehend what they mean or how they should be used, either in education policy or journalism.
Several years ago, the Los Angeles Times hired an outside researcher to help devise its own method for interpreting the teacher evaluation data provided by the district. The high-profile publication of the "Grading the Teachers" project led to much debate in the education and journalism communities.
In a recent issue brief, Center for American Progress senior education policy analyst Diana Epstein and co-author Raegen Miller (CEP's associate director for education research) suggested publicizing evaluation results for individual teachers would do more harm than good.
Epstein and Miller were blunt in their criticism of a groundbreaking -- and controversial -- Los Angeles Times project that ranked thousands of the city's teachers by their multi-year test scores.
By building its own interpretative model, the newspaper crossed the line from reporting to research, the authors contended. The Los Angeles Times (and others that follow its lead) should be held to a more rigorous standard, they argued.
"If journalists attempt to do their own analyses of value-added data, they should follow the same standards that researchers do when protecting human subjects," Epstein and Miller wrote. "This means that data are de-identified and individual names are never published."
For the evaluations to be useful, the teachers have to be willing partners, and publicizing their names along with the results will only make teachers less willing to engage in the process, the authors contended.
Doug Smith, the data reporter for the Los Angeles Times project, said his biggest issue with the report was that it was "doctrinal."
The issue brief suggested there might be negative consequences to this kind of reporting, but those were assumptions, rather than known facts, Smith said.
In reality, there have been no obvious negative consequences to publishing the names of the teachers and their ratings, Smith said.
"The district's test scores increased by about the same amount they have in previous years -- there was no significant difference," Smith said. "There were no reports of parents storming the schools and demanding different teachers for their children, throwing the campuses into turmoil."
The Los Angeles Times gave teachers the chance to review their own rankings prior to the list being published, as well as a chance to add written comments to their personal page on the paper's Web site. (The New York Times is offering teachers a similar opportunity to comment and correct errors.)