Over the past month or so, in newspapers and local-news websites all around the country, public high schools and school districts have been trumpeting reports about how they've done on various national rankings of high schools. For instance, here's Bill Runey, principal of Attleboro High School in Massachusetts. "We're really proud of this," he said in a press release put out by the school district. He was referring to the fact that Attleboro had been ranked 1,947th in the nation on the Washington Post's annual ranking of "America's Most Challenging High Schools."
On a local level, school rankings long have been the sort of thing city magazines thrive on, along with their "best of" issues that purport to tell readers where to buy the best burger in the city or get the best waxing. In a single metropolitan area (or even in a single state), rankings of public schools may have some utility if they are done thoughtfully, using sensible metrics. Parents might be able to use that information to find an affordable residence near good schools, while still leaving themselves within reasonable reach of their place of employment. It's harder to fathom the logic for ranking high schools nationwide. Few are the families who will move out of state or across the country on the basis of claims about school quality.
Without taking away from whatever credit Runey and Attleboro High School deserve for their achievements, let's call national rankings of high schools what they are: nonsense. There is no way to say, with any degree of accuracy at all, where any given high school ranks in relation to others in terms of how good it is or how challenging it is. And the claim that Attleboro High School, which was not even fully accredited as recently as seven years ago, is now in the top ten percent of America's high schools -- among the most challenging -- seems improbable, at best.
And yet, every year since 1998, Jay Mathews, an education journalist at the Washington Post, has been putting together a ranking of what he calls "America's Most Challenging Schools," or the Challenge Index. For years, this national list was published by Newsweek, which was owned by the Washington Post Company. When the Post sold off Newsweek in 2010, it kept the Mathews index for itself. Newsweek then produced its own ranking, which has been continued by the Daily Beast. And, of course, US News & World Report, an organization famous for fueling Americans' obsessions with rankings (colleges, law schools, hospitals, etc.) started its own high-school list, too.
All of these lists have flaws that stem from the inherent absurdity of presuming to rank schools around the country according to how good or challenging they are. And they all come in for criticism. Recently, Matthew Di Carlo, a senior research fellow at the Albert Shanker Institute, took a critical look at the Newsweek/Daily Beast and US News rankings, finding some good and some bad features in each of them.
But it's the Mathews "Challenge Index" that has given rise to the sharpest criticism over time (see here, here, and here, for example) because of its methodology, which is reductionist in the extreme. It uses only one factor to calculate its rankings: It divides the number of Advanced Placement (AP), International Baccalaureate (IB), and Cambridge (AICE) exams taken at each school by the number of graduating seniors. Note that the numerator is not even the number of such exams passed, but merely the number taken. So, a given school can rise on the list by increasing the number of its students who take "advanced" classes.
Conversely, schools that are more discerning and thoughtful about which students ought to be taking AP classes end up suffering in the rankings. So, the list produces nonsensical anomalies such as high schools with very low graduation rates ranking much higher on the "Challenge Index" than excellent schools that don't game the ranking system, or that, like Scarsdale High School, have joined the growing list of schools that have eliminated AP courses so that, as Bruce Hammond puts it, "students and teachers could rediscover their passion and creativity" once freed of what is too often a rigid and stultifying AP curriculum.
To their credit, US News and Newsweek/Daily Beast, which also use AP and IB courses as a measure, have made their rankings more sophisticated and reasonable by also adding other measures of a school's quality, such as (in the Daily Beast's case) graduation rates and college-acceptance rates, and (in the case of US News) performance on state accountability tests and the proficiency rates of a school's least advantaged students on those tests. For explanations of their methodologies, see here for the Daily Beast and here for US News.)
Despite steady criticism over the years, Mathews has retained and defended the simple formula he uses to calculate his Challenge Index, refusing to factor in other appropriate measures of school quality beyond the number of students taking advanced classes. (His only concession has been to add a separate list of schools, what he calls "The Catching Up Schools," that takes into account how impoverished the student body is, as measured by the percentage of students who quality for federal lunch subsidies. He also now notes that information in a separate column on his main ranking, along with the percentage of graduates who passed at least one "college-level" test during their high school career, but does not factor those data into his rankings.) Because Mathews otherwise insists on only using AP and IB exams as his measure, the Challenge Index typically comes in for the sharpest criticism of all these rankings. The essential criticisms can be summarized as follows:
1) The inherent impossibility of measuring relative quality in schools. Quality is a very subjective matter, especially in something as intangible as education. And using a simple measure to rank thousands of schools certainly cannot capture the relative quality of schools or indicate which are better than others.