I thought I already knew what was wrong with US News’ rankings of “Best Colleges,” so I was slow to reach for Malcolm Gladwell’s recent piece in the New Yorker, “The Order of Things” (Feb. 14 and 21). But (to use our past president’s wonderful locution), I misunderestimated Gladwell’s contribution, a portion of which I will share here.
People who follow higher education know that US News’ rankings rely heavily on inputs, not outputs (e.g., not the learning the schools impart but the amount of resources spent), and that they use estimates of reputation for a good part of the ranking (22.5%).
But the problems with US News’ rankings apparently go deeper or at least are more complex. Gladwell argues that it is impossible to come up with a single ranking of heterogeneous institutions (as US colleges are) on multiple dimensions — as US News tries to do — without making “implicit ideological choices.” He says that those choices mean that schools that enable more students to get better educations are always going to be low on the list.
To be specific: universities that currently rank in the middle of US News’ list can’t improve their rankings, for two reasons. A University of Michigan sociologist who studies rankings has found that the university presidents who take the reputation survey (some are expected to “evaluate” more than 200 peer institutions) depend heavily on the existing US News rankings for their evaluations! In other words, the reputation process is circular.
Second, student selectivity swamps measures of effectiveness. Here’s how it happens. US News does have what Gladwell calls an “efficacy” measure, “graduation rate performance.” Since graduation rates depend largely on the selectivity of the incoming students, this measure “compares a school’s actual graduation rate with its predicted graduation rate given the socioeconomic status and the test scores of its incoming freshman class.”
If the graduation rate is higher than expected, the difference raises a school’s score, because the school is graduating more students than would get through on the basis of selectivity alone. (There might be some question about this as a measurement of efficacy, but that’s not my point right now.)
The problem, says Gladwell, is that “no institution can excel at both.” For example, Yale is so high on the selectivity scale (it’s ranked first among national universities) that its “predicted graduation rate” is 96. Thus, its efficacy rate can’t be more than four, and it’s actually two. In contrast, Penn State, which has the lowest ranking of the top 50 national universities, is not as selective as Yale. But it does very well on the graduation measure; its expected graduation rate is 73% and its actual graduation rate is 85%, giving it an “efficacy” score of 12, the highest in the top 50.
But US News gives twice as much weight to selectivity as to efficacy — a completely arbitrary choice and, according to Gladwell, the wrong measure in terms of social benefit (although from the perspective of the student seeking prestige, it may be the right choice).
Finally, the rankings leave out price. Although Gladwell doesn’t recalculate the top 50 universities with price as a factor, he does so with law schools, since an Indiana University law professor has conveniently laid out the chief US News criteria in a spreadsheet. The expected schools are there, led by the University of Chicago, Yale, and Harvard. If Gladwell makes price a factor and gives it equal weight with the US News’ other criteria for law schools, two new schools pop up on the list. The upstarts are Brigham Young University and the University of Colorado.
Gladwell suggests that a school should be rewarded for being affordable, but this is beyond the pale for US News. As a result, says Gladwell, “the Yales of the world will always succeed at the US News rankings because the US News system is designed to reward Yale-ness.”
