The Washington PostDemocracy Dies in Darkness

U.S. News changed the way it ranks colleges. It’s still ridiculous.

Harvard University occupies the second spot among national universities in U.S. News & World Report's ranking. (Charles Krupa/AP)

U.S. News & World Report has made some changes in the formula for the latest version of its famous annual college rankings.

It has dropped data for admission rates — which focused attention on the most highly selective schools — and put some focus on low-income students. Critics who have dismissed the rankings for obsessing on exclusivity and giving short shrift to social mobility might see that as positive movement.

But, alas, the problems that made these influential rankings questionable in the past are not gone. As I wrote last year, the success of a data-based ranking is, obviously, the quality of the data. If you put junk in, you get junk out.

And that's pretty much what you get with most rankings of schools. The folks doing the ranking decide what is important to them or their audience, and, for some reason, consumers and schools themselves put a great deal of stock in the outcome of ever-changing, questionable methodology.

Princeton and Williams still top U.S. News college rankings — but new formula scrambles the annual lists

So let's look at what went into the new U.S. News college rankings, which shuffled some of the top 20 schools and created more movement in schools below those. Wealthy schools continued to have an advantage in the rankings.

Princeton University remained the No. 1 national university, as it has for seven previous years, and Williams College was the top national liberal arts college, as it has been for the previous 15 years. (You can see below for more of the rankings, if so inclined.)

But the University of California at Riverside, which had almost the same six-year graduation rate for Pell and non-Pell students, rose 39 places — to a tie at No. 85. Georgia State University climbed 36 spots to a tie at No. 187. And Howard University jumped 21 spots, to No. 89.

With a simple change in their formula, U.S. News rankers have helped to alter the image of many schools. That's powerful.

This is how U.S. News says it came up with the new rankings, with some commentary:

1) GRADUATION RATES: 35 percent

This looks at the retention and graduation of students within six years — and is up from a 30 percent weight the year before. The new 35 percent weight includes a new metric: the graduation rates of students whose low family income qualifies them for Pell grants. But how much does it count in the overall rankings? A total of 5 percent: 2.5 percent for Pell grant graduation rates and another 2.5 percent for Pell Grant graduation rates compared with all other students.

The magazine does not, naturally, discuss one of the big problems with graduation rates: They don't consider transfer students, and there are estimates that up to one-third of students transfer at some point in their college career. That alone raises the question of the validity of a graduation rate metric.

2) FACULTY RESOURCES: 20 percent

This looks at class size, faculty salary, faculty with the highest degree in their fields, student-faculty ratio and proportion of faculty who are full time. This relates, obviously, to the wealth of the institution. Rich schools have an advantage.

3) EXPERT OPINION: 20 percent

This comes from a U.S. News survey of “top academics” — presidents, provosts and deans of admissions — asked to rate the academic quality of peer institutions. Nearly 24,400 high school counselors are also surveyed.

It's hard to take seriously a ranking that has such a subjective metric. Presidents, provosts and deans of admissions are generally too busy with their own institutions to actually learn about the important details of their peers. Numerous top-ranking administrators have told me for more than a decade that they don't answer the survey or that they have someone else in their office do it. That's not a scientific survey of administrators, to be sure, but neither are the U.S. News rankings.

It is also worth noting that the responses of high school counselors, who do actually keep track of what is going on at colleges, have a 5 percent weight, down from 7.5 percent in the year before. The rest of the expert opinion share comes from the “top academics."

4) FINANCIAL RESOURCES: 10 percent

Obviously, this metric, which speaks to the programs and services a school can offer, benefits wealthy ones.

5) STUDENT EXCELLENCE: 10 percent

This is down from 12.5 percent last year. What does it mean? High school records of students — test scores and grades. There's something new for the latest rankings in this area: Acceptance rate, which had a 1.25 percent weight in last year's ranking, has been removed to make room for the social-mobility indicators. It wasn't much before.

6) ALUMNI GIVING: 5 percent

Again, wealthy schools that graduate more students because they have more resources benefit more from this.

So, in the end, what do you really have?

For the record, these are the top 10 national universities in the latest rankings — and notice the many ties:

1. Princeton University

2. Harvard University

3. Columbia University

3. Massachusetts Institute of Technology

3. University of Chicago

3. Yale University

7. Stanford University

8. Duke University

8. University of Pennsylvania

10. Johns Hopkins University

10. Northwestern University

And these are the top 10 for national liberal arts colleges:

1. Williams College

2. Amherst College

3. Swarthmore College

3. Wellesley College

5. Bowdoin College

5. Carleton College

5. Middlebury College

5. Pomona College

9. Claremont McKenna College

10. Davidson College

This is the magazine's explanation for its methodology:

Outcomes (35 percent, up from 30 percent in 2018)
More than one-third of a school's rank comes from its success at retaining and graduating students within 150 percent of normal time (six years). It receives the highest weight in our rankings because degree completion is necessary to receive the full benefits of undergraduate study from employers and graduate schools. We approach outcomes from angles of social mobility (5 percent), graduation and retention (22 percent), and graduation rate performance (8 percent).
Social mobility: New this year, we factored a school's success at promoting social mobility by graduating students who received federal Pell Grants (those typically coming from households whose family incomes are less than $50,000 annually, though most Pell Grant money goes to students with a total family income below $20,000). See below the two measures that factor into social mobility.
Scores for the new social mobility indicators were then adjusted by the proportion of the entering class that was awarded Pell Grants because achieving a higher low-income student graduation rate is more challenging with a larger proportion of low-income students.
As a result of adding indicators for social mobility into the 2019 Best Colleges rankings, when combined with the graduation rate performance, U.S. News takes economic diversity into account in indicators that comprise 13 percent of the rankings.
Our other outcome measures include:
Graduation and retention rates: The higher the proportion of first-year students who return to campus for sophomore year and eventually graduate, the better a school is apt to be at offering the classes and services that students need to succeed. This has two components:
The graduation rate indicates the average proportion of a graduating class earning a degree in six years or less; we considered first-year student classes that started from fall 2008 through fall 2011. First-year retention indicates the average proportion of first-year students who entered the school in the fall 2013 through fall 2016 and returned the following fall. Graduation is given four times more weight than retention. We weighted it at 22 percent total, down from 22.5 percent in 2018.
Graduation rate performance: We compared each college's actual six-year graduation rate to what we predicted for its fall 2011 entering class. The predicted rates were modeled from admissions data, proportion of undergraduates awarded Pell Grants, school financial resources, and national universities' math and science, or STEM, orientations. We weighted it at 8 percent, up from 7.5 percent in 2018.
The graduation and retention rate numerical ranking published on usnews.com for the 2019 Best Colleges is based on a school's total score in the following four ranking indicators: average six-year graduation rates, average first-year retention rates, Pell Grant graduation rates and Pell Grant graduation rates compared all other students. Previously, the graduation and retention rate numerical ranking published on usnews.com was based on a school's total score in these two ranking indicators: average six-year graduation and average first-year retention rates.
Faculty Resources (20 percent)
Research shows the greater access students have to quality instructors, the more engaged they will be in class and the more they will learn and likely graduate. U.S. News uses five factors from the 2017-2018 academic year to assess a school's commitment to instruction: class size, faculty salary, faculty with the highest degree in their fields, student-faculty ratio and proportion of faculty who are full time.
Expert Opinion (20 percent, down from 22.5 percent in 2018)
We survey top academics – presidents, provosts and deans of admissions – asking them to rate the academic quality of peer institutions with which they are familiar on a scale of 1 (marginal) to 5 (distinguished). To get another set of important opinions, U.S. News also surveyed nearly 24,400 counselors at public, private and parochial high schools from all 50 states and Washington, D.C, via emails provided to U.S. News by MDR, a division of Dun & Bradstreet.
Academic reputation matters because it factors things that cannot easily be captured elsewhere. For example, an institution known for having innovative approaches to teaching may perform especially well on this indicator, whereas a school struggling to keep its accreditation will likely perform poorly.
Financial Resources (10 percent)
Generous per-student spending indicates that a college can offer a wide variety of programs and services. U.S. News measures financial resources by using the average spending per student on instruction, research, student services and related educational expenditures in the 2016 and 2017 fiscal years. Spending on sports, dorms and hospitals does not count.
Student Excellence (10 percent, down from 12.5 percent in 2018)
A school's academic atmosphere is influenced by the selectivity of its admissions. Simply put, students who achieved strong grades and test scores during high school have the highest probability of succeeding at challenging college-level coursework; enabling instructors to design classes that have great rigor.
New for 2019, acceptance rate (1.25 percent in last year's ranking) has been completely removed from the ranking calculations to make room for the new social mobility indicators.
Also, we reduced the weight of the two remaining student excellence factors assessing the fall 2017 entering class – standardized tests and high school class standing.
Standardized tests: U.S. News factors admissions test scores for all enrollees who took the mathematics and evidence-based reading and writing portions of the SAT and the composite ACT. The SAT scores used in this year's rankings and published on usnews.com are for the new SAT test administered starting March 2016.
We weighted standardized tests at 7.75 percent, down from 8.125 percent in 2018.
Schools sometimes fail to report SAT and ACT scores for students in these categories: athletes, international students, minority students, legacies, those admitted by special arrangement and those who started in summer 2017. For any school that did not report all scores or that declined to say whether all scores were reported, U.S. News reduced its combined SAT/ACT percentile distribution value used in the ranking model by 15 percent. This practice is not new; since the 1997 rankings, U.S. News has discounted under these circumstances because the effect of leaving students out could be that lower scores are omitted. U.S. News also footnotes schools that declined to tell U.S. News whether all students with SAT and ACT test scores were represented.
If the combined percentage of the fall 2017 entering class submitting test scores is less than 75 percent of all new entrants, its combined SAT/ACT percentile distribution value used in the rankings was discounted by 15 percent. U.S. News has also applied this policy in previous editions of the rankings.
High school class standing: U.S. News incorporates the proportion of enrolled first-year students at National Universities and National Liberal Arts Colleges who graduated in the top 10 percent of their high school classes. For Regional Universities and Regional Colleges, we used the proportion of those who graduated in the top quarter of their high school classes.
We weighted this at 2.25 percent, down from 3.125 percent in 2018.
Alumni Giving (5 percent)
This is the average percentage of living alumni with bachelor’s degrees who gave to their school during 2015-2016 and 2016-2017. Giving measures student satisfaction and post-graduate engagement.
— U.S. News & World Report

Alas, some of the key problems with the initiative remain. And that means the junk-in/junk-out scenario still holds.

The problem with the 2018 U.S. News & World Report rankings: Junk in, junk out

Glad to see U.S. News playing down inputs (selectivity) and emphasizing outputs (graduation rates, commitment to serving all economic classes). That said, rankings are still ridiculous when it comes to choosing a college. Select the one you like among those with a high four-year graduation rate. All the institutions graduating 90 percent or more within six years are outstanding.

Loading...