One of the more intriguing areas of international relations research over the past decade has been the potent effect that global performance indicators — ranking exercises such as the OECD Better Life Index, the annual State Department human rights reports, or the Legatum Institute’s Prosperity Index — have on policymakers and the public.
It might seem as though many of these rankings and blacklists are simply PR exercises in attention-getting — and many of them are — but they still matter. Ranking announcements generate news stories. Politicians and bureaucrats do not like learning that they are falling behind in rankings or getting named and shamed on blacklists. Some countries might actually alter their policies to avoid such stigmas.
This is not hypothesis, but fact. Political science research suggests that “performance indicators can influence state policy outputs, especially when they are based on systematic monitoring, are comparative (and especially quantitative), are wielded by a respected actor or group/organization of actors, and are widely disseminated.” A raft of follow-up research bolsters the idea that global performance indicators matter.
There is a dark side to these efforts, however, as Alexander Cooley and Jack Snyder have noted in their work. Actors will try to game the ranking schema. Dubious methodologies and normative biases can color the rankings in political ways. Sometimes they matter too much. Shenanigans within the World Bank’s Ease of Doing Business Index forced the bank’s chief economist Paul Romer to resign in 2018. By 2020, its perverse effects persuaded the bank to cancel the 2020 exercise.
Two stories from recent weeks suggest deeper flaws with global performance indicators that should give scholars and policymakers pause — not rejection, but pause — about these rankings.
In 2019, Johns Hopkins University partnered with the Economist Intelligence Unit and the Nuclear Threat Initiative to create the Global Health Security Index. According to its website, “The GHS Index is the first comprehensive assessment of global health security capabilities in 195 countries.” The initiative received funding from a host of prestigious foundations, including the Gates Foundation, to set it up.
Created less than a year before the start of the coronavirus pandemic, the GHS Index should have been the ideal way to predict how countries would handle the novel coronavirus. The index predicted that the two most prepared countries would be the United States and the United Kingdom. Whoops.
Cullen S. Hendrix of the Peterson Institute for International Economics examines what the GHS index got wrong. He finds zero correlation between the GHS Index and covid-19 mortality, when one would have hoped to have seen a negative relationship. His short article is worth reading in full, but his principal takeaway is clear enough: “What the architects of GHS appear to have underestimated — or have miscalculated about the specific nature of the COVID-19 pandemic — is the importance of political leadership, social consensus on the necessity of bearing containment costs, and geography.”
At least the GHS Index was a good-faith effort to rank something that had not been ranked before. What happens when a long-trusted data source goes off the rails?
Political scientists have long relied on the Polity data set for the coding of regimes as democratic, autocratic or somewhere in between. It has been a staple of statistical analysis in comparative politics and international relations for generations. If it starts going wonky in its coding, a lot of research becomes much harder to accomplish.
It’s safe to say that Polity is starting to go wonky in its coding of the United States. During the Trump years, Polity started downgrading the U.S. democracy score. The aggregated sub-components range between +10 (full democracy) to -10 (totalitarian dictatorship). From 1974 to 2016, the United States was at 10.
Then Donald Trump was elected and Polity started downgrading the United States to an 8. Another downgrade occurred in 2019, to a 7, and then 2020 led to another downgrade to a 6. Polity’s rationale was the continued weakening of executive branch constraints.
Reasonable people can and do disagree over just how much unchecked power Trump exercised as president. Polity is hardly alone in downgrading the U.S. score.
The problem is the internal consistency of Polity’s coding. In lowering the U.S. score to a 6, Polity was asserting that in 2020, the United States was less democratic than the United States was at any time since 1825. Now I’m not an American politics scholar, but I am vaguely familiar with U.S. history. Any schema that codes the United States today as less democratic than before the elimination of slavery and the extension of the voting franchise beyond White property-holding men over the age of 21 has some serious flaws.
The comparative coding is equally problematic. According to Polity, Hungary remained a full democracy with a coding of 10.
Polity’s coding is even worse than that, however, because after the 2020 election, which saw record turnout, Polity downgraded the United States again because of further weakening in executive constraints. The reasoning:
Trump adamantly refused to concede the election, claiming widespread fraud resulting from the expanded voting procedures. President Trump systematically and vehemently proceeded to undermine the integrity of the voting process, filing over sixty lawsuits designed to nullify the results in four swing states. … Having failed to reverse the election results, the President organized a rally of supporters on January 6, 2021, near the White House to stop the certification of the Electoral vote by Congress. Trump exhorted the crowd and called for them to march on Capitol Hill to stop the certification. The crowd forced their way into the Capitol building and delayed, but did not stop, the certification of the election result. The House of Representatives passed a second impeachment against President Trump for inciting insurrection on January 13, 2021. President Joseph Biden was inaugurated on January 20, 2021.
This description is accurate and yet contradicts the claims of weakened executive constraints. Trump tried to enhance his power at the expense of other branches. The result was a complete rejection of his claims in the judicial branch, an impeachment of Trump by the legislative branch, and Trump’s replacement by Joe Biden in the executive branch.
The folks at Polity are trying to send a message, and to be fair to them, it’s not an unimportant one. The events of Jan. 6 will resonate for quite some time. But by Polity’s own coding schema, this latest move makes zero sense. I am hardly the only political scientist to think so — see Paul Poast and Ben W. Ansell for more.
Rankings matter. But the easier it becomes to question the methodology of these rankings, the less power they have. Polity is in danger of seeing its reputation compromised — and, in the process, undercutting political scientists who have relied on its data.