The Washington PostDemocracy Dies in Darkness

Fact, fiction, and social science replication

A recent scandal will lead to calls for greater transparency in political science data. But that's been an ongoing trend.

A scientist working.
Placeholder while article actions load

Earlier this week, the New York Times caught up with the Michael LaCour imbroglio that everyone else (including Spoiler Alerts) was talking about last week. And, as per usual, the Times added some texture and additional facts (including the point that LeCour has lawyered up) to the story.

But there was also something vaguely off about Benedict Carey and Pam Belluck’s story, particularly these sections:

The case has shaken not only the community of political scientists but also public trust in the way the scientific establishment vets new findings. It raises broad questions about the rigor of rules that guide a leading academic’s oversight of a graduate student’s research and of the peer review conducted of that research by Science….
The scientific community’s system for vetting new findings, built on trust, is poorly equipped to detect deliberate misrepresentations. Faculty advisers monitor students’ work, but there are no standard guidelines governing the working relationship between senior and junior co-authors….
“It is simply unacceptable for science to continue with people publishing on data they do not share with others,” said Uri Simonsohn, an associate professor at the Wharton School of the University of Pennsylvania. “Journals, funding agencies and universities must begin requiring that data be publicly available.”….
Survey data comes in many forms, and the form that journal peer-reviewers see and that appears with the published paper is the “cleaned” and analyzed data. These are the charts, tables, and graphs that extract meaning from the raw material — piles of questionnaires, transcripts of conversations, “screen grabs” of online forms. Many study co-authors never see the raw material.

These excerpts are not totally off-base  — someone skillfully committed to faking data to generate an extraordinary result could very possibly get that article through a peer-reviewed process. It happens across scientific disciplines, not just in political science.

But the NYT’s depiction of how the process works is incomplete. For one thing, while fraud is possible, the ability to do it and get away with it is inversely proportional to the significance of the finding. As the co-founders of Retraction Watch noted in their New York Times op-ed last Friday:

Journals with higher impact factors retract papers more often than those with lower impact factors. It’s not clear why. It could be that these prominent periodicals have more, and more careful, readers, who notice mistakes. But there’s another explanation: Scientists view high-profile journals as the pinnacle of success — and they’ll cut corners, or worse, for a shot at glory.

Actually, the explanation seems pretty straightforward to me: even if the incentive to use fraudulent means to get a paper into a top-tier journal is high, so is the incentive to falsify high-impact papers. So yes, someone can get away with this, but the risk of getting caught increases with the impact of the paper. I suspect that many would prefer to catch this kind of thing ex ante, but ex post is much better than not at all. Or, to be all political science-y about it, fire alarms are more efficient than police patrols in detecting this sort of bad faith error.

The article also misses some recent changes in the way many of the social sciences disciplines are trying to be more transparent about data and methods. The easier it is for other researchers to replicate findings, the narrower the window of fraud opportunities. Twenty years ago, the situation was grim enough for leading lights of the discipline to call for change.. As recently as a decade ago, it was possible to publish statistical or experimental papers in social science journals without making available any data whatsoever beyond the tables and figures in the paper itself.

Now, however, most top-tier journals have submission guidelines similar to what the American Political Science Review states here:

If your manuscript contains quantitative evidence and analysis, you should describe your procedures in sufficient detail to permit reviewers to understand and evaluate what has been done and — in the event the article is accepted for publication – to permit other scholars to replicate your results and to carry out similar analyses on other data sets. With surveys, for example, provide sampling procedures, response rates, and question wordings; calculate response rates according to one of the standard formulas given by the American Association for Public Opinion Research, Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. For experiments, provide full descriptions of experimental protocols, methods of subject recruitment and selection, payments to subjects, debriefing procedures, and so on. In any case involving human subjects, the editors may require certification of appropriate institutional review and/or conformity with generally accepted norms….
In addition, authors of quantitative or experimental articles are expected to address the issue of data availability. You must normally indicate both where (online) you will deposit the information that is necessary to reproduce the numerical results and when that information will be posted (such as “on publication” or “by [definite date]”). You should be prepared, when posting, to provide not only the data used in the analysis but also the syntax files, specialized software, and any other information necessary to reproduce the numerical results in the manuscript. Where an exception is claimed, you should clearly explain why the data or other critical materials used in the manuscript cannot be shared, or why they must be embargoed for a limited period beyond publication.

(The American Economic Review has similar guidelines.)

The 2014 TRIP survey of U.S.-based international relations scholars suggests that international relations scholars, at least, are starting to comply with data availability requests. Less than 10 percent of respondents who generated new quantitative data said that they would not make it available to others; more than 70 percent of respondents had already made their data available to others.

Seventy percent is not a hundred percent, and an informal poll of colleagues suggests that a lot of kinks remain in the system. Ohio State University political scientist Bear Braumoeller told me: “these changes are very recent. I’ve had an RA working on replications of studies done in the past 10 years, and the frequency with which he is unable to replicate even very recent ones is appalling.” And the University of Rochester’s Hein Goemans pointed out to me that even journals that require authors to deposit replication files often only have broken or empty links.

Furthermore, there are legitimate conflicts of interest regarding some areas of data transparency. Simply making the raw data from surveys freely available is a nonstarter: there’s a reason that Institutional Review Boards exist, and one of them is to ensure that the privacy risks to survey respondents, interviewees, and experimental subjects are kept to a minimum. The goal should be to replicate, as Jonathan Ladd notes. Furthermore, the costs of facilitating transparency in qualitative work is considerably higher than in quantitative or experimental papers. If this shift towards transparency disadvantages qualitative scholarship, that’s a problem. Finally, it’s also true that building data sets involves massive fixed costs. The researchers who painstakingly build them — particularly the untenured — deserve some degree of temporary monopoly over their data, in order to facilitate their chance to publish.

Still, the impression the Times story gives is that the social sciences have done nothing to address the prospect of fudging or falsifying data to earn a publication or three. And that’s not really accurate. As political science research has had greater real world impact, a parallel trend has been towards greater ability to replicate existing empirical work. We’re progressing towards a situation where most political scientists are transparent about their methods and data most of the time. That’s not perfect, and it won’t stop dedicated con men in their tracks, but it’s a significant improvement over even the recent status quo.

The LaCour episode will, hopefully, prod things further in that direction. As will the inevitable Hitler-in-the-bunker video:

New revelations are already emerging about LaCour and the myriad debates on What It All Means will continue. It would be unfortunate, however, if this story left the impression that political science has been static on these issues. It has not.