Under the heading, “Some evidence on the (non)robustness of political science results to replication,” Allen Dafoe writes:
In a recent blog entry you wrote “I have a general feeling (supported by data collected by researchers such as Uri Simonsohn) that lots of shaky research gets published, and I have a lot of anecdotal evidence, but that’s about it.”
I wanted to share some slightly more systematic anecdotal evidence. In this article, Science Deserves Better [in the journal PS: Political Science & Politics], in the section “Robustness of Published Results to Replication”, I summarize a survey I did of political scientists about their experience replicating other work. The primary takeaway is that about 50% of respondents reported that the results of the study they tried to replicate were not robust. This is probably an underestimate of the non-robustness of published research in political science, because responses to this question were biased towards studies for which replication materials are available, which are probably higher quality studies. (Wicherts, Bakker, and Molenaar (2011) report that willingness to share data is positively associated with the strength of the evidence and the quality of the reporting of statistical results.)
All that is published does not replicate;
Often have you heard one state:
Many a man his work create
But by outside to inflate:
E’en in APSR do errors infiltrate.
Had you been as wise as great,
Young in limbs, in judgement late
Your answer had not been in plate
Fare you well, some other date.