The Washington PostDemocracy Dies in Darkness

No, science’s reproducibility problem is not limited to psychology


Scientists with pitchforks, torches and battering rams are threatening to storm the SOS headquarters in the wake of our item Thursday headlined "Many scientific studies can't be replicated. That's a problem."

Their message: It's just the psychologists! Not us! Our work is super-reproducible! Don't lump us in with those squishy psycho-babbling crypto-phrenologists! (I'm paraphrasing liberally.)

The dyspeptic scientists have a point: The study published Thursday in the journal Science looked only at psychology experiments. Our headline generalized the issue. We also originally ran a stock photo of a couple of people doing a chemistry experiment; we have since swapped that out with a photo of lead author Brian Nosek.

Although I wouldn't nominate that headline for a Pulitzer, I think I'll stick with it, because the issues raised by this study go beyond psychology. Other sciences have had reproducibility issues as well. See our story from earlier this year: "Reproducibility at last: The new scientific revolution."

The leaders of the new study picked psychology to scrutinize because they are themselves psychologists. Their concerns -- and the concerns of other reformers such as John Ioannidis, author of the famous paper "Why most published research findings are false" -- go to the incentive systems and professional pressures that affect all scientists. It's not just psychologists who are under pressure to find significant results that can be published in prestigious journals. It's not just psychologists who engage in data dredging.

[Why science is so hard to believe]

And by the way: If you made a list of major science retractions, controversies and embarrassing episodes in recent years, you'd see a full range of disciplines in the mix. From our earlier story:

Consider “cold fusion”: In 1989, two scientists claimed to have achieved nuclear fusion at room temperature, previously considered impossible. It was a bombshell announcement — but no one else could replicate their work. Cold fusion didn’t take off because mainstream scientists realized it wasn’t real.
A more recent case involved “arsenic life.” In 2010 a paper in Science suggested that a bacterium in Mono Lake, Calif., used arsenic instead of phosphorus in its genetic code and represented a new form of life. Rosemary Redfield, a scientist, cast doubt on the conclusion, and other researchers couldn’t replicate the finding. The consensus is that it was a misinterpretation.
In early 2014, the scientific world was rocked by a tragic case in Japan. A young scientist, Haruko Obokata, claimed to have found evidence for a phenomenon called “STAP,” stimulus-triggered acquisition of pluripotency — a way to manipulate ordinary cells to turn them into stem cells capable of growing into a variety of tissues.

Last year we ran a ton of stories about the BICEP2 experiment. That was physics at its heroic best: They found, apparently, gravitational waves from the dawn of time. It made such a huge splash that people were talking about Nobel Prizes in the offing. But the "discovery" got walked back in a big way. It may yet hold up on further observation, but that was a case of scientists getting ahead of their facts and holding a premature triumphant press conference.

[Planck flings dust at BICEP2: No discovery of gravitational waves from the big bang]

But now then: Are psychology experiments more likely than, say, chemistry experiments or physics experiments to have issues with reproducibility? Ioannidis told me yes, probably so.

“I think on average physics and chemistry would do better. I don’t know how much better," he said.

Maybe someone should try to constrain the differences between the physical sciences and the social sciences. Perhaps physics and chemistry will do their own version of the reproducibility study?

Read more:

Hundreds of scientists ask Science to stop publishing a smorgasbord of stereotypes

Two scientific journals accepted a study by Maggie Simpson and Edna Krabappel

‘Fraudulent’ peer review strikes another academic publisher -- 32 articles questioned

Sexism in science: Peer editor tells female researchers their study needs a male author