There is, writes Daniele Fanelli in a recent issue of Nature, something rotten in the state of scientific research—"an epidemic of false, biased, and falsified findings" where "only the most egregious cases of misconduct are discovered and punished." A research fellow at the Institute for the Study of Science, Technology and Innovation at the University of Edinburgh, Fanelli is a leading thinker in an increasingly alarming field of scientific research: one that seeks to find out why it is that so much scientific research turns out to be wrong.
For a long time the focus has either been on industry funding as a source of bias, particularly in drug research, or on those who deliberately commit fraud, such as the spectacular case of Diederik Stapel, a Dutch social psychologist who was found to have fabricated at least 55 research papers over 20 years. But an increasing number of studies have shown that flawed research is a much wider phenomenon, especially in the biomedical sciences. Indeed, the investigation into Stapel also blamed a "sloppy" research culture that often ignored inconvenient data and misunderstood important statistical methods. In a now-famous 2005 paper, Stanford University's John Ioannidis, a medical mathematician, laid down a statistical gauntlet by arguing that most published research findings were false.
"There's little question that the [scientific] literature is awash in false findings—findings that if you try to replicate you'll probably never succeed or at least find them to be different from what was initially said," says Fanelli. "But people don't appreciate that this is not because scientists are manipulating these results, consciously or unconsciously; it's largely because we have a system that favors statistical flukes instead of replicable findings."
This is why, he says, we need to extend the idea of academic misconduct (currently limited to fabrication, falsification, or plagiarism) to "distorted reporting"—the failure to communicate all the information someone would need to validate your findings. Right now, he says, we're missing all the "unconscious biases, the systemic biases ... the practices, mistakes, and problems that hardly ever count as fraud, even though they have a very important—and probably the largest—effect on creating technically false results in the literature."
One particularly challenging bias is that academic journals tend to publish only positive results. As Isabelle Boutron, a professor of epidemiology at René Descartes University in Paris, points out, studies have shown that peer reviewers are influenced by trial results; one study showed that they not only favored a paper showing a positive effect over a near-identical paper showing no effect, they also gave the positive paper higher scores for its scientific methods. And Boutron has herself found extensive evidence of scientists spinning their findings to claim benefits that their actual results didn't quite support.
"We need a major cultural change," says Fanelli. "But when you think that, even 20 years ago, these issues were practically never discussed, I think we're making considerable progress."
