Like many, science journalist Carl Zimmer has become concerned about the ability of science to correct itself; at least on a culturally acceptable timescale brought about by instant media publication of studies.
Arsenic life is one example, though clearly the system worked. While the article got published it was criticized quite rapidly and bloggers have caught stealth Creationist papers that peer reviewers missed too.
Replication failures have become an increasing concern; Greg Critser recently wondered if this was limited to the life sciences and if perhaps an over-dependence on 'out of the box' technology is leading to "artifactual" results.
Life from arsenic is one thing; it gave science bloggers something to be sanctimonious about but it wasn't hurting anyone. Obviously cancer studies are a much more serious issue. It isn't fraud, in most cases, when studies cannot be replicated. People make mistakes - but the look elsewhere effect shows us mistakes are easier to make when you want to find a result. It isn't just the life sciences. Climate scientists are willing to believe some farmer's thermometer reading from 25 years ago if it matches what they expected to see but if it did not, they are more skeptical. Physicists are usually so concerned about the look elsewhere effect they are downright afraid to announce discoveries - on July 4th they announced the Higgs boson with a far higher degree of confidence than most other science has and still there was a debate about whether to call it a discovery and wait until later. Journalists went ahead and invented teleportation, regardless of the hesitance of scientists.
Zimmer discusses one biologist who was unable to replicate a study and how it led her to create the Reproducibility Initiative, which is out to verify results and provide a sort of cultural reward for it.
Will that sort it work? Some journals have agreed to add a badge to all papers that have been replicated through the Reproducibility Initiative. "Think of it as a scientific Good Housekeeping seal of approval. It shows that you care enough about the science—and are confident enough in your own research—to have someone else see if it holds up," Zimmer writes.
Good Scientist! You Get a Badge. By Carl Zimmer, Slate
