Statistical significance is valid, in the right hands, but in the wrong hands it does nothing but undermine trust in science. And it is almost always in the wrong hands.
Want to believe a weedkiller can cause cancer or that liberals will have prettier daughters than conservatives? You don't need science for those to get into a peer-reviewed journal, you only need statistical significance. Dredge hard enough, and you can link anything to anything because using modern technology you can detect anything in anything, and if you believe in a world where risk does not matter, only a molecule of any hazardous substance is the same as bathing in it, 'the dose does not make the poison' and you can put out a press release touting that chewing gum is killing the planet and get it written up by a journalist.
Such abuse of data, torturing it until it confesses to what you want, is why a group writing in Nature asked to "retire statistical significance", me included.
Credit:
V. Amrhein et al.
How did we get here?Because Reagan loved science
I argue that President Ronald Reagan caused the problem. He loved science more than any President since Jefferson, he believed that science was one of the few things government should invest in. And he intended to make sure more of it got funded. But he wanted a metric for performance. His team came up with publication in a peer-reviewed journal as the standard. Journals had been around long before that but they were not mentioned in the beatified tones they are now. When Krebs discovered the secret to what is now called the Krebs cycle his paper was rejected by Nature because they already had a backlog of articles and he sent it to Enzymologia instead. It made no difference in the work, he still got a Nobel prize, because then journals were just a place where academics put things they wanted others to read. They were not lauded by media as a gold standard for being science fact.
After Reagan boosted science funding and told agencies peer-reviewed papers were required, some learned to game the system by passing peer review using arcane tools like statistical significance. Statistical significance is easy to manipulate and hard to debunk because a lot of underlying data can't be easily replicated. Lack of legitimate peer review is how papers claiming a weedkiller turns frogs gay or that hurricanes with female names are more dangerous got media attention. It's why every month Harvard School of Public Health produces some new nonsense claiming food diaries mean they can declare a new Miracle Diet or Scary Chemical, without doing any science at all.
I'm a statistician. My motto is 'I haven't read your paper yet but I'm virtually certain your methods are flawed&your results are wrong.'
— Stephen John Senn (@stephensenn) April 9, 2014
That is my take. However, a recent article says long before that it was a social science quest to look as legitimate as the physical and life sciences that led to them embracing P-hacking, HARKing (Hypothesis After Results Are Known) and general data dredging and then later it bled into epidemiology to such an extent that our own taxpayer-funded National Institute of Environmental Health Sciences is among the world's leading culprits in public disillusionment toward science.
They have a point. Social sciences want to feel like science but often don't.(1) They want to express certainty about behavior even if they are only doing surveys of white undergraduate psychology students. The secret sauce of significance testing interesting or even powerful effects.
Embrace uncertainty - because certainty is where good data go to die
We weren't the first to ask to have this nonsense stop. Bruce Bower notes that Dr. Geoffrey Loftus, editor of Memory&Cognition, tried in 1993 to get scholars to stop claiming experimental results were real just because they are statistically significant.
Like statistically significant hooey about a molecule of formaldehyde being an endocrine disruptor, if such 1990s nonsense has lasted this long, it is probably here to stay.
NOTE:
(1) Science has a theoretical foundation. No social science does. Yet scientists who otherwise engage in critical thinking often adorn those fields in a halo for reasons no one can fathom. I was once lambasted for criticizing an anthropology 'paper' that was nothing but an informal, uncontrolled survey on the Web. Even the simple arithmetic in it was wrong. Yet the conclusion was 'men are jerks' and when I noted the flaws, the progressives who dominate the life sciences attempted to shout me down as a male jerk. Maybe she is a well-liked insider or has some issues that others know about that meant protection, but beyond that I have no guess except that politics infects nearly everything. The paper was still garbage and that is why the author had to pay to publish it.
