An influential psychological theory, borne out in hundreds of experiments, may have been debunked, writes Daniel Engber, and then "how can so many scientists have been so wrong?"

It sounds like a clarion call for concern about the scientific method. Yet it isn't. Instead, it was something that was rather predictable to anyone who knows how most modern (post-1960s) psychology experiments are conducted - and that in most cases they never are conducted, at least not in the way people who took science in school understand experiments.

There are many reasons why public understanding (and acceptance) of science is not where we might like it, and chief among them is the willingness of journalists to let anyone call themselves scientists - and to use the word "theory" even when it clearly is not. Political science? Well, it has science in the name. Economics? Sure, the New York Times pays a guy to blog about that, so it must be science. And then there is psychology.

As you recall from high school, science has a theoretical foundation. What does psychology lack? A theoretical foundation. Yet what do you see over and over in newspaper and magazine articles about psychology? The word "theory" about things that are perhaps a hypothesis but more often correlational speculation. Journalists love to write about correlation papers because it is easy and often topical. As sure as you are reading this, next February there will be a whole raft of psychology papers on the 'science of romance' in time to make the Washington Post before Valentine's Day - and leading the charge will be Social Psychology, which has seen numerous instances of fraud and ridicule and outright error - and then a lack of anything resembling replication, like the instance mentioned in the first paragraph, which has to do with "ego" influence. 

Psychologists don't have a theoretical foundation in the way that scientists understand it (think gravity and evolution) but they nonetheless use the word theory a lot, and one of the ways they have used it was in creating the bedrock in one flavor of psychology - ego deflation.

What did biologists know about the social psychology experiment? That there’s no way drinking a glass of lemonade would make a difference in how the brain functions. People may think it does - that weird guy who runs the CrossFit video empire and thinks Coke causes diabetes but Pepsi does not certainly has all kinds of odd beliefs about nutrition, but he has the weight of evidence on his side. Who created the weight of evidence? All those researchers who specialize in the field  

It's understandable why psychologists in the field would have confirmation bias, but why have so many journalists been fooled for so long? Engber answers it when he writes:

The effect has been recreated in hundreds of different ways, and the underlying concept has been verified via meta-analysis. It’s not some crazy new idea, wobbling on a pile of flimsy data; it’s a sturdy edifice of knowledge, built over many years from solid bricks.

He's not alone, statistical analyses have been stumping people for a long time, so much so that the American Statistics Association disavows p-values as a tool for accuracy. And then when it comes to scholars, sometimes they are just wrong and sometimes they just don't see their own confirmation bias, but sometimes they are going out of their way to manufacture an effect.

I'm a statistician. My motto is 'I haven't read your paper yet but I'm virtually certain your methods are flawed  &  your results are wrong.'

— Stephen John Senn (@stephensenn) April 9, 2014

Meta-analyses don't verify anything - I can do an unweighted random effects meta-analysis and create organic food that is more nutritious or pregnant men. Nor do 3,000 citations. All sciences have some bandwagon effect but areas like social psychology are overrun with it. Anything that gets mainstream media attention suddenly gets overrun with papers claiming to replicate it.

As the actual good science journalist Gary Taubes put it in an interview here:

I used to joke with my friends in the physics community that if you want to cleanse your discipline of the worst scientists in it, every three or four years, you should have someone publish a bogus paper claiming to make some remarkable new discovery — infinite free energy or ESP, or something suitably cosmic like that. Then you have it published in a legitimate journal ; it shows up on the front page of the New York Times, and within two months, every bad scientist in the field will be working on it. Then you just take the ones who publish papers claiming to replicate the effect, and you throw them out of the field. A way of cleaning out the bottom of the barrel.

And then p-hacking (keep torturing the data until it gives you the statistical significance you want) and HARKing (Hypothesizing After Results are Known) are really common.  The statistical website FiveThirtyEight even has a tunable program that lets you prove the economy is better or worse if Republicans or Democrats are in office. Partisan journalists have written whole books using social psychology papers to show that their political party is more intelligent, smarter, and more ethical. One psychologist can even show you his political party will have prettier daughters.

None of that can scientifically tell you if a hypothesis is true - what science wants to know - but it can provide a publishable result - what a whole bunch of psychologists of the last 30 years have been more concerned about.

There is good news: As I predicted years ago, young scholars - who went into the field expecting to do science and had to be alarmed the field was mostly surveys of college students and either shoddy or manipulated statistics - are tripping up their elders. Will the young man who tripped up the ego deletion experiment get tenure? If so, then it shows psychology may actually be trying to become science. 

So we will have fewer articles claiming the following
- Women evolved to shop
- Spiritual people are brain damaged
- Conservatives are motivated by fear and cognitively unable to think with complexity
- Liberals are faking it and get more conservative when they drink

A good rule of thumb when reading weak observational studies, psychology surveys and, in modern times, a shocking number of epidemiology papers: "If an estimate is statistically significant, it’s probably an overestimate."