Almost two decades ago, the Sokal affair occurred. A physicist tired of philosophical pseudoscientific gibberish and set out to write a paper of barely comprehensible jargon and get it in a peer-reviewed journal. It dutifully espoused philosophy as superior to science and then used science terminology for legitimacy. It was, in short, the perfect example of what was wrong.
Paul Feyerabend is dead and so is postmodernism, now you can just generate rubbish in an automated postmodernism generator and dispense with being creative, but its techniques are still with us. In psychology, Feyerabend's belief that any crazy thing is possible is being shown true.
Many psychologists, frustrated by high-profile statistical snake oil, see the issue and are frustrated by it. In 2011, a psychologist named Joseph P. Simmons and colleagues set out to use real experimental data to prove an impossible hypothesis: that listening to The Beatles’ “When I’m Sixty-Four” makes people younger. They recruitd a small sample of undergraduates to listen to either The Beatles song or one of two other tracks, then administered a questionnaire asking for a number of random and irrelevant facts and opinions—their parents’ ages, their restaurant preferences, the name of a Canadian football quarterback, and so on. The result: By strategically arranging their data and carefully wording their findings, the psychologists “proved” that randomly selected people who hear “When I’m Sixty-Four” are, in fact, younger than people who don’t. This p-hacking, torturing the data until it says what you want, sounds ridiculous but it is common.
Using surveys, almost anything is possible. It isn't just psychology, epidemiology now suffers from retrospective statistical fishing expeditions, and forget sociology. If you read about some popular chemical that is supposedly implicated in X, you can be confident someone was dunking a critter in a bucket of chemicals until it got a tumor - and then they threw out all of the results that didn't match their belief.
Scientists have long been guilty of cutting messy corners - Galileo did it, perhaps Mendel did it. But psychology, especially fields like social psychology, are accepted by almost no one except journalists in mainstream media and other social psychologists because the issue is that severe.
It's a crisis of confidence because few care in those fields care about becoming more scientific, they just want to say something edgy and clever enough to get on a TED talk.
Jerry Adler at Pacific-Standard has a terrific overview of the problem: The Reformation: Can Social Scientists Save Themselves?
