How often have you read a headline like “New Study Shows That Coffee May Cause Cancer”? The problem with such studies is that many are wrong. When other researchers try to replicate the same findings in a subsequent study, the relationship disappears. This is known as the replication crisis in science. It first received widespread attention in 2005 when Dr. John Ioannidis published a paper provocatively titled “Why Most Published Research Findings Are False.”
Despite increased focus and concern in the scientific community, the crisis continues and shows no signs of abating. Drs. Michael Muthukrishna and Joseph Henrich recently published a perspective paper in Nature Human Behaviour in which they claim that persuading scientists to work within a theoretical framework would reduce the problem of replication.
A Quick Explanation
The reasons for the replication crisis are many and complex, including fraud, but one major cause is confirmation bias. To understand this, imagine a journal of luck science. Every week, this journal publishes a paper by recent lottery winners on what method they used to win. Some read the winning numbers in tea leaves, some used a coin flip, while others used relatives’ Social Security numbers.
Out of the millions of people who play the lottery there usually will be one or more winners by pure chance every week. No method is involved. That’s why if you tested the winners’ methods the following week, they would all fail to replicate their success.
The problem is that the luck journal publishes only the winners. The losers won’t publish because failure is not very interesting. This produces a publication bias for random results that happen to be correct by chance.
The Hockey Stick Graph
The most famous case of such a biased finding was a study published in 1999 by Dr. Michael Mann. It showed 900 years of flat temperature followed by 100 years of rapid warming. Its shape quickly led it to be nicknamed the Hockey Stick Graph, and it played a key role in the 2001 U.N. Intergovernmental Panel on Climate Change Third Assessment Report.
It is perhaps the most important graph in the history of science, because it was instrumental in scaring the living daylights out of politicians. Until that study, the climate skeptics were on par with the global-warming alarmists. The science debate was still raging, and no definite winner had been declared. After politicians saw that graph, they were convinced that humans were destroying the climate. Without it, the Paris Climate Agreement and the Green New Deal would be unthinkable.
The problem is that it was a bogus study, constructed in much the same manner as the hypothetical luck journal of lottery winners. Mann had the equivalent of millions of lottery tickets at his disposal, and so he was guaranteed to find whatever result he was looking for.
In 2006, the result was finally settled when highly qualified statisticians led by Dr. Edward Wegman produced a report for Congress showing that Mann’s method could reproduce a graph of any desired shape. It was junk science.
Improved Framework
Although the study by Muthukrishna and Henrich primarily targeted psychology and clinical medicine, their ideas apply equally to climate science, which is characterized by a lack of coherent methodology. [perfectpullquote align=”right” bordertop=”false” cite=”” link=”” color=”” class=”” size=”24″]…“climatology,” no such field of science officially exists.[/perfectpullquote]
Despite the common usage of the term “climatology,” no such field of science officially exists. It is a highly interdisciplinary hodgepodge of intersecting expertise from a multitude of fields, with no overarching theoretical framework. That is why so many researchers resort to vague and vacuous methods like “scientific consensus” to determine truth.
That is also why bogus science like the hockey stick graph could not only easily pass through peer review but also be allowed to influence public policy without proper quality control and independent verification.
As such, the replication crisis runs deeper than anyone may suspect. In this time of fake news and dismal trust in the media, science still is held in high regard by most. But that may not continue into the future.
