What irreproducible results mean for the law of scientific evidence

• Author: Jason M. Chin
• Pages: 157-166

157

g Summer 2016

ha irreucible esus mean 

he la  cieniic eience

JASON M. CHIN

In 2015, Brian Nosek and several collaborators reported the results of

perhaps the most important scientic study of the year– it was a study

they performed by copying the work of others. That is not as contradictory

as it sounds, or at least it should not be.

While the conventional wisdom is that most scientic ndings have

been vetted and reproduced many times before they reach scientic con-

sensus, the reality is that such rigour is exceedingly rare. Nosek sought

to remedy this situation, wrangling up a group of 270 other researchers

(collectively the Open Science Collaboration, or OSC) and attempting to

redo 100 psychology experiments already published in leading peer-re-

viewed journals.1 The question: Would these re-creations nd the same

results as the initial studies?

Prior to the OSC’s endeavour, the conventional wisdom was that, yes,

published science contains some false positives, but they are a small

minority and quickly identied through a robust self-correction process.

The OSC’s results suggest that this conventional wisdom is wrong –

only 36 percent of the studies in the sample were reproducible. This

nding surprised even the alarmists, who long suggested the number

was closer to 50 percent.2 The OSC’s results also garnered widespread

1 Open Science Collaboration, “Estimating the Reproducibility of Psychological Sci-

ence” (2015) 349 Science 943.

2 John PA Ioannidis, “Why Most Published Findings Are False” 2:8 PLoS Med e124.