Improving Expert Evidence: The Role of Open Science and Transparency.

AuthorChin, Jason M.


  1. Introduction 369 II. From Crisis to Renaissance: A Brief History of the Open Science Movement 373 III. The Evidence-Based Evidence Movement 378 IV. Six Parallel Challenges 382 A. A Flawed Incentive System 382 B. Excessive Flexibility 384 C. Motivated Reasoning 385 D. Epistemological Immodesty 386 E. A Preoccupation with Eminence 387 F. Intentional and Negligent Misbehaviour 388 V. Open Science Lessons for Legal Actors 389 A. Adopting Open Scientific Practices: The Role of Expert Witnesses 390 B. Open Science and the Duty to Fairly Present the Case: The Role of the Prosecution 396 C. Open Science and Expert Evidence Law: The Role of the Gatekeeper 400 VI. Conclusion: Improving Trust and Efficiency in Expert Evidence 407 I. INTRODUCTION

    The way in which science is conducted and communicated is fundamentally changing. Scientists and journals are increasingly adopting practices aimed at making science more transparent, reproducible, and democratic. (1) This article will demonstrate several parallels between this movement in science--the open science movement--and similar trends in expert evidence law. In particular, the genesis of many aspects of the open science movement was the realization that longstanding practices had failed, allowing spurious findings to reach general acceptance. (2) A similar pattern has been observed in several classic fields of expert evidence. (3) These parallels have significant consequences for law--a field where flaws in its truth-determining mechanisms have contributed to grave miscarriages of justice. (4) This article's central thesis is that open science-inspired reforms align with, and further, the ideals of expert evidence: these reforms help produce knowledge that is susceptible to critical evaluation.

    The open science movement responded to the discovery of several results, many previously seen as robust and well-established, that could not be independently reproduced. (5) While these examples could at first be disregarded as outliers, the scientific community has come to acknowledge that they reflect an endemic problem. For instance, large-scale attempts to reproduce established social scientific findings have only succeeded about 40-60% of the time and have reported considerably weaker findings. (6) Reflecting these surprising (non)findings, a survey of 1,576 scientists in the journal, Nature, reported that 90% of those surveyed believed that science had a reproducibility problem. (7) Over 50% of researchers reported having failed to reproduce another's finding. (8)

    These meta-scientific revelations have begun to inform and inspire reform. For simplicity, we use the term "open science movement" to refer to the totality of these developments. The movement, however, is broader than what we will focus on in this article, with an earlier (but ongoing) part of the campaign focused on access to scientific articles (i.e. removing paywalls to publicly-funded research). Rather, we will devote most of our analysis to transparency and openness as ways to improve the rigour of research methods by reducing undisclosed flexibility. This component of the movement has taken on various names in the literature, such as the "replicability" (9) crisis and the "credibility revolution." (10)

    In parallel with the open science movement, the forensic sciences (i.e. science designed to answer legal questions) and scientific evidence in court more broadly have also been subjected to increased scrutiny. (11) Although open science and evidence law have almost never been explicitly linked, (12) many of the issues are remarkably similar. For instance, the challenges in science flow from cognitive biases that focus scientists on the data that confirm their hypotheses at the expense of those that do not (despite both sets of data having equal evidential value). (13) These are the very biases present in expert evidence that have been uncovered by academics, (14) and reports from peak bodies of scientists and jurists convened to address the failures of the criminal justice system. (15) Moreover, both movements are associated with a lack of transparency (16) and a preoccupation with eminence over methodology. (17)

    It is meaningful that mainstream scientists are being accused of many of the same practices that have resulted in wrongful convictions in law. Specifically it means that it is insufficient that expert witnesses be directed to behave like scientists. Instead, they should behave like open scientists. In other words, expert witnesses should be expected to behave as scientists should be expected to behave: candidly sharing the results of research, avoiding appeals to status, and skeptically scrutinizing their own work and that of others. (18) By embracing these norms (rather than simply mainstream ones), expert witnesses can provide evidence that is both more trustworthy and more susceptible to rational evaluation. (19)

    Parts II and III will go on to describe the geneses of the open science movement and the evidence-based evidence movement, respectively. Part IV will analyze these movements, identifying six points of comparison. These similarities suggest mutually applicable reforms--a commitment to transparency and openness can improve the accuracy of both science and expert evidence. Part V then delves into legal reforms. Part VI concludes with two ancillary benefits of open expert evidence: improved trust and efficiency

    Before delving into the substance of our article, we should provide a brief caution. While we will suggest that insights from the open science movement have much to offer fact-finding in court, we note that science and law do not share all of their values. Importantly, law must balance other interests, like procedural fairness, adversarial imbalance, and finality. (20) For instance, courts should be sensitive to the fact that a wellheeled corporate defendant facing a product liability claim would often be expected to have access to more sophisticated, case-relevant scientific evidence than the plaintiff. On the other hand, the criminally accused often face resource constraints, making it difficult to hire a rebuttal expert. As a result, we do not suggest holding all parties to the highest standard of open scientific evidence.

    Rather, as explained in Parts V and VI, openness of the foundations and application of expert knowledge simply results in evidence that is more susceptible to rational evaluation. Indeed, as noted above, the open science movement is sometimes referred to as the "credibility revolution" (21) because the reforms transcend value-laden categorizations of science and non-science, applying across fields of knowledge generation. Similarly, in law, academics have criticized rules that require slippery taxonomies of expert evidence (e.g. those that would give different scrutiny to science as opposed to what a court might characterize as non-science, or put less weight on evidence simply because it was generated for the purposes of litigation). (22) Openness, as we will demonstrate, cuts across quantitative and qualitative disciplines. Put simply, and as the meta-scientific findings demonstrate below, openness makes the expert opinion's strengths and weaknesses more apparent and can therefore promote justice.


    If a team of research psychologists were to emerge today from a 7-year hibernation, they would not recognize their field. Authors voluntarily posting their data. Top journals routinely publishing replication attempts, both failures and successes. Hundreds of researchers preregistering their studies. Crowded methods symposia at many conferences. Enormous increases in sample sizes. Some top journals requiring the full disclosure of measures, conditions, exclusions, and the rules for determining sample sizes. Several multilab replication efforts accepted for publication before any data were collected. Overall, an unprecedented focus on replicability. What on earth just happened? (23) The open science movement was spurred by a surprising number of reports of published studies proving to be irreproducible (another thrust of open science is dedicated to making paywalled scientific journals available to the public, especially when the underlying research was publicly funded). (24) In other words, researchers attempted to recreate the findings of previous studies, but found inconsistent or considerably smaller effects. Such incidences were concentrated in pre-clinical and clinical medical research (25) and psychology. (26) Many fields, however, are struggling with the reproducibility of their findings. (27) These failures to reproduce results inspired large-scale systematic studies (mentioned in Part I), finding that studies published in eminent journals regularly proved irreproducible. (28)

    These demonstrations of systemic problems within science raised difficult questions. Most fundamentally, why were these studies, which carried the indicia of good science (e.g. testing with a low reported error rate and publication in leading journals) (29) nevertheless false positive findings?

    Here, we note that many of these discoveries of false positives were not entirely surprising to some, in particular, sociologists of science, who long warned that science was more human and error prone than most realized. (30) Importantly, however, large-scale replication attempts (in the past, replications of previous work were rare) gave teeth to the sociological concerns and encouraged rigorous meta-scientific research into the precise weaknesses in science. (31)

    One of the most widely-studied of those weaknesses is what has come to be known as "researcher degrees of freedom" or "questionable research practices" (QRPs). (32) These are undisclosed choices that researchers can use to increase their chances of finding a result that meets conventional levels of statistical certainty: "In the course of...

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT