Luck of the Draw III: Using AI to Extract Data About Decision-Making in Federal Court Stays of Removal. (Canada)

• Date: 22 March 2024
• Author: Rehaag, Sean
• Published date: 22 March 2024

Introduction

I. Context

II. Methodology

A. Web-Scraping Using Python

B. Docket and Docket Entry Screening Using Regex

C Docket Entry Categorization and Data Extraction Using GPT-3

D. Docket Level Logic Using Pandas and Final Dataset

E. Data Verification

E Data Analysis

G Limitations

III. Findings

A. Outcomes in Stay of Removal Motions, Overall and by Year

B. Outcomes in Stay of Removal Motions, by Judge Deciding the Motion

C. Outcomes in Stay of Removal Motions, by City Where Judicial Review Was Filed

D. Outcomes in Stay of Removal Motions, by Type of Underlying Application

E. Variance in Judge Stay Grant Rates Is Not Fully Attributable to City, Year, and Case Type

IV. Discussion & Conclusions

A. Variance in Stay Grant Rates Across Judges

B. Variance in Stay Grant Rates Across Cities

C New Computational Legal Research Tools and Bulk Access to Court Materials

Appendix A: Example of Online Federal Court Docket

Appendix B: Statistical Analysis

Introduction

In Canada's deportation regime, the final procedure available to stop removal once all other recourses have been exhausted is to apply to the Federal Court for a stay of removal. These applications are typically heard on an expedited basis, days or even hours before individuals are scheduled to be put on a plane.

Everyone involved in stays of removal, including lawyers for individuals and the Department of Justice, as well as the Federal Court justice deciding the issue, are confronted with challenging tasks. They have little time to prepare or review materials, hearings tend to be short, and decisions must be made quickly. Often, individuals who are about to be removed are experiencing a crisis--many in immigration detention--with all the attendant difficulties that poses. Levels of stress for counsel can be high, which can make it difficult to secure counsel, as many lawyers are understandably disinclined to take on these cases. (1)

These challenges combine with the enormous stakes of stay of removal decision-making. Individuals applying for stays assert that their removal would result in irreparable harm, sometimes including persecution, torture, or even death. (2) Recent jurisprudence has also placed increasing weight on stay proceedings as a site for ensuring that the deportation regime complies with the Canadian Charter of Rights and Freedoms (Charter)? This means that the stakes are high, not just for individuals, but also for the legal system.

While some immigration and refugee law procedures have attracted substantial scholarly attention, there is comparatively little research on stays of removal. This is likely due to methodological challenges. Until recently, decisions in Federal Court stays of removal were mostly unpublished, which made traditional legal doctrinal analysis difficult. Happily, in 2018, the Federal Court began publishing most stay decisions. Several scholars are in the process of completing research projects on this new body of published stay decisions. (4)

This article offers a quantitative overview of all Federal Court stay of removal decision-making over the past ten years to help provide context for ongoing research on the new body of published stay decisions. It does so through computational legal research methods that build on two prior research projects about the luck of the draw in Federal Court decision-making. (5) Specifically, the article leverages a form of computational natural language processing using a large language model machine learning process (GPT-3) to extract data from online Federal Court dockets. It then reviews patterns in outcomes in thousands of stay of removal applications identified through this process.

The patterns in Federal Court stay of removal applications revealed through this research appear to be troubling. From 2012 to 2021, the rates at which stays of removal were granted varied dramatically depending on which justice was assigned to hear the case: some justices deciding large numbers of cases granted stays over 80% of the time, while others granted stays less than 10% of the time. The concern is not just with outliers. Rather, there was a wide range in stay grant rates across many justices. In other words, there appears to be a large unexplained variance in stay of removal grant rates depending on which justice decides the application--similar to the findings of large unexplained variance in rates at which different justices granted leave in refugee law applications for judicial review highlighted in the prior two Luck of the Draw studies. (6)

This finding points to the possibility of inconsistent and arbitrary outcomes in high-stakes decision-making. Absent some kind of reasonable explanation for the variance in stay grant rates across justices (e.g., the Federal Court reveals that cases are assigned to particular justices after some sort of screening that would result in some justices hearing, on average, stronger or weaker claims), the Federal Court should take measures to encourage more consistency in stay decision-making. In the meantime, it is risky for courts to rely heavily on stays of removal to ensure that removal complies with constitutional procedural justice concerns because, in practice, this form of time-pressured decision-making appears to generate results that raise concerns about arbitrary outcomes that vary depending on which justice decides the case.

The article also aims to demonstrate how technological development in the field of artificial intelligence--and specifically large language models accessed through simple application programming interfaces--is now sufficiently accessible that legal scholars with modest coding skills can (and should) pursue empirical research projects that would have been cost-prohibitive or technically challenging only a few years ago. There are many concerns about how such technology is being deployed in asymmetrical and rights-limiting ways, especially in the border control setting where this technology has been used to limit access to asylum, to facilitate removals, and to otherwise enhance the power of the state at the expense of (mostly racialized) migrants. However, this article demonstrates that this same technology can be deployed in ways that shift the object of scrutiny from the movement of marginalized people to flaws in human-made legal decision-making, thereby increasing transparency and potentially enhancing access to rights. As this article argues, the main barrier to such rights-enhancing use of machine learning is that access to bulk legal data is currently restricted largely to commercial actors and lawyers representing clients with deep pockets, such as Department of Justice lawyers. Stay of removal decision-making is an excellent example of this problem: the terms of service of the websites where these decisions are published all prohibit bulk and programmatic access--meaning that if one wants to study stay of removal decision-making at scale, one must find methodologies that do not rely on published decisions. This study does so by extracting data from court metadata (specifically online court dockets) that are made available by the Federal Court on a website that does not prohibit programmatic access. So, in addition to making the code used for this project publicly available to assist non-commercial researchers with other similar projects, another output of this project is a large dataset of hundreds of thousands of Federal Court dockets that is being made available for non-commercial researchers.

The key contributions of this article, then, are: (1) to describe a method of deploying machine learning tools to extract legal data at scale from online court metadata in a context where working programmatically with the text of decisions themselves is impossible, and to share the code used for the research so that it can be used by other researchers; (2) to share the data on hundreds of thousands of Federal Court online dockets for use in other research; and (3) to explore patterns in a form of legal decision-making that is high stakes, high volume, and understudied.

The article begins by briefly setting out the law and process for applications for stays of removal for readers who may be unfamiliar with this area of law. Next, the article describes the research methodology used for this study. Then the article sets out the findings of the study. Finally, the article offers several recommendations and conclusions.

I. Context

Canada's deportation regime involves many different processes through which non-citizens may seek to remain in the country. These include refugee claims, pre-removal risk assessments, Immigration and Refugee Board processes to contest inadmissibility, humanitarian and compassionate applications, requests for temporary resident permits, and requests for administrative deferrals of removal--as well as administrative appeals for some of these procedures and judicial review. While any given non-citizen will likely not have access to all these processes, most (although not all) do have access to some kind of process to challenge their removal. In some of these processes, individuals benefit from not being removable while the process is ongoing, but for others, removal can occur even while the matter is pending. (7)

When all the other processes to which an individual has recourse that prevents or delays removal have been exhausted, the final process available is to apply for a stay of removal from the Federal Court. Stays of removal are a form of interlocutory relief. That is, a motion for a stay of removal is not a freestanding application to remain in Canada. Rather, it is a request for an injunction to delay removal pending the outcome of some other process. For example, a person who is not entitled to an automatic stay of removal might seek a stay of removal pending the determination of a judicial review of a refugee decision or pending a judicial review of an administrative deferral of removal request.

The...