PREDICTING EMPLOYMENT NOTICE PERIOD WITH MACHINE LEARNING: PROMISES AND LIMITATIONS.

• Date: 01 June 2020
• Author: Dalian, Samuel,Touboul, Jonathan,Lain, Jason,Sfedf, Dan
• Published date: 01 June 2020
• Author: Dalian, Samuel

Rapid advances in data analysis techniques--particularly for predictive algorithms--have opened the door for radically new perspectives on legal practice and access to justice. Several firms in North America, Asia, and Europe have set out to use machine-learning techniques to generate legal predictions, raising concerns regarding ethics, reliability and limits on prediction accuracy, and potential impact on case law development. To explore these opportunities and challenges, we consider in depth one of the most litigated issues in Canada: wrongful termination disputes and, more specifically, the question of reasonable notice determination. Beyond the thorough analysis of this question, this paper is also intended to act as a road map for non-technicians (and especially lawyers) on the application of artificial intelligence (AI) methods, illustrating both their potential benefits and limitations in other areas of dispute resolution.

To achieve these results, we first created a large dataset by annotating historic cases related to employment termination. This dataset proved useful for assessing the predictability of reasonable notice of termination, that is, the accuracy and precision of AI predictions. In particular, it helped identify the degree of inconsistency in notice period cases, incidentally exposing the limitations of legal predictions. We then developed predictive algorithms to estimate notice periods based on details of the employment period and investigated their accuracy and performance. Moreover, we thoroughly analyzed these algorithms to better understand the judicial process, and in particular to quantify the weight and influence of case-specific features in the determination of reasonable notice. Finally, we closely analyzed cases that were poorly predicted by the algorithms to understand the judicial decision-making process and identify inconsistencies--a strategy that will ultimately yield a deeper practical understanding of case law.

This project will open the door to the development of an access-to-justice project and will provide users with an open-access platform for employment legal help (www. MyOpenCourt.org).

Les progres rapides des techniques d'analyse de donnees --les algorithmes predictifs en particulier--ont ouvert la porte a des avenues radicalement nouvelles en matiere de pratiques juridiques et d'acces a la justice. Plusieurs cabinets d'avocats d'Amerique du Nord, d'Asie et d'Europe ont entrepris d'utiliser des techniques d'apprentissage statistique (machine learning) pour predire et generer des conclusions d'ordre juridique, ce qui souleve des preoccupations concernant l'ethique, la fiabilite et les limites de la precision de ces conclusions, ainsi que leur impact potentiel sur le developpement de la jurisprudence. Pour explorer ces possibilites et ces defis, nous examinons en profondeur l'une des questions les plus litigieuses au Canada : les licenciements abusifs et, plus particulierement, la question de la determination du preavis raisonnable. Au-dela de l'analyse approfondie de cette question, cet article se veut egalement une feuille de route pour les non-techniciens (et surtout les avocats) sur l'application des methodes d'intelligence artificielle (IA), illustrant a la fois leurs avantages potentiels et leurs limites dans d'autres domaines de la resolution des litiges.

Pour atteindre cette fin, nous avons d'abord collige un vaste ensemble de donnees en annotant les cas historiques de congediement injustifies. Cet ensemble de donnees s'est avere utile pour evaluer la previsibilite d'un preavis raisonnable de licenciement, c'est-a-dire la precision des predictions de l'IA En particulier, cette approche permet de determiner le degre d'incoherence et de variation des cas de preavis, en exposant incidemment les limites de ses conclusions legales. Nous avons developpe des algorithmes predictifs afin d'estimer les delais de preavis en fonction de la duree de l'emploi et avons etudie leur precision et leur performance. De plus, nous avons procede a une analyse approfondie de ces algorithmes afin de mieux comprendre le processus judiciaire, et en particulier de quantifier le poids et l'influence des caracteristiques propres a chaque affaire dans la determination du preavis raisonnable. Enfin, nous avons analyse de pres les cas mal predits par les algorithmes d'IA afin de mieux comprendre le processus decisionnel judiciaire et d'en determiner les incoherences--une strategie qui permettra en definitive d'approfondir la comprehension pratique de la jurisprudence.

Ce projet ouvre la voie au developpement d'un projet d'acces a la justice a plus grande echelle et fournira aux utilisateurs une plateforme en libre acces d'aide juridique en droit du travail (www.MyOpenCourt.org).

Introduction I. Method of Data Collection A. Legal Text Milling B. Composition of the Database C. Inherent Fluctuations in Notice Calculation II. Predictability of Reasonable Notice Calculation A. Selecting Relevant Criteria and Identifying Correlated Attributes B. Predicting Notice Periods Based on Length of Service C. Beyond Employment Duration: Adjustments According to Other Bardal Factors 1. Prediction Error and Adjustments with Additional Factors 2. Refining die Prediction: Cumulative Role of Other Bardal Factors a. C haracter of Employment b. Employee Qualification c. Age d. Year of Judgment Mid the Evolution of the Case Law e. Availability of Similar Employment f. All Bardal Factors Combined for Prediction 3. The Failure of More Advanced Machine-Learning Algorithms to Reduce die Prediction Error a. Decision Trees and Random Forests b. Neural Networks III. Error Analysis: Machine Learning and Inconsistency in the Calculation of Notice Conclusion Introduction

Machine learning, or artificial intelligence (AI), relies on computing capabilities to analyze and extract patterns within vast amounts of information, and to derive statistical models from these patterns to predict associations between features and outcomes. For the field of law, these techniques offer opportunities for developing predictive tools capable of evaluating the odds of winning cases (1) or estimating damages. (2) From a theoretical standpoint, the application of AI in law has the potential to shed new light on how legal decisions are made by illuminating the evolution of case law and the consistency (and predictability) of judicial decisions. (3) Moreover, providing access to efficient AI systems could create invaluable tools for improving access to justice, currently a prominent issue in North America. (4) From a practical standpoint, AI systems would provide critical information for litigants insofar as determining litigation outcomes is key in helping them decide whether they should settle or litigate, by enabling them to identify their Best Alternative to a Negotiated Agreement (BATNA). (5) It is important to note, however, that while the prospect of applying AI in the legal field has come with high expectations, it has also raised important ethical concerns, (6) in particular regarding the reliability (7) and explicability of predictions. (8)

Accordingly, this piece is intended to present an overview of the application of AI and statistical methods to legal data, and more particularly to the determination of reasonable notice for workplace dismissals. It will build on previous statistical studies that have attempted to decipher judicial logic regarding the determination of notice. (9) That said, it is important to note that machine learning and statistical techniques are not applicable to every sub-field of law. The most promising areas are legal questions in which the court's determination lends itself to identifying a discrete set of factors, and for which sufficient historical data exist. In this regard, notice calculation seems particularly well suited to the application of data science insofar as it is a fact-driven area that relies on a set of factors defined by a landmark case (Bardal). (10)

We have therefore explored in depth the predictability of notice determination in order to give insight into the judicial process. To this end, we have constructed an extensive dataset of wrongful termination employment cases in Canada. The database is structured around legally relevant factual predictors--the features that judges take into consideration in determining notice and that are available prior to the formulation of any judgement. This extraction and normalization of features from legal texts into a predefined structured data source opens the way to data analysis and the conception of predictive algorithms. For each case, we gathered a variety of factors, including all the well-known Bardal factors--namely character of employment, age, duration, availability of other employment, experience, and qualifications--as well as external factors, such as the name of the judge, the employment industry, and the gender of the employee.

The project sought to answer a set of predictive and descriptive questions. Specifically, we focused on three main sets of questions: (1) What is the predictability of notice periods? How precise and accurate can predictions be? What is the percentage of total outcomes predicted correctly? What is the uncertainty of these predictions? (2) Can AI help identify the relative weight of the factors taken into consideration by judges and the way each factor contributes to the notice period determination? and (3) Can AI help to identify inconsistencies in case law?

This paper is embedded into a wider endeavour aimed at developing an open access prediction project for employment litigation in Canada, with the ultimate goal of improving access to justice for all individuals. (11) The project is conducted by the Conflict Analytics Lab, a consortium focused on the application of AI to dispute resolution, established in 2018 as a forum for collaboration between academic institutions and industry partners...