The use of A.I. in the Employment Tribunals

AI in the Employment Tribunals

The legal landscape is undergoing a monumental shift, propelled by the relentless march of Artificial Intelligence. At the forefront of this revolution are Large Language Models (LLMs), which are not merely augmenting but actively redefining how legal information is processed and leveraged. A pioneering study by Joana Ribeiro de Faria, Huiyuan Xie, and Felix Steffek called “Information extraction from employment tribunal judgments using a large language model” “illuminates this transformation, demonstrating how GPT-4, one of the most advanced LLMs, achieves "high accuracy in legal information extraction" from judgments of the UK Employment Tribunal (UKET). This isn't just academic theory; this research holds "significant implications for legal research and practice," promising to "revolutionise the way legal information is processed and utilised".

Unlocking Vast Legal Knowledge with A.I. in the Employment Tribunals

Court judgments are incredibly dense repositories, brimming with intricate case specifics and the foundational reasoning behind judicial rulings. Traditionally, gleaning crucial information from these documents has been a labour-intensive, manual process. However, as Ribeiro de Faria, Xie, and Steffek assert, LLMs like GPT-4 are making automatic information extraction "increasingly feasible and efficient". The UKET, with its vast collection of publicly accessible judgments, presented an ideal testing ground for their innovative exploration.

The researchers structured their investigation around two core extraction challenges:

Comprehensive General Extraction

This task involved GPT-4 meticulously extracting eight crucial aspects that are invaluable to both seasoned legal professionals and the general public alike:

• The facts of the case.

• The claims made by all parties.

• References to legal statutes, acts, regulations, provisions, and rules, including specific numbers and articles.

• References to precedents and other court decisions.

• The general case outcome.

• The general case outcome is summarised into one of four distinct labels: "claimant wins," "claimant partially wins," "claimant loses," or "other".

• Detailed order and remedies.

• The essential reasons for the decision encompass both procedural and substantive elements.

• Focused Analysis for Predictive Power - A more concentrated effort zeroed in on facts, claims, and the labelled outcomes. This targeted extraction was explicitly designed to lay the groundwork for a tool capable of predicting the outcome of employment law disputes.

GPT-4's Astounding Accuracy

The results of Ribeiro de Faria, Xie, and Steffek's stringent manual verification process are nothing short of extraordinary. GPT-4 demonstrated exceptional accuracy in pulling legal information from the complex UKET judgments.

• Remarkably, it achieved a perfect 100% accuracy in extracting references to legal statutes and precedents.

• Near-perfect accuracy was recorded for claims (0.981) and general case outcomes (0.996).

• Even for seemingly more intricate tasks like extracting detailed case outcomes and formulating reasons for decisions, GPT-4 maintained an impressive accuracy of 0.996 for both.

• While slightly lower, the accuracy for facts (0.942) and general case outcomes summarised into one of four labels (0.912) remained very strong, still exceeding 0.9.

This "overall high accuracy" underscores GPT-4's immense potential in processing vast quantities of legal documents efficiently. The researchers attribute this success, in part, to their use of rigorous prompt engineering techniques, iteratively refining instructions to guide GPT-4 to extract precise and contextually relevant information. For example, they specifically instructed GPT-4 on how to handle multiple parties or ensure detailed order extraction.

Predictability and "Procedural Truth"

Beyond mere extraction, the study by Ribeiro de Faria, Xie, and Steffek also rigorously explored the suitability of this extracted data for a subsequent case outcome prediction task. Their analysis carefully distinguished between two critical concepts:

• Procedural Predictability: A system that accounts for all procedural actions and events of the parties involved.

• Substantive Predictability: A system that deliberately excludes procedure-based decisions, focusing solely on the merits of the legal dispute.

Of the 260 cases analysed, approximately half (124 cases, or 47.7%) were deemed suitable for a computational prediction task, with longer judgments proving to be more informative for this purpose. This finding suggests substantial potential for training and evaluating powerful predictive models using LLM-curated data.

However, the authors also critically acknowledge inherent limitations. A key insight is that even in "substantive" judgments, outcomes can be significantly influenced by procedural factors, such as a respondent's failure to present evidence or respond (as per Rule 21 of the UKET Rules of Procedure). This introduces the concept of "procedural truth," a familiar concept in legal education, which can diverge from the actual truth of a dispute. Furthermore, Ribeiro de Faria, Xie, and Steffek point out that the current method of extracting facts directly from judges' final decisions might inadvertently introduce biases, as judges already possess knowledge of the case outcome when they draft their judgments.

A More Accessible and Equitable Legal Future for AI in the Employment Tribunals

The implications of this groundbreaking research are far-reaching. By contributing to the development of "accurate and open dispute resolution systems," Ribeiro de Faria, Xie, and Steffek aim to directly address the "current knowledge imbalance". In today's legal landscape, employers often benefit from superior access to legal information and advanced systems compared to employees. The creation of publicly accessible predictive systems, offered at little to no cost, could profoundly "level the playing field," fostering greater equity in employment law disputes.

This pioneering work underscores that LLMs are not merely tools for enhancing efficiency; they are poised to "enhance legal research and practice, improving case management and informing judicial decision-making processes". As LLMs continue their rapid evolution, their integration into the legal profession is set to "herald a new era of legal AI where LLMs play a critical role in shaping the future of the legal profession and research". The study by Ribeiro de Faria, Xie, and Steffek not only provides vital insights but also releases an accurate and publicly available dataset (as part of the Cambridge Law Corpus), ensuring the responsible and beneficial development of these robust legal A.I. in the Employment Tribunals.