AI Empowers Lawyers, A Study
In short, GPT-4 brings low performers up to par, and provides large and consistent increases in speed on various legal tasks - without harming quality.
A quick reminder that Mackrell International (MI) has partnered with AltaClaro to offer an AI prompting course for lawyers – see the AltaClaro PR on the MI site and the attached syllabus for the course for further details. If your firm has interest in joining, contact Keith Heddle (MI Managing Director).
As you should know, even GPT-3.5 has a 'general understanding of the legal domain'. How does that boost human legal analysis? On November 7, we received lower-bound conclusions in "Lawyering in the Age of Artificial Intelligence", the published results of the first randomized controlled trial on legal tasks. In short, GPT-4 brings low performers up to par, and provides large and consistent increases in speed on various legal tasks - without harming quality.
Here are the key points, with pages from the study indicated:
Randomized Controlled Trial Design (pp.13-15)
The study used a randomized controlled trial involving 60 law students from the University of Minnesota. Participants were assigned to complete four typical legal tasks —and randomly assigned to complete two of the four tasks with the assistance of AI (GPT-4). This design aimed to measure the impact of AI on legal analysis quality and speed, controlling for individual skill variations.
Training on AI Use (p.14)
Importantly, before beginning the tasks, all participants underwent training on effectively using GPT-4 for legal tasks. The training covered general AI usage principles and specific applications in litigation and transactional contexts. It aimed to equip participants with the skills to harness AI for legal analysis.
Effect of AI on Performance (pp.4,16-17,30)
Participants reported increased satisfaction when using AI and believed their ability to use it improved throughout the study. The study further closely replicated results from non-legal domain research. It found that AI slightly improved the quality of participants' legal analysis but notably increased speed across all tasks. The greatest benefits were seen among lower-skilled participants, suggesting that AI could be an equalizer in legal services. However, benefits were not uniform across tasks, indicating that discernment is needed in applying AI.
Survey Results and Implications (pp.31-39)
The subjective participant impressions coupled with the objective findings indicate a transformative potential for AI in law. Moreover, it is important to note that this potential may very well be understated considering that AI capabilities continue to rapidly accelerate. The results prompt the authors to speculate widely on potential needs to reevaluate legal education, law practice, judicial work, and client-lawyer interactions in the context of AI technologies.