Harnessing the power of Large Language Models for
legal document review

The RAG Report: Can Large Language Models be good enough for legal due diligence?

There is a huge opportunity for the application of Generative AI to legal work. At Addleshaw Goddard we have been embracing this new technology over the past two years, building out our understanding and AI capabilities. Following months of research into the validity of Large Language Models and a Retrieval Augmented Generation approach to legal, we are sharing our findings to provide insight into the realities of using Generative AI in law.

This comprehensive research report, which we believe is the first of its kind from a law firm, sets out the work carried out by our Innovation Group to develop and test a robust method of using Large Language Models to review documents in the context of an M&A transaction legal due diligence project. 

Our research includes learnings from developing AGPT, Addleshaw Goddard's internal LLM based solution for chat and document review, the steps being taken to develop an M&A transaction specific platform and the optimisation process taken to get our RAG solution from a performance of 74% accuracy to upwards of 95%.

 

 

 

   

 

 

 

 

Share Print

Some of the key findings

95%

20%

16%

9%

Accuracy

IMPROVEMENT

RECALL ACCURACY

IMPROVEMENT





Through optimised retrieval techniques and improved prompting approaches, we can increase the accuracy of LLMs in commercial contract reviews from 74% to 95%, on average.

By employing an optimised retrieval approach we can improve the identification of provisions by an average of ~20%.

Including language instructing LLMs to pay attention to specific keywords improved recall accuracy by ~16%.

Use of a Follow Up Prompt gives an average accuracy improvement of 9.2% across all configurations.

95%

20%

Accuracy

IMPROVEMENT



Through optimised retrieval techniques and improved prompting approaches, we can increase the accuracy of LLMs in commercial contract reviews from 74% to 95%, on average.

By employing an optimised retrieval approach we can improve the identification of provisions by an average of ~20%.

16%

9%

RECALL ACCURACY

IMPROVEMENT



Including language instructing LLMs to pay attention to specific keywords improved recall accuracy by ~16%.

Use of a Follow Up Prompt gives an average accuracy improvement of 9.2% across all configurations.

Download a copy

If you are interested in learning more, complete the form below to access a copy of the report.

Our expertise

AI teaser

Our Innovation and Legal Technology Team

Our passion in this area is driven by our clients and their business challenges. We develop, test, invest in and embrace new technology on a continuous basis in order to enable the smart delivery of legal services and creation of clever solutions to clients, both faster and more cost-effectively.

Find out more