Recurrent neural networks trained on investment treaty texts can automatically draft new clauses of lawyer-like quality. They will not replace negotiators, but may make their task of composing a compromise text easier.

In 2016 the science fiction short film “Sunspring” based on a movie script written entirely by artificial intelligence (AI) made the headlines. Its creators had trained a deep learning algorithm – a so-called recurrent neural network (RNN) – on the screenplays of dozens of science fiction movies to produce it. The dialogues are grammatically perfect and sensible enough, albeit somewhat cryptic occasionally, to the extent that an uninitiated observer could have mistaken the artificial creation for an artistic one. At the same time, the story lacks coherence and the exchanges are repetitive, which the authors attribute to recurrent patterns in science fiction movies of characters trying to understand their environment.

“Sunspring” thus epitomizes the achievements, but also the shortcomings of deep learning RNNs. They are surprisingly good at producing correct, sensible and genre-specific texts, yet they do not (yet) match the creative qualities and cohesion of real human work. So what is the role of these RNNs for automatic text production in other domains – such as law?

Legal texts have legal consequences. That, in turn, risks setting the bar for automated text production in law very high. While we can find “Sunspring” or Shakespearing plays written by AI entertaining, we would not want to have the same algorithm enshrine our rights and obligations in a contract or treaty.

But even in the legal domain, automated text production does not have to be perfect to be useful. Similar to the reliance on a template or a document prepared by a junior associate, AI can facilitate the work of senior lawyers or negotiators by allowing them to start from a draft rather than from scratch.

In international investment law, for instance, states typically use model agreements to write treaties. This leads to battle-of-the-forms as the parties bargain over whose template is to become the basis of negotiation. RNNs can alleviate this power struggle by producing consensus language faithfully reflecting the prior treaty practice of both states as starting point for negotiations. Similarly, if states wanted to negotiate a multilateral investment treaty, RNNs could be used to derive a text that consolidates the practice of close to 200 states into a single document.

In short, not only is there room for useful, if not perfect, automated text production using deep learning RNNs in law, but these algorithms can perform tasks difficult to accomplish for humans such as distilling common language from 3000 investment treaties.

Predicting investment treaty texts

Dmitriy Skougarevskiy and I have developed a framework and application to automatically draft international investment agreements. In a recent paper presented at the 2017 International Conference on Artificial Intelligence and Law and building on earlier work presented at the 2016 JURIX conference, we introduce a framework which seeks to harness the power of current RNN algorithms for legal text production while mitigating their shortcomings. Users can test our application on a dedicated website to predict, amongst others, the China-United States bilateral investment treaty (BIT) currently under negotiations.

So, in non-technical terms, what did we do? We began by training a deep learning RNN algorithm on the corpus of close to 1600 English-language BITs, which we had collected as part of our www.mappinginvestmenttreaties.com project. The algorithm then predicted new treaty texts – letter by letter. While initial results were encouraging, the output suffered from the same shortcomings as the short film “Sunspring”: our texts were repetitive reflecting mainstream language of treaties concluded in the 1990s (when up to four investment treaties were signed per week) and lacked coherence having multiple articles on subrogation but none on dispute settlement for instance.

To remedy these shortcomings, we enhanced our framework by training the algorithm on individual articles e.g. on “expropriation” or “national treatment” rather than entire treaties. Furthermore, we artificially inflated the share of more recent treaties in our training sample so that the algorithm would not only pick up patterns in the majority of agreements, but also trends in BITs of a more recent vintage.

We assessed the success of our new framework by comparing predicted to real agreements. Our diagnostics showed that our improvements successfully aligned the design of predicted texts with actual agreements. Clauses that were virtually absent in predicted agreements of our base model, such as public policy exceptions, now appeared in a similar share of real and predicted agreements.

Filtering output to match bargaining power

To turn the automated production of treaty texts into a useful application for predicting future agreements or for generating a compromise draft in current negotiations, we introduced a user-directed filter. That filter selects the one clause amongst hundred of predicted clauses that textually most closely corresponds to the treaty practice of a selected party or combination of parties.

A predicted “fair and equitable” treatment clause that is 100% inspired by prior Canadian practice will therefore link fair and equitable treatment to the international minimum standard of treatment in line with prior Canadian BIT practice. Once the user decreases the input to say 30% Canadian and 70% German treaty practice as filter, that reference will disappear given that German treaties do not traditionally link “fair and equitable treatment” to the customary international law minimum standard.

The future of automated legal text production

The framework presented here is work in progress. Improvements still need to be made in understanding when the RNN produces truly novel texts, rather than just replicating existing language. In addition, further research is required to assess the varying performance of filters to explore, for instance, why more sophisticated word embedding approaches seems to work less well than simple Jaccard distances in selecting candidate articles that accord with legal intuition.
The framework and application we present on our website is also just one possible use case for the deployment of RNNs in automated legal text production. Instead of states’ bargaining power, our filter could also be adapted to predict agreements of different levels of investment protection. Furthermore, instead of international treaties, the same framework could be trained on contracts to predict agreements between private parties.

We are thus just at the beginning of a wider deployment of RNN-based AI in law. While one has to be careful not to overstate its potential, it is equally important to acknowledge that even if robots cannot write treaties as well as humans, there are useful applications for deep learning algorithms for legal document production that will make the life of lawyers and negotiators easier and are therefore worth pursuing.