{'fraction_non_alphanumeric': 0.026095563754161794, 'fraction_numerical': 0.005309097453432917, 'mean_word_length': 4.991105121293801, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Law of Large Documents: Understanding the Structure of Legal Contracts Using Visual Cues

Large, pre-trained transformer models like BERT have achieved state-of-the-art results on document understanding tasks, but most implementations can only consider 512 tokens at a time. For many real-world applications, documents can be much longer, and the segmentation strategies typically used on longer documents miss out on document structure and contextual information, hurting their results on downstream tasks. In our work on legal agreements, we find that visual cues such as layout, style, and placement of text in a document are strong features that are crucial to achieving an acceptable level of accuracy on long documents. We measure the impact of incorporating such visual cues, obtained via computer vision methods, on the accuracy of document understanding tasks including document segmentation, entity extraction, and attribute classification. Our method of segmenting documents based on structural metadata out-performs existing methods on four long-document understanding tasks as measured on the Contract Understanding Atticus Dataset.


INTRODUCTION
Businesses interact through contracts, and legal departments in many companies are stretched thin managing requests for information about these contracts. Companies need timely reporting on the date contracts are up for renewal, the length of cancellation periods, how much money they should bill, and other details that are embedded in the contract prose. Finding these answers quickly in a large collection of documents, some of which can be hundreds of pages long, can lead to a lot of frustration (and billable hours). At Lexion, we use natural language processing methods to perform this work automatically, extracting key pieces of information from contracts for lawyers to find at a glance [4].
The most powerful recent methods for document understanding are computationally expensive, and they are limited in the length Figure 1: Legal counsel needs to quickly find which state or country's laws will govern an agreement, but this information is often hidden in long documents in sections like "Miscellaneous". In this example contract from CUAD [6], the desired answer, "State of Massachusetts", is broken across a page boundary, with the page number "4" in between. To extract this result correctly, a model would need information about the structure and formatting of the document to know that it should discard the page break, newlines, page number, and any other extraneous information common in legal documents such as headers and footers. OCR metadata provides the rich information necessary to find the correct answer span in difficult cases like this one.
of text they can process at once. To leverage these state-of-theart model architectures, researchers have developed strategies to segment documents and feed them piecemeal into models. However, these strategies typically discard the document-level context that is essential for complex natural language tasks on long documents. We develop a method that makes use of state-of-the-art models while also maintaining information about the document's structure and formatting. Key to this method is the use of features from Optical Character Recognition (OCR) above and beyond the document's raw text (Figure 1).
We evaluate our method on a publicly-available dataset of commercial legal contracts (Section 3). Our method (Section 4) outperforms prior work on the dataset for four key contract understanding tasks (Section 5). We also quantify the effect of the OCR features and measure the impact of document length on the difficulty of the task (Section 6).

RELATED WORK 2.1 Long Document Understanding
When working with long documents, many models are limited in the length of text they can consider at once. To overcome this limitation, the simplest method is to keep only the portion of the document that fits and discard the rest, but many legal documents contain important information that can be located anywhere in the document. Another method is to split the text into equal-sized sections and input them into the model separately, for example using a sliding window over the documents so that each segment fits within the 512-token maximum length of BERT-based models [6]. However, splitting documents into sections without any contextual information makes downstream tasks much more difficult (see Subsection 6.1).
Most work on long documents circumvents the challenge of splitting documents into meaningful sections by using datasets with clearly demarcated sections, such as Wikipedia articles [1,2], but determining section boundaries is difficult on a corpus of documents with inconsistent structure like legal documents. Research has consistently shown better results on document understanding tasks after splitting documents into sections [2,5,12,13]. As we will show, representing document structure is essential to the section splitting process, and therefore to achieving strong results on downstream tasks.

OCR
Converting images into text via OCR is a precursor to many NLP tasks, but modern OCR tools provide much more than just textual information. OCR methods produce rich metadata including page partitioning and placement, font style and size, justification, and level of indentation, which become features that allow NLP models to incorporate the document-level context that is essential for understanding long documents.
However, OCR features can be difficult to integrate into NLP pipelines. Existing strategies include encoding them as special characters in models like BERT [3], creating encodings to represent the spatial location of each character [8], or using representations particularly suited to structured data such as graphs [7].

DATASET
Legal documents are especially challenging for NLP tasks because of their length. We use the Contract Understanding Atticus Dataset (CUAD), which contains 510 English-language commercial legal contracts from the public domain [6]. Each contract is labeled by trained annotators for 41 attributes. Some of these attributes are phrases that must be extracted from the contract as written (entity extraction), and other attributes are yes-or-no values that can be answered from the contract text (classification). Documents in CUAD are longer than those in most NLP datasets, which are often composed of sentences [10,14] or short social media posts like tweets limited to 140 characters [11,15]. Even document-length corpora consist of documents that are much shorter than legal documents: the average length of a Wikipedia article, for example, is 620 words [9], compared to CUAD's average length of 9,594 words. Detailed statistics on CUAD are included in Appendix B.
Since CUAD does not provide section labels, we report section splitting results on our proprietary corpus of contracts, evaluated against section labels created by expert annotators.

EXPERIMENTAL SETUP
Our end-to-end system is outlined in Figure 2 below, beginning with a raw PDF contract and ending with a predicted answer for a document understanding task. In order to provide focused inputs to our information extraction models, we split documents into coherent sections of several types: clauses, sub-clauses, headers, and footers ( Figure 3). While clauses and sub-clauses contain the information that lawyers are interested in parsing from contracts, it is necessary to detect other types of sections like headers and footers as well because they contain noise that interrupts the flow of text, making it challenging for models to extract the correct values. We first trained a baseline model for section splitting which takes textual and linguistic features as inputs, with no visual or structural information. This model struggled to accurately identify sections that a human could easily distinguish using visual cues not available to our baseline model.
With the hypothesis that the rich visual data available through OCR would improve performance on section splitting, we tested ablations of our baseline model to measure the impact of four types of visual cues: • Page layout features: the proximity of text to various regions of the page (e.g. the beginning, end, center, or margin of a page).
• Text placement features: the placement of text on the page (e.g. is it centered or aligned). • Visual grouping features: which words in the text are grouped together into a paragraph, list, or table. • Stylistic features: whether the words are stylized (e.g. bold, italic, or underlined).
We find that these visual features improve performance for all section types, with the largest impact on footer detection. We detail these results in Section 5.

Metrics
We evaluate our models using precision, recall, and F1 score because many contract attributes suffer from class imbalance. For example, only 15 of the 510 contracts in CUAD (3%) forbid price changes (the "Price Restriction" attribute). If a lawyer wanted to find these 15 contracts, a model that returned none of them would achieve 97% accuracy, while the recall of such a model would be a more meaningful 0%.

Section Splitting
To evaluate the impact of visual cues on the accuracy of our section splitting method, we run multiple experiments in which we activate different groups of visual features, and then measure the accuracy on a held out dataset. We compare each group's performance to a baseline model that relies only on text-based linguistic features and low-level layout information, with no access to visual cues from OCR features.
We observe that visual cues have a strong impact, which varies depending on the section type (Table 1). At a high level, it is clear that footer detection gains the most from OCR features, but clauses and sub-clauses see significant improvements as well. The results for headers are mixed: recall improves, but at a cost to precision. We look more closely at the reasons for these results in Subsection 6.2.

Document Understanding Tasks
Using the outputs of the section splitting process, we select relevant sections and input them into downstream models trained to extract key metadata from contracts. The four tasks we evaluate reflect four common questions lawyers need to know about their contracts: • Which contracts are still active? ("Expiration Date", framed as an entity extraction task to find the correct date in the contract text) • Which state or country's law governs this contract? ("Governing Law", framed as an entity extraction task to find the correct location in the contract text) • Can I terminate this contract? ("Termination for Convenience", framed as a classification task to answer yes or no correctly based on the contract text) • Which contracts will survive a merger or acquisition? ("Anti-Assignment", framed as a classification task to answer yes or no correctly based on the contract text) We evaluate against two competing approaches: • Expert rules: Regex-like rules written by trained annotators referencing a corpus of legal documents that does not include the CUAD contracts. For example, an annotator might determine that if the phrase "may terminate at will" appears in a contract, that contract's value for the "Termination for Convenience" attribute is "yes". Because of the differences between the contracts in the annotators' reference corpus and the contracts in CUAD, these rules have high precision but low recall, since they cannot generalize to unseen language in a new corpus. • DeBERTa: The previously best-performing model on CUAD [6], a DeBERTa-xlarge model fine-tuned for the question answering task. We compare to the reported precision at 80% recall for four attributes.
Results for the full end-to-end tasks are detailed in Table 2. Our method, which uses rich visual cues to split contracts into sections for use on downstream tasks, offers the best balance of precision and recall across the four tasks. Additionally, examples of correct and incorrect model predictions on each task are included in Appendix A. To quantify the impact of document length on the difficulty of a document understanding task, we train a CRF model to extract Governing Law from contract segments of increasing length. For each length, we draw a fixed-size window around the correct answer span in every contract. We use a random offset before and after the answer span so that answers are not always located in the same place. As shown in Figure 4, as document length increases, performance sharply falls. Entity extraction on long documents is a difficult task without the benefit of visually-aware, structural features that help the model locate the answer within a document.

Impact of OCR Features
Our results show strong performance on end-to-end document understanding tasks, but to what extent can we attribute that performance to the model's understanding of document structure via OCR features, as opposed to the task setup or model architecture?
To quantify the impact of visual cues on one of our downstream tasks, we compare an Anti-Assignment classifier trained using sections that were obtained with and without the help of visual cues. As shown in Table 3, the downstream model performs even better when it has access to document structure via OCR features. Finally, we provide a more detailed analysis of the performance for each section type from Table 1: 6.2.1 Clauses. Clauses benefit the most from style features. This is intuitive, since clauses very frequently start with a phrase that acts as a heading, set apart in bold or underlined. The visual grouping features also had a positive impact, since clauses often appear together in paragraphs of text.

Sub-clauses.
Sub-clauses benefit from the same features as clauses, but it is noteworthy that the impact is lower, particularly for recall. Sub-clause detection is generally a harder problem, even for human annotators, due to deeply nested sub-clauses and a wider variety of possible ways to format them.

Headers.
Headers do not see a large benefit from OCR features. In fact, page layout features hurt precision on headers, since they can be misleading when clauses and sub-clauses appear at the beginning of a page like a header. Similarly, text placement features are also less effective because the model tends to overfit to the placement of logos and auto-generated stamps like "DocuSign Envelope Id", which always show up in the same location. To address this weakness, we plan to add more diverse data and regularization in addition to richer features.

Footers.
Footers benefit the most from OCR features. This is not surprising, since footers consistently appear at the end of the page, and are much more common than headers. Accurate footer detection is particularly useful since footers often interrupt the flow of clauses, which hurts downstream models that rely on clean section text.

CONCLUSION
We discuss a method for extracting valuable metadata from legal documents, which are especially challenging due to their length and their diverse structure and formatting. We show that visual cues are a key component in a successful document understanding pipeline, from section splitting to entity extraction and classification.
This work explores only a subset of the types of visual cues available. Future work might explore additional features such as font information, spacing between lines and blocks of text, superscripting, subscripting, and color. Given the impact of the visual cues we examined, we expect these features to further improve performance on document understanding tasks.

A MODEL PREDICTIONS
We list examples of correct and incorrect model outputs for the four end-to-end tasks: Expiration Date (Table 4), Governing Law (Table 5), Termination for Convenience (Table 6), and Anti-Assignment (Table 7). "Relevant Section Text" refers to the output of our section splitting model, which identifies clauses and sub-clauses and then labels which sections are relevant to a given downstream task. Table 8 lists detailed statistics on the contracts in CUAD.

B DATASET B.1 Details on CUAD
CUAD provides both PDF and text versions of contracts. For our experiments, we use the PDF versions so that we can extract OCR metadata from the formatting and structure of the contracts.

B.2 Attributes Chosen from CUAD
While CUAD includes labels for the two most useful contract metadata attributes, Document Name and Parties, we choose not to evaluate using these labels because both of these attributes occur multiple times throughout each document, and the label offsets in CUAD do not consistently come from the same location in the document. For example, some Document Name labels are selected from an Appendix rather than the beginning of the document, making evaluation on these labels misleading.

B.3 Test Set
To facilitate future work on CUAD, we list the filenames of the 51 documents in our test set in Table 9. by the Borrowers, in accordance with the terms of this Agreement is less than $5,000,000 and any payment, or any part thereof, on account of any of the Obligations is invalidated, declared to be fraudulent or preferential, set aside, rescinded or must otherwise be restored or returned by the Agent or the Lenders upon the insolvency, bankruptcy, liquidation, dissolution or reorganization of any Borrower or upon or as a result of the appointment of a receiver, intervenor or conservator of, or trustee or similar officer for any Borrower, or any substantial part of this property, or otherwise, all as though such payment had not been made.
full text bold text ×

TERM AND TERMINATION (a) This Agreement and the license granted under this
Agreement shall remain in effect perpetually as long as fees are paid by Sparkling in accordance with the Fee Schedule and the Agreement is not otherwise terminated in accordance with this Section.
bold text bold text ✓ Table 5: Model Predictions for "Governing Law"

Relevant Section Text
Correct Answer

Model Prediction
14. Governing Law. This Agreement shall be construed and interpreted in accordance with the laws of the State of Ohio, without recourse to any principles of law governing conflicts of law, which might otherwise be applicable. bold text bold text ✓

Relevant Section Text
Correct Answer Model Prediction 8.2 Termination. Either Party may terminate this Agreement for the material breach or default of any of the terms or conditions of this Agreement by the other Party upon thirty (30) days' written notice and the opportunity to cure during such notice period; and such termination shall be in addition to any other remedies that it may have at law or in equity. Additionally, LBIO may terminate this Agreement if MD Anderson is declared insolvent or enters into liquidation or has a receiver or an administrator appointed over all or any part of its assets or ceases or threatens to cease to carry on business, or a resolution is passed or a petition presented to any court for the winding up of the Party or for the granting of an administration order in respect of MD Anderson, or any proceedings are commenced relating to the insolvency or possible insolvency of MD Anderson.

Model Prediction
VII. ASSIGNMENT. Neither this Agreement nor any rights or obligations or licenses hereunder may be assigned, pledged, transferred or encumbered by either party without the express prior written approval of the other party, except that either HEMISPHERX or SCIEN may assign this Agreement to any successor by merger or sale of substantially all of its business or assets to which this Agreement pertains, without any such consent. Any assignment in violation hereof is void.

Yes
Yes ✓ ASSIGNMENT: NOW, THEREFORE, for good and valuable consideration, the receipt and adequacy of which are hereby acknowledged, Seller does hereby transfer, sell, assign, convey and deliver to Backhaul all right, title and interest in, to and under the Assigned Intellectual Property, including, without limitation, the Trademarks and Patents set forth on Schedules A and B hereof, respectively, and all goodwill of the Purchased Business associated therewith. Seller hereby covenants and agrees, that from time to time forthwith upon the reasonable written request of Backhaul or Buyer, that Seller will, at Backhaul's cost and expense, do, execute, acknowledge and deliver or cause to be done, executed, acknowledged and delivered, each and all of such further acts, deeds, assignments, transfers, conveyances and assurances as may reasonably be required by Backhaul or Buyer in order to transfer, assign, convey and deliver unto and vest in Backhaul title to all right, title and interest of Seller in, to and under the Assigned Intellectual Property.

No
Yes × (no relevant section found) Yes No ×

Assignments and Transfers by Seller
Trusts. The provisions of this OMA shall be binding upon and inure to the benefit of the Seller Trusts and their respective successors and assigns. A Seller Trust may transfer or assign, in whole or from time to time in part, to one or more liquidating trusts its rights hereunder in connection with the transfer or resale of Stock held by such Seller Trust, provided that such Seller Trust complies with all laws applicable thereto and provides written notice of assignment to GWG promptly after such assignment is effected, and provided further that such liquidating trust and each beneficiary thereof executes a joinder to this OMA effective as of the date of such assignment or transfer.

No
Yes × 133 Assignment. Neither Party shall assign this Development Agreement or the obligations contained herein without the express written consent of the other Party.

Yes
Yes ✓ 15.7 Successors and Assigns. This Agreement shall inure to the benefit of and be binding upon the Parties and their respective successors and assigns, including, but not limited to, any chapter 11 or chapter 7 trustee; provided, however, that this Agreement may not be assigned by any of the Parties without the prior written consent of the other, provided further that notwithstanding the foregoing, GA and Tiger may each collaterally assign this Agreement and their rights thereunder to their respective lenders.