Better Call CLAUSE

Abstract

Overview

CLAUSE advances legal reasoning evaluation through systematic analysis of contractual discrepancies. Built upon two established legal corpora (CUAD and ContractNLI), our framework represents the first fully AI-generated legal contradiction dataset.

Figure 1: CLAUSE pipeline showing Data Preparation, Discrepancy & Benchmark generation, and Evaluation phases.

Key Contributions

Dataset

CLAUSE implements a comprehensive taxonomy of modifications featuring two fundamental dimensions:

Perturbation Categories

Dataset Statistics

Evaluation Tasks

We define three hierarchical tasks to assess LLM capabilities in detecting and reasoning about legal discrepancies:

Key Results

We evaluated four leading LLMs: GPT-4o-mini, Gemini 2.0 Flash, Gemini 2.5 Flash, and LLaMA 3.3 70B Instruct.

Figure 2: Comparison of model performance showing Miss and Extra rates for CUAD and NLI datasets across L1 (zero-shot) and L2 (one-shot) evaluation levels.

Key Findings

• Performance Hierarchy: Gemini-2.5 leads with highest F1-scores (63%+ on CUAD), driven by a high-recall (90%+), low-precision strategy. All models perform significantly better on CUAD than the more complex NLI dataset.
• Category Challenges: Omission_Legal proves exceptionally challenging, with GPT-4o-mini achieving only 31% F1 and LLaMA-3.3 dropping to 9.3% F1 on NLI. Legal discrepancies are consistently harder than in-text contradictions.
• Explanation Quality Gap: Models generate fluent explanations (Clarity 4.0+) but with low Completeness (<2.0), indicating surface-level coherence without substantive legal depth.
• Law Citation Weakness: Even the best model (Gemini 2.5) achieves below 14% accuracy in semantically matching legal citations, revealing fundamental gaps in external legal knowledge retrieval.

Eval 1: Binary Detection Results

Data Examples

In-Text Contradiction Example

{
  "file_name": "DovaPharmaceuticalsInc_10-Q_Promotion_Agreement.txt",
  "perturbation": [{
    "type": "Omissions - In Text Contradiction",
    "original_text": "'Detail(s)' shall mean a Product presentation during a face-to-face sales call... Neither e-details, nor presentations made at conventions...",
    "changed_text": "'Detail(s)' shall mean a Product presentation... Presentations made at conventions, exhibit booths, shall constitute a Detail.",
    "explanation": "The term 'Detail' is redefined to include presentations at conventions, directly contradicting the original definition that explicitly excluded them.",
    "location": "Section 1.19",
    "contradicted_location": "Section 4.2.2",
    "contradiction_exists": "YES"
  }]
}

Legal Contradiction Example

{
  "file_name": "PharmagenInc_Endorsement_Agreement.txt",
  "perturbation": [{
    "type": "Ambiguities - Ambiguous Legal Obligation",
    "original_text": "All HDS' uses of Celebrity Attributes... shall be subject to the prior written approval of Celebrity via his agent...",
    "changed_text": "All HDS' uses of Celebrity Attributes... should aim to obtain the prior approval... A reasonable effort should be made...",
    "contradicted_law": "Lanham Act - False Endorsement",
    "law_citation": "15 U.S. Code § 1125(a)",
    "law_url1": ["https://www.law.cornell.edu/uscode/text/15/1125"],
    "law_explanation": "The modified text creates ambiguity around the approval process, potentially leading to usage without explicit consent, violating the Lanham Act."
  }]
}

Citation

If you find this work useful, please cite our paper:

@inproceedings{roychoudhury2025clause,
    title={Better Call CLAUSE: A Discrepancy Benchmark for Auditing LLMs Legal Reasoning Capabilities},
    author={Roy Choudhury, Manan and Chandramouli, Adithya and Anand, Mannan and Gupta, Vivek},
    booktitle={Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL)},
    year={2026}
}

Acknowledgements

We are grateful to Rui Heng Foo for his invaluable assistance with the data generation pipelines. We extend our sincere thanks to Arizona State University's Lincoln Center for Applied Ethics, along with their law professors and partners, for lending their expertise to advise on our perturbation categories and to authenticate our perturbed contracts for real-world relevance. We also thank the ASU Law and Pre-Law students for volunteering their time and expertise. Finally, we acknowledge the CoRAL Lab at Arizona State University for providing the computational resources that made this research possible.