LegalHalluLens: Typed hallucination audit finds 38-40 pp gap

LegalHalluLens, a framework published on arXiv and submitted on 16 Jun 2026 by Lalit Yadav and Akshaj Gurugubelli, delivers a fine-grained audit of hallucinations in legal AI and a calibrated multi-agent debate pipeline to reduce fabrications. The authors measure performance over CUAD and report results across 510 contracts and 249,252 clause-level instances.

What does LegalHalluLens measure?

LegalHalluLens breaks hallucinations into four legally motivated claim categories and reports both aggregate and typed failure modes, exposing differences aggregate metrics hide. The four categories are numeric, temporal, obligation/entitlement, and factual claims, and the paper shows an aggregate hallucination rate near 52% while revealing a within-model gap of approximately 38-40 percentage points between obligation/numeric and temporal claims over CUAD.

The framework also introduces the Risk Direction Index, a scalar designed to reduce omission-versus-invention bias and to show that two systems with matched 52% aggregate hallucination rates can nonetheless carry opposite RDIs. The typed profiles and RDI aim to give compliance officers an actionable signal rather than a single averaged error rate.

How does the calibrated multi-agent debate pipeline perform?

A debate pipeline calibrated to both magnitudes and directions, and informed by typed profiles and RDI, reduces fabricated detections and matches stronger APIs while using a much smaller backbone. The authors report that the debate pipeline reduces fabricated detections by 45% and achieves per-category gains that track the typed diagnosis, and that it matches commercial APIs with a substantially smaller backbone of 4B active parameters.

LegalHalluLens structures debate with Skeptic challenges and asymmetric gates targeted at measured failure modes rather than generic tuning. The paper shows those targeted mechanisms outperform generically tuned debate, and that the diagnostics produced by typed profiles and RDI serve directly as calibration inputs for the multi-agent system.

Why does this matter?

A 52% aggregate hallucination rate gives a false sense of sameness; LegalHalluLens demonstrates concrete, directional failure modes that affect procurement and accountability decisions. Exposing an approximately 38-40 pp gap between claim types matters for legal workflows where numeric or obligation claims may be relied on differently than temporal claims, and the Risk Direction Index lets organizations distinguish whether a system tends to omit or invent information.

The calibrated debate pipeline provides a practical mitigation: it reduces fabrications by 45% and can reach parity with larger commercial APIs while running on a 4B active parameter backbone, suggesting cheaper models can achieve comparable safeguards when informed by typed diagnostics.

What to watch

Watch whether typed hallucination profiles and the Risk Direction Index appear in procurement language, compliance checklists, or agent design guidelines, since the authors explicitly position the framework to support direction-aware procurement, accountability, and agent design for legal AI deployed in the wild. Also watch for replication beyond CUAD, the dataset used here (Hendrycks et al., 2021), and for further evaluations comparing RDIs across different vendors.

Details and provenance: the paper, "LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI," is arXiv:2606.18021 and was published at the Second Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD) at ICML 2026. The authors evaluated 510 contracts and 249,252 clause-level instances and report an aggregate hallucination rate of ~52%, a within-model gap of approximately 38-40 percentage points, and a 45% reduction in fabricated detections from the calibrated debate pipeline.