T2D-Bench: Benchmarking LLMs for Type 2 Diabetes Evidence

T2D-Bench is a reproducible benchmark and evidence-gated evaluation framework for assessing whether large language model outputs meet explicit, graph-checkable evidence requirements for Type 2 Diabetes. The dataset and framework were submitted to arXiv on 23 Jun 2026 and test LLM outputs against a multi-layer clinical-lifestyle knowledge graph across 100 structured vignettes.

What is T2D-Bench and how is it built?

T2D-Bench is built on a multi-layer clinical-lifestyle knowledge graph that combines a biomedical spine (UMLS, DrugBank, SIDER), computable American Diabetes Association Standards of Care rules, and lifestyle knowledge linked by a mechanistic bridge to glycemic laboratory effects. The benchmark encodes explicit, graph-checkable evidence requirements so that LLM outputs can be tested for both guideline compliance and biologically grounded lifestyle claims. The authors describe an "evidence gate" that detects unsupported omissions and applies constrained revision to bring outputs into verifier-level compliance with the benchmark's evidence requirements.

How did LLMs perform on T2D-Bench?

Across 100 structured vignettes covering diagnosis, medication safety, and adversarial lifestyle conflicts, baseline LLM outputs failed benchmark-defined evidence-path checks in 35% of cases for GPT-4o-mini and 33% for GPT-4o. The benchmark exposes both unsupported claims and missing evidence paths that link clinical or lifestyle advice to glycemic laboratory effects. The paper reports that the evidence gate can both detect these failures and use constrained revision to correct outputs so they meet the benchmark's verifier-level constraints.

Why does this matter?

LLMs can produce clinically fluent recommendations while failing to satisfy guideline constraints or explicitly justify lifestyle-related glycemic claims. The benchmark makes such unsupported clinical omissions explicit and measurable. The authors summarize the consequence plainly: "These results show that computable evidence constraints can make unsupported clinical omissions explicit, measurable, and correctable in diabetes-focused LLM outputs." That matters for clinicians, tool builders, and regulators who need auditability and traceability of claims linking lifestyle or medication advice to lab effects.

How does the benchmark differ from typical LLM evaluation?

T2D-Bench focuses on evidence-path checks rather than only surface-level correctness or fluency. It couples established biomedical resources (UMLS, DrugBank, SIDER) with computable ADA Standards of Care rules and a mechanistic bridge to laboratory outcomes, enabling graph-based verification of whether an LLM's recommendation is supported by explicit, checkable evidence. The framework also includes mechanisms to revise model outputs under constraint when evidence gaps are detected.

What the paper includes and where it appeared

The submission on arXiv (arXiv:2606.24145) is seven pages with two figures and two tables and was accepted as a poster at the AMIA 2026 Annual Symposium. The authors are Saba A. Farahani, Hung Cao, Ramesh Jain, and Amir M. Rahmani.

What to watch

Watch for implementations of evidence gates in clinical LLM toolchains and follow-up work that applies the T2D-Bench approach to other chronic conditions. The next concrete signals will be public code or datasets tied to the benchmark, and demonstrations of the constrained revision step operating in deployed clinical-assistant prototypes.