MedCalc-Pro: medical-calculation benchmark and LLM agent

MedCalc-Pro, a new medical-calculation benchmark and agent framework, was submitted to arXiv on 3 July 2026 by Siran Zhao, Ruihui Hou, Ziyue Huai, Chennuo Zhang and Tong Ruan. The project includes 2,268 real-world clinical cases and evaluates three task settings: single-calculator, multi-calculator, and nested-calculator scenarios.

What does MedCalc-Pro contain and test?

MedCalc-Pro contains 2,268 real-world clinical cases, covers 77 medical calculators and spans 14 clinical departments, and it tests three progressively challenging task settings: single-calculator, multi-calculator, and nested-calculator. The benchmark is explicitly designed to move beyond simplified, single-calculator queries and to simulate clinical workflows where multiple calculators are needed, calculations are nested in scale, or the user query does not specify the target calculator.

The dataset and task design aim to reflect realistic clinical complexity. Single-calculator items map one patient case to one calculator. Multi-calculator items require joint evaluation across multiple tools. Nested-calculator items require chained or hierarchical calculations where intermediate results feed into subsequent tools.

How does the agent framework work, and how did it perform?

The proposed agent framework supports multi-tool selection and nested-tool calling, and it suppresses parameter error propagation through structured validation and evidence review; in systematic comparisons across open-source, closed-source, and medical-specialized LLMs, the framework achieves the best performance across all three task settings. The authors present the agent as a more generalizable solution than existing frameworks and methods, specifically for complex clinical scenarios where tool choice is not explicit and intermediate results matter.

Key elements of the framework include mechanisms for selecting multiple calculators when needed, calling tools in nested sequences, and applying structured validation plus evidence review to reduce errors that would otherwise propagate through chained calculations. The paper reports systematic comparisons across three LLM categories: open-source models, closed-source models, and medical-specialized models, and concludes the new framework outperforms those baselines on the single-calculator, multi-calculator, and nested-calculator tasks.

Why it matters

MedCalc-Pro raises the bar on medical-calculation evaluation by moving from toy problems to realistic, multi-step clinical tasks where tool selection and error propagation matter. That matters for any application that uses LLMs to assist clinicians with dosing, scoring, or risk calculations because real clinical work often requires choosing between multiple calculators or combining them in sequence. Demonstrating a framework that explicitly handles multi-tool selection and nested calls, while adding structured validation and evidence review, addresses practical failure modes that simpler benchmarks and tool-using agents do not exercise.

The benchmark's scale, 2,268 cases across 77 calculators in 14 departments, creates a wider testbed for models and frameworks meant to operate in clinical settings. The comparison across open-source, closed-source, and medical-specialized LLMs provides an initial view of where existing approaches fall short and where an agent-layer can add value.

What to watch

Follow whether groups reproduce the paper's claim that the proposed agent framework "achieves the best performance across all three task settings" when evaluated on the full 2,268-case MedCalc-Pro benchmark, and whether code or data releases appear alongside the arXiv submission. Also watch for independent evaluations that report per-calculator or per-department performance, which would show where multi-tool selection or nested calls remain most error-prone.

MedCalc-Pro appears on arXiv as arXiv:2607.02879, with a DOI at https://doi.org/10.48550/arXiv.2607.02879 and a submission date of 3 Jul 2026. The paper frames the benchmark and the agent framework as tools to test and improve LLM behavior in more realistic medical-calculation contexts.