ORAgentBench benchmark: LLM agents on 107 OR tasks pass rates

ORAgentBench, published on arXiv on 18 Jun 2026 by Jiajun Li and seven coauthors, is an execution-grounded benchmark that tests whether large language model agents can solve end-to-end operations research work. The suite contains 107 human-reviewed tasks across diverse operational scenarios; agents must write and run solution code and their submissions are evaluated by hidden validators for schema validity, hard-constraint feasibility, and normalized objective quality. Experiments with fourteen frontier agent-model configurations found the best agent passed 35.51% of all tasks and 20.59% of hard tasks.

What is ORAgentBench and how is it structured?

ORAgentBench is an execution-grounded benchmark of 107 human-reviewed operations research tasks, each packaged in an isolated environment with a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents are expected to produce executable solutions: they must write and run solution code and produce a submission that the benchmark’s hidden validators can check for schema validity, hard-constraint feasibility, and normalized objective quality. The benchmark is designed to test the full workflow from operational artifacts to validated decisions rather than just isolated modeling or solving steps.

The paper frames the benchmark against prior OR evaluations that often decouple modeling from solving or rely on pre-formalized or text-only instances. ORAgentBench instead bundles artifacts and validators so an agent’s output is scored on whether it meets the exact operational requirements, not merely whether it looks plausible in text.

How did current LLM agents perform on the benchmark?

Fourteen frontier agent-model configurations were evaluated on the benchmark; the top-performing agent passed 35.51% of all tasks and 20.59% of hard tasks. Many submissions that were technically feasible still fell below the required quality threshold, and agents frequently failed hard tasks. Failure analysis in the paper attributes most errors to strategic weaknesses: missed operational rules, brittle problem formulations, weak construction of feasible solutions, and insufficient solution improvement.

The authors also tested the effect of OR-specific procedural skills. Those procedural skills increased feasibility on hard tasks, but did not reliably improve overall solution quality or raise the pass rate. The experiments therefore separate three failure modes: getting a syntactically valid submission, producing a feasible solution that satisfies hard constraints, and delivering a solution that attains the normalized objective quality needed to pass.

Why it matters

The benchmark exposes a gap between generating plausible optimization code and delivering dependable operational decisions that stand up to formal validators. A best-case pass rate of 35.51% across 107 tasks shows that current autonomous LLM agents are far from reliable OR practice, particularly on hard tasks where the top pass rate was 20.59%. For organizations that require validated, constraint-satisfying operational decisions, these results underline that agents today are not a drop-in replacement for human OR expertise.

ORAgentBench also reframes evaluation: practical OR systems need not only modeling and solving capabilities, but robust procedural workflows that handle messy data, configuration artifacts, and strict validators. The benchmark’s emphasis on execution and hidden validation highlights where agent development must focus next.

What to watch

Look for agent iterations that raise the overall pass rate well above the current 35.51% and improve normalized objective quality on feasible solutions. Progress on robust formulation and iterative improvement routines that reduce the strategic failure modes named in the paper would be the clearest signal that LLM agents are approaching dependable OR practice. Researchers and vendors should also publish evaluations showing agents passing a larger share of the hidden validators for schema, feasibility, and objective quality.