Capability Frontier: Benchmarks Miss 82% of LLM Performance

The arXiv paper The Capability Frontier: Benchmarks Miss 82% of Model Performance, submitted 25 Jun 2026, shows standard single-model, single-run evaluations substantially understate what language models can achieve when multiple models and multiple generations are optimally combined. The authors construct a Capability Frontier, a Pareto frontier over models and generations, and test 21 LLMs across 16 benchmarks spanning coding, reasoning, medicine, factuality, instruction following and agentic tasks.

What did the authors measure?

The Capability Frontier is the set of best achievable performances at each cost level under an oracle that can route queries across multiple models and select among multiple generations; the paper compares that frontier to each benchmark's top-performing single model at matched cost. The authors assembled results from 21 models on 16 widely used benchmarks and explicitly correct two biases: underestimation caused by single-model evaluation and overestimation caused by taking maxima over noisy samples.

The construction requires pairing models and sampling budgets to form a Pareto frontier of accuracy versus cost, then measuring how much headroom that frontier reveals compared with conventional single-run reporting. The paper also runs controlled probabilistic simulations to explore how the distribution of query topics affects the gap between oracle routing and the best single model.

How large is the gap between benchmarks and the Capability Frontier?

Correcting for single-model evaluation alone yields a 54% error rate reduction, and adding the correction for single runs produces an 82% improvement in achievable performance, with state-of-the-art accuracy matched at an 85% reduction in cost. Those are the paper's headline empirical numbers from their study of 21 LLMs across 16 benchmarks.

Beyond the headline, the authors show the improvement is not uniform: the gains are larger on heterogeneous, multi-domain workloads. Their probabilistic simulations demonstrate that higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing and the best single model, which explains why multi-domain benchmark suites show larger recoverable headroom.

Why it matters

The paper implies collective LLM capabilities are substantially underestimated by common reporting practices, which can mislead both researchers and deployers about model readiness and cost trade-offs. If multiple cheaper or specialized models plus selective sampling can match or exceed a single reported SOTA at far lower cost, procurement choices, red team planning, and risk assessments change: evaluations that present only a single-model, single-run number may hide cheaper paths to higher real-world accuracy.

The authors’ quantification — specifically the 54% error rate reduction from correcting single-model evaluation and the 82% improvement after also correcting for single runs — gives concrete scale to that claim and ties it to measurable evaluation procedures rather than vague theorizing.

What to watch

Look for follow-up work and evaluation suites that publish multi-model Pareto frontiers or that report matched-cost comparisons, since those would validate whether the Capability Frontier approach holds outside the paper's 21-model, 16-benchmark sample. Also watch for benchmarks and leaderboards adopting cost-matched, multi-generation reporting: that is the concrete signal that evaluation practice is shifting.

The paper’s simulations also point to a testable hypothesis: as benchmark suites increase topic entropy, the gap between oracle routing and single-model top-lines should grow, which would further motivate multi-model evaluation.

The Capability Frontier paper is available on arXiv as arXiv:2606.26836 and frames evaluation as an operational, cost-aware decision rather than a single-number trophy.