Inference Compute and Frontier LLM Evaluation: arXiv 2026

Jessica McFadyen and four co-authors submitted a paper on 16 Jun 2026 arguing that how much compute is available at test time, and how it is used, strongly shapes frontier LLM evaluation. The study evaluates up to 12 frontier language models across seven challenging benchmarks and tests three simple inference-scaling interventions.

What did the authors test and how?

The paper evaluated up to 12 frontier language models on seven benchmarks spanning software engineering, mathematics, medicine and cybersecurity, using three controlled inference-scaling interventions: larger token budgets, context compaction, and repeated submission attempts. The interventions were applied either guided by the model itself or using minimal correctness feedback. The submission lists Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei and Cozmin Ududec as authors and the manuscript runs 34 pages with 4 figures.

The setup isolates how changing inference-time compute and protocol choices affects measured performance. The benchmarks include named tasks such as FrontierMath, Humanity's Last Exam, TerminalBench and a cybersecurity benchmark. The interventions are deliberately simple so the results reflect protocol and compute allocation rather than exotic tooling.

How did inference compute change benchmark results?

Larger token budgets substantially improved performance across multiple domains, the authors find, and repeated submission broadly improved results as well. Newer models reached higher performance at large budgets, unlocking harder tasks and solving them more reliably. Benchmarks varied in which inference-scaling method helped most: the benefit of larger token budgets, external feedback, and parallel attempts depended on the benchmark.

Put plainly, fixed-budget evaluations can understate frontier capability as models advance: low scores sometimes reflect a restrictive evaluation protocol rather than a models limits. The paper presents three headline findings. First, expanding token budgets raises scores on cybersecurity, FrontierMath, Humanity's Last Exam and TerminalBench. Second, the gap between models widens when higher inference budgets are allowed, with newer models showing greater gains at scale. Third, no single inference-scaling intervention dominated across all benchmarks; repeated submission helped broadly, but other methods had benchmark-specific value.

Why it matters

Evaluations shape how model capability is perceived and compared. If scores shift meaningfully with token budgets, compaction strategies or retry policies, then single-budget reports can mislead researchers, policymakers and safety reviewers. The authors therefore argue that benchmark scores are protocol-dependent and that capability should be reported as a function of inference-time compute. They ask that evaluations specify protocol choices explicitly and compare model generations over a large shared compute range at matched budgets, especially in safety- or policy-relevant settings.

This recommendation changes what a responsible evaluation looks like: researchers and evaluators must decide whether a reported low score is a property of the model or of the testing protocol. The paper shows that that decision matters across domains from mathematics to cybersecurity.

What to watch

Will benchmark authors and evaluation suites adopt the papers recommendation to report capability across a range of inference-time compute budgets and to match budgets when comparing model generations? The concrete signal to track is whether future benchmark releases publish performance curves across token budgets, retry policies and compaction strategies rather than single-budget summaries.

The paper is available on arXiv as arXiv:2606.17930 and provides a controlled empirical argument that evaluation protocols need as much scrutiny as model architectures when assessing frontier LLM capability.