CORE-Bench: Life After Benchmark Saturation, v1.1 Findings

The CORE-Bench benchmark continues to yield actionable signals after accuracy saturation, according to an arXiv paper submitted on 23 Jun 2026 (arXiv:2606.26158). The paper introduces CORE-Bench v1.1 and an out-of-distribution suite called CORE-Bench OOD, surfaces threats to construct validity, and reports a statistically significant speedup by about a factor of two from human-agent collaboration in a small randomized experiment.

What did the authors change in CORE-Bench?

They introduced CORE-Bench v1.1 and CORE-Bench OOD to address construct validity and out-of-distribution generalizability, while preserving the original task focus on computational reproducibility of scientific code. The paper uses CORE-Bench Hard as a case study and explains that when accuracy saturates, retiring the benchmark misses opportunities to measure six other dimensions of agent performance: construct validity issues such as shortcuts, out-of-distribution generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and uplift from human-agent collaboration.

The authors name specific artifacts: CORE-Bench v1.1, an improved benchmark, and CORE-Bench OOD, an out-of-distribution task suite. They argue these additions help detect validity threats that become visible only with more capable agents, and they position v1.1 and OOD as tools to measure multiple operational qualities that an accuracy-only lens would overlook.

How did the paper measure performance after saturation?

The team measured efficiency, reliability, model performance, scaffold performance, and human-agent uplift even though accuracy had saturated on existing tasks. They show that CORE-Bench v1.1 still produces meaningful measures for these dimensions and that the out-of-distribution suite tests generalizability beyond the original benchmark distribution.

The paper reports a small-scale randomized experiment on real-world computational reproducibility tasks, finding a statistically significant speedup by about a factor of two when humans worked with agents. The authors note this is likely an underestimate because one-fifth of human-only reproductions reached the time limit before completing. Beyond the experiment, the paper analyzes threats to construct validity, for example shortcuts that are hard to anticipate with less capable agents, and it evaluates the relative impact of the scaffold used to run agents versus the underlying model.

Why it matters

Measuring only accuracy drives benchmark churn and hides operational weaknesses that matter in practice. By documenting six nonaccuracy dimensions and supplying v1.1 and an OOD suite, the authors provide a concrete alternative to accuracy-centric evaluation, one that can reveal shortcuts, reliability failures, and where human collaboration adds value. Teams building or adopting agents for computational reproducibility will gain a clearer sense of performance trade-offs: a model that scores well on saturated accuracy metrics can still fail on efficiency, reliability, or out-of-distribution tasks.

The paper also supplies empirical evidence that humans plus agents can materially speed up real-world reproducibility work, which changes cost and workflow calculations for researchers and tool builders responsible for reproducing scientific code.

What to watch

Watch for broader adoption of CORE-Bench v1.1 and CORE-Bench OOD by the reproducibility and benchmarking community and for follow-up experiments that scale the randomized study to more tasks and participants. A larger trial that reduces the fraction of human-only runs hitting a time limit would clarify the true magnitude of human-agent uplift.

Authors and provenance: the paper, "Life After Benchmark Saturation: A Case Study of CORE-Bench," was submitted on 23 Jun 2026 to arXiv as arXiv:2606.26158 and lists authors including Nitya Nadgir, Sayash Kapoor, Kangheng Liu, Peter Kirgis, Matilda Orona and Arvind Narayanan among others.