Measuring Intelligence Beyond Human Scale, arXiv paper (2026)

The arXiv paper "Measuring Intelligence Beyond Human Scale," submitted 8 Jul 2026 as arXiv:2607.07040, proposes replacing absolute benchmarks with a relative measurement paradigm in which models generate public challenges that separate other systems. The paper, authored by Jerry Han, Rafael Moschopoulos, Ella Colby, Vishrut Goyal, Andrew Tu, Kia Ghods, Mark Braverman and Elad Hazan, describes protocols that aim to scale evaluation "beyond the human frontier."

What is the proposed measurement paradigm?

The paper answers this directly: evaluation moves from absolute, human-authored benchmarks to relative measurement where models create challenges and aggregation yields an adversarial psychometric rating system. In short, systems are rated by how well they can separate peers on model-generated tasks, rather than by performance on fixed human-designed tests.

The authors argue human-authored benchmarks saturate, and "above human capability, examiners may not know which tasks are both hard and verifiable." Their proposed approach has models produce public challenges that can distinguish the capabilities of other systems, then aggregate outcomes into a rating that can scale with system capability.

How do the authors say the protocols work?

They describe practical protocols intended to reduce incentives for private-information attacks, support judge-free adjudication, and naturally scale with agent capabilities. The paper emphasizes designs that limit benefits from keeping solutions private and that permit adjudication without human judges, enabling evaluation to continue as systems outgrow human oversight.

The framework is instantiated across two domain types in the paper: verifiable domains, where correctness can be definitively checked, and open-ended, non-verifiable domains, where verification is harder. The authors illustrate how model-generated evaluation can operate in both settings and continue to measure systems "beyond the human frontier."

Why it matters

Benchmarks that rely on human designers hit a ceiling when models exceed human experts. The paper offers a mechanism to keep measurement aligned with rising capabilities by making evaluation itself an output of the systems under test. If adopted, this shifts some control of test creation to the modeled agents, potentially preserving evaluative resolution as capabilities grow.

That matters for researchers and evaluators who need scalable, verifiable comparisons between systems when no human can reliably design or judge suitably hard tasks. The protocols the authors describe also try to anticipate incentive problems, such as private-information attacks, which would otherwise undermine comparative measurement.

What to watch

Track two concrete signals the paper itself highlights: the pending registration of the arXiv-issued DOI via DataCite noted in the submission metadata, and further instantiations of the framework across verifiable and open-ended, non-verifiable domains. The authors present both the conceptual framework and practical protocols; evidence of implementations or community adoption will indicate whether the approach can actually scale evaluation beyond human capability.

The paper opens with the question, "How can we measure intelligence beyond human capability?" and proposes an "adversarial psychometric rating system" as the route forward; the upcoming DOI registration and any published instantiations will be the primary milestones to follow.

References and metadata: arXiv:2607.07040, submitted 8 Jul 2026; authors Jerry Han, Rafael Moschopoulos, Ella Colby, Vishrut Goyal, Andrew Tu, Kia Ghods, Mark Braverman, Elad Hazan; arXiv-issued DOI via DataCite (pending registration).