AgentLens: production-assessed benchmark for coding agents

AgentLens is a production-assessed benchmark that evaluates entire interactive code-agent trajectories rather than reducing a run to a single pass/fail bit. Submitted to arXiv on 7 Jul 2026 as arXiv:2607.06624 by seven authors, the benchmark pairs formal verification with LLM-written trajectory reviews and side-by-side comparisons, and the authors release the benchmark as open source.

What is AgentLens and what does it measure?

AgentLens measures the full trajectory of a coding agent: how it follows instructions, uses tools, verifies its own work, recovers from mistakes, and communicates with users. The opening definition in the paper states that most code-agent benchmarks reduce a run to a single bit, while AgentLens evaluates that whole trajectory, producing a readable explanation for why each score was assigned.

The project authors are Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, and Sergey Nikolenko. The paper is available on arXiv (arXiv:2607.06624) and includes a DOI link at 10.48550/arXiv.2607.06624.

How does AgentLens evaluate an agent run?

AgentLens combines formal verification, LLM-written trajectory reviews, and side-by-side comparisons to turn each run into a human-readable review plus a formal score. The benchmark pairs objective checks where they exist with LLM-generated narrative reviews, so each run yields both a formal check and an explanation of why the score is what it is. The paper emphasizes that this design makes AgentLens useful for ranking models and for diagnosing detailed behavior.

The authors describe using AgentLens for three concrete production tasks: diagnosing model behavior, comparing successive versions of their own agent, and catching product regressions in a nightly evaluation pipeline. Each of those uses relies on the combination of automated verification and narrative trajectory reviews to surface regressions or behavioral changes that a single pass/fail metric would miss.

Why does this approach matter?

AgentLens matters because users of coding agents experience the full interaction trajectory, not just the final answer. By evaluating how an agent reasons, retries, tool-uses, and explains itself, the benchmark surfaces issues that binary success metrics hide. That lets teams debug model behavior, validate incremental updates, and detect regressions in continuous evaluation, according to the authors.

This approach changes the unit of evaluation from a pass/fail outcome to a documented interaction, which is especially relevant for production deployments where recoverability, verification, and explanation are operational concerns. The paper positions AgentLens as useful beyond ranking models: the benchmark is explicitly used to diagnose behavior and to support a nightly pipeline for regression detection.

What did the authors release and where?

The paper states the authors release the benchmark as open source at the URL included in the submission. The arXiv record lists the full-text PDF and TeX source, making the methodology and implementation available for inspection and reuse. The submission metadata shows the paper was submitted on 7 Jul 2026 to the cs.AI category on arXiv.

What to watch

Watch whether other teams adopt AgentLens for their own continuous evaluation or nightly regression pipelines, and whether its combination of formal checks plus LLM-written trajectory reviews surfaces different failures than existing pass/fail benchmarks. Also watch the open-source repository for code, data, and integration examples the authors provided in the paper's linked materials.