Agentic Review Systems: benchmark with GPT-5.5 hits 83% pairwise

Benchmarking Agentic Review Systems, submitted to arXiv on 18 Jun 2026, evaluates how full agentic review systems perform on real research papers and synthetic error tests. The paper compares two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline, across six LLMs spanning frontier and efficient models.

What did the authors test and on which systems?

The paper measures two outcomes: whether AI reviews track paper quality using external signals, and whether systems detect injected errors with known ground truth. The authors evaluated OpenAIReview, coarse, Reviewer3, and a zero-shot baseline across six LLMs; they also built a perturbation benchmark that injects four categories of errors into papers across eight arXiv subject classes.

The evaluation covered both paper-level signals (citations and acceptance decisions) and error-level detection. The perturbation benchmark supplies ground-truth faults so the authors could measure detection recall. The study includes a public deployment of OpenAIReview and reports real-user feedback from that deployment.

How well did the systems perform on paper quality and error detection?

Every system scored above chance on pairwise accuracy when judged against external signals, and the best configuration, OpenAIReview paired with GPT-5.5, reached 83.0% pairwise accuracy. For error detection, the strongest configuration, again OpenAIReview with GPT-5.5, achieved 71.6% recall for injected errors.

The union of detections across all six models reached 83.3% recall, indicating models discovered different errors and that combining model outputs can increase coverage. Despite these strengths, the paper notes substantial room for improvement: the top configuration still missed roughly 28.4% of injected errors.

What did real users say about deployed AI reviews?

OpenAIReview saw positive skew in user votes on its comments, with votes favoring its comments at a ratio of 1.44 to 1. The most common user complaints centered on false positives and minor nitpicks rather than hostile rejection of the system. The paper contains 11 pages, 7 tables, and 4 figures documenting the benchmarks and deployment observations.

Why it matters

The evaluation shows agentic review systems can already track human quality signals and find many real or injected problems, but they are not flawless. An 83.0% pairwise accuracy and a 71.6% detection recall for the strongest configuration mean these systems can assist triage and error finding while leaving substantial gaps. The 83.3% recall achieved by aggregating six models suggests practical improvements may come from better multimodel design rather than only scaling a single model.

These results matter because peer review faces pressure from AI-assisted research and rising submission volumes; automated review tools that reliably surface issues or rank submissions could shift workloads and reviewer priorities. Positive user votes for OpenAIReview in public deployment indicate that reviewers may accept AI comments when they are helpful, but complaints about false positives show a need for precision and better calibration.

What to watch

Look for studies or releases that clarify which of the six LLMs the authors used and how each model's strengths differ, since this paper shows model diversity improves recall but does not name all models. Also watch for follow-up work that reduces false positives and nitpicks in deployed comment streams, and for attempts to operationalize model unions or ensembles to approach the 83.3% union recall in production.

Bibliographic note: the paper is arXiv:2606.19749, titled "Benchmarking Agentic Review Systems," authored by Dang Nguyen, Wanqing Hao, Yanai Elazar, and Chenhao Tan, submitted 18 Jun 2026.