ARC-AGI-1: Agent harnesses hit 67.25% pass@2 at $0.62

An Explorer-Definer pipeline reached 57.50% pass@2 on the public 400-task ARC-AGI-1 set at $0.25 per task. A Reflective Orchestrator augmentation pushed that to 67.25% pass@2 at $0.62 per task, all using the open-weight model DeepSeek V3.2 in non-thinking mode and without ARC-specific fine-tuning.

What did the paper achieve on ARC-AGI-1?

On the public 400-task ARC-AGI-1 evaluation set, an Explorer-Definer pipeline reached 57.50% pass@2 at $0.25 per task, and a Reflective Orchestrator reached 67.25% pass@2 at $0.62 per task. Both experiments ran under a strict budget with no benchmark-specific training and used DeepSeek V3.2 in non-thinking mode.

The paper contrasts those results with a 15.50% one-shot baseline, reporting that the two architectures together lift that baseline by roughly 52 percentage points. The authors also report that removing the pipeline’s think tool reduces pass@2 by 5.75 percentage points, and that an unbiased pass@1 analysis shows a +9.81 percentage-point lift from the orchestrator.

How do the Explorer-Definer pipeline and Reflective Orchestrator work?

The Explorer-Definer pipeline separates pattern discovery and executable transformation synthesis into two explicit agent stages: an Explorer for pattern discovery and a Definer for synthesizing executable transformations. The design intentionally avoids ARC-specific fine-tuning and relies on architecture to recover capability under a strict cost budget.

The Reflective Orchestrator augments this pipeline by autonomously exploring new transformations when previous hypotheses fail on training pairs. The orchestrator implements adaptive re-exploration to broaden generation when early candidates do not solve tasks, and the authors report that this adaptive approach produces an unbiased pass@1 lift of +9.81 percentage points while matching selection-mediated pass@2 gains.

The paper diagnoses the system as generation-bound rather than selection-bound, noting that selection via training-pair accuracy captures about 95% of the candidate ceiling. That diagnosis motivates the orchestrator’s focus on expanding generation rather than improving ranking.

Why it matters

These results show substantial ARC-AGI-1 gains can come from agentic architecture and search strategies rather than heavier test-time compute or benchmark-specific fine-tuning. Raising pass@2 from a 15.50% one-shot baseline to 67.25% at modest per-task costs demonstrates that careful decomposition of discovery and synthesis plus adaptive re-exploration recovers much of the task capability under a strict budget.

The distinction the authors draw between generation-bound and selection-bound failure matters for research priorities: if selection captures ~95% of the candidate ceiling, allocating compute and design to produce more and broader candidate generations is the clearer path to further gains.

What to watch

Watch for replications that expand generation budgets or iterate adaptive re-exploration strategies to confirm the diagnosis that broader generation, not better ranking, drives improvements. The paper’s concrete signals to follow are unbiased pass@k lifts (the paper reports +9.81 pp at pass@1) and whether architectures that increase generation breadth produce additional pass@2 gains beyond the reported 67.25%.