FirstResearch: Auditable Question Formation for LLM Agents

FirstResearch, submitted to arXiv on 6 Jul 2026, introduces a framework aimed at making the first research question proposed by an LLM scientific-discovery agent inspectable before it is executed. Its core artifact, the Research Question Certificate, records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule.

What is the Research Question Certificate?

The Research Question Certificate is a structured, first-principles record that captures the building blocks of a proposed research question: primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule. The certificate is intended to make the proposed question inspectable before downstream execution, so a scientist can examine the mechanism, falsifier, and assumptions the LLM used to derive the question.

The paper frames the certificate as the framework's central artifact and reports that a certificate-centered core is the strongest component in ablation tests. The authors make code, prompts, saved outputs, and reproduction scripts available at the URL provided in the submission.

How did FirstResearch perform in evaluations?

FirstResearch outperformed controlled prompt-level baselines on ten LLM-agent research topics under a primary DeepSeek-blind-judge protocol, and an independent Gemini-2.5-Flash rescore preserved the system-level ranking. In the Gemini-2.5-Flash rescore of the same 40 baseline packages, FirstResearch scored 4.86 out of 5, versus 4.38 out of 5 for the strongest baseline, with Pearson agreement of 0.865 on average score.

A focused ablation further highlights the certificate effect: certificate-only scoring reached 4.90 out of 5 under DeepSeek and 4.88 out of 5 under Gemini-2.5-Flash, while removing certificates caused scores to drop below 1 out of 5 under both judges. The evaluation therefore ties system-level improvement to the presence of explicit, inspectable derivation artifacts. The authors stress these results are preliminary and use LLM judges rather than human domain experts.

Why it matters

FirstResearch ties a practical auditing mechanism to a core problem in LLM-assisted scientific discovery: the first research question can sound plausible while hiding the mechanism or assumptions that a scientist needs to inspect. By forcing the agent to emit explicit definitions, assumptions, a falsifier and a minimal decisive test, the approach creates a concrete audit trail. The reported scores and the ablation imply those constraints materially change how questions are formed and evaluated by LLM judges.

This matters for teams that plan experiments or literature synthesis with LLM agents, because an inspectable question reduces the chance that an agent will drive downstream work based on unstated or untestable assumptions. The paper itself frames the claim narrowly: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable.

What to watch

The next concrete signals will be human domain expert evaluations of the same certificates and independent reproductions using the provided code, prompts and reproduction scripts. Confirmation that human expert scores mirror the DeepSeek and Gemini-2.5-Flash rankings would substantially strengthen the paper's preliminary claim.