Autoformalization: Agent Instructions to Policy-as-Code

Adam Mondl, Matthew Maisel and John H. Brock submitted a paper on 25 Jun 2026 (arXiv:2606.26649) describing an autoformalization pipeline that translates agent prompts, multi-component tool descriptions and natural-language policy documents into formally verified policies. The authors say the output policies are written in the Cedar Policy Language and that the pipeline uses an LLM-based generator-critic loop. The paper was accepted at the Second Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD), ICML 2026.

What did the authors build?

They built an autoformalization pipeline that converts three input types—agent prompts, MCP tool descriptions, and natural-language policy documents—into formally verified policies expressed in the Cedar Policy Language. The paper frames this as an alternative to probabilistic guardrails and hand-coded symbolic enforcement, and reports that, on the MedAgentBench benchmark, the autoformalized policies "cover substantially more of the source natural-language specification" than prior hand-coded symbolic enforcement.

The submission characterizes existing approaches as either probabilistic (fine-tuned classifiers and prompt-based steering) offering no formal guarantees, or as hand-coded symbolic enforcement that does not scale. The pipeline addresses that gap by producing formally verified policies that scale across broader specifications.

How does the autoformalization pipeline work?

The pipeline feeds agent prompts, tool descriptions and policy documents into an LLM generator-critic loop that outputs policies in Cedar. The generator produces candidate formalizations and the critic evaluates or filters those candidates, producing a set of formally verified policies as the result.

The authors present this LLM-driven generator-critic loop as the core mechanism: the generator drafts formal policy artifacts from natural language, while the critic checks and refines those drafts toward formal verification. The paper positions Cedar Policy Language as the target representation for those policies and evaluates the pipeline against MedAgentBench to measure coverage of source specifications.

Why it matters

Autoformalization could close the gap between informal safety requirements and enforceable policy-as-code by scaling formal policy creation beyond hand-written symbolic rules. If an LLM-driven pipeline can reliably emit Cedar policies that capture more of a natural-language specification, teams operating in high-stakes domains can move from probabilistic guardrails to formally verified enforcement without hand authoring every rule. That reduces manual labor and addresses the lack of formal guarantees in many current deployments.

How did the pipeline perform in evaluation?

On the MedAgentBench benchmark the authors report that their autoformalized policies cover substantially more of the source natural-language specification than prior hand-coded symbolic enforcement. The paper highlights this comparative coverage as its primary empirical claim, using MedAgentBench as the evaluation corpus.

What to watch

Look for the workshop presentation at AIWILD, ICML 2026 and any released code or examples tied to arXiv:2606.26649. The next concrete signals will be reproducible artifacts showing the generator-critic loop producing Cedar Policy Language policies for new policy documents and independent measurements of formal verification outcomes on standardized benchmarks such as MedAgentBench.