Reasoning Consistency Scanning: 60-transcript audit InspectScout

Silvia Santano submitted Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations to arXiv on 8 Jul 2026. The paper formalizes "reasoning consistency", defines a six-subtype taxonomy, builds a validated benchmark of 60 transcripts, implements a scanner for InspectScout, and reports results across four generator models and three evaluations.

What is reasoning consistency scanning?

Reasoning consistency scanning is a method for detecting whether a model's stated chain-of-thought is logically consistent with its answer, using only the transcript of the interaction. The approach treats consistency as distinct from faithfulness; consistency can be assessed post hoc from transcripts with no experimental intervention, while faithfulness requires controlled interventions to test whether the stated reasoning reflects the actual process.

The paper presents a formal definition of consistency and a six-subtype taxonomy of inconsistency to categorize mismatches between stated reasoning and answers. That taxonomy is part of the framework Santano proposes for routine audits of safety-evaluation transcripts.

What did Santano build and test?

Santano produced three concrete artifacts: a formalization of reasoning consistency, a validated benchmark of 60 adapted transcripts, and a working scanner integrated into InspectScout that targets consistency in safety-evaluation transcripts. The 60 transcripts were manually adapted from InstrumentalEval outputs and the empirical evaluation covers outputs from four generator models across three inspect_evals evaluations.

The paper reports that "reasoning inconsistency is present, detectable, and varies systematically across both models and task types." Santano implemented the scanner to operationalize the taxonomy and used the benchmark to validate detection on real evaluation transcripts. The benchmark and experiments tie the conceptual taxonomy to measurable outcomes in InspectScout.

How does this differ from prior work on chain-of-thought?

Unlike prior work that focuses on faithfulness, which requires interventions to judge whether a model actually used the stated reasoning process, reasoning consistency scanning restricts itself to what can be measured from transcripts alone. Santano positions consistency as a more tractable, audit-friendly property that can be checked retrospectively. The benchmark adapts InstrumentalEval outputs for manual validation, and the experiments use inspect_evals to compare behaviour across models and task types.

Why it matters

A consistency-first audit gives evaluators a practical lever: it lets teams flag logical contradictions and subtype them without running costly interventions. Finding and categorizing inconsistency in transcripts creates a lower-effort signal that can surface problematic outputs during safety evaluations. Santano’s scanner for InspectScout turns taxonomy into an operational tool, so audit pipelines that already collect transcripts can run consistency checks as part of model assessment.

What to watch

Look for public releases of the 60-transcript benchmark, the taxonomy details and the scanner implementation for InspectScout; Santano’s submission to arXiv lists those as the main contributions. The next milestone will be whether other evaluations adopt the taxonomy or report similar systematic variation across models and task types when they apply the scanner to inspect_evals-style transcripts.

References and factual anchors: arXiv:2607.07229, submitted 8 Jul 2026; author Silvia Santano; benchmark of 60 transcripts adapted from InstrumentalEval; six-subtype taxonomy; scanner implemented for InspectScout; results reported across four generator models and three inspect_evals evaluations.