Self-CTRL Reinforcement Learning: Improves LM Self-Consistency

Self-CTRL, introduced in a paper submitted on 16 Jun 2026 by Itamar Pres, Laura Ruis, Melat Ghebreselassie, Belinda Z. Li and Jacob Andreas, is a training method that optimizes consistency between a language model's self-explanations and its behavior. The authors report that consistency training raised the correlation between self-reported and behaviorally measured biases from R^2 = 0.24 to R^2 = 0.64 and improved refusal prediction by a third-party auditor from 36% to 92%.

What is Self-CTRL and how does it work?

Self-CTRL is a training procedure that enforces agreement between a model's explanations and its actions by updating explanations to better predict behavior or by updating behavior to better match explanations. The method explicitly optimizes for consistency between a language model's self-explanations and its behavior on related inputs, rather than only supervising behavior or explanations independently.

The paper applies this approach in two domains. First, a formal probabilistic reasoning task where models must imitate a family of biased samplers and then report the associated biases. Second, a constitutional AI domain in which models must describe when they will refuse or comply with user requests. The authors present experiments and analyses across those settings in a 34-page manuscript with 12 figures.

What were the key results?

On the probabilistic reasoning task, Self-CTRL increased the correlation between self-reported and behaviorally measured latent biases from R^2 = 0.24 to R^2 = 0.64 on a set of held-out distributions, a gain the authors say matches the generalization achieved with direct ground-truth supervision. In the constitutional AI experiments, Self-CTRL produced rules that better described the model's behavior on held-out requests, improving refusal predictions of a third-party auditor model from 36% to 92%.

The paper also reports that updating behavior (the opposite direction of alignment) reduced HarmBench failure rate from 15.0% to 0.5% while not substantially increasing refusal on harmless prompts. Those three concrete numbers — R^2 0.24 to 0.64, auditor refusal predictions 36% to 92%, and HarmBench failure 15.0% to 0.5% — anchor the empirical claims across the two domains.

Why it matters

Language models that faithfully describe their own behavior can be audited, understood, and trusted more easily, and Self-CTRL provides a concrete recipe for improving that match. The method ties interpretability and safety metrics together: the same consistency objective both raises a statistical measure of explanation fidelity (R^2) and moves a safety benchmark (HarmBench) in the desirable direction. For developers and auditors, that suggests training objectives that explicitly link explanations to actions can produce measurable and complementary gains.

What to watch

The next signals to follow are replication and scope. See whether the R^2 and HarmBench gains replicate on other model families and datasets beyond the two domains studied, and whether third-party auditors continue to benefit when models are trained with consistency objectives. The paper is available on arXiv as arXiv:2606.18327 and includes experiments and figures that detail the reported results.