Feedback Manipulation Regularization (FMR): 98% Misalignment Cut

Benjamin Poole and Minwoo Lee submitted Feedback Manipulation Regularization on 8 Jul 2026, an algorithm-agnostic technique that uses evaluative feedback as a corrective signal to align imitation-learning policies. The authors adapted Safety Gymnasium environments for alignment evaluation and report improved aptitude and up to a 98% reduction in misalignment across a range of imitation learning algorithms.

What is Feedback Manipulation Regularization (FMR)?

FMR is an algorithm-agnostic regularization method that treats evaluative feedback as a corrective signal during single-stage offline training, aimed at improving alignment of imitation learning policies. It departs from multi-stage pipelines common in language-generation work by combining demonstrations and feedback directly in fully sequential decision-making environments rather than the contextual bandit framing.

The proposal frames evaluative feedback not as a separate post-processing stage but as a manipulation of the training loss to nudge policies toward aligned behavior. The paper positions FMR as compatible with multiple imitation learning algorithms and suitable for offline settings where training uses logged demonstrations and feedback.

How did the authors test FMR and what did they measure?

Poole and Lee adapted Safety Gymnasium environments as a principled testbed for alignment evaluation and applied FMR across a range of imitation learning algorithms. The core empirical claim in the abstract is twofold: FMR improves aptitude, and it achieves up to a 98% reduction in misalignment.

The experiments emphasize offline, fully sequential decision-making scenarios rather than contextual bandit setups typical of language-model alignment work. The abstract also states FMR remains robust in limited data regimes, including cases with scarce aligned demonstrations and uninformative noisy demonstrations.

How does FMR differ from existing alignment approaches?

FMR contrasts with prevalent pipelines that combine human demonstrations and evaluative feedback in multiple stages designed for contextual bandit problems in language generation. Instead, FMR integrates feedback into a single-stage offline objective tailored to sequential decision-making. That single-stage framing is the main methodological divergence claimed in the paper's abstract.

The method is described as algorithm-agnostic, meaning it is presented as applicable across different imitation learning algorithms rather than tied to one specific architecture or training recipe.

Why it matters

Offline alignment for sequential decision-making is a distinct technical challenge from bandit-style reward learning. Embedding evaluative feedback directly into the offline imitation objective offers a path to train agents that behave in line with human evaluative signals without requiring multi-stage re-ranking or separate reward modeling pipelines. If the reported up to 98% reduction in misalignment holds across more environments and tasks, that scale of improvement would materially lower the burden of post-hoc corrections and human oversight in certain offline settings.

FMR's claimed robustness in limited-data regimes is also significant: alignment methods that still work when aligned demonstrations are scarce or noisy address a common bottleneck for real-world deployments where high-quality labeled trajectories are expensive.

What to watch

Follow-up evaluations beyond Safety Gymnasium, and replication of the "up to a 98% reduction in misalignment" claim across other sequential environments, will be the key signals to validate FMR's generality. Evidence that FMR integrates cleanly into existing imitation learning stacks across different algorithms would confirm its algorithm-agnostic promise.

Authors and citation details The paper is titled "Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning," authored by Benjamin Poole and Minwoo Lee, submitted 8 Jul 2026 and available as arXiv:2607.07859. The abstract states the method, testbed adaptation, and the empirical headline of up to a 98% reduction in misalignment.