RL-Finetuned VLMs: Robustness and CoT Consistency, ICML 2026

On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs, a paper published July 2026 for the ICML venue, finds that reinforcement learning finetuning raises visual-reasoning benchmark accuracy while creating new failure modes in reasoning traces and robustness.

The paper is authored by eight researchers — Rosie Zhao, Anshul Shah, Xiaoyu Zhu, Xinke Deng, Zhongyu Jiang, Yang Yang, Joerg Liebelt, and Arnab Mondal — with Rosie Zhao marked as affiliated to Harvard University and Yang Yang to OpenAI, and both Rosie Zhao and Yang Yang noted as having done work while at Apple.

What are the paper's main findings on robustness and CoT consistency?

The authors find that RL finetuning improves benchmark accuracy but introduces an "accuracy–faithfulness trade-off": finetuning raises benchmark scores while eroding reliability of the chain-of-thought and robustness to contextual shifts. They show that simple, controlled textual perturbations, such as misleading captions or incorrect chain-of-thought traces, cause substantial drops in both robustness and model confidence, and that these effects become more pronounced when chain-of-thought consistency is measured across open-source multimodal reasoning models.

The study contrasts open-source RL-finetuned models with closed models, reporting that closed models exhibit similar failure modes yet sustain markedly greater robustness and reasoning consistency, a gap the authors interpret as a shortcoming in current open-source RL finetuning rather than an inherent limit of the task.

How do interventions like augmentation and reward shaping affect performance?

Adversarial augmentation improves robustness but does not by itself prevent faithfulness drift, the paper finds. Introducing a faithfulness-aware reward can restore alignment between answers and reasoning, however combining that reward with adversarial augmentation risks training collapse onto shortcut strategies and leaves robustness elusive.

In short, augmentation helps robustness in isolation, faithfulness-aware rewards help align answers with CoT, but the two together can produce perverse optimization outcomes that hurt robust, faithful reasoning.

How did the authors situate this work in prior research?

The paper points to earlier analyses of chain-of-thought dynamics and VLM training recipes as context. It references a February 24, 2026 study on CoT trace dynamics and a June 5, 2025 paper on improving vision language model CoT reasoning, which argued that datasets dominated by short annotations produce weak generalization for longer explanations. Those prior works motivated the present focus on the reliability and faithfulness of CoT outputs after RL finetuning.

Why it matters

The findings challenge the dominant evaluation lens that prizes benchmark accuracy alone. If RL finetuning can boost scores while degrading the faithfulness of reasoning traces and sensitivity to textual perturbations, then deployed multimodal systems risk producing confident but ungrounded explanations. The contrast between open-source and closed models further implies that training protocols, not task fundamentals, drive much of the observed gap in robustness and CoT consistency.

What to watch

Follow whether follow-up work adopts combined evaluation protocols that jointly measure correctness, robustness, and CoT faithfulness, and whether open-source training recipes incorporate faithfulness-aware reward signals without triggering shortcut collapse. The paper itself was published July 2026 at ICML and its authors highlight the practical tension between augmentation and reward shaping as the next experimental frontier.

References in the paper include a February 24, 2026 analysis of CoT trace dynamics and a June 5, 2025 paper on VLM CoT training that motivated the authors' focus on longer rationales.