Verbal Reinforcement Learning: Closing the Feedback Loop

Yanwei Cui and six coauthors submitted "Closing the Feedback Loop: From Experience Extraction to Insight Governance in Verbal Reinforcement Learning" to arXiv on 16 Jun 2026, and the paper was accepted to the ICML 2026 RLxF: Reinforcement Learning from World Feedback Workshop (RLxF@ICML 2026) in Seoul, South Korea. The paper targets training-free verbal reinforcement learning, where LLM agents learn from objective world feedback by extracting verbal rules and injecting them as context, updating behavior without changing model parameters.

What did the paper propose?

The paper proposes a three-layer architecture, rules, evidence, and skills, connected by a feedback-driven curation loop to close an insight governance gap. The authors identify a core "retention-forgetting dilemma": retaining stale insights causes negative transfer, while discarding them causes catastrophic forgetting when conditions recur. To navigate that dilemma the paper sets out four requirements: outcome-driven evaluation, persistent structured evidence, non-monotonic knowledge lifecycle, and compositional governance.

The three layers map to concrete functions. Rules distill experience from world outcomes. Evidence logs track rule reliability across episodes. Skills decide which rules to apply, how to resolve conflicts, and when to abstain. The feedback-driven curation loop ties those layers together to manage rules over time.

How does the governance loop work?

Rules capture distilled experience from world outcomes, evidence logs record each rule's reliability, and skills govern selection, conflict resolution, and abstention; the curation loop feeds back reliability and outcomes to update which rules are used. In practice the paper frames world feedback as objective signals such as dynamic task outcomes, market returns, or demand forecasts. Training-free verbal reinforcement learning extracts verbal rules from those signals and injects them into agent context, so the governance loop must decide which extracted rules remain useful as the environment changes.

The architecture treats knowledge as non-monotonic: rules can be promoted or retired based on evidence rather than only accumulated. That persistence of structured evidence is intended to prevent two failure modes the authors name: negative transfer due to stale retained insights, and catastrophic forgetting when useful past signals vanish from short-term memory.

How was the approach evaluated?

The authors used financial forecasting as a case study, where world feedback is naturally abundant, noisy, and non-stationary, and show that accumulated experience either degraded performance below the zero-shot baseline or dramatically improved accuracy and risk-adjusted returns depending on whether the curation loop was present. The paper therefore uses a domain with repeated, shifting conditions to stress the retention-forgetting tradeoff and to demonstrate that the presence of the curation loop flips outcomes between harmful and beneficial use of past experience.

The paper emphasizes that the same accumulated experience produced opposite effects with and without the governance loop, illustrating the practical cost of underinvesting in insight governance compared with experience extraction alone.

Why it matters

LLM agents trained to adapt via context rather than parameter updates are attractive in environments where feedback is plentiful but patterns shift. The paper exposes a governance hole: extraction-heavy methods can hurt agents when environments are non-stationary. By proposing explicit evidence logging and a skills layer to manage rule application, the authors offer a concrete path to reduce negative transfer and avoid catastrophic forgetting while retaining useful past signals for when conditions recur.

That matters for applications that rely on objective, changing signals such as market returns or demand forecasts, because poor governance can flip a system from better-than-zero-shot to worse-than-zero-shot performance.

What to watch

Look for the RLxF@ICML 2026 workshop presentation in Seoul for technical details and examples from the financial forecasting case study. Also watch for any released code or evidence logs from the authors that demonstrate the rules, evidence, and skills interactions under noisy, non-stationary feedback.

The paper is identified on arXiv as arXiv:2606.17591 and was submitted on 16 Jun 2026.