Causal Reinforcement Learning: arXiv primer by Bareinboim et al

Elias Bareinboim, Junzhe Zhang and Sanghack Lee submitted "An Introduction to Causal Reinforcement Learning" to arXiv on 23 Jun 2026 (arXiv:2606.24160), arguing that causal inference and reinforcement learning address the same counterfactual relations and should be studied together. The 3,015 KB submission frames a unified formal treatment and introduces several new classes of learning that extend standard RL modes.

What is causal reinforcement learning?

Causal reinforcement learning is the authors' proposed unifying view that explicitly combines causal inference and reinforcement learning so agents can reason about counterfactuals and exploit causal structure in environments. The paper defines any RL environment as decomposable into autonomous mechanisms with causal invariances, modeled parsimoniously as a structural causal model, and uses that formalization to broaden what RL can learn and ask.

The authors contrast the disciplines by function: causal inference supplies principles for answering counterfactual questions when data for the unrealized reality are absent, while reinforcement learning provides methods to learn a policy that optimizes a measure such as reward or regret through trial and error. They argue the overlap—counterfactual relations—creates novel learning opportunities when mathematized together.

How does the paper unify RL and causal inference and what new modes does it introduce?

The paper places standard RL settings inside a structural causal model and unifies online, off-policy, and causal calculus learning under a single treatment, then introduces additional learning classes including generalized policy learning, where to intervene, imitation learning, and counterfactual learning. By showing that any standard RL problem implicitly encodes a structural causal model, the authors treat multiple learning modalities as aspects of the same formal framework.

Specifically, the manuscript notes that environments can be decomposed as a collection of autonomous mechanisms with different causal invariances, and that modeling these as structural causal models lets researchers reason across modes of learning that previously appeared unrelated. Beyond that unification, the paper names novel and pervasive classes of learning settings that expand the scope of counterfactual learning, calling the combined field causal reinforcement learning or CRL.

How does the paper position CRL relative to existing RL tasks?

The authors map familiar RL tasks into causal terms and then extend them: online and off-policy learning and causal calculus learning are recast as operations over the underlying structural causal model, while generalized policy learning, imitation learning, and counterfactual learning are presented as natural extensions that exploit interventions and counterfactual reasoning. This reframing means problems like "where to intervene" become formal parts of policy design rather than informal heuristics.

The paper argues these extensions let practitioners ask new questions, for example which interventions change long-run outcomes, or how to learn from demonstrations when distributional shifts correspond to changes in causal mechanisms rather than mere covariate shifts.

Why it matters

Bringing causal inference and reinforcement learning into a single mathematical framework changes what agents can conclude from data and how they generalize. If environments are represented as structural causal models, policies can leverage causal invariances to transfer across settings, reason about unseen contingencies, and answer counterfactuals that standard RL cannot. That matters for any application where interventions alter underlying mechanisms rather than just observed distributions.

What to watch

Look for follow-on work that formalizes and benchmarks the named learning classes, especially generalized policy learning and counterfactual learning, and for empirical studies testing whether modeling environments as structural causal models improves policy transfer. The arXiv submission date is 23 Jun 2026 and the paper is available as arXiv:2606.24160 for researchers to build on.

References

Paper: "An Introduction to Causal Reinforcement Learning," Elias Bareinboim, Junzhe Zhang, Sanghack Lee, arXiv:2606.24160, submitted 23 Jun 2026 (3,015 KB).