Generalist agents memory: arXiv paper defines what to store

Khurram Yamin and five coauthors submitted a paper titled "What Must Generalist Agents Remember?" to arXiv on 17 Jun 2026 (arXiv:2606.18746). The paper develops a formal account of the information a generalist agent must store in memory to act near-optimaly across multiple environments and goals.

What does the paper prove?

The paper proves two core claims: first, when two domains share an observational bottleneck but require incompatible optimal actions, any uniformly near-optimal policy must induce distinct memory distributions at that bottleneck. Second, if an agent's memory contains enough information to estimate values for related goals, that memory can be used to approximately reconstruct the agent's local transition dynamics. Together these formal results show memory is required for domain disambiguation and for recovering transition information used in planning.

The authors state this leads to a separation theorem: sufficiently successful agents cannot rely only on current state observations, but must preserve domain-relevant information in memory. They frame memory as a substrate supporting three capabilities: domain disambiguation, transition-model reconstruction, and planning.

How do the results work?

The first result relies on a setup where two domains share an observational bottleneck yet demand incompatible optimal actions; in that setting the paper shows any policy that is uniformly near-optimal across both domains must produce different memory distributions at the bottleneck. The second result shows a converse direction: memory that suffices to estimate values for related goals contains enough signal to approximately reconstruct local transition dynamics.

The technical contribution is a formal characterization of memory requirements rather than an empirical evaluation. The paper frames its theorems around observational bottlenecks, value estimation from memory, and reconstruction of local dynamics, tying these elements into a single account of what memory must encode for generalist performance.

Why it matters

The paper gives a crisp theoretical answer to a practical question about generalist agents: observation alone can be provably insufficient when environments share compressed perceptual channels but need different actions. That forces designers to treat memory as an explicit resource that must encode domain identity or task-relevant histories. By showing memory can support approximate transition-model reconstruction, the work connects memory design to downstream planning capabilities rather than treating memory as mere short-term context.

Framing memory this way shifts evaluation: success on multi-domain goals implies constraints on the distribution of memory states, not only policy performance on isolated tasks.

What to watch

Look for follow-up work that applies these formal conditions to concrete architectures and training regimes, testing whether learned memory representations indeed induce the distinct bottleneck distributions the paper requires. Also watch for empirical studies that use value-based memory signals to reconstruct local dynamics, validating the paper's second result in practice.

Authors and bibliographic details: the paper is authored by Khurram Yamin, Namrata Deka, Maitreyi Swaroop, Albert Ting, Jeff Schneider, and Bryan Wilder and is available on arXiv as arXiv:2606.18746, submitted 17 Jun 2026. The abstract summarizes the conclusion succinctly: memory is described as "the substrate that supports domain disambiguation, transition-model reconstruction, and planning for generalist agents."