Memory in the Loop: In-Process Retrieval hits ~100µs latency

Yusuf Khan and Carlo Lipizzi's paper Memory in the Loop (arXiv:2607.05690, submitted 6 Jul 2026) shows that placing a memory store inside the agent's reasoning loop reduces memory-answer latency to roughly 100 microseconds and collapses the per-step cost that makes in-loop retrieval expensive. The authors measure in-process store operations with p50 latencies between 80 and 165 microseconds and find redundant actions fall to 0.0 of 12 at in-process speed versus 7.2 of 12 at a 110 millisecond cloud round trip.

How does moving memory inside the loop change agent behaviour?

An in-process store answers in about 100 microseconds, three orders of magnitude faster than networked stores, and that speed converts the store from an intermittently consulted tool into extended working memory available every step. The paper contrasts in-process answers at ~100 microseconds with networked stores that answer in tens to hundreds of milliseconds, and shows a monotonic rise in redundant actions as latency increases: 0.0 of 12 redundant actions at in-process speed, 7.2 of 12 at a 110 ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079).

They demonstrate this end-to-end across four GPT-5-class models under a bounded window: recall improves from 0/5 without in-loop memory to between 3.6 and 4.8 out of 5 with in-loop memory. The authors also report that every write persisted: "The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the agent's read policy, not the store." An instructed restate-every-reply baseline also achieves perfect recall but at a token cost that grows with the working set.

What were the measured bottlenecks and how can they be fixed?

Embedding, not retrieval, is the dominant per-step cost when the store sits across the network: networked embedding takes roughly 200 to 400 milliseconds, dwarfing retrieval in many setups. Pairing the in-process store with a small local embedder returned the complete operation to a measured ~40 microseconds, the paper reports. The authors place store op p50 between 80 and 165 microseconds; with a local embedder the whole read-write-embed cycle drops toward ~40 microseconds.

The paper also quantifies the worst-case inflation: when retrieval is expensive, end-to-end latency per step can inflate by up to 83x. Prior designs limited retrieval to once per turn or relied on serving-layer scheduling. Khan and Lipizzi argue that those choices manage the cost rather than questioning it: the true lever is where the store lives.

Why it matters

If a memory store is fast enough to be directly available every step, the paper argues the agent treats it as extended working memory and changes its behaviour accordingly. That reduces redundant internal actions and improves recall without forcing instruction-heavy workarounds like restating context every reply. The work also relocates the engineering focus: reducing round-trip retrieval latency is valuable, but eliminating the embedding-network bottleneck by running a small local embedder is necessary to unlock sub-millisecond in-loop memory.

This affects anyone building language agents that reason in multiple steps: moving the store in-process and localizing embedding can change architecture choices, token costs, and how read and write policies are designed. The authors show that failures blamed on the store often trace back to the agent's read policy, not data loss in the store.

What to watch

Check whether local embedding solutions reproduce the ~40 microsecond full-cycle figure when paired with in-process stores, and whether other agent benchmarks replicate the redundant-action reductions (0.0 of 12 at in-process versus 7.2 at 110 ms). Also follow whether practical agent stacks adopt in-process stores and small local embedders at scale, and whether read-policy design shifts away from workarounds like restating every reply.