Akashic MemAttention: Low-overhead LLM service, +10.2 pts

Akashic, presented in an arXiv paper submitted 7 Jul 2026, is a low-overhead memory system for LLM inference built around a mechanism the authors call MemAttention. The paper, authored by Yang Liu, Zhaokai Luo, Huayi Jin, Ruozhou He, Chenchen Hong, Zhiyong Wang, Yifei Liu, Yunfei Gu, Chentao Wu, and Junhao Hu, evaluates Akashic across four representative workloads and three model sizes and reports measurable gains in accuracy and serving efficiency.

How does Akashic work?

Akashic organizes context into bounded chunks and models semantic relationships across chunks using MemAttention, while applying hardware-software co-designed memory placement to co-locate likely co-retrieved chunks. In plain terms, the system avoids repeatedly rewriting the full interaction history by preserving cross-chunk evidence, and it reduces retrieval fragmentation and I/O overhead by placing chunks that are likely to be fetched together near each other in memory.

The paper frames the motivation succinctly: "replaying the full history for every request quickly becomes impractical." Long contexts raise prefill cost, can exceed context limits, and can bury task-relevant evidence in irrelevant content. Akashic’s chunking plus cross-chunk semantic modeling aims to preserve the evidence an LLM needs without resending or reconstructing the entire history on every request.

How much does Akashic improve serving and outputs?

Across the evaluated workloads and model sizes, Akashic improves task accuracy by up to 10.2 points, increases throughput by up to 1.21x, and raises sustainable request rate by up to 1.88x over strong prior memory baselines. Those three concrete numbers come directly from the paper’s abstract and summarize the empirical gains the authors report.

The improvements are presented as upper bounds observed in the experiments: up to 10.2 points on task accuracy, up to 1.21x throughput, and up to 1.88x sustainable request rate. The authors tie these gains to both MemAttention’s ability to preserve cross-chunk evidence and to the hardware-software co-design that minimizes retrieval overhead by co-locating likely co-retrieved chunks.

Why it matters

Long-lived LLM-based agents and multi-turn systems accumulate context across interactions, tool calls, and sessions. Replaying full histories on every query inflates prefill costs and can push models past context limits; it also mixes relevant and irrelevant content in a way that can degrade output quality. Akashic addresses these operational pain points by keeping context bounded yet semantically connected, cutting unnecessary I/O and preserving the task-relevant signals the model needs to produce better outputs.

That combination of software-level memory modeling and hardware-aware placement matters because it targets both the algorithmic source of lost evidence and the practical bottleneck of retrieval overhead. The paper’s reported gains in accuracy, throughput, and sustainable request rate make this a systems-oriented proposal rather than a purely algorithmic tweak.

What to watch

See whether the Akashic approach generalizes beyond the four representative workloads and three model sizes evaluated in the paper, and whether the authors release code or detailed hardware-placement recipes that others can reproduce. Real-world adoption will hinge on how the chunking, MemAttention scoring, and co-location policies perform at larger scale and in production agent workflows.