CONCORD: Sparse Asynchronous Aggregation for Device-Cloud RAG

CONCORD, an asynchronous sparse aggregation framework introduced in an arXiv preprint submitted 13 Jun 2026 by Xuedong Hu, Zhiqing Tang, Zhi Yao, Tian Wang and Weijia Jia, targets device-cloud retrieval-augmented generation under document isolation. The paper addresses a setting where private documents remain on edge devices while public knowledge sits in the cloud, and where privacy or policy forbids raw document exchange.

What CONCORD does

CONCORD reframes the cloud not as a continuously synchronized co-generator but as an asynchronously arriving evidence source. The framework introduces two key mechanisms. Waiting debt control decides, at each decoding step, whether to continue waiting for remote participation based on the observed return of waiting. Certificate-guided minimal supplementation requests only the remote evidence needed to determine the current greedy token decision. When a step consults the cloud, CONCORD preserves the same greedy token as dense dual-end aggregation. Steps that do not consult the cloud commit locally without remote evidence.

The paper positions this approach against existing methods that rely on frequent remote synchronization and dense evidence transfer, which the authors say limit throughput under realistic latency and bandwidth conditions.

Evaluation and key results

The authors evaluate CONCORD on two benchmarks: Natural Questions and WikiText-2. On Natural Questions, CONCORD improves end-to-end throughput over baselines by 1.66×, and on WikiText-2 it improves throughput by 2.15×. The framework reduces per-token communication by over two orders of magnitude, while maintaining comparable answer quality and comparable perplexity to the baselines reported. The manuscript is listed as to be published in IEEE ICWS 2026.

Why it matters

Device-cloud RAG setups must balance privacy, latency and bandwidth. CONCORD attacks this trade-off by minimizing remote evidence transfer and making remote participation conditional and sparse. That combination targets two practical pain points: high per-token communication costs and reduced throughput when networks introduce latency. For deployments that keep private documents on-device, the framework implies lower network use and higher token-level throughput without sacrificing the greedy-decoding behavior that dense dual-end aggregation provides when remote evidence is consulted.

What to watch

The paper is slated for IEEE ICWS 2026, which should surface peer-review details and implementation notes. Watch for released code or system-level measurements that reproduce the reported 1.66× and 2.15× throughput gains and the claim of per-token communication cut by over two orders of magnitude.