Bayesian Uncertainty for Agentic RAG, tested on HotpotQA

Louis Donaldson, Connor Walker, Koorosh Aslansefat and Yiannis Papadopoulos submitted a proof-of-concept paper on 1 Jul 2026 that applies Bayesian uncertainty propagation to Agentic Retrieval-Augmented Generation pipelines. The study evaluates planner, evaluator and generator uncertainty signals on StrategyQA and HotpotQA using GPT-3.5-Turbo and GPT-4.1-Nano and reports performance via AUROC, AUARC, ECE and Brier Score.

What did the paper propose and test?

The paper defines an uncertainty-aware Agentic RAG framework in which planner, evaluator and generator stages each produce uncertainty signals, and those signals are combined via a Bayesian Network to yield system-level uncertainty and node-level failure indicators. The authors implemented the method as a proof-of-concept and ran experiments on two multi-hop question-answering datasets, StrategyQA and HotpotQA, with evaluations conducted using GPT-3.5-Turbo and GPT-4.1-Nano.

The framework extracts uncertainty from semantic divergence and generator self-evaluation, then propagates those measures through the Bayesian Network. The goal is to estimate when multi-stage reasoning pipelines may fail and to provide actionable indicators about which pipeline node is likely responsible.

How did Bayesian propagation perform on HotpotQA and StrategyQA?

Bayesian propagation proved more effective on HotpotQA, where uncertainty accumulates across multi-hop reasoning stages, while StrategyQA revealed limitations tied to miscalibration and unreliable upstream signals. The authors evaluated discrimination, selective prediction and calibration using Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Accuracy-Rejection Curve (AUARC), Expected Calibration Error (ECE) and Brier Score.

The paper attributes HotpotQA gains to explicit multi-hop structure that lets uncertainty signals compound across planner, evaluator and generator steps, making system-level risk easier to detect via Bayesian combination. StrategyQA, by contrast, exposed cases in which upstream signals were poorly calibrated, reducing the effectiveness of propagated uncertainty for selective prediction and calibration metrics.

Why it matters

Reliable uncertainty estimates are essential for deploying agentic RAG systems in settings where errors carry real costs. The paper positions Bayesian uncertainty propagation as a mechanism to surface both system-level risk and node-level failure indicators, which could guide human intervention or automated safeguards. The authors also note the approach is preliminary and must be validated in industrial contexts, naming Offshore Wind maintenance decision support as an example target domain for future work.

This matters because multi-stage pipelines are common in retrieval-augmented systems, and current practice often lacks principled ways to combine heterogeneous uncertainty signals across planning, retrieval and generation stages. A Bayesian Network offers a clear, probabilistic method to fuse those signals and produce interpretable diagnostics.

What are the study's limits?

The paper is explicit that the results are proof-of-concept and dataset-dependent. StrategyQA results showed the approach can fail when upstream signals are miscalibrated or unreliable, indicating that signal quality at each node remains a gating factor. The authors call the mechanism "promising but preliminary" and recommend further validation.

What to watch

Look for follow-up validation in industrial case studies and replication on additional multi-hop benchmarks. The authors flag Offshore Wind maintenance decision support as a target for future validation, and subsequent work that reports real-world deployment metrics or extends signal extraction for better calibration would be a direct test of the approach.

Metadata: the paper was submitted to arXiv as arXiv:2607.00972 on 1 Jul 2026 and was prepared for submission to the 7th International Conference on Maintenance and Intelligent Asset Management (ICMIAM 2026). The evaluation uses GPT-3.5-Turbo and GPT-4.1-Nano and measures AUROC, AUARC, ECE and Brier Score.