Qwen 2.5 7B vs XGBoost: detecting LLM epistemic blind spots

Akshat Dasula, Prasanna Desikan and Jaideep Srivastava submitted a paper on 17 Jun 2026 that tests whether a large language model can recognize the limits of its knowledge on structured clinical data. The authors compare Qwen 2.5 7B and XGBoost, and report concrete failures of LLM verbalized confidence and methods to produce patient-specific reliability estimates.

What did the authors test and what were the headline findings?

The paper compares Qwen 2.5 7B against XGBoost on a clinical tabular prediction task and measures attribution divergence and calibration. The authors report that the LLM’s verbalized confidence is near-constant, ranging 0.856 to 0.937, even when accuracy varies between 49% and 75.3%. They also show an inverse difficulty effect: the LLM’s accuracy drops to 64.8% when XGBoost is 99% correct, yet the LLM matches XGBoost (73.8% vs. 73.1%) when XGBoost is moderately uncertain.

The paper frames these results as a cold start problem for LLMs on structured data and presents four primary findings: vacuous verbalized confidence, the inverse difficulty effect, the impact of few-shot and SHAP-derived feature evidence, and a cross-model calibrator that maps attribution divergence to reliability.

How did interventions change LLM behaviour and metrics?

Few-shot examples combined with SHAP-derived feature evidence reduced attribution disagreement and improved accuracy without retraining. Specifically, the Attribution Disagreement Score (ADS) fell from 1.54 to 0.38, while accuracy rose from 49% to 75.3% after these super-additive interventions. The authors further introduce a cross-model calibrator that uses attribution divergence signals to estimate LLM reliability; this calibrator reduced expected calibration error from 0.254 to 0.080, replacing the uninformative verbalized confidence with patient-specific reliability estimates.

In short, adding model-agnostic feature evidence and few-shot context substantially aligned the LLM with XGBoost and improved both disagreement and predictive performance on the task the authors studied.

Why it matters

The paper exposes a concrete failure mode: "LLM verbalized confidence is epistemically vacuous," a short phrase the authors use to describe confidence that tracks prompt format rather than correctness. That matters because clinical settings require per-patient reliability, not a generic confidence signal. Demonstrating that cross-model attribution divergence can produce patient-level reliability estimates without model internals or repeated inference suggests a practical path to reduce epistemic uncertainty when deploying LLMs on structured data.

The findings also show LLMs can underperform even when a conventional model is near-perfect, so treating LLM outputs as self-aware risk assessments would be unsafe without external calibration.

What to watch

Look for follow-up work evaluating the cross-model calibrator across other LLM architectures and clinical datasets and for replication of the ADS reductions when different feature-attribution methods are used. The authors note their approach requires no access to model internals, so confirmation on other LLMs would test how general attribution-divergence signals are as reliability predictors.

The paper was accepted at EIML@ICML 2026 and is available on arXiv (submitted 17 Jun 2026), providing the detailed experiments and metrics above.