SALT benchmark evaluates LLM uncertainty in long-form generation

SALT, a new benchmark released on arXiv (arXiv:2607.03870) and submitted 4 July 2026 by Ido Amit, Ido Galil and Ran El-Yaniv, provides six procedurally generated tasks with single deterministic long textual ground truths so researchers can evaluate uncertainty at unit-level resolution in long-form generation. The authors say the suite enables correctness, calibration and ranking measurements without external judges; code is available alongside the paper and the submission was accepted to the 43rd International Conference on Machine Learning (ICML 2026).

What is SALT and how does it work?

SALT is a six-task benchmark built from procedurally generated problems that each have one deterministic long textual ground truth, enabling "unit-level evaluation of correctness, calibration, and ranking without external judges." The benchmark deliberately uses zero-noise labels because the paper argues uncertainty evaluation is highly sensitive to label imperfections and long-form benchmarks typically depend on fallible labels. By construction SALT measures errors at resolutions from atomic units (tokens or minimal units) up to coarser line-level units, letting experiments probe where confidence signals break down.

What did experiments across 50+ LLMs reveal?

The authors evaluated more than 50 LLMs and extracted three headline findings: confidence functions vary in which uncertainty aspect they dominate; "confidence ranking largely breaks at atomic resolution," even when separability appears at coarser line-level units; and two distinct drivers cause future errors during generation. First, errors can propagate from corrupted prefixes and this propagation is dominated by global context correctness. Second, there is a bounded degradation effect tied to increasing answer-context length. The paper also reports a trade-off introduced by reasoning: using Chain-of-Thought prompting or internalized reasoning improves accuracy while degrading confidence ranking.

The controlled, atom-level interventions SALT enables were central to separating the two drivers of future errors. By corrupting prefixes and varying answer-context lengths, the authors could attribute propagation effects to global context correctness and measure the bounded degradation independently. These controlled manipulations rely on SALT's single-answer deterministic ground truths so that any unit-level mistake is unambiguous.

Why it matters

Reliable uncertainty at token and atomic scales is essential for risk-sensitive applications that cannot discard whole generations when only a small fragment is wrong. SALT shows that confidence signals that look useful at line-level can fail at atomic resolution, which undermines mitigation strategies that assume fine-grained confidence ranking holds. The trade-off between reasoning and confidence ranking is especially consequential: methods that raise accuracy via Chain-of-Thought or reasoning-internalization may make it harder to spot the model's remaining mistakes using its own confidence estimates.

What to watch

Follow publications and presentations from ICML 2026 where the authors will present SALT and experiment details; the paper's code release is available with the arXiv submission. Researchers should watch for follow-up work that applies SALT-like deterministic tasks to new uncertainty metrics or that attempts to restore atomic-level ranking while preserving reasoning gains.

Paper and authors: "Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth," Ido Amit, Ido Galil, Ran El-Yaniv, arXiv:2607.03870, submitted 4 July 2026, accepted to ICML 2026. The authors report experiments across 50+ LLMs and a six-task benchmark named Single-answer Atomic Long-form Target (SALT).