ForecastBench-Sim: Simulated-World Forecasting Benchmark

ForecastBench-Sim, introduced in an arXiv paper submitted on 17 Jun 2026 by Jaeho Lee, Nick Merrill and Ezra Karger (arXiv:2606.18686), is a simulated-world forecasting benchmark built on Freeciv game rollouts. The benchmark hands forecasters a fixed world report, asks about hidden future states, then continues the simulation to score those forecasts.

What is ForecastBench-Sim?

ForecastBench-Sim is a forecasting benchmark that replaces slow, sparse real-world outcomes with immediately resolvable simulations drawn from Freeciv, a turn-based strategy game modelled on the Civilization series. The paper describes a pipeline that produces structured world snapshots for forecasters and then resumes game rollouts to verify outcomes. The release artifacts and scoring protocol accompany the 15-page paper, which contains five main figures and six appendix figures and was given a spotlight presentation at Forecasting as a New Frontier of Intelligence, the Workshop on AI Forecasting at ICML 2026.

How does the benchmark generate and score questions?

The benchmark generates questions by sampling game rollouts and presenting a fixed world report, a structured snapshot of the current game state. Forecasters answer questions about hidden future states; ForecastBench-Sim then continues the simulation and scores forecasts against the resolved states. Because the world is simulated, the authors say the same setup can generate continuous or binary forecasting questions at arbitrary time horizons. The design also supports paired intervention worlds to produce conditional or causal questions and can produce resolved examples of rare or disruptive outcomes.

What kinds of tasks and evaluations does it include?

ForecastBench-Sim includes multiple question families and a scoring protocol aimed at studying probabilistic reasoning under dynamic world states. The paper describes question families for unconditional forecasting, conditional or causal comparisons using paired interventions, and tasks emphasizing rare events. The authors report validation slices drawn from model evaluations and an anonymized human pilot as part of the benchmark's initial evaluation. The accompanying artifacts include code, data and media linked from the paper's repository information.

Why it matters

ForecastBench-Sim gives researchers control over outcome frequency, time horizon and interventions, which addresses three persistent problems in real-world forecasting benchmarks: slow outcome resolution, sparsity of tail events, and difficulty scoring counterfactual questions. By using Freeciv rollouts, the benchmark produces immediately resolvable, repeatable examples that let teams iterate faster on probabilistic forecasting methods and test causal reasoning in paired-world setups.

What to watch

Watch for public release artifacts and code linked from the paper's deposits, and for follow-up evaluations that expand the model and human validation slices the authors describe. The next concrete signal will be community uptake of the pipeline and whether researchers publish comparative results using ForecastBench-Sim alongside real-world forecasting benchmarks.

Paper and submission details: the work appears as arXiv:2606.18686, submitted 17 Jun 2026, authored by Jaeho Lee, Nick Merrill and Ezra Karger. The document is 15 pages long and includes five main figures and six appendix figures, and it received a spotlight presentation at the Forecasting as a New Frontier of Intelligence workshop at ICML 2026.