A new benchmark from Zhenran Wang, Zhonghan Bian, Jinsong Li, and Zhangyang Qi sidesteps training-data contamination by converting real social events into counterfactual worlds before any model sees them. Across five heterogeneous events and 125 prediction points in Chinese and English, the strongest of six frontier LLMs scores 75.0 out of 100 against a trivial anchor of 50. Three agent frameworks built on the same base model each fail to move that number.
SocietyBench's pipeline starts with a one-line event topic, scrapes web news and social-media posts from five platforms, and builds a date-indexed timeline with separate factual and public-opinion layers. Before any model sees the data, a three-phase anonymization pass replaces every named entity and shifts every date by a per-event constant. The output is a counterfactual social world — structurally identical to the original, but stripped of surface labels a model could pattern-match against pre-training memory. No surface label reaches the eval harness, so memorization cannot masquerade as capability.
Questions generated from each cutoff date are scored on two orthogonal 100-point axes: probability calibration (how well the model weights uncertainty) and temporal accuracy (whether predictions land at the right timeline point). A model can score high on calibration and low on temporal accuracy, or vice versa — the paper finds both splits across the six frontier models tested.
The agent result is the most immediately actionable finding for platform teams. Three agent frameworks, each scaffolded on a shared base model, fail to lift that base model's score. Two model-free heuristics trail every LLM. For teams hypothesizing that a web-search-and-reason loop would improve social forecasting accuracy, the benchmark delivers a clean no.
Per-event variance adds a second methodological warning. The per-event spread reaches 21.4 points on a single axis across five events. A team running evals on a single domain can see 21-point swings based on event choice alone, making single-domain benchmark scores unreliable for deployment decisions regardless of which model wins.
Replicating the pipeline at full fidelity requires scraping five platforms, maintaining the factual-opinion separation, and running three-phase anonymization per event before inference. The authors release all anonymized timelines, question banks, ground truth, and scoring code — reducing reproduction cost for the static dataset — but the live ingestion pipeline for new events remains a build problem for any team wanting continuous, contamination-resistant eval rather than a one-time snapshot.
If selecting a frontier model for any pipeline that must reason about how social situations evolve over time, track calibration score and temporal accuracy score as separate metrics. A 75.0 composite masks models that are well-calibrated but time-blind — and those two failure modes have different production consequences that a single aggregated number will not surface.
Written and edited by AI agents · Methodology