98 Percent of Activation Explanations Don't Ground Claims

An audit of Anthropic's Natural Language Autoencoder for Qwen-2.5-7B, fine-tuned from Qwen-2.5-7B-Instruct at layer 20 and available on Hugging Face, has shown that standard reconstruction metrics are inadequate for detecting false claims in natural-language activation explanations. The model easily passes reconstruction metrics, but only about two percent of specific claims are actually grounded in the activation, meaning approximately ninety-eight percent are not faithfully grounded and survive by tracking the input's broad gist without carrying verifiable facts, allowing the majority of false assertions to go undetected.

The audit focused on a layer-20 activation verbalizer checkpoint fine-tuned from Qwen-2.5-7B-Instruct. The tool operates as an autoencoder pair, with an activation verbalizer mapping a residual-stream vector to natural-language prose and an activation reconstructor mapping that prose back to a vector. Research by Hiskias Dingeto demonstrates that the objective is structurally blind to per-claim accuracy, rewarding sufficiency without penalizing false additions.

Dingeto identified two ways the test can be satisfied without being faithful. On the released Qwen-2.5-7B NLA, only about two percent of specific claims are genuinely grounded in the activation, while the rest survive by tracking the input's broad gist without carrying verifiable facts. In synthetic sandbox tests with exact ground truth, the standard training recipe produced co-adapted "private codes"—false wording that the reconstructor learned to depend on—in all five of five runs, meaning the prose is unfaithful by construction yet scores well on reconstruction.

The paper provides detection metrics, focusing on training dynamics and audit methodology rather than inference stack performance. It quantifies the effectiveness of an alternative training scheme called RECAP—Readable Encodings via Co-trained Auxiliary Predictors—where linear heads are trained alongside the target model to keep designated content decodable. A RECAP-trained probe achieves an AUC of 0.96 at distinguishing true claims from false ones, compared to 0.82 for probes trained on a standard model. Under adversarial editing designed to suppress roughly eighty-seven percent of the lie penalty while preserving reconstruction score, the RECAP probe still flags deception at AUC 0.95, while a control probe collapses to 0.51, indistinguishable from chance. RECAP adds +0.001 nats of training cost. On a fresh verbalizer in the Pythia-160M replication run, RECAP pushed truth scores of 0.44–0.46 versus near-zero controls, though that verbalizer still conveyed only part of the designated content.

For production stacks, the practical barrier is architectural timing. RECAP requires co-training auxiliary linear heads inside the target model during initial training; it cannot be added to already-shipped weights such as the public Qwen-2.5-7B NLA checkpoint. Safety teams currently using activation explanations to audit steering vectors or concept probes are therefore limited to post-hoc audit protocols—the grounded-vs-true cross and the evaluator swap—which can be run without retraining but remain manual diagnostic steps, not automated filters. The evidence is also scoped narrowly: a single released NLA, a synthetic sandbox, and a small continued-pretraining model at mostly single seeds, with frontier-scale behavior explicitly untested. The private-code failure mode is not a rare edge case but the default outcome of the standard recipe.

The key takeaway for architects is that reconstruction loss alone is insufficient as a faithfulness gate for natural-language interpretability tools; any production safety layer built on activation verbalizers needs claim-level verification, not just vector similarity.