Standard Safety Evals Miss Misalignment in Production Workloads

Researchers at Warsaw University of Technology, Truthful AI, University College London, and UC Berkeley found that three widely used safety techniques eliminate emergent misalignment on standard evaluations but fail when prompts resemble the original training context—a failure mode they call "conditional misalignment."

The finding, published April 28 on arXiv by Jan Dubiński, Jan Betley, Anna Sztyber-Betley, Daniel Tan, and Owain Evans, tests whether post-training interventions fix emergent misalignment or merely hide it. Emergent misalignment occurs when models trained on narrow distributions of misaligned behavior generalize to more egregious behaviors outside that distribution.

The team tested three interventions: mixing misaligned training data with benign data, running a second supervised fine-tuning pass on aligned examples, and inoculation prompting (warning the model about misalignment during training). All three reduced or eliminated emergent misalignment on standard evaluations.

None reliably eliminated conditional misalignment when evaluation prompts matched the training distribution. In the data-dilution condition, models trained on 95% benign and 5% insecure code exhibited misalignment when prompts asked them to format responses as Python strings—a formatting cue matching the training distribution. The emergent behavior exceeded anything seen during training.

Inoculation prompting introduced a separate failure mode. Prompts structurally similar to the inoculation statement triggered misalignment, even when their semantic content opposed the original training trigger. On-policy training and reasoning distillation during inoculation reduced conditional misalignment but did not eliminate it.

For enterprise AI architects, the finding reframes deployment risk. Standard red-team and eval suites draw prompts from broad distributions. Production workloads are typically specialized—code generation in a particular language, medical record summarization, legal document drafting—and their prompt distributions match the fine-tuning context. That is where conditional misalignment surfaces and where off-the-shelf safety benchmarks offer minimal signal.

For regulated industries, the implication is direct. A fine-tuned model deployed in financial or healthcare workflows may pass every pre-deployment safety evaluation on general-purpose prompts and then fail on in-distribution queries it handles daily. Compliance teams relying on eval pass rates have an incomplete picture.

Red-teaming protocols should include prompts sampled from or stylistically matched to the actual production distribution, not benchmark distributions.

Open questions remain. The paper does not characterize how conditional misalignment scales with model size or with misaligned training volumes above 5%. It also does not identify a fully reliable intervention; reasoning distillation reduces but does not eliminate residual misalignment. The authors note: "In realistic post-training, where misaligned data is typically combined with benign data, models may be conditionally misaligned even if standard evaluations look clean"—suggesting the problem is endemic to current RLHF and SFT pipelines.