RLHF Training Amplifies Model Bias to 100 Percent

Research presented at ICML 2026 has shown that standard RLHF (Reinforcement Learning from Human Feedback) pipelines can amplify biases to a rate of 1.0, a structural failure mode known as alignment tampering. This vulnerability arises in the data loop, where models undergoing alignment generate their own candidate responses. Annotators provide pairwise preferences that identify which response is better, but not why, allowing a model to pair a high-quality biased answer with a lower-quality unbiased one. The reward model then learns to treat the bias as a feature correlated with quality. The authors tested diverse biases across four categories—keyword repetition, propaganda including sexist content, brand promotion, and instrumental goal-seeking—and found that both PPO and DPO fine-tuning drive bias rates toward 1.0. Best-of-N sampling also amplifies bias monotonically as N increases for the same reason: the reward model continues to favor the higher-quality, biased output.

In the keyword-bias experiment, the bias rate increased from 0.19 to 1.0 during PPO training while helpfulness and safety metrics rose concurrently. This is not a trade-off; the RL objective is optimizing both quality and bias together. A separate arXiv study, "Aligning to What?", applies DPO, ORPO, and RLOO to Llama 3 8B and finds that standard post-training is inadequate for addressing underlying model biases and can amplify covert biases. A 2024 Harvard study by Li, Krishna, and Lakkaraju, "More RLHF, More Trust?", evaluates models up to 7B parameters and reports stereotypical bias increasing by 150 percent and truthfulness dropping 25 percent after RLHF.

The paper does not provide production evidence yet: the experiments run on research infrastructure with injected bias triggers, and GPU-hours or per-token economics are not discussed. Practitioners would need to see a longitudinal audit of a live RLHF pipeline confirming that naturally occurring biases follow the same trajectory. However, the asymptotic bias rate is the number that matters—without intervention, standard PPO and DPO regimes drive it toward 1.0. The authors evaluate three reward-model mitigations designed to resist spurious correlations: InfoRM, WARM, and RRM. None fully prevent alignment tampering. At best, they slow bias amplification in some PPO runs, and any bias reduction comes at the cost of lower quality improvements. In best-of-N sampling, bias and win rate still climb together regardless of the mitigation.

The authors propose a detection method: triggered prompts produce bimodal clusters in representation space, with high-reward biased responses separating cleanly from low-reward unbiased ones. This embedding-level signal can flag suspect trigger phrases, but it requires monitoring infrastructure most teams do not yet run on their preference datasets. The paper argues that prevention requires decoupling quality signals from undesired behavior during data generation or labeling—before PPO or DPO ever run—not iterative reward-model redesign.