RLHF and RLVR pipelines compress model diversity. Every reward step nudges outputs toward the highest-probability correct answer, eroding distribution breadth until the model is reliable but repetitive. That works for question-answering benchmarks. For downstream RL training that depends on varied rollouts to generate useful reward signal, it's a direct tax. For tasks where creative or exploratory generation matters, it quietly degrades performance.
A new paper from Ananya Sahu, Mohit Bansal, and Elias Stengel-Eskin (published August 7) introduces CreativeInstruct: a single-model, instruction-tuning method to recover lost diversity without surrendering quality gains from post-training. The mechanism is a learnable control token—[StartCreativity]—injected during supervised fine-tuning. The model learns that this token predicts creative, base-model-like generation. At inference time, whether to activate it is a choice, not a sampling parameter.
![CreativeInstruct mechanism: a single SFT stage teaches the model that [StartCreativity] predicts creative, base-model-like generation; no inference overhead at deploy time.](https://cdn.sanity.io/images/bxbpomp2/production/d8b0d58b8fb910d42be3d46667062ee7806b3e6e-535x808.png?w=1200&fit=max&auto=format)
No inference overhead. Multi-model ensembles and distillation have been the workaround for diversity collapse, but they multiply deployment cost and complicate serving. CreativeInstruct runs as a single checkpoint. On narrative generation benchmarks, it matches or exceeds the diversity of multi-model and distilled baselines—confirmed in both automatic metrics and human evaluation.
| Approach | Model Checkpoints | Inference Overhead | Diversity vs Baselines | Requires Re-tuning |
|---|---|---|---|---|
| CreativeInstruct | Single checkpoint | None | Matches or exceeds multi-model & distilled baselines | Yes — SFT stage on creative/non-creative pairs |
| Multi-model Ensemble | Multiple checkpoints | Multiplied (N× serving cost) | High | No (uses existing checkpoints) |
| Distillation | Single checkpoint (post-distillation) | None | Lower than CreativeInstruct | Yes — distillation training run |
The paper also proposes a new diversity metric based on graph edit distance over narrative structure. Existing lexical and semantic measures (n-gram overlap, cosine similarity) miss structural variation: two stories with different plots but similar vocabulary score as less diverse than they are. Graph-edit-distance metrics operate on discourse structure and catch narrative variation those measures skip.
Human annotators rated CreativeInstruct outputs as more creative than standard post-trained baselines in 70.3% of cases. This is notable because it reflects open-ended generations, not synthetic benchmarks, and the gain came without quality loss on the post-trained model's core strengths.
The RL finding is most actionable. When GRPO (group relative policy optimization, widely used in reasoning model training) was applied to a CreativeInstruct checkpoint rather than a standard post-trained one, math performance improved 4% on AMC and 5% on MATH. The mechanism: more diverse initial rollouts generate richer reward signal, letting GRPO find better policies before collapse. The starting point matters.
What's hard: this requires a supervised fine-tuning stage on labeled creative/non-creative pairs to teach the model what [StartCreativity] predicts. Teams training post-training from scratch can insert this; teams with existing distilled or RLHF'd checkpoints must weigh whether the quality-diversity tradeoff justifies re-tuning. The paper reports no experiments at very large scales, and it's unclear whether the 4–5% RL gains persist as model capacity and training compute grow.
For teams building reasoning agents where RL is part of the loop: diversity collapse from standard post-training isn't a background inefficiency—it's a ceiling on what GRPO or similar can achieve. CreativeInstruct provides a documented path to raise that ceiling at no inference cost.