Self-Generated Replay Cuts Catastrophic Forgetting in Fine-Tuned Models

A paper posted May 25 by researchers at NYU—Martin Marek, Dongkyu Cho, Shikai Qiu, Rumi Chunara, Pavel Izmailov, and Andrew Gordon Wilson—shows that language models can sample from their own distribution to generate replay data that nearly eliminates catastrophic forgetting without storing exemplars from prior tasks. For teams running continual fine-tuning pipelines, external exemplar buffers may be optional.

Self-generated samples drawn from the model's own distribution serve as effective substitutes for stored training examples during sequential fine-tuning. When a model is fine-tuned on a new task, interleaving gradient updates with forward passes on self-generated text preserves prior capability at near-exemplar-replay quality. No separate generative model, no data pipeline, no curated replay buffer—the model is its own archive.

The paper identifies three forgetting regimes. First: capacity constraint. Models pretrained close to saturation cannot absorb new tasks without overwriting prior knowledge; self-generated replay does not fix saturation-induced forgetting. Second: optimization tradeoff. When capacity is available, low learning rates reduce forgetting but require substantially more training steps, a bottleneck well-known from domain-adaptive fine-tuning runs. Third: replay as tradeoff-breaker. With self-generated replay, high learning rates eliminate the forgetting penalty, collapsing what was a two-variable optimization problem into a single decision.

For a fine-tuning pipeline: if your base model is not capacity-saturated and you need sequential adaptation across domains or task types, run high-learning-rate fine-tuning while piping self-generated continuations as replay signal. Replay costs only inference compute on the model you are already training—no separate buffer management, no dependency on original training data, which is often unavailable at production fine-tuning time when starting from a public checkpoint.

This matters for agent system designers. Dynamic agent deployments require incremental instruction-following updates—adapting to a new tool schema, changed output format, or expanded domain—without degrading core reasoning or prior-task adherence. The standard workaround, separate fine-tuned adapters per task, multiplies governance surface: each adapter needs regression testing, versioning, and routing logic. If self-generated replay works at production scale, single-model continual adaptation becomes viable, shrinking the need for growing model zoos.

Two caveats matter. The capacity saturation finding is a hard stop: if you are fine-tuning a model already heavily adapted—domain-specific continued pre-training followed by instruction tuning—saturation is a real risk and replay will not compensate. Measuring remaining capacity is non-trivial, and the paper does not offer a production-ready diagnostic. Second, the "nearly eliminates" qualifier carries weight; practitioners need benchmark numbers by task type and model scale before relying on this for latency- or accuracy-sensitive production workloads.

Earlier self-synthesis rehearsal approaches such as SSR (arXiv 2403.01244) required the base LLM to generate synthetic instances via in-context learning and a separate refinement step. This paper's framing is simpler: the model samples from its own distribution directly, with no auxiliary protocol. The tradeoffs—diversity of generated samples, alignment to prior task distribution—are the natural next evaluation for teams considering adoption.

Route sequential fine-tuning through self-generated replay before investing in exemplar storage infrastructure. But run a capacity audit on your base checkpoint first—saturated models need a different intervention entirely.