Multiverse Computing published a distillation recipe this week that breaks the two-GPU-cluster bottleneck: cache the teacher's top-100 logits once offline, swap in a fused chunked KL loss, and run long-context distillation recovery on a single H200 instead of a rack. The paper, co-released on Hugging Face, targets teams compressing open-weights models like Qwen, GLM, or Kimi — where the teacher weighs 2.8 trillion parameters and demands 3 TB of VRAM just to load.
The cost driver is straightforward to articulate, ruinous in practice. Standard online distillation keeps both teacher and student in GPU memory simultaneously. At each training step the teacher runs a full forward pass and produces a probability distribution across its entire vocabulary. For gpt-oss-120b at vocabulary size 201,088, sequence length 32K, and batch size 4, that teacher-probability tensor is roughly 50 GB in bfloat16 before a single gradient computes. Add student weights, activations, optimizer states, and the per-step memory spike hits around 250 GB — well above the 141 GB ceiling of an H200 or B200.
The first fix is offline distillation. Run the teacher once, cache the top-100 most probable tokens per sequence position, store those logits on disk, then evict the teacher from GPU memory entirely. The student trains against the cache indefinitely. The same cache is reusable across ablations: swapping student architectures, learning rates, or sequence lengths no longer requires rerunning a trillion-parameter model.
The second fix is the fused chunked KL loss. The paper benchmarks three mathematically equivalent formulations. Dense KL expands the cached top-100 teacher logits into a full vocabulary-size grid and compares it against the student's equally dense log-probability grid, holding two full vocab × sequence tensors in memory simultaneously. Forward-chunked KL keeps the teacher sparse and computes the loss over one sequence slice at a time, eliminating the dense teacher tensor and achieving the fastest throughput; the liability is that the student's full logits grid materializes for the backward pass, so memory still scales with sequence length. The fused chunked KL goes further by fusing the output projection directly into the loss loop. It never builds the student's full logits tensor at all: it projects hidden states to logits for one chunk, folds the result into the running loss accumulator, discards the chunk, and moves on. The backward pass recomputes each chunk on the fly rather than storing it. The trade-off is additional FLOPs for recomputation; the gain is peak VRAM drops from roughly 250 GB to 128 GB.
| Method | Teacher tensor | Student logits tensor | Peak VRAM | Throughput | Backward pass |
|---|---|---|---|---|---|
| Dense KL | Full vocab × sequence grid | Full vocab × sequence grid | ~250 GB | Baseline | Standard — all logits retained |
| Forward-chunked KL | Sparse (top-100 tokens) | Full vocab grid (backward) | Reduced vs. dense; scales with sequence length | Fastest | Standard — student full logits stored |
| Fused chunked KL | Sparse (top-100 tokens) | Never materialized | ~128 GB | Lower than forward-chunked (extra recompute FLOPs) | Recomputes each chunk on the fly |
That 128 GB figure is meaningful: it fits inside a single H200 with room for student model weights, removing the requirement for multi-node tensor parallelism. This makes long-context healing runs — fine-tuning a compressed student to recover capabilities at extended sequence lengths — practical on single-GPU infrastructure for the first time.
For architects evaluating this, the integration surface is narrow. Neither change requires modifying model architecture. The offline cache is a preprocessing step against any teacher checkpoint. The fused chunked loss is a drop-in replacement for the KL term in any standard distillation training loop. Multiverse Computing produced Hypernova 60B using this pipeline; Nvidia's Nemotron 3 Puzzle 75B used comparable compression. Both models show the production trajectory: start with a frontier teacher, distill to deployable size, recover quality through cheap multi-ablation training.
The constraint that remains is the top-K approximation. Caching only 100 logits per position discards the tail of the teacher's distribution. For most tokens — high-confidence, peaked distributions — this matters little. For ambiguous positions where the teacher spreads probability mass across hundreds of candidates, a top-100 cache is lossy. Teams compressing models for narrow domains where tail tokens carry semantic weight should validate recovery quality on their specific eval sets before committing to production.