Aggressive KV-cache eviction cuts context costs but degrades models. They read from partial history, loop and degenerate, burning tokens before producing useful answers. Researchers at Seoul National University introduce KV-Rescue, a training-free framework that recovers lost accuracy by pairing an evicted base model with a lightweight full-context helper.
The core insight: eviction creates an information gap, not a capability gap. Testing an evicted Qwen2.5-Math 7B and a full-context Qwen2.5-Math 1.5B on identical problems, an oracle selecting the better answer recovered 79% of the accuracy gap. If eviction degraded capability, the 1.5B would be too weak to help. It isn't, because it sees full context.
| Configuration | Model | KV Context | Outcome |
|---|---|---|---|
| Base (degraded) | Qwen2.5-Math 7B | Partial (evicted) | Accuracy drops due to partial history |
| Helper | Qwen2.5-Math 1.5B | Full context | Stronger on problems where base degrades |
| Oracle selection | Best of both above | Both | 79% of accuracy gap recovered |
Sampling harder from the evicted model doesn't close the gap. Best-of-N inference improves scores but plateaus because all candidates come from the same partial cache.
KV-Rescue addresses the gap structurally. At each step, the evicted base and full-context helper generate candidates. A process reward model selects which candidate extends the shared trajectory. An online detector using token entropy and compressibility signals eliminates degenerate base-model outputs before they corrupt the trace. No fine-tuning required.
Results span five math benchmarks on Qwen2.5-Math 7B and 72B models. At eviction budget B=64, KV-Rescue recovers 87% of lost accuracy. On MATH500 at budget 128, it surpasses full-KV pass@1 at N=4 and improves as N increases. Stopping degeneration cuts base-model token generation by 43%.
| Metric | Value | Condition |
|---|---|---|
| Accuracy gap recovered | 87% | Eviction budget B = 64 |
| MATH500 vs. full-KV pass@1 | Surpasses | Budget B = 128, N = 4; improves as N increases |
| Base-model token generation | −43% | Degeneration detector active |
Operational cost: the helper is a separate 1.5B-parameter model running alongside the 7B base. The PRM is a third model. Total cost scales with N and per-step budget across both, so teams tune N against latency constraints. The method covers math-reasoning benchmarks; code and long-document performance are not addressed.
For architects running aggressive KV eviction and seeing accuracy cliffs, this is a training-free fix: deploy a small helper model and domain-matched PRM.