DepthWeave-KV cuts LLM cache memory by 8.3x without retraining

DepthWeave-KV, as described in an arXiv paper by Cordoba et al., achieves 8.3x key-value cache compression and 72.8 tokens per second decode throughput at 64K context without retraining the base model, positioning itself as a drop-in, calibration-free alternative to full key-value materialization. The paper does not specify the serving stacks used for integration, nor the specific models or GPU hardware behind the benchmarks. Practitioners will need to see a reproducible evaluation on open-weight models such as Llama 3 or Qwen2.5 within a standard inference engine, including p50/p99 latency percentiles and batch-size scaling curves, before considering the 72.8 tok/s figure transferable to their own stacks.

DepthWeave-KV's mechanism divides KV states across neighboring transformer layers into shared low-rank channel bases, with low-frequency components amortized across depth, and sparse, token-specific residuals for attention-sensitive behavior. A token-conditional depth router assigns higher reconstruction rank to instruction-bearing and retrieval-critical tokens, avoiding a uniform budget across every position and layer. During generation, attention-output probes track reconstruction error online and adapt compression dynamically. This method bypasses the deployment friction of post-training compression schemes like SVDq or KVTC, which often require representative holdout sets and lengthy profiling.

The authors claim near-full-cache task quality despite the 8.3x memory reduction, evaluated on LongBench, Needle-in-a-Haystack, L-Eval, and long-form QA and summarization benchmarks. At 64K context, KV cache capacity grows linearly with sequence length, making 8.3x reductions crucial for fitting within a single node. DepthWeave-KV differs architecturally from recent alternatives: xKV uses Centered Kernel Alignment for up to 8x compression and reports a 4.23x end-to-end speedup in its own benchmarks, while MiniCache achieves roughly 41% memory reduction and about 5x throughput improvement in its separate evaluation. DepthWeave-KV pursues more aggressive compression via per-token routing and online adaptation, though no head-to-head experiments against these methods have been published.

As the paper is pending peer review, independent validation will be required. Open questions include how attention probes and the depth router affect batching dynamics and tail latency under load, whether synthetic needle tests translate to real-world document collections, and how much of the reported 72.8 tok/s figure is owed to the online error tracking itself. Until the compute overhead of those probes is isolated from the end-to-end speedup, architects cannot assume the same gains at high batch sizes or on consumer-grade GPUs. The transferable pattern is token-conditional depth routing; the risk is deploying a calibration-free cache compressor before its batching behavior is characterized in standard serving stacks.