Researchers at UC Santa Barbara and Hanbat National University published CoinRAG on August 7, 2026, a KV cache optimization for retrieval-augmented generation that replaces chunk-level reuse with sub-chunk "information nugget" caching. On LongBench multi-hop question answering benchmarks, it achieves 5.3% relative F1 improvement — 41.7 vs. 39.6 — against Standard RAG and Cache-Augmented Generation baselines, while holding prefill inside a P99 time-to-first-token budget of 100 ms.
| System | LongBench F1 | Relative F1 Gain | P99 TTFT Budget |
|---|---|---|---|
| Standard RAG (baseline) | 39.6 | — | Not reported |
| Cache-Augmented Generation (baseline) | 39.6 | — | Not reported |
| CoinRAG | 41.7 | +5.3% | ≤ 100 ms |
Cache-Craft (arXiv 2502.15734) quantifies the problem in production: prefill consumes 55.4% of total inference time on one cluster (Sys-X) and 76% on another (Sys-Y). Redundant chunk processing costs over 12 billion tokens per month on LLaMA-3-70B across eight A100 GPUs — roughly $50,000 in GPU time. Retrieved chunks contain answers alongside substantial irrelevant context, so caching the full chunk caches the noise along with the signal.
| Metric | Value | Context |
|---|---|---|
| Prefill share of total inference time — Sys-X | 55.4% | One production cluster |
| Prefill share of total inference time — Sys-Y | 76.0% | Second production cluster |
| Redundant chunk tokens per month | >12 billion | LLaMA-3-70B, 8 × A100 GPUs |
| Estimated GPU cost of redundant prefill | ~$50,000 / month | LLaMA-3-70B, 8 × A100 GPUs |
CoinRAG caches below the chunk level. Offline, an LLM extracts text-span-based "information nuggets" from each chunk — compact units isolating relevant assertions. KV caches for these nuggets are precomputed and stored. At inference, two-stage retrieval narrows candidates at chunk level, then selects query-relevant units within those chunks. The system assembles a contextualized KV representation by composing precomputed nugget caches with a chunk-level context signal, giving the model cross-chunk coherence without encoding full chunks.
Chunk-level systems face a trade-off: hit the 100 ms time-to-first-token SLA with fewer chunks or exceed it to answer multi-hop questions. CoinRAG shifts that frontier by making each retrieval unit smaller and more semantically dense. The same latency budget fits more relevant evidence because each nugget uses fewer tokens — and fewer prefill FLOPs — than the chunk it came from.
Competing approaches exist. CacheBlend (EuroSys '25) reuses any precomputed chunk cache regardless of position and selectively recomputes KV values for a small subset of tokens to restore cross-chunk attention, achieving 2.2–3.3× time-to-first-token reduction over full recompute. It ships as LMCache, an open-source vLLM integration. CacheClip uses an auxiliary model to identify which tokens to recompute, reaching 3.33× prefill speedup while retaining 85.2–91.1% of full-attention performance. CoinRAG avoids recomputation by precomputing at finer grain from the start.
| System | Caching Granularity | TTFT / Prefill Improvement | Quality Retention | Requires Fine-tuning | Open-source Integration |
|---|---|---|---|---|---|
| CacheBlend (LMCache) | Chunk-level; selective KV recompute for cross-chunk attention | 2.2–3.3× TTFT reduction vs. full recompute | Restored via selective recomputation | No | Yes — vLLM (LMCache) |
| CacheClip | Token-level; auxiliary model selects tokens to recompute | 3.33× prefill speedup | 85.2–91.1% of full-attention performance | No (aux model only) | Not specified |
| CoinRAG | Sub-chunk nuggets; no recomputation at inference | Within 100 ms P99 TTFT on multi-hop QA | +5.3% F1 over baseline (41.7 vs. 39.6) | Yes — nugget-aware fine-tuning required | Not specified |
Trade-offs exist. Nugget extraction requires an offline LLM pass over every document before serving. Nugget-aware fine-tuning means the target model is no longer off-the-shelf. Deployment teams on high-churn corpora face cache invalidation pressure — every document update triggers re-extraction and re-caching. The paper evaluates on three LongBench multi-hop QA datasets; single-hop factoid retrieval and summarization tasks are not covered.
CoinRAG suits workloads where multi-hop QA dominates, document churn is low, and one fine-tuning run is feasible. CacheBlend (via LMCache) offers lower barrier entry for existing vLLM stacks with mixed retrieval patterns. The 5.3% F1 gain under a hard 100 ms P99 budget is measurable, but the fine-tuning dependency makes it a system commitment.