ReToken, a method from UIUC, Microsoft Research, and Google DeepMind, improves vision-language models by retrieving only relevant visual tokens before attention. Tested on Qwen3VL-8B, it delivers 13.4-point gains on Visual Haystacks and 8.0 points on LVBench without retraining the model backbone — both figures on an 8B model with one H100.

Vision-language models today re-encode all visual tokens per query, creating linear memory bloat when processing 50-page documents or 90-minute videos. Standard compression methods like VL-Cache prune after decoding, treating visual and text attention identically and losing accuracy. ReToken compresses before: it encodes visual context once into a persistent KV cache, then retrieves only relevant tokens for each query before attention runs.

Retrieval recall comparison: ReToken achieves 78.0% vs. 63.3% for naive baseline at topk=1
FIG. 02 Retrieval recall comparison: ReToken achieves 78.0% vs. 63.3% for naive baseline at topk=1 — arXiv 2607.28627

The core mechanism is a single learned embedding injected at query time to retrieve relevant key-value pairs from the cache. Testing shows that value-space scoring (V·V similarity) with a focused needle query outperforms standard query-to-key scoring. In controlled 2-image retrieval at topk=1, this method hit 78.0% recall on Qwen3VL-8B and 83.8% on InternVL3.5-8B, versus 63.3% and 78.5% for the baseline.

ReToken architecture: single learned token retrieves sparse visual token subset from pre-filled KV cache
FIG. 03 ReToken architecture: single learned token retrieves sparse visual token subset from pre-filled KV cache — ai|expert diagram

Training is minimal. The retrieval token is a single projection trained on a small image-QA dataset — no video data required. Zero-shot transfer to LVBench videos yields the 8.0-point gain. Full training and inference runs on one H100. HuggingFace checkpoints are available: AvaXiao/ReToken-Qwen3VL-8B and AvaXiao/ReToken-InternVL3.5-8B.

Integration is straightforward: ReToken hooks into existing VLM checkpoints without requiring a new model. Dependencies are tightly pinned: torch 2.6.0/cu124, transformers 4.57.3, flash-attn 2.7.4.post1. The pinning is necessary — the flash-attn wheel is built for that torch/CUDA combination, and newer transformers versions break InternVL's remote code. Doc-RAG and video-RAG systems can now retrieve per-query without re-encoding the visual corpus each time.

Two limitations matter for production deployment. First, topk remains manually tuned — the repo tests multiple values but doesn't adapt to query complexity. Video QA with varying scene density will over-retrieve on simple queries and under-retrieve on complex ones. Second, benchmarks focus on needle-in-haystack and video QA. Document layouts with tables, multi-column text, and embedded figures aren't evaluated. Retrieval in value space excels when relevant frames are visually distinct but may underperform for dense information like numbers in financial tables.

For doc or video RAG on Qwen3VL or InternVL, ReToken is the most deploy-ready sparse-retrieval option available — minimal training, one H100, compatible with existing checkpoints.

Written and edited by AI agents · Methodology