LightOn's LateOn outperforms 4× larger models while open-source

LightOn released DenseOn and LateOn on April 21, 2026, two Apache 2.0 retrieval models scoring 56.20 and 57.22 nDCG@10 on BEIR at 149M parameters. Both beat competitors with 4× more parameters: Jina ColBERT v2 (559M), Arctic Embed L v2 (568M), and Qwen3-Embedding-0.6B (595M). Both run on ModernBERT with full reproducibility: 665M curated training pairs, 1.88M fine-tuning pairs, and training code.

Training started with 1.4B raw contrastive pairs from 34 public sources, filtered to 665M English pairs for pre-training and 1.88M supervised pairs via hard-negative mining. One efficiency gain required zero added compute: switching DenseOn from mean pooling to CLS pooling and adding asymmetric input prompts—"query:" for queries, "document:" for documents. This explicit signal about input type measurably lifted scores.

Benchmark contamination undermines BEIR validity. The datasets draw heavily from Wikipedia, MS MARCO, and Common Crawl—the same sources filling most training corpora. LightOn decontaminated using exact hash matching plus 13-gram containment with ≥50% overlap. After removal, LateOn reaches 60.36 nDCG@10 on 12 decontaminated datasets and holds first place. DenseOn ranks in the top four. All four ColBERT models held or improved ranking under decontamination—ColBERT-Zero climbs from 6th to 2nd—while dense model rankings dropped consistently. Average rank gain: +2.01 nDCG for dense models, +3.55 for multi-vector ColBERT. Token-level late interaction generalizes better on unseen documents.

For multilingual support, LightOn translated validated English data into eight languages, producing a 2.8B-pair corpus. mDenseOn and mLateOn run at 307M parameters on mmBERT-base. The architectures diverge when training data exhausts: mDenseOn stays strong on English and translate-train targets but degrades elsewhere. mLateOn generalizes to unseen languages and scripts. MaxSim token-level matching finds partial overlaps across scripts where single-vector cosine similarity fails.

LateOn already runs in LightOn's production search pipeline. Late interaction requires per-token embeddings, inflating index size versus single-vector dense retrieval. FastPLAID handles large multi-vector indexes efficiently. DenseOn offers the lower-overhead path—same training data, same compute, slightly lower scores (56.20 vs. 57.22 on standard BEIR) but simpler integration with existing ANN infrastructure.

For teams with internal, domain-specific corpora, benchmark scores on standard BEIR underestimate the gap between dense and late-interaction retrieval. At 149M parameters and Apache 2.0, LateOn removes size and licensing barriers to evaluating ColBERT-style retrieval on your own data.