Cornell's Co-LMLM Matches GPT-4o-Mini by Storing Facts in a Database

Cornell's Co-LMLM, a 360M-parameter continuous-query limited memory language model, achieves lower perplexity than vanilla transformers trained on 40× more data and scores on SimpleQA in line with GPT-4o-mini and Claude Sonnet 4.5, not by packing facts into weights but by externalizing them to a continuous-key knowledge base during pretraining. The architecture replaces symbolic, relational lookups of prior limited memory language models with flexible vector queries over a database of roughly 54.6 million textual knowledge triplets, preventing the model from memorizing factual tokens it must retrieve.

The stack starts with an annotation pipeline tagging factual spans in arbitrary corpora, not just Wikipedia. Over the OLMo2 Wikipedia corpus and FineWeb-Edu—about 3 billion tokens total—a GPT-4o seed distills a LLaMA-3.1-8B-Instruct annotator, inserting special `[dblookup('Entity', 'Relationship') -> Value]` tokens into training sequences. During pretraining, knowledge values tied to those lookups are masked from the cross-entropy loss, so the 360M model learns to predict that a fact is needed and how to query for it, but never learns to emit the fact from parameters. At inference, the custom HuggingFace `LlamaForLMLM` class generates autoregressively until it produces a lookup token; a FAISS index then retrieves the nearest continuous key via cosine similarity over `all-MiniLM-L6-v2` embeddings, using a match threshold of 0.6. The retrieved human-readable text value is concatenated back into the context, and generation proceeds.

Operationally, knowledge lives in the KB rather than the weights, allowing corrections or deletions with simple database edits—no retraining, no risk of catastrophic forgetting of adjacent facts, and no NPO-style unlearning artifacts. The prior 382M-parameter LMLM already matched LLaMA2-7B on factual precision benchmarks like FactScore and T-REx, and Co-LMLM extends this to free-form, non-Wikipedia text while keeping retrieval outputs fully attributable.

The barriers to production are significant and specific. The annotation pipeline alone demands GPT-4o API calls followed by a two-day run across 64 A6000 GPUs to process just 3 billion tokens, making dataset construction a capital-intensive step that must be repeated for each new domain. Retrieval quality is bounded by the `all-MiniLM-L6-v2` embedding model and the annotator's coverage; the arXiv paper leaves open the fallback behavior when the FAISS search returns no match above the 0.6 threshold, a failure mode that standard RAG usually absorbs via parametric knowledge the base model still retains. Most critically, Co-LMLM requires pretraining from scratch with the LMLM recipe; it is not a fine-tune, adapter, or post-hoc retrieval wrapper. All experiments are also capped at the 360M–382M scale, so the 40× data-efficiency claim and the latency cost of interleaved generation-and-retrieval remain unproven at the 7B-parameter scale where most production services operate.

For architects building under GDPR, regulatory fact-correction, or rapid unlearning requirements, the pattern to steal is baking retrieval tokens into pretraining itself so the model is structurally incapable of memorizing what it may later be forced to forget.