Embedding pretraining beats model size in 1,215 entity-matching tests

A new paper from researchers at the University of Amsterdam, Amsterdam UMC, and TU Berlin/BIFOLD ran 1,215 fine-tuning experiments to isolate what drives performance gains in entity matching: architecture, model backbone, or pretraining objective. Prior benchmarks conflated all three.

The factorial design tested three matcher architectures (bi-encoder, cross-encoder, generative), three Qwen3 variants with different pretraining objectives, and three model sizes across nine datasets. Every combination was trained and measured, eliminating the default assumption that "bigger model, better results."

For teams running bi-encoders in production, pretraining objective matters more than model size. Embedding-oriented Qwen3 variants deliver stronger initialization and favorable representation geometry. Swapping in a base or instruction-tuned variant degrades accuracy measurably, even at larger parameter counts. Teams fine-tuning sentence-transformer encoders on domain data must select pretraining objective deliberately.

Cross-encoders consistently outperform bi-encoders because they jointly encode the record pair rather than embedding each independently. Joint attention catches subtle field-level differences bi-encoders miss when compressing each record into a single vector. Larger Qwen3 variants narrow the gap partially but do not close it. This reflects a hard latency/accuracy tradeoff: cross-encoder inference at query time is slower, but the precision advantage is architectural—not addressable by scaling the bi-encoder alone.

Generative matchers do not universally outperform cross-encoders. Practitioners defaulting to LLM-as-judge or GPT-4 prompts often assume otherwise. Their advantage concentrates narrowly in distribution shift: when record schemas diverge from training data or in cross-dataset transfer. For in-distribution matching on fixed schemas, generative matchers carry higher compute cost without accuracy gain. They solve the messy tail, not the default case.

Larger models rely more heavily on shortcut learning—surface correlations in training data—and do not reliably improve matching performance. For pipelines with small training sets or uneven schema coverage, scaling backfires: it produces overconfident matchers that fail on subtle duplicates.

Code, training scripts, evaluation data, and all 1,215 run results are available at github.com/Jantory/llm-trained-matcher, making results directly reproducible against your own datasets.

For fixed-schema, in-distribution matching, a cross-encoder with an embedding-pretrained backbone is the defensible default. Generative matchers are targeted tools for cross-schema and transfer scenarios. Scaling model size is not a substitute for selecting the right pretraining variant.