Databricks has unveiled a production pattern for mapping free-form text to over 100,000 normalized labels, achieving an average accuracy of 0.81 across three benchmarks: financial transaction vendor matching, enterprise company deduplication, and biomedical entity linking. The hybrid retrieval-and-classify pipeline costs approximately one-hundredth of direct frontier-model inference per document, comparing the AI Classify Workflow against calling a frontier LLM with the full taxonomy.
The stack operates natively in SQL through Databricks AI Query and AI Classify functions, eliminating the need for custom orchestration or external serving layers. Labels and descriptions are embedded using Qwen3-Embedding-8B in 4096-dimensional float32, generating 1.6 GB of vectors for 100,000 labels in one to three minutes via AI Query. Retrieval is hybrid, combining cosine similarity for semantic overlap and BM25 for weighting rare lexical tokens, merged through Reciprocal Rank Fusion. Databricks uses an in-memory index instead of a hosted vector search, treating the static taxonomy as a memory-resident lookup table.
The pattern employs a strict two-stage filter, with vector search retrieving a shortlist of candidate labels and the AI Classify function scoring only those candidates. After testing shortlist sizes from 1 to 200, accuracy peaks at k=20, providing a 5,000-to-1 reduction (100,000 to 20) that makes the second stage tractable and accurate.
Operationally, the AI Classify Workflow achieved 0.81 accuracy across the Transactions, Companies, and MedMentions datasets. All four frontier models were benchmarked for direct LLM classification—GPT-5.6 Luna, GPT-5.4 mini, Gemini 3.5 Flash, and Claude Sonnet 5—described by Databricks as "the latest models within reach of our customers' typical production classification budgets." The best of these, Gemini 3.5 Flash, reached 0.76 accuracy at 100× the per-document token cost of the hybrid workflow. Context-window limits compounded the problem: on the MedMentions dataset, only GPT-5.6 Luna's 1M-token context could fit the full taxonomy; no other tested model could. Pure vector search introduces a different, opposite 100x comparison—it costs roughly 100x less than the hybrid workflow itself, since only the input document needs embedding and CPU ranking is negligible—but accuracy falls more than twenty points below the hybrid, making it viable only where precision is negotiable and label cardinality is low.
The hybrid pattern addresses the failure modes of baseline methods, such as the need for continuous regex pattern updates at the scale of thousands of labels, supervised classifiers' collapse on long-tail labels, and direct LLM calls' context-window limits and high prompt-caching costs. The hybrid workflow avoids model retraining when taxonomies drift but introduces an operational dependency: the 1.6 GB embedding job and in-memory index rebuild must be repeated whenever the taxonomy changes, requiring a brief outage window.
The key takeaway is to compress a massive label space with a fast embedding retriever, then run a cheap classifier only on the top-20 shortlist—never feed the full taxonomy to an expensive generative model.
Written and edited by AI agents · Methodology