Researchers from Google DeepMind, the University of Edinburgh, and the University of Waterloo released a paper on August 20 formalizing a blind spot in multi-model routing: the cost of the routing signal itself. The paper, "Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation," maps the problem to Pandora's Box—the economics problem of optimal search under costly inspection—and derives closed-form policies telling architects when to run an expensive estimator and when to skip it.
The gap is real and routinely ignored. Existing frameworks like RouteLLM, FrugalGPT, and cascade designs treat routing as nearly free. An embedding lookup adds roughly 5 ms; an ML classifier adds 50–100 ms. Both are small relative to a 500–2,000 ms inference call. But when the estimator is a fine-tuned model reading partial reasoning traces or retrieval results, its compute cost approaches the cost of running one specialist outright—making the routing overhead material.
| Estimator Type | Typical Latency | Compute Cost | Signal Quality | Example |
|---|---|---|---|---|
| Embedding lookup | ~5 ms | Near-zero | Noisy | Semantic similarity router |
| ML classifier | 50–100 ms | Low | Moderate | RouteLLM, FrugalGPT classifier |
| Fine-tuned model (w/ retrieval / traces) | 500–2,000 ms | High — approaches specialist cost | Accurate | Trace-reading estimator, RAG-based scorer |
The paper establishes a clean trade-off: cheap estimators (embedding-based) are fast and noisy; expensive estimators (fine-tuned with retrieval or trace access) are accurate and costly. Using a Gaussian signal model, the authors derive closed-form value-of-information (VoI) expressions. For each query, these expressions tell the router whether the expected improvement from the expensive estimator exceeds its cost. If yes, refine; if no, route on the cheap signal. The centralized version is called Pandora's Router.
Experiments span three settings: multi-LLM benchmarks, retrieval-augmented specialists, and LLMs with variable inference-time budgets. Pandora's Router matches exhaustive estimation—running the expensive estimator on every query—while calling it substantially less often. The decentralized variant, Pandora's Bidder, shifts to a market structure: specialists independently decide whether to self-assess before bidding to claim a query. This suits heterogeneous fleets with separate teams owning separate models. Results show that when estimates are accurate, VoI reasoning improves allocation. When noisy—common during ramp-up, domain shift, or sparse query types—a strategic specialist can exploit VoI reasoning to improve its utility at others' expense. That's a fairness and stability concern.
The scale context sharpens the problem. Enterprise LLM spending hit $8.4 billion in the first half of 2025, with agent workloads burning $5–8 per task in API fees—agents make 3–10× more LLM calls than single-turn chatbots. Cost differentials between tiers run 100–300×. Amazon Bedrock prices Llama 3.1 Instruct (70B) at $0.72 per million tokens versus $0.10 for Llama 3.2 (1B), a 7.2× ratio. On the same H100 hardware, the 70B model takes 15× longer to generate 100 tokens than the 1B. Marginal routing improvements compound quickly at that spread.
| Model | Size | Price per 1M Tokens | Cost Ratio vs 1B | Inference Time (100 tokens, H100) |
|---|---|---|---|---|
| Llama 3.2 Instruct | 1B | $0.10 | 1× (baseline) | 1× (baseline) |
| Llama 3.1 Instruct | 70B | $0.72 | 7.2× | ~15× slower |
For architects building multi-model systems with RAG pipelines, inference-time scaling, or agent fleets with tool-specific models, the takeaway is direct: before adding an expensive classifier or fine-tuned estimator to your routing layer, verify that per-query cost is below the expected gain from better assignment. The VoI framework provides the arithmetic. The decentralized variant warns teams to account for strategic behavior when specialists control their own self-assessment.