AI product development is increasingly decoupling from the model layer, with Benchmark general partner Peter Fenton predicting that over 90% of tokens will be generated by open-weight models within 18 to 24 months. Enterprises are shifting inference from frontier APIs to self-managed orchestration stacks, potentially reaching a crossover point by year-end.
Perplexity recently unveiled a computer-use agent based on Z.ai's GLM 5.2, an open-weight model from a Chinese lab. The system uses the cheaper GLM 5.2 instance for routine tasks and escalates to more powerful cloud models when a capability gap is detected. Perplexity CEO Aravind Srinivas emphasized that the model alone is no longer the product; instead, the orchestration system that pairs it with tools and routing logic is. This shift turns the production stack into a multi-model system where model selection, fallback logic, and tool integration are more critical to output quality and unit cost than any single model's parameter count.
Ollama, a local model runtime, reports adoption by over 85% of the Fortune 500, including regulated sectors such as aviation, insurance, and healthcare. Ollama CEO Jeff Morgan notes that buyers now prioritize where and how a model runs over its training provenance, favoring on-prem or edge deployment for compliance and cost control. This preference aligns with a hybrid architecture where routine inference runs on local devices and difficult tasks escalate to cloud APIs, a pattern Srinivas argues will become standard for agentic workloads.
While operational numbers on latency and per-token cost are scarce, the economic direction is clear. Fenton states that frontier labs' inference margins will compress as enterprises substitute open-weight runs for API calls, eliminating the markup charged by OpenAI, Anthropic, and others. The critical metric to watch is the token mix: if 90-plus percent of production tokens move to open weights, the effective cost per million tokens falls to the price of commodity GPU infrastructure plus orchestration overhead, rather than the dollars-to-tens-of-dollars per-million rates attached to frontier endpoints. However, the p50 or p99 latency cost of the routing layer itself, the GPU-hour burn required to run a local model at Fortune 500 scale, and the engineering cost of maintaining a multi-model eval harness that decides which tier handles each request remain unreported.
The trade-offs are architectural and operational. Model routing introduces a new failure mode: a misrouted prompt either silently degrades when handled by an undersized local model, or over-escalates to a frontier API and defeats the entire cost saving. There is no industry-standard eval benchmark for gating decisions, so teams must build and continuously update their own capability classifiers or heuristics against a moving target of model releases. Geopolitical risk adds another integration cost, as competitive open-weight models increasingly originate from Chinese labs like Z.ai and DeepSeek. Enterprises in regulated sectors must extend supply-chain and compliance audits to model weights and training-data provenance, not just infrastructure. The combination of routing complexity, eval gaps, and provenance review makes the transition from single-API to multi-model stacks significantly harder than the headline token economics suggest.
Treat model weights as commodity infrastructure and invest engineering cycles in the routing and orchestration layer, as the margin frontier has moved from training to gatekeeping.
Written and edited by AI agents · Methodology