Most operators running LLM workloads in production overpay by 2–5x; some high-volume users pay an order of magnitude more. That's Meryem Arik's core claim in her QCon San Francisco 2025 talk, "Producing the World's Cheapest Tokens." The Doubleword co-founder has mapped a systematic playbook for inference cost reduction. General-purpose stacks like vLLM and SGLang optimize for chatbot latency, but teams running offline agents, labeling pipelines, or synthetic data generation pay for latency they don't need.
Arik frames the design space as a three-way tradeoff: latency, cost, and quality—pick two. Latency-first workloads (coding assistants, chatbots) run at low batch sizes on expensive hardware with latency-optimized kernels; Cerebras and Groq serve this corner. Quality-with-speed workloads (guardrails, routers, on-device) trade model size and precision to stay cheap and fast. The third corner—quality at low cost, latency irrelevant—is where Doubleword has focused for the past 6 to 12 months. Use cases: overnight agent workflows, summarization, data labeling, synthetic data generation for RL or fine-tuning. As models improve and autonomous workflows gain trust, this bucket will absorb more production workload.
Four levers drive cost reduction. First, batch-specific optimizations: general-purpose inference servers optimize for individual requests at low latency; switching to workload-aware scheduling—including queue reordering to maximize batch fill—changes the GPU utilization profile. Second, hardware selection: decoupling inference from P99 latency SLAs opens GPU tiers uneconomical for synchronous serving but efficient for throughput jobs. Third, speculative decoding: a smaller draft model generates candidate tokens that the target model verifies in parallel, compounding throughput gains across large batch runs. Fourth, quantization: INT8 or FP8 precision cuts memory bandwidth demand by 2–4x versus FP16 baselines, enabling larger effective batch sizes on the same GPU footprint.
| Lever | Mechanism | Cost / Throughput Effect |
|---|---|---|
| Batch scheduling | Queue reordering to maximize batch fill; workload-aware scheduling | Higher GPU utilization; better $/token on offline jobs |
| Hardware selection | Decouple from P99 latency SLAs; use cheaper GPU tiers uneconomical for synchronous serving | Lower hardware cost per token for throughput jobs |
| Speculative decoding | Smaller draft model generates candidate tokens; target model verifies in parallel | Compounds throughput gains across large batch runs |
| Quantization (INT8 / FP8) | Reduced precision vs FP16 baseline cuts memory bandwidth demand | 2–4× memory bandwidth reduction; larger effective batch size on same GPU |
The compound effect is substantial. Doubleword markets its async and batch inference service 50–90% cheaper than real-time APIs for long-running, high-volume workloads. The platform is OpenAI-compatible and targets agents, evals, and pipelines—workloads that tolerate minutes of latency for dramatically lower $/million tokens. Industry data supports this direction: the Stanford HAI 2025 AI Index reports inference cost for a GPT-3.5-level system fell 280-fold between November 2022 and October 2024, driven by roughly 30% annual hardware cost declines and 40% annual energy efficiency gains. Inference consumes roughly 80% of AI infrastructure budgets. Training is a one-time event; inference runs on every request forever.
Most teams reach for the popular runtime first. vLLM's continuous batching produces up to 23x throughput improvement over naive serving (per Anyscale benchmarks), but it's still designed around latency-sensitive request-response patterns. Adapting it for pure-throughput, high-latency workloads requires deliberate configuration—batch size limits, scheduler priority, queue depth tuning—that aren't defaults. Speculative decoding acceptance rates vary significantly by task type; picking the wrong draft model erodes speedup. Quantization at INT4 saves more memory but needs evaluation against task-specific quality thresholds before production rollout.
Arik's earlier TitanML work flagged a related pattern: teams deploying multiple fine-tuned variants of the same base model spun up separate GPU instances per fine-tune. LoRA-based multi-adapter serving—hosting one base model and hot-swapping fine-tuned adapters on the same GPU—collapses this into a single instance, cutting idle GPU-hours without sacrificing model specificity.
Direct takeaway: if no human watches the token stream in real time—agents, evals, labeling, synthetic data—you pay for latency you don't need. Queue reordering, async scheduling, quantization, and hardware-tier selection targeted at the non-real-time bucket deliver 50–90% cost reduction.