Martin Spier, who leads ChatGPT Performance at OpenAI, framed the core problem at QCon AI as explosive user growth crashing into agentic development at scale. ChatGPT hit one million users in five days from launch and the engineering teams serving it never fully caught up. Agentic development made that catch-up structural.
The speed problem is not primarily GPU-bound. A single user request passes through client processing, conversation loading, context assembly, tokenization, routing, inference, streaming, and observability before a token reaches the user. Any step can bottleneck. If a task requires 30 model requests, an extra second per request compounds. Reducing repeated work throughout the system matters as much as making the model faster in isolation. Shaving one second per request off the orchestration layer matters as much as raw model throughput.
OpenAI's response: a Rust-based agentic harness connecting models, tools, and the user environment. The key design choice for cache efficiency treats all model-visible history as append-only. New messages, tool results, and environment state append at the end rather than insert earlier. Tools surface in deterministic order. Approval policies and runtime settings apply at execution time, not embedded in definitions. The prefix seen by the model stays stable across turns, boosting prompt-cache hit rates. GPT-5.2 API pricing embeds the incentive: 90% discount on cached input tokens ($1.75/1M standard) gives that design discipline financial weight.
| Token type | Price per 1M tokens | Discount vs. standard |
|---|---|---|
| Standard input | $1.75 | — |
| Cached input | $0.175 | 90% |
OpenAI spent $5.02 billion on Azure inference alone in the first half of 2025. In June 2026, engineers developed a software-only optimization—no new hardware, no architectural overhaul—that cut inference costs by more than 50%. Applied to ChatGPT's logged-out visitor traffic, it reduced the Nvidia GPU count serving that segment to roughly a couple hundred. The technique remains undisclosed; analysis points to KV cache reuse, quantization, and smarter request batching as likely contributors. Whether gains transfer to paid API tenants running agentic workloads—the most compute-intensive traffic—remains open.
| Metric | Value / Detail |
|---|---|
| Azure inference spend (H1 2025) | $5.02 billion |
| Cost reduction achieved | >50% |
| Optimisation approach | Software-only — no new hardware or architectural overhaul |
| GPU count serving logged-out traffic (after) | ~a few hundred Nvidia GPUs |
| Likely techniques (analyst assessment) | KV cache reuse, quantization, smarter request batching |
Agentic coding adds second-order pressure: regressions land faster. Teams shipping with coding agents sit one abstraction layer removed from understanding what they pushed. Humans no longer know all details of a change before it goes live. That compresses the detection window before regression compounds. OpenAI's answer: meet automation with automation. Deploy always-on AI agents for profiling, regression detection, and continuous optimization. The performance team is building telemetry and tooling that agents can read directly, rather than waiting for humans to analyze flame graphs.
Standard benchmarks—fixed 1k-in/8k-out or 8k-in/1k-out token pairs—don't capture the multi-turn, heavy-tailed, tool-interleaved shape of agentic traces. The metrics that matter shift: end-to-end trace latency, time-to-first-answer-token across a full agent session, cache hit rate across turns, and scheduler behavior under high concurrency of short output bursts. KV cache pressure from long-running sessions and scheduler pressure from high output request volumes are the load characteristics that matter; single-turn benchmarks miss both.
| Dimension | Standard Benchmarks | Agentic Workloads |
|---|---|---|
| Token profile | Fixed pairs (e.g. 1k-in/8k-out or 8k-in/1k-out) | Multi-turn, heavy-tailed, tool-interleaved |
| Primary latency metric | Single-request latency | End-to-end trace latency |
| Time-to-first-token scope | Per single request | Per full agent session |
| Cache measurement | Not captured | Cache hit rate across turns |
| Concurrency pattern | Not captured | Scheduler under high concurrency of short output bursts |
| KV cache pressure | Not captured | Long-running sessions |
| Scheduler pressure | Not captured | High output request volumes |
Inference serving for agentic workloads is a systems design problem with multipliers. Fix the harness before you tune the model.