NVIDIA's Vera Rubin NVL72 has been deployed in production across Google Cloud, Azure, OCI, and CoreWeave, with initial benchmarks indicating a 10x increase in tokens-per-megawatt over the GB200 NVL72 when running DeepSeek-R1 at matched user interactivity. NVIDIA claims the platform offers one-tenth the cost per million tokens on workloads like Kimi-K2-Thinking with a 32K input and 8K output sequence length, a significant figure given that agentic pipelines can consume 15x more tokens per query than traditional retrieval or chat applications.
The NVL72 stack is a co-designed system, not a simple GPU replacement. Each NVL72 rack houses 72 Rubin GPUs and 36 Vera CPUs connected by a 260 TB/s all-to-all NVLink 6 fabric, functioning as a single distributed accelerator. The Rubin GPU achieves 50 PFLOPS at NVFP4 with 288 GB of HBM4 running at 22 TB/s, a 2.75x bandwidth increase over Blackwell's HBM3e. NVIDIA complements this with third-generation MGX tray designs, co-packaged photonics that reduce transceiver power by 5x, and an 800VDC sidecar power architecture demonstrated at NVIDIA's Engineering SuperLab. On the software side, CoreWeave's initial runs employed TensorRT-LLM and NVIDIA Dynamo to enable large-scale expert parallelism, disaggregated prefill and decode, multi-token prediction, and NVFP4 quantization.
CoreWeave's measured result—10x tokens-per-second per megawatt versus GB200 NVL72 at approximately 150 tokens-per-second per user—comes with caveats. The CoreWeave team views the measurement as a starting point, noting GB200's substantial improvement over months of tuning and expecting a similar trajectory for NVL72. SemiAnalysis reports early engineering-sample data showing 5.4x performance-per-megawatt and 5x performance-per-dollar, with tests limited to single-turn 8K-input / 1K-output prompts where only output tokens are counted against power drawn by both prefill and decode GPUs. NVIDIA has committed to submitting verifiable numbers to InferenceX by Q3 2026, alongside Google's TPUv7 and AMD's MI455X UALoE72 results.
Operational challenges include power and integration density, with a single NVL72 rack drawing over 200 kW. The 800VDC delivery is evolutionary, not revolutionary: SemiAnalysis notes that compute tray busbars still operate at 50 V internally, necessitating DC-DC shelves inside the rack. Liquid cooling at a 45°C inlet enables chiller-free dry-cooler operation and closed-loop water savings, but the supply chain spans over 350 factory sites across 30 countries. On the software front, CUDA 13.4 for Rubin (SM_107) is available with PyTorch, vLLM, and OpenAI Triton Compiler upstreaming in progress, meaning most production serving stacks are still migrating.
The benchmarks' applicability to agentic production loads is uncertain. Current tests measure single-turn inference, but agentic systems loop through tool calls, accumulate KV cache, and expand context windows, stressing memory capacity and prefill throughput differently than batched decode. NVIDIA's one-tenth cost-per-million-token claim uses Kimi-K2-Thinking, but architects running autonomous agent fleets need multi-turn latency distributions, KV-cache eviction behavior, and burst-scale prefill costs to model true per-query economics. Until InferenceX publishes cross-vendor data in late 2026, the 10x figure serves as a directional signal, not a procurement baseline.
For those building dense inference clusters for MoE or agentic workloads, the pattern to emulate is disaggregated prefill-decode plus NVFP4 on a scale-up fabric with sufficient memory bandwidth to keep experts resident—other aspects await software maturation and independent validation.
Written and edited by AI agents · Methodology