Netflix has published details of how it keeps GPU fleets running under large-scale AI training and inference workloads, treating hardware failure not as an infrastructure nuisance but as a distributed-systems problem to be engineered around. The company reports more than 99.95% GPU-node availability and a job success rate that climbed from approximately 98% to more than 99.5%, according to InfoQ's account of Netflix's engineering writeup.
The core issue, as InfoQ describes it, is that a large training job can run for weeks across hundreds of GPUs with every worker operating in lockstep — so a single failed GPU, a degraded network adapter, or one slow machine can stall or drag down the entire job. A conventional cloud health check confirms a virtual machine is running and reachable, but Netflix found that tells you nothing about whether its GPUs can actually sustain a distributed workload: a GPU can answer a status query normally and still fail once memory is allocated or sustained computation starts. A network adapter can likewise be present but running with fewer links than expected, letting collective operations continue at severely reduced bandwidth while looking healthy from the outside.
Netflix built its GPU compute platform on Kubernetes, using NVIDIA DCGM for device-level checks covering GPU errors, memory, interconnect, and thermal conditions, alongside checks on the state and count of high-speed Elastic Fabric Adapter interfaces. A Kubernetes node agent translates these checks into node conditions that controllers can act on. The checks are built to fail closed — if a diagnostic tool hangs, the resulting timeout marks the machine unhealthy rather than letting a broken check silently report success — and Netflix keeps fallback paths through kernel logs for cases where the normal GPU diagnostic mechanism itself breaks.
Detection is only the first half of the design. Netflix's controllers respond progressively rather than immediately swapping out a whole machine: a transient problem initially just blocks new work while existing jobs keep running; if the condition persists, the machine is drained and its workloads moved elsewhere. Remediation options range from killing a single problematic job or resetting a GPU up to rebooting or fully replacing the machine, and this integrates with Karpenter's node auto-repair capabilities, with Netflix layering on policies for how long to tolerate a given failure type and how much of the cluster can be disrupted at once.
Because hardware failures can't be eliminated, Netflix's training platform is built to make recovering from them cheap. Large jobs use checkpointing and elastic workers so training resumes from the last healthy checkpoint after a machine is lost, with automatic retry on failure — while application-level failures are classified separately so the system doesn't keep blindly restarting jobs that are failing because of a genuine software bug rather than hardware. Netflix also treats the training job itself as a distributed system to monitor, looking not just for dead machines but for a single slow rank dragging down the rest of a synchronous job — a GPU doesn't have to fail outright to tank throughput.
On the inference side, Netflix is using NVIDIA's Multi-Instance GPU capability to let smaller workloads share a GPU with memory-level isolation between tenants, improving utilization versus dedicating a full GPU per workload, and the company says it's working toward Kubernetes Dynamic Resource Allocation for more expressive, topology-aware placement of GPUs, network adapters, and CPUs within a machine. InfoQ's report also notes Meta's parallel work in this space: Meta uses its own hardware diagnostics, automated remediation, checkpointing and rapid recovery, with a focus on detecting silent data corruption across very large AI clusters, and Meta reports that hardware failures accounted for more than 66% of training interruptions in its Llama 3 experience — a figure that underscores why both companies are investing this heavily in failure handling rather than just buying more capacity.
What's notably absent from InfoQ's account is any cost figure for this reliability layer — no number on engineering headcount, infrastructure overhead, or what the alternative failure rate would have cost Netflix in wasted training time. The piece also doesn't specify cluster size, model scale, or which workloads (training versus inference) saw which improvement, leaving the 98%-to-99.5% jump somewhat underspecified as to scope.
For teams running their own large GPU clusters, the operative lesson is that liveness checks are not health checks: a machine that answers pings is not the same as a machine that can sustain collective communication at full bandwidth, and closing that gap requires device-level diagnostics, fail-closed timeouts, and graduated remediation built directly into the scheduler rather than bolted on after the fact.