How Netflix Split Autoscaling Into Two Systems

Netflix runs 30,000+ Flink jobs across multiple AWS regions, most auto-generated by their Data Mesh platform. Keeping those pipelines right-sized demands autoscaling. After years running two separate autoscalers in parallel, Netflix learned that the metrics source determines the failure mode—the two systems fail in opposite directions.

The first autoscaler is homegrown, built around 2019 on Mantis, Netflix's streaming-job platform. It works outside-in, consuming cluster-level signals from Atlas, Netflix's telemetry system: CPU, network, Kafka lag, input rate, consume rate per job. It calculates catch-up time from lag, combines CPU and network thresholds with performance history, and uses input-rate regression to decide scaling. Because it runs as an independent Mantis job, Flink platform failures don't touch it. Each Mantis node handles a shard of the Flink fleet with no custom coordination. Across thousands of pipelines, it delivered 25–45% resource savings.

The limitation is metric granularity. The homegrown autoscaler controls one knob: total TaskManager count. Every operator in a job scales together. This works for single-operator pipelines moving records between Kafka topics. It breaks on multi-operator stateful DAGs with branches, joins, and terabytes of state—the jobs Ads, recommendations, and live-events teams run. Each new topology required custom logic. No general pattern emerged.

Worse: the system only sees what its metrics show. A fully-loaded job might report normal CPU, leaving the autoscaler blind. A networking migration silently changed how Atlas counted traffic. Some metrics undercounted load. The gap stayed hidden until production broke.

The Apache Flink community autoscaler is Netflix's second system. It works inside-out, reasoning from the job itself. Its mechanism: True Processing Rate (TPR). For each subtask, Flink tracks how much of each second it spends working versus blocked or idle. TPR divides observed throughput by that busy fraction to extrapolate capacity at full load. An operator handling 700 records/sec while busy 70% of the time has a TPR of 1,000 records/sec. Starting from source operators, the autoscaler walks the job DAG and uses each operator's TPR, input/output ratios, and target utilization to compute the parallelism each vertex needs—so no operator becomes the bottleneck. Per-job configuration—stabilization windows, thresholds, scaling behaviors—travels with the job, not the platform.

The tradeoff: this autoscaler lives inside the Flink runtime. Flink platform issues can directly affect it in ways an external system wouldn't. Scaling is expensive regardless: Netflix takes a savepoint, stops the job gracefully, and restarts at the new parallelism. For large stateful jobs with terabytes of state, that cycle takes several minutes.

Netflix runs both autoscalers in production now. Migration toward the OSS autoscaler is underway. The lesson: custom infrastructure maintenance costs compound invisibly. The networking-migration incident—where Atlas metrics silently lost accuracy—shows how external-observer autoscalers accumulate hidden dependencies on systems they don't control.

For teams running inference pipelines or event-driven workloads with heterogeneous operator graphs, the choice is the same Netflix faced: coarse external signals scale simply but cap out at single-knob control. Internal per-subtask metrics unlock per-operator precision at the cost of tighter coupling to the runtime.