Together AI shipped autoscaling for its Dedicated Model Inference product. Teams now set replica bounds and target inference-native metrics instead of CPU-oriented signals that fail under LLM workloads. Configuration is a single CLI PATCH command.
The billing model charges per replica-minute, making idle GPUs a direct cost. Engineers face two failure modes: over-provision and pay for GPUs at 15% utilization, or under-provision and watch p95 latency collapse when traffic outpaces batching capacity. A replica at its concurrency ceiling doesn't degrade gradually—it queues, and TTFT can jump from 200 ms to 15 s. Classic CPU autoscaling and reactive cloud rules don't fix this.
Classic autoscaling fails at a specific point: a GPU reports 60% utilization while its request queue backs up, because utilization measures arithmetic intensity, not queue depth. LLM cold starts take several minutes. A new replica must be placed on a GPU node, pull tens of gigabytes of weights into VRAM, and warm up. By the time traffic spikes, a scale-out triggered at the spike's edge is already too late. The system must act on leading signals, not lagging ones.
Together AI's control loop is proportional: desired_replicas = ceil(N × observed / target). If the target is 8 in-flight requests per replica and observed count is 16, the system wants 2× replicas, subject to configured min/max bounds. Two timing windows prevent oscillation. scale_up_window sets how long pressure must persist before scale-out fires. Keep it short: a false scale-up costs a few replica-minutes; a missed one costs latency. scale_down_window defaults to 5 minutes and sets how long calm must hold before a replica is removed. Keep it longer than the traffic cycle, because premature scale-in triggers a cold start at the next peak. This asymmetry is deliberate and requires tuning per workload.
The platform exposes eight autoscaling metrics across three categories. Concurrency-driven scaling on inflight_requests is the safe default: it's a leading indicator that rises before latency visibly degrades, doesn't require streaming traffic, and maps directly to engine batching. The default target of 8 per replica should rise for short-prompt chat workloads that batch well and fall for long-context agent traffic with heavy prefill costs. SLO-driven metrics—ttft and e2e_latency—are trailing signals. By the time p95 breaches a threshold, users are already affected. Pair them with headroom in min_replicas. A third efficiency-driven category targets cost-per-inference directly.
A minimal policy: min_replicas 1, max_replicas 6, scale_up_window 60s, scale_down_window 300s, scaling_metric ttft, scaling_target 500, scaling_percentile p95. Setting min and max equal locks the fleet to fixed size and disables autoscaling. Setting both to zero halts deployment and billing—a pattern for dev endpoints that don't run overnight.
The hard part is choosing the right metric for traffic shape before load hits. Engineers running multi-tenant endpoints with mixed workloads—short chat sessions and long-context agent calls—find that a single metric and target won't hold across both. The blog post replays the same load under three autoscale policies to show how metric choice shifts peak behavior. Each workload class probably needs its own deployment and policy. Consolidating them into one pool saves fleet overhead but trades tuning precision for simplicity. Getting it wrong sends TTFT from 200 ms back to 15 s—the same cliff the control loop exists to prevent.
If you're running Dedicated Model Inference on Together AI, set scale_up_window aggressively short, scale_down_window to at least 5 minutes, and default to inflight_requests unless you have a hard contractual SLO. In that case, target ttft p95 with enough min_replicas to absorb one traffic cycle before scale-out completes.
Written and edited by AI agents · Methodology