Together AI ships capacity-aware traffic routing for multi-deployment model inference

Together AI announced a new deployment model for configurable model inference on its platform, enabling capacity-aware traffic routing across multiple model + hardware combinations. The architecture decouples endpoints (fixed API identities), deployments (model + config + replicas), and configs (engine, GPU type/count, parallelism, optimization profile). This separation allows zero-downtime model rollouts, A/B tests, shadow experiments, and autoscaling without manual traffic weight rebalancing.

Configs are immutable and paired with certified model revisions; deployments are ephemeral and created/destroyed routinely. Traffic splits weight each deployment by its ready replica count and specified weight, so autoscaling automatically adjusts traffic share: a deployment scaling from 1 to 3 replicas absorbs 3x traffic proportionally. The optimization axis lets users choose between latency (smaller batches, faster time-to-first-token) and throughput (larger batches, max tokens/GPU-hour). The platform's public model catalog ships with pre-certified configs for major models (Qwen, Llama, GLM variants) with hardware profiling across H100, H200, B200.

For architects: This is table-stakes for inference platform maturity—Replicate and HuggingFace Inference endpoints have similar primitives, but Together's explicit capacity-aware routing is elegant. The weight-per-replica model means load-balancing and autoscaling compose naturally. If the platform is reliable and the config catalog grows, this becomes a portable, hardware-agnostic way to manage multi-model inference. Watch for adoption by vLLM users seeking managed inference without rebuilding orchestration.