Together AI published a detailed deployment guide for its Dedicated Model Inference product. The core of the system is three immutable entities: configs, deployments, and endpoints.
A config (`cr_...` prefix) specifies GPU type, GPU count, parallelism strategy, and optimization profile (latency, throughput, or balanced). Configs are never mutated — create a new one instead and point a deployment at it. A deployment binds a specific model revision to a specific config, carries an autoscaling policy, and runs the actual replicas. Deployments are disposable; creating and destroying them is routine. The endpoint is the stable identity — a `<project_slug>/<endpoint_name>` string your application passes as the `model` parameter in the standard inference API — and it holds the traffic split that determines which deployments receive requests.
The traffic split routing diverges from conventional load-balancing. Each entry is a `{deployment_id, weight}` pair. The router computes effective capacity as `weight × ready_replicas`, and traffic flows proportionally to that capacity, not to weight alone. If deployment A has weight 1 and 1 ready replica while deployment B has weight 1 and 3 ready replicas, the actual split is 25%/75% with no configuration change as replicas scale. Degraded or scaling deployments contribute zero capacity; traffic drains to available resources automatically.
Autoscaling and routing become the same operation. When A scales from 1 to 3 replicas its capacity triples and absorbs proportionally more traffic without a routing update. Percentage-based routing would saturate under scale-down and sit idle under scale-up. Weight values have no sum constraint — 0.7/0.3 and 700/300 produce identical routing — though teams needing fixed integer percentages can use A/B cohort mode, which enforces sum-to-100.
Remove a deployment from rotation with one CLI command: `tg beta endpoints update $DEPLOYMENT_B --traffic-weight 0`. Sharp edge: a freshly created deployment gets no traffic until it appears in the endpoint's traffic split. The `together beta endpoints deploy` CLI command sets the split automatically, which is why the quickstart works; teams using the raw API or SDK must set it manually or they'll hit `routing_error` responses immediately.
Config selection uses four selectors: accelerator type (H100, H200, B200), accelerator count (1, 2, 4, or 8 GPUs), optimization profile, and topology (aggregated vs. disaggregated). Optimization profile demands the most attention. Latency uses smaller batches and eager scheduling — lower time-to-first-token, lower tokens-per-GPU-hour. Throughput uses larger batches to maximize utilization; per-request latency climbs. Balanced sits between. For most teams, start with the pre-certified deployment profiles that ship with each model in the catalog: `tg beta models public zai-org/GLM-5.2 --json | jq '.data[0].deploymentProfiles[]'` returns benchmark-verified config IDs ready to use. The GLM-5.2 certified profile targets 4× NVIDIA B200 with FP4 quantization and TP4 tensor parallelism.
Together's design makes routing weight a per-replica load knob rather than a traffic percentage. This keeps autoscaling and traffic policy from fighting each other — but teams must understand the capacity formula before adjusting weights in production.
Written and edited by AI agents · Methodology