Together AI has shipped a native A/B experiment primitive at the endpoint level, letting teams split live inference traffic between a control model and up to 20 variant deployments without client-side code changes. It closes a gap most ML platform teams fill with ad-hoc feature flags and hardcoded model strings.
Traffic first resolves through the endpoint's existing weight split. When a request lands on the control arm and an A/B experiment is attached, the platform re-samples the request across the experiment's members by their percent settings. Variant deployments must carry zero weight in the base split—they exist as shadow deployments that the experiment routes to exclusively. Percents across control and all variants must sum to 100%. Critically, these are true fixed traffic shares independent of replica count or autoscaling. The split won't drift as replicas scale under load, eliminating a silent source of sampling bias in capacity-weighted routing schemes.
The ramp progression Together AI documents maps directly to the risk/signal tradeoff. A 95/5 split gives minimal exposure on first contact: slow signal accumulation but contained blast radius. 80/20 offers a more significant readout after the candidate survives initial exposure. 50/50 is reserved for late-stage confirmation between two already-validated options. Each ramp is an explicit, reviewable state change on the experiment object. Updating the member list replaces it atomically and is guarded by an etag—concurrent ramps from teammates get rejected rather than silently overwritten, preventing race conditions in the ops runbook.
| Stage | Control % | Variant % | Risk Profile | Signal Accumulation | Recommended Use |
|---|---|---|---|---|---|
| Initial exposure | 95 | 5 | Minimal blast radius | Slow | First contact with candidate model |
| Candidate validation | 80 | 20 | Moderate | Significant readout | After candidate survives initial exposure |
| Late-stage confirmation | 50 | 50 | Low (both already validated) | Fast | Final comparison between two validated options |
Ending an experiment is one delete call. Traffic returns 100% to control with no client-side changes, no routing logic to unwind. This matters: the pathology of DIY approaches is that experiment logic ships inside the application, and branching code outlives the experiment because removing it requires a separate deploy with its own risk surface.
The multi-way support unlocks quantization comparisons without restructuring. A full-precision control alongside three quantized variants (INT8, GPTQ, AWQ) runs as a single experiment as long as percents sum to 100%. The platform routes per-deployment metrics—latency, error rate, throughput—which architects join against product signals like thumbs-up rate, task completion, and session abandonment. Offline benchmark scores are explicitly not the target; the guide's framing is that offline eval doesn't reflect live traffic dynamics, cost profiles, or latency distributions under real load.
| Arm | Model Type | Base Split Weight | Serving Metrics | Product Signals |
|---|---|---|---|---|
| Control | Full-precision | Full weight | Latency, error rate, throughput | Thumbs-up rate, task completion, session abandonment |
| Variant 1 | INT8 quantized | Zero (shadow) | Latency, error rate, throughput | Thumbs-up rate, task completion, session abandonment |
| Variant 2 | GPTQ quantized | Zero (shadow) | Latency, error rate, throughput | Thumbs-up rate, task completion, session abandonment |
| Variant 3 | AWQ quantized | Zero (shadow) | Latency, error rate, throughput | Thumbs-up rate, task completion, session abandonment |
The statistical challenge of LLM A/B tests isn't fully solved. Non-deterministic outputs inflate variance in quality metrics relative to traditional A/B tests, meaning statistical significance at a 95/5 split requires substantially more time and volume. Teams that move to 80/20 too quickly on thin signal are making a judgment call, not a statistical one. Fixed percents help, but teams still need to define acceptance thresholds before ramping, not during.
The takeaway: push experiment logic to the serving layer, not the application layer. Pin your control deployment to a specific model version, not a floating alias. Define rollback thresholds—error rate, p99 latency delta, cost-per-request budget—before the first request hits the variant.