MISA-T Cuts RL Rollout Iteration Time 22.8% on vLLM

Researchers from multiple institutions published MISA-T on August 11, a routing-layer admission policy that replaces per-workload heuristics with three coordinated mechanisms — adaptive session admission, workload-aware KV-capacity allocation, and residency-time-aware KV accounting — to schedule mixed RLVR, RLHF, and agentic rollouts on shared asynchronous inference service. On Step3.7 and Qwen3.6-35B-A3B, MISA-T beats a sweep-tuned cache-aware vLLM Router by 53.3% and 43.6% rollout throughput respectively. In a matched 50-iteration Step3.7 experiment it delivers 35.6% throughput increase and cuts mean iteration time by 22.8% while keeping the trainer's workload mixture intact.

The core problem: prefix-aware routing, the current standard in vLLM production stacks, routes requests to replicas that already hold their prefix in KV cache. This works when all requests resemble each other. Post-training pipelines mixing RLVR (short verifiable sequences, high turnover), RLHF (moderate multi-turn, human-paced feedback), and agentic rollouts (long multi-turn sessions with tool-call gaps) on the same cluster produce three workload classes with radically different KV-residency times. Long-residency agentic sessions fill blocks and evict RLVR tokens mid-rollout, stalling the trainer's sample pipeline.

MISA-T sits at the routing layer and tracks each session's expected residency time before admission. It gates admission by whether sufficient KV capacity exists to serve the session's full trajectory without displacing workloads closer to completion. Capacity allocation is workload-class-aware: RLVR sessions receive tight blocks matched to their short sequences, while agentic sessions are assigned regions that tolerate tool-call gaps without triggering eviction. The residency-time accounting component adjusts estimates dynamically as observed session durations diverge from prior.

MISA-T achieves its throughput gains against a sweep-tuned baseline, not a default vLLM Router config. The comparison team ran hyperparameter sweeps on the cache-aware router before measuring, so the gap represents genuine algorithmic improvement over a well-configured system. In the rollout-only ablations, prefix-cache hit rates remain high alongside throughput gains, confirming MISA-T does not trade cache efficiency for admission throughput.

For inference platform teams, the regime where this matters most is asynchronous post-training at scale: an actor model on vLLM generating rollouts while a separate training cluster runs FSDP or Megatron with GRPO. The rollout step is the critical-path bottleneck. A 22.8% reduction in mean iteration time compounds across thousands of training steps. Systems like RollArt (Alibaba, 96 H800 + 32 H20 GPUs) and BiDiRL (bidirectional rollout/training resource sharing) show rollout throughput directly throttles training throughput. MISA-T targets the scheduling gap inside the rollout pool that neither system addresses.

The paper evaluates two models: Step3.7 (a dense reasoning model) and Qwen3.6-35B-A3B (a MoE model with 3.6B active parameters per token). The throughput delta is larger on Step3.7 (53.3%) than Qwen3.6-35B-A3B (43.6%), likely reflecting tighter KV-budget conditions in dense models. The paper does not report GPU count or cluster topology, making extrapolation to larger multi-node deployments uncertain. The vLLM Q2 2026 RL roadmap already lists adaptive throughput/latency profiles for RL rollout long-tail and RDMA-based artifact transport as open items. MISA-T addresses the scheduling layer between those two concerns but does not solve cross-cluster weight-sync overhead.

For post-training pipelines mixing RLVR, RLHF, and agentic rollout on shared vLLM replicas, MISA-T's routing-layer approach requires no changes to the training framework or inference engine — only to the router that dispatches sessions to replicas.