LMSYS 发布 Miles v0.1：
把前沿模型后训练拆成可验证、可扩展的工程流水线。

‹ Back to Blog‹ Back to BlogContents

- The Miles RL Loop

- Rollout

- Fast Agentic Rollout by SGLang

- Fully Async RL

- Agentic Environments

- Token-In-Token-Out (TITO)

- Efficient Rollout Routing Replay (R3)

- Training

- Low-precision Training

- Memory Efficiency & Disk Offload

- Two Training Backends

- Weight Update

- Verified Day-0 Model Support

- Other Post-Training Recipes

- LoRA RL

- On-Policy Distillation (OPD)

- Zero-KL Alignment

- Code Quality Principle

- Miles-Diffusion

- Multi-Hardware Support

- Example: Training GLM-5.2 on Terminal-Use Tasks with 64 NVIDIA GB300 GPUs

- Acknowledgement

- References

Miles v0.1: Production-level Post-training
Miles TeamAugust 18, 2026
We present Miles v0.1, a full-stack production-ready system for frontier post-training, the successor to our first Miles release [1]. Building upon slime's clean design, Miles optimizes every stage in the RL training loop around a simple principle: verified, clean, and customizable everywhere. With accuracy, efficiency, reliability, and scalability as first-class goals, Miles aims to make frontier-scale RL accessible to researchers and developers alike. In this blog post, we will walk through Miles end-to-end.

The Miles RL Loop

An RL training job in Miles is a loop over the following stages:

- Rollout — SGLang engines generate trajectories. In agentic RL, each multi-turn rollout session interacts with its own isolated environment that executes actions and produces the reward.

- Training — completed trajectory groups are consumed by the trainer (NVIDIA Megatron-LM or FSDP), which computes the RL loss and updates the policy.

- Weight update — new weights are synchronized back to the rollout fleet with minimal interruption to in-flight rollouts.

In what follows, we will go through each element in the loop to highlight how we make Miles accurate, efficient, reliable, and scalable.

Figure 1. The Miles fully async RL loop.

Rollout

Every rollout in Miles is generated by SGLang. Building upon SGLang's native inference efficiency, Miles unlocks the full agentic training workflow: multi-turn sessions, tool execution, sandboxed environments, and token-faithful trajectory capture.

Fast Agentic Rollout by SGLang

Miles provides fast agentic rollout by natively integrating SGLang, which is optimized for long, multi-turn generation. Agentic trajectories vary widely in length, and each new turn reuses most of the previous context. By default, Miles uses the SGLang router, which keeps all turns of a session on the same SGLang engine and DP rank (if DP attention is enabled) to reuse the cached prefix, while assigning new sessions to the least-loaded rank so that a few long trajectories do not overload part of the fleet. The SGLang router also reserves KV-cache capacity for long sessions beforehand. These features ensure a balanced and stable rollout concurrency, keeping cache-hit rate high in agentic training.

Fully Async RL

For long-context, tool-use, and agentic workloads, rollout time is mainly determined by a handful of stragglers. A synchronous schedule exacerbates the straggler issue, because the trainer has to sit idle until the slowest trajectory in the batch returns, and the rollout engines have to wait until the optimizer finishes model update. Miles' fully asynchronous RL eliminates such mutual-blocking by allowing rollout engines to generate persistently: rollout generation stays continuously in flight while the trainer consumes completed groups and updates the model. Neither side is blocked by the other.

Figure 2. How fully async RL handles the long tails. Different colors indicate different weight versions; the green fragments represent tool-call time.

Scheduling operates at sample granularity: each completed trajectory immediately frees a slot, keeping generation concurrency stable despite large differences in trajectory length. Completed groups enter a bounded data buffer that decouples rollout throughput from training cadence. This buffer also forms a customizable policy boundary where users decide which samples to keep, retry, discard, or reject as stale without modifying scheduling or execution.

To make evaluation asynchronous, Miles provides three evaluation modes, distinguished by where model weights come from.

- Shared-engine evaluation uses the rollout fleet and temporarily pauses new submissions, making it suitable for small debug sets without extra GPUs.

- Dedicated evaluation uses a separate GPU fleet loaded from checkpoint snapshots, allowing training and rollout to continue uninterrupted.

- External evaluation passes a checkpoint directory to any user-provided evaluator, including non-SGLang services.

All three modes associate results with the checkpoint and training step that produced them. If evaluation finishes several steps later, Miles reports that lag rather than misattributing the result; failures skip the evaluation point instead of terminating training.

Figure 3. Evaluation modes: sharing the rollout engines gates generation, while a checkpoint snapshot hands the eval to a dedicated fleet or an external service without stopping training.

Agentic Environments

In agentic RL, much of the work happens in an isolated environment. Training a coding agent, for example, means giving the model its own sandbox for each task: it runs commands, edits files, reads what comes back, and a test suite at the end decides whether the task was solved. The environment holds that state, executes the actions, and runs the verifier that produces the reward. Miles runs many such episodes at once, records each trajectory in a form the trainer can learn from, and carries the verifier's result through as the reward.

An environment reaches Miles through a plug-in point. Miles offers multiple plug-in points at different levels of the rollout stack, so an environment can take over as much or as little of the rollout as it needs. We ship integrations for existing ecosystems, including Harbor, HUD, NeMo Gym, OpenEnv, and Prime Intellect Verifiers. You can even bring your own environments, which plug in through the same points.

The sandboxes run on any backend of your choosing. We support several, including AgentENV, Daytona, E2B, and Modal. Each episode gets a fresh sandbox, built from that task's own image, so nothing is shared between episodes and nothing stays resident after a run.

Miles v0.1 ships maintained end-to-end recipes for agentic coding and terminal tasks. They are validated on real hardware and can be launched as they stand.

Token-In-Token-Out (TITO)

In multi-turn agentic RL, model outputs pass through message parsing, tool execution, and chat-template rendering before entering the next turn. This process can change tokenization, prune historical reasoning, or reserialize tool calls, causing the trainer to see a different token context from the one actually used during rollout.

The TITO session server in Miles [6] preserves the exact token IDs generated by the model. On each new turn, it tokenizes only the newly appended messages and merges them with the existing prefix. This allows the complete trajectory to be assembled into one contiguous training sample, preserving the original rollout log probabilities while loss-masking tokens that are not generated by the model. For each model family, TITO is validated through CPU round-trip tests and real SGLang GPU sessions, safeguarding the token-level exactness necessary for R3, OPD, and zero-KL alignment.

Building on TITO, Miles is developing a training recipe for black-box agent harnesses such as Claude Code and Codex. These harnesses spawn subagents and compact context at runtime, so the number of trajectories per task is dynamic and unknown in advance. Miles therefore records the full trajectory tree through the session server, and on the training side applies the loss normalization required for varying batch sizes, keeping gradient scale consistent.

Figure 4. Token-in, token-out. The session server keeps the exact token ids the engines produced, so the trainer sees the tokens the model actually generated even when the harness is an opaque black box.

Efficient Rollout Routing Replay (R3)

MoE RL is sensitive to tiny numerical differences between rollout and training: a single top-k routing flip changes both the token computation and the expert weights that receive the gradients. To resolve this issue, Miles' Rollout Routing Replay (R3) records SGLang's expert routing results during rollout and replays them during training. Such replay is handled efficiently in SGLang, adding only minimal overhead compared with normal routing.

Training

Trainer is the backbone of RL. In Miles, we ship numerous trainer optimizations that make RL stable, fast, and resource-efficient.

Low-precision Training

Miles supports rollouts in NVFP4, MXFP4, MXFP8, and FP8; end-to-end training recipes for NVFP4, MXFP8, and FP8; and INT4 quantization-aware training (QAT) [3] for most models. Earlier posts cover the FP8 [2] and INT4 QAT [3] recipes in depth.

Capturing Blackwell's low-precision throughput in RL requires more than swapping GEMM dtypes: quantization on the rollout and training sides must agree, or the mismatch accumulates across weight updates into policy divergence. To this end, we build the Blackwell-native MXFP8 and NVFP4 recipes [9] as end-to-end precision contracts across the stack: MXFP8 runs rollout, forward, and both gradient GEMMs with hardware blockwise scaling; NVFP4 quantizes MoE expert weights per-token with online activation scaling to avoid batch-dependent quantization artifacts; a bit-exact quantizer contract keeps training and rollout kernels seeing the same quantized values, with fine-grained flags to keep sensitive layers in BF16. The recipes are verified with low KL divergence between SGLang and Megatron-LM. All low-precision configurations keep reward curves closely tracking the BF16 baseline while reducing rollout time.

Memory Efficiency & Disk Offload

Training a 744B-parameter model asynchronously on 16 nodes poses a significant memory issue. Miles ships with sophisticated memory optimization that large-scale runs depend on, starting with the optimizer: optimizer states can be offloaded to CPU or node-local NVMe and streamed back per bucket during each optimizer step, so they never need to be resident on the GPU at once. In the example that we demonstrate at the end of this blog post, this NVMe optimizer-state streaming is what lets the GLM-5.2 optimizer fit alongside the training engine on 32 GB300 GPUs. Beyond offloading, Miles also enables a series of other GPU and CPU memory footprint reductions, saving 30+ GB HBM memory per GPU in the example GLM-5.2 run, and hundreds of GB of CPU memory per node when the trainer is colocated with the rollout engines.

Two Training Backends

Miles supports two training backends behind one interface: NVIDIA Megatron-LM and PyTorch FSDP. A single launch flag picks which of them owns the model on the GPU; a backend implements five methods, and nothing above that line changes.

- Megatron-LM is the default, and the backend the model recipes are written against. It splits the model internally across tensor, pipeline, context, expert and expert-tensor parallelism, and supports CPU and NVMe optimizer offload, per-bucket optimizer state streaming, and checkpoints that are parallelism-agnostic, so the layout can change later without reconverting.

- FSDP trains the model's own HuggingFace implementation under PyTorch FSDP2. Config and weights load straight from the HuggingFace directory, so there is no conversion step and no architecture flags to write, and parallelism is data parallel only. Architectures that need small corrections register them as adaptation specs rather than forking the model code.

Weight Update

In each training step, after the optimizer updates the model weights, the new weights have to be synchronized across all rollout engines. When training and rollout run on different GPUs, weight synchronization can become a major pipeline bottleneck. Miles provides two optimized paths for diverse deployment settings. With P2P weight transfer [5], training ranks re-shard each weight bucket for the target SGLang layout and write only the required shards directly into rollout-rank memory over RDMA. This avoids broadcasting the full model to every rank and uses multiple training ranks as senders in parallel. The benefit grows with model size and expert parallelism: for Kimi-K2 1T, P2P reduced weight-update time from 53.3 seconds to 7.2 seconds.

For rollout fleets without direct NCCL or RDMA connectivity, Miles supports disk-delta updates. Consecutive RL steps often change only a small fraction of the model's bytes, so Miles only needs to publish the delta-weights against the previous policy version instead of transferring a full checkpoint. Rollout engines apply and verify the delta-weights while generation continues, then pause only to load the combined weights into SGLang. Usually only 2% of parameters are included in the delta-weights for BF16 rollout, and 0.5% for FP4 rollout. In a GLM-4.7-Flash run, this reduces the per-update payload from 62.4 GB to 0.69–0.83 GB while keeping the generation pause within 3–5 seconds.

Verified Day-0 Model Support

Miles lands new frontier models on the day their weights become public, together with SGLang. Kimi-K3 [13], DeepSeek-V4 [10], Inkling [12], Qwen3.8 [14] and NVIDIA Nemotron 3 Ultra [11] were all trainable in Miles on release day, because the inference path in SGLang and the RL recipe in Miles are brought up in parallel rather than one after the other. Day-0 coverage is not limited to one architecture family either: it spans dense and MoE models, hybrid attention, and multimodal inputs. Beyond day 0, almost every open frontier model runs on Miles, including DeepSeek-V3.2, Kimi-K2.6, Qwen3.8, GLM-5.2, Gemma-4, GPT-OSS and so on. For each supported model, we ship CI-guarded recipes in Miles. The full list can be found here.

Other Post-Training Recipes

Beyond the core RL loop, Miles is a general post-training platform. The rollout engines, trainer, and weight-update paths described above are shared components rather than RL-specific machinery, and they compose into other training paradigms: LoRA RL, on-policy distillation (OPD), and supervised fine-tuning (SFT), which runs the same trainer with an SFT loss and no rollout engine at all. This section walks through LoRA RL and OPD, where most of the Miles-specific work lives, and closes with Zero-KL Alignment, which eliminates the numerical mismatch between rollout and training so that on-policy data is genuinely on-policy.

LoRA RL

Miles enables end-to-end LoRA reinforcement learning across training, weight synchronization, and SGLang rollout, for both LLMs and diffusion models. The base model remains frozen and resident, while Miles trains and synchronizes only the adapters, and SGLang applies the latest adapter during rollout.

Miles supports dense and MoE models including GPT-OSS, GLM, Kimi K2/K3, and Inkling. For GLM-5.2 744B-A40B, the trainable parameters occupy just 212 MB, around 0.014% of the size of the model's BF16 base weights—significantly reducing training costs.

Alongside saving memory, LoRA also brings efficiency advantages in RL training:

- Less time for each trainer step: the trainer computes weight gradients and optimizer updates only for the small low-rank adapters, reducing gradient computation, optimizer workload, memory traffic, and distributed communication, thereby improving overall training efficiency.

- Lightweight and efficient weight synchronization: Miles only needs to synchronize the updated LoRA adapter. It packs distributed TP and EP fragments into a small number of flat collectives, then transfers the serving-ready tensors directly to colocated SGLang workers, without staging them in host memory. Because only the LoRA weights are transferred, this approach significantly reduces transfer time.

Miles also provides a multi-LoRA RL backend for training and serving independently configured LoRA adapters on a shared base model. This experimental backend updates only changed adapters and routes mixed-adapter rollouts through SGLang, laying the foundation for a Tinker-compatible platform with lower compute, memory, communication, and operational cost.

On-Policy Distillation (OPD)

Miles supports on-policy distillation [8], which allows a student to learn from teacher guidance on its own rollout distribution. The reverse KL is implemented as a "reward," so users can optionally complement the distillation with conventional GRPO/PPO-style rewards. In terms of how the reverse KL is approximated, users can also choose between computing only from the student's sampled token, or from a set of tokens with top-K probabilities.

For long-context workloads, Miles uses sparse per-position teacher scoring: the teacher scores only the candidate tokens required at each generated position. This avoids unnecessary construction, communication, and parsing overhead. Miles also offers configurable candidate selection and weighting strategies for a range of OPD recipes.

In a Qwen3.5-35B-A3B self-distillation experiment on a B200 node, with only the reverse KL "reward", OPD reduced the average rollout length from 18.6K to ~6K tokens, while improving held-out DAPO performance from 84.6% to 89.5%. This demonstrates OPD's ability to transfer behaviors with minimal effect on benchmark performance.

Figure 5. On-policy distillation. The teacher scores the tokens the student generated, and the reverse-KL signal drives the update, optionally alongside a task reward.

Zero-KL Alignment

Miles supports Zero-KL Alignment, minimizing numerical differences between rollout and training engines by aligning attention, GEMM, operator precision, batching behavior, and other execution details, in addition to model weights. In supported configurations, the training–rollout log-probability difference can stay exactly zero.

Zero-KL Alignment is currently only available for the Qwen 3 model family, but we plan to support more model architectures, kernels, and parallelism configurations soon.

Code Quality Principle

By now, we have covered the entire RL training loop in Miles, and all those features are built around our code quality principle: Miles should be easy to read and extend. The training driver is intentionally written like pseudocode, while major components, like rollout functions, data sources, losses, and rewards, sit behind small, typed interfaces. The rollout stack is divided into agent, generation, and rollout layers, so an environment or agent framework can replace only the layer it needs while reusing everything else. This makes it straightforward to build a custom RL training loop without forking or rewriting Miles.

Miles-Diffusion

Miles' design philosophy extends beyond LLMs to diffusion models. In Miles-diffusion, sglang-diffusion serves as the high-performance rollout engine and returns the full denoising trajectory with per-step log-probs. An FSDP2 trainer consumes selected SDE steps and optimizes the RL objective under hybrid shard and SP. Inherited from Miles' design, Flow-GRPO, DiffusionNFT, and SFT are all unified within a single trainer with pluggable loss, rollout, and reward components. Initial supported models include SD3.5, Qwen-Image, Wan2.2, LTX-2.3, Cosmos3, and Minimax-H3, with curve-verified recipes for each.

On the rollout side, sglang-diffusion rollout organizes generations into microgroups, reusing encoder results and batching samples. A 5.87× speed-up is observed over plain serial generation on Qwen-Image at 512×512 with group size 16. Each microgroup is then deserialized and reward-scored as its own async stream to overlap with others. On the training side, optional deterministic mode enables bitwise reproducibility, which verifies recipe quality together with documented curves and end-to-end CI. To address training-inference mismatch and keep precision-sensitive DiT parameters, FSDP2 is patched to enable fine-grained per-parameter dtype control.

Multi-Hardware Support

Miles runs the same training loop on NVIDIA and AMD GPUs. NVIDIA coverage spans A100 through GB300, the hardware behind the reference run and the Blackwell low-precision recipes in this post. AMD support is native ROCm through HIP and RCCL [4], [7]: Miles runs on MI300X through MI355X with the same SGLang rollout integration and dedicated Docker images, and CI exercises both vendors alike, running end-to-end training tests on NVIDIA and AMD runners.

Example: Training GLM-5.2 on Terminal-Use Tasks with 64 NVIDIA GB300 GPUs

As an end-to-end example of agentic RL training, we demonstrate how we use Miles to train a GLM-5.2 744B-A40B model on terminal-use coding tasks using fully asynchronous RL across 64 NVIDIA GB300 GPUs, with 32 GPUs for rollout and 32 GPUs for training. For maximal performance, we set training parallelism to TP 2 / PP 4 / CP 4 / EP 8 and inference parallelism to TP 8 / EP 8, with multi-token prediction (MTP) enabled. Each multi-turn terminal agent runs in its own sandbox using the OpenEnv integration in Miles. In this reference run, we use a 65k max sequence length and a batch size of 64.

- The reference run completes 100 terminal-bench-like coding task rollout steps stably, sustaining roughly 4.5 minute training steps, with the rollout weights lagging on average 1.7 steps behind the trainer.

- Rollout and training are almost fully overlapped via sample-level asynchronous scheduling, DP-aware and load-balanced routing, and a balanced SGLang server config. Rollout reaches a 96% prefix-cache hit rate with stable concurrency.

- Thanks to the substantial memory optimization enabled by Miles, we can fit asynchronous training of a 744B model into 64 GPUs, hosting the training engine on only 32 GB300 GPUs.

The full launch script for this run can be found here.

Figure 6. Metrics in the GLM-5.2 agentic RL training example.

Acknowledgement

We would like to thank all community contributors of Miles and SGLang for their invaluable contributions and continued support ❤️

References

The Miles blog series

[1] Introducing Miles — RL Framework To Fire Up Large-Scale MoE Training

[2] Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL

[3] Squeezing 1TB Model Rollout into a Single H200: INT4 QAT RL End-to-End Practice

[4] ROCm Support for Miles: Large-Scale RL Post-Training on AMD Instinct™ GPUs

[5] Updating 1T parameters in seconds — P2P weight transfer in Large Scale Distributed RL

[6] No Token Left Behind: Demystifying Token-In-Token-Out in Miles

[7] Bringing DeepSeek-V4 Flash RL Training to AMD Instinct MI355X GPUs with Miles

[8] OPD Support in Miles

[9] Towards Blackwell-Native 8-bit and 4-bit RL: End-to-End MXFP8 and NVFP4 RL in Miles

Day-0 model support with Miles RL

[10] DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles

[11] SGLang and Miles Add Day-0 Support for NVIDIA Nemotron 3 Ultra for Long-Running Autonomous Agents

[12] SGLang and Miles Add Day-0 Support for Inkling, a Frontier Multimodal Model

[13] SGLang and Miles Add Day-0 Support for Kimi K3

[14] SGLang and Miles Add Day-0 Support for Qwen3.8