Off-Policy Drift in LLM RL

Note: This post is a work-in-progress and may contain mistakes or miss recent work. If you spot an error or know of something I should cover, please send me an email. If you find this post useful, please cite it using the BibTeX at the end. This post was written with assistance from Claude Code, Codex, and GPT-5.2 Pro.

Updates in v2 (2026-05-18). Corrected method descriptions for R3, IcePop, QUATRO, MinPRO, SAPO (gate vs. shape), GSPO stop-gradient scoping, OAPL denominator and critic, KL-cov formula, R²VPO, and DISPO. Replaced the stale "13 advantage estimators" claim with the current 14 (added gdpo). Added the two missing *_token_icepop presets to the rollout-correction table. Pinned all verl-source claims to verl-project/verl@70744059 (the repo migrated from volcengine/verl in early 2026). Re-wrote the DeepSeek-V3.2 and Ring-1T production-recipe paragraphs against the actual paper Sections 3 and 2.3, and added a CompassMax reference (corrected to arXiv:2512.07710 — the original prompt had a different paper's ID). Added a Mar–May 2026 paper round-up, a diagnostic decision tree, a comparison map of every method on the same axes, and a notation map clarifying $\rho_t$ vs. $r_t$. Open-problems section rewritten as five falsifiable conjectures.

Modern LLM reinforcement learning is secretly off-policy (Yao et al. 2025). Even frameworks advertised as "on-policy" have a gap between the policy that sampled the data and the policy that computes gradients on it. The causes are many: separate engines for inference and training running at different precisions, different attention kernels, different tensor-parallel configurations, MoE routing decisions that flip between forward passes, async pipelines where the policy moves before stale rollouts are consumed, and multi-epoch training over the same batch. Each source independently shifts the log-probabilities that the gradient depends on, and they compound. The result is corrupted gradients that can collapse training.

This post walks through how mismatch arises, how to detect it, how to correct it with importance sampling and rejection sampling, and how policy loss functions interact with staleness. The treatment follows the rollout correction framework in verl, with references to the relevant source files throughout.

Why LLM RL training is secretly off-policy

The two sources of staleness

In a standard LLM RL pipeline, the full importance weight between the current training policy $\pi_\theta$ and the rollout (behavior) policy $\mu_{\theta_{\text{old}}}$ decomposes into two factors (Zheng et al. 2025):

The first factor is training-inference mismatch: even when the weights are identical ($\theta = \theta_{\text{old}}$), the rollout engine $\mu$ and the training engine $\pi$ produce different log-probabilities. Precision differences (FP8 vs BF16), different attention backends (FlashAttention vs PagedAttention), different parallelism strategies. This factor should be 1 everywhere. In practice it is not.

The second factor is policy staleness: the weights have moved since the rollout was generated. This happens in multi-epoch PPO (the policy updates between epochs but trains on the same rollout data), in one-step async pipelines (generation of batch $N$ overlaps with training on batch $N-1$), and in fully async pipelines (multiple training steps may occur before fresh rollouts arrive).

Both factors corrupt the gradient. Policy gradient methods assume the data was sampled from the current policy. When that assumption breaks, the gradient points in the wrong direction, and the error compounds across training steps. This is the mechanism behind RL collapse (Li & Liu 2025; Yao et al. 2025).

Notation map

Three policies show up everywhere in this post. The literature is not consistent about which symbol means which, so it is worth pinning notation explicitly:

• $\mu_{\text{rollout}}$ — the rollout engine's distribution over tokens (vLLM, SGLang, etc.) at the rollout-time weights.
• $\pi_{\text{old}}$ — the training engine's distribution over tokens at the same rollout-time weights $\theta_{\text{old}}$ (i.e., the "actor forward pass" recomputed log-probs).
• $\pi_\theta$ — the training engine's distribution at the current weights $\theta$.

From these we get three ratios:

• Engine-mismatch ratio: $\rho_t^{\text{infer}} = \pi_{\text{old}}(y_t)/\mu_{\text{rollout}}(y_t)$ — same weights, different engines. Should be 1; in practice is not.
• Policy-staleness ratio: $r_t(\theta) = \pi_\theta(y_t)/\pi_{\text{old}}(y_t)$ — same engine, different weights. This is the standard PPO ratio.
• Full importance weight: $w_t(\theta) = \pi_\theta(y_t)/\mu_{\text{rollout}}(y_t) = \rho_t^{\text{infer}} \cdot r_t(\theta)$ — current policy on training engine vs. behavior policy on rollout engine.

In the rollout-correction sections below, a bare $\rho_t$ means the engine-mismatch ratio $\rho_t^{\text{infer}}$. In the policy-loss sections, a bare $r_t$ means the staleness ratio. Wherever the post uses $\rho_t$ without qualifier in equations imported from a particular paper, the convention follows that paper.

Root causes in detail

The two-factor decomposition above identifies two abstract sources of drift. In practice, six concrete root causes produce this drift, and understanding each determines whether the right fix is a systems change, an algorithmic correction, or both.

Floating-point precision: BF16 vs FP16 vs FP8

BF16 is the default training dtype for most modern LLM stacks, and for good reason: its 8-bit exponent matches FP32's dynamic range, which simplifies mixed-precision recipes. But BF16 has only 7 bits of mantissa (versus FP16's 10), and that coarser rounding is the single largest contributor to training-inference mismatch in many production pipelines. When the inference engine and the training engine both operate in BF16 but use different fused kernels, different reduction orders, or different intermediate accumulation widths, the rounding errors do not cancel — they compound through the softmax into systematically different token probabilities.

Qi et al. (2025) make the sharpest version of this claim: switching both the rollout engine and the training engine to FP16 can nearly eliminate the mismatch with minimal code changes. FP16's extra 3 mantissa bits are enough to make the residual rounding error negligible relative to other noise sources. The cost is a narrower dynamic range (5-bit exponent), which requires careful loss scaling — but that is a solved problem in most frameworks.

At even lower precisions the picture is worse. FP8 and INT8 quantized rollouts are attractive for throughput (roughly 2x over BF16 on modern accelerators), but they introduce a structured distribution shift: the quantized actor is not just a noisy version of the full-precision policy but a systematically different one. FP8-RL (Qiu et al. 2026) and Jet-RL (Xi et al. 2026) both demonstrate that a mixed BF16-train / FP8-rollout stack is not truly on-policy, and that the resulting mismatch must either be corrected with importance sampling or eliminated by unifying precision across both sides. QuRL (Li et al. 2026) goes further, framing the quantized actor as a permanently distinct behavior policy and designing adaptive clipping ranges around the full-precision/quantized ratio.

Zhang et al. (2026) adds a nuance: the failure is not purely about arithmetic precision. A fixed numerical discrepancy between rollout and training can be tolerable early in training and then suddenly become destabilizing later, because the optimizer moves into a region of parameter space where that discrepancy aligns with a steep loss landscape. In other words, mismatch is coupled to optimization state. This is why purely static precision alignment (switching to FP16 once at the start) works most of the time, but occasionally fails in ways that a reactive intervention — like learning-rate scheduling triggered by response-length surges — can catch.

Kernel and framework divergence

Even with identical dtypes, rollout engines and training engines typically use different software paths: different fused attention kernels, different reduction orders in layernorm and softmax, different batch-scheduling policies, and different memory layouts. These differences are invisible at the API level (both engines accept the same weights and produce "the same" outputs) but visible at the floating-point level, where they produce systematically different log-probabilities.

He et al. (2025) argue that what practitioners call "nondeterminism" in LLM inference is mostly a misnomer. The real culprit is batch dependence and floating-point non-associativity: the same prompt produces different logits depending on what other prompts share its batch, because fused kernels reduce across the batch dimension in an order that varies with batch composition. This is deterministic given the batch but not batch-invariant, and since RL training and RL rollout almost never use the same batching strategy, the two engines diverge even when everything else matches.

Yuan et al. (2025) provide a broader catalog of numerical nondeterminism sources in LLM inference, including thread-scheduling in CUDA reductions, non-deterministic cuBLAS GEMM algorithms, and radix-cache-induced prefill chunking differences. While their focus is inference rather than RL, the root causes they identify are exactly the ones that create rollout-training mismatch: every source of numerical variation in the inference engine is a source of off-policy drift in the RL pipeline.

The systems community's response was a sequence of increasingly strict equivalence projects. The vLLM + TorchTitan bitwise consistency article (2025) audited every kernel invocation in the forward pass until rollout and training produced bitwise-identical outputs. The SGLang deterministic inference effort (2025) post achieved reproducible inference across requests by fixing chunked prefill, CUDA graph replay, and radix cache behavior. Both demonstrate that eliminating kernel divergence is possible but expensive: the constrained kernels are slower, and the restrictions (fixed batch sizes, no dynamic batching, deterministic CUDA graphs) limit the serving optimizations that make rollout fast in the first place.

For practitioners, the implication is pragmatic: full kernel-level equivalence is achievable but usually not worth the throughput cost. The more common approach is to accept some kernel divergence and correct it algorithmically — which requires measuring the divergence first (see the diagnostic toolkit below) and then applying IS or RS correction proportional to the observed gap.

Parallelism strategy mismatch

A particularly common and often overlooked divergence source is that the rollout engine and the training engine use different parallelism strategies. The rollout engine typically uses multi-GPU tensor parallelism (TP) for low-latency generation, while the training engine runs with TP=1 under Fully Sharded Data Parallel (FSDP) or a different TP degree under Megatron. Even with identical kernels and identical dtypes, different TP configurations produce different outputs because the all-reduce operations that merge partial results across GPUs use floating-point addition, which is not associative.

Concretely, a TP=4 rollout partitions each matrix multiply across 4 GPUs, computes partial results, and reduces them with an all-reduce. The training engine at TP=1 computes the same matrix multiply on a single GPU with a different reduction tree. The two results differ in the low bits of the mantissa, and these differences propagate through layernorm, softmax, and the autoregressive chain to produce measurably different token probabilities. The gap is typically smaller than the BF16/FP16 precision gap but large enough to matter over long sequences.

Zhang et al. (2025) isolate this problem and propose Tree-Based Invariant Kernels (TBIK): custom reduction kernels whose output is bitwise identical regardless of TP size. TBIK achieves this by fixing a canonical reduction tree (independent of the number of GPUs) and padding to ensure identical accumulation order. Integrated into both vLLM and FSDP, TBIK eliminates the TP-induced component of mismatch entirely. The throughput cost is modest (single-digit percent) because only the reduction kernels are constrained, not the matrix multiplies themselves.

For pipelines that cannot adopt TBIK, the alternative is to match TP configurations across rollout and training — but this is often impractical, since the optimal TP for generation throughput (high TP, low latency) differs from the optimal TP for training throughput (low TP, high batch size under FSDP). The mismatch is a direct consequence of the engineering reality that serving and training have different parallelism sweet spots.

MoE routing discrepancies

For Mixture-of-Experts models, the "policy" is not just a probability distribution over tokens; it includes the router's discrete decisions about which experts process each token. A small numerical perturbation in the router logits can flip the top-$k$ expert selection, replacing one expert's parameters with another's. Unlike the continuous perturbations from precision or kernel differences, this is a discontinuous change. The model suddenly computes with a different subnetwork, and the resulting log-probabilities can shift by much more than the initial numerical perturbation would suggest.

This makes MoE routing mismatch qualitatively different from dense-model mismatch. In a dense model, a small logit perturbation produces a small output perturbation (the computation is Lipschitz). In an MoE model, the same small perturbation can cross a routing boundary and produce an arbitrarily large output change. Standard importance sampling can correct for the resulting ratio, but the ratio itself can be extreme, leading to high variance or outright rejection of the affected sequences.

R3 / Routing Replay (Ma et al. 2025) addresses this by recording the routing distributions produced by the inference engine and replaying them during training. Instead of letting the training engine's router make fresh decisions (which may differ due to numerical divergence), R3 forces the training engine to use the same expert assignments that the rollout engine used while still letting the softmax over routing logits keep its gradient flow. This eliminates the discontinuity: the training forward pass now uses the same computational pathway as the rollout, so the remaining mismatch is only the continuous kernel/precision component.

IcePop (introduced inside the Ring-1T report (Ling Team 2025), §2.3.2) takes a lighter-weight approach: it operates on the probability ratio $k = \pi_{\text{train}}(y_t;\theta_{\text{old}})/\pi_{\text{infer}}(y_t;\theta_{\text{old}})$ between the training and inference engines (same weights) for the sampled token, and applies a double-sided mask: $M(k) = k$ if $k \in [\alpha, \beta]$, otherwise $0$. Tokens outside the band contribute zero gradient; tokens inside the band retain the calibration coefficient. The default reasoning-RL band is $[\alpha, \beta] = [0.5, 5.0]$, and the paper reports that only about 1–2 per mille of training tokens get masked in this configuration. Routing flips are one cause of the ratio tail (especially in MoE), but the mask criterion is the probability ratio itself, not router top-k disagreement.

Industrial deployments confirm the severity of the problem. DeepSeek-V3.2 (§3.1) introduced Keep Routing — "preserve the expert routing paths used during sampling in the inference framework and enforce the same routing paths during training" — and reports the technique has been in their RL pipeline since DeepSeek-V3-0324. Note that Ring-1T itself does not use routing replay: instead, it freezes the MoE router bias during RL (§2.3.4) and relies on IcePop's per-token probability-ratio mask to catch the residual divergence. For MoE models, some form of routing alignment is not optional. Generic IS correction without routing awareness is insufficient.

Asynchronous training lag

Policy staleness from asynchronous pipelines is the most classical form of off-policy drift: the rollout policy is literally an older checkpoint. In one-step async (overlap generation of batch $N$ with training on batch $N-1$), the lag is exactly one gradient step — usually mild. In fully async pipelines like AReaL (Fu et al. 2025), the lag can accumulate across many training steps, producing importance ratios that are heavy-tailed and gradient estimates that are high-variance or biased.

But there is a subtler form of staleness that operates even within a single synchronous batch: minibatch staleness. When a rollout batch is split into multiple minibatches, the policy updates after each minibatch, so by the time the trainer reaches minibatch $k$, the current policy has moved $k-1$ gradient steps from the policy that generated the data. This happens in two common settings. Multi-step PPO uses a single epoch with multiple minibatches — off-policy from the second minibatch onward. Multi-epoch PPO makes multiple passes over the same batch — off-policy from the second epoch onward, and compounding with minibatch splits within each epoch. Both are the same mathematical problem as async staleness, just at a finer timescale. DeepSeek-V3.2 formalizes this with off-policy sequence masking: it computes the KL divergence between the current policy and the sampling policy for each sequence, and masks out negative-advantage sequences whose divergence exceeds a threshold. Critically, the divergence is computed using the sampling probabilities returned by the inference engine, so it captures both engine mismatch and minibatch staleness.

The distinction between these two forms of lag matters for the choice of correction. Classical async staleness is predictable: you know how many gradient steps have elapsed since the rollout, and you can bound the expected drift. Minibatch-reuse staleness is data-dependent: the drift depends on the gradient magnitude of the intervening updates, which depends on the advantage signal, which depends on the reward distribution. This is why per-sequence masking (as in DeepSeek-V3.2) or per-sequence adaptive clipping (as in QUATRO (Lee et al. 2026)) tends to outperform global staleness bounds: the amount of drift varies across sequences within the same batch.

Quantized inference as automatic off-policy

Quantized rollout deserves its own discussion because it is simultaneously the most common mismatch source and the one most often overlooked. When the rollout engine generates with INT8, FP8, or W8A8 (8-bit weights, 8-bit activations) kernels and the training engine computes gradients from full-precision log-probabilities, the pipeline is off-policy by construction. The quantized actor is not an approximation of the training-time policy. It is a different policy, one whose softmax outputs are shifted in structured ways by the quantization grid.

FlashRL (Yao et al. 2025) was among the first to frame this explicitly: 8-bit rollouts can dramatically cut generation cost, but the resulting mismatch must be treated as off-policy data, not ignored. FP8-RL (Qiu et al. 2026) provides a complete production stack — W8A8 inference components, FP8 KV cache, and explicit IS correction integrated into the verl ecosystem — and demonstrates that low-precision RL can match BF16 baselines, but only when the correction mechanism is part of the design from day one. Jet-RL (Xi et al. 2026) makes the stronger architectural claim: the right solution is a unified precision flow across rollout and learning, not post-hoc correction of a precision gap.

The tradeoff is throughput versus correction overhead. Quantized rollout with IS correction is typically faster than full-precision rollout without correction, because the throughput gain from quantization (1.5-2x) exceeds the cost of the extra forward pass needed for IS ratio computation. But the correction adds engineering complexity and monitoring burden: you need to track the quantization-induced mismatch separately from other mismatch sources to know whether your IS weights are doing useful work or just adding noise. QuRL (Li et al. 2026) proposes a cleaner abstraction: treat the quantized actor as a permanent, first-class behavior policy with its own trust region, rather than as a defective copy of the training policy that needs patching.

These sources compound. A typical setup with FP8 rollouts, BF16 training, TP=4 inference, TP=1 training, and 4 PPO epochs experiences the precision gap, parallelism mismatch, and multi-epoch staleness simultaneously. The rollout correction framework treats them uniformly through the importance weight decomposition above.

Identifying mismatch — a diagnostic toolkit

Before correcting mismatch, you need to measure it. verl provides a hierarchy of metrics for this, from per-token signals to aggregate health indicators. The off-policy metrics live in rollout_corr_helper.py; debug rollout-vs-actor probability metrics are in verl/utils/debug/metrics.py; gradient-variance proxies are in verl/trainer/ppo/metric_utils.py. Where this section names a verl function or preset, the claim is verified against verl-project/verl@70744059 (2026-05-18).

The per-token log-importance ratio

The basic signal is the per-token log importance ratio:

Here, $\pi_{\text{old}}$ denotes the log-probabilities recomputed by the training engine (the "actor forward pass"), and $\pi_{\text{rollout}}$ denotes the log-probabilities recorded during rollout generation. If these two quantities agree everywhere — if $\log\rho_t \approx 0$ for all tokens — then the policies match and no correction is needed.

To measure this signal without applying any correction, configure verl's rollout correction in metrics-only mode:

# Metrics only, no correction applied
rollout_corr_config = RolloutCorrectionConfig.disabled()

Or equivalently via YAML config (which is already the default in rollout_correction.yaml):

algorithm:
  rollout_correction:
    rollout_is: null
    rollout_rs: null

This logs all diagnostic metrics without modifying any gradients or masks, giving you a clean baseline measurement.

Off-policy diagnostic metrics

The raw per-token $\log\rho_t$ is useful but hard to aggregate. The metrics below (computed in compute_offpolicy_metrics() in rollout_corr_helper.py) summarize the distribution shift into scalars you can track on a dashboard.

KL divergence (k1 estimator). The simplest divergence measure:

This is the Monte Carlo estimate of $D_{\text{KL}}(\pi_{\text{rollout}} \| \pi_{\text{old}})$. Unbiased but can have high variance.

K3 KL. A lower-variance alternative from Schulman (2020):

where $\rho_t = \exp(\log\rho_t)$. Always non-negative, with better numerical properties than k1.

Chi-squared divergence. More sensitive to tail behavior. At the token level:

And at the sequence level, using the product of per-token ratios:

Chi-squared divergence directly relates to the variance of importance-weighted estimators. When $\chi^2_{\text{token}} \gt 1.0$, the IS weight distribution is heavy-tailed enough to substantially inflate gradient variance — for estimating a constant, the IS variance exceeds twice the on-policy level.

Perplexity gap and PPL ratio. The absolute and relative differences in perplexity between rollout and training engines:

A PPL ratio above 1.0 means the training engine is less confident than the rollout engine (higher perplexity); below 1.0 means the opposite. A ratio slightly above 1.0 is the expected default even when both engines implement the same policy, because $\text{PPL}_\text{rollout}$ on its own samples measures the entropy $H(\pi_\text{rollout})$, while $\text{PPL}_\text{old}$ on those same samples measures the cross-entropy $H(\pi_\text{rollout}, \pi_\text{old}) = H(\pi_\text{rollout}) + D_\text{KL}(\pi_\text{rollout} \| \pi_\text{old})$. By Gibbs' inequality the KL term is non-negative, so any numerical discrepancy between the two engines pushes the ratio above 1.0. A ratio substantially above 1.0, or any ratio below 1.0, therefore signals a systematic gap (precision differences, kernel divergence, TP mismatch, or routing discrepancies) beyond the baseline information-theoretic floor.

IS weight health and rejection sampling metrics

Once you enable IS or RS correction, additional metrics (from compute_is_metrics() in rollout_corr_helper.py) track the health of the correction itself.

Effective Sample Size (ESS). The fraction of samples carrying meaningful weight:

where $\tilde{w}$ denotes the normalized IS weights (mean 1). An ESS of 0.5 means your effective batch size is half the actual batch size. Below 0.3, most of your compute is wasted on samples that contribute almost nothing.

IS weight statistics. Mean, standard deviation, min, and max of the IS weights. Healthy weights have mean near 1.0 and low std. A mean far from 1.0 suggests systematic bias; high std means a few samples dominate.

Truncation fractions. The fraction of IS weights clipped by the truncation threshold. High truncation fractions mean severe mismatch that IS alone cannot handle.

RS masked fraction. For each rejection sampling option (and overall), the fraction of sequences whose response masks were zeroed out. This tells you how much data you are throwing away.

Debug probability metrics

For deeper investigation, verl provides debug metrics (in verl/utils/debug/metrics.py) comparing raw probability distributions, all logged under the training/ namespace:

• training/rollout_actor_probs_pearson_corr: Pearson correlation between rollout and actor probabilities (exponentiated log-probs) across all tokens. On-policy training should see correlations above 0.99. Below 0.95 means something is seriously off in the backend.
• training/rollout_probs_diff_mean and training/rollout_probs_diff_max: mean and maximum absolute difference in probabilities. The max catches rare but extreme discrepancies.

Gradient variance proxy metrics

When IS weights are applied, they can inflate gradient variance beyond what the advantage signal warrants. verl tracks this via compute_variance_proxy_metrics() in metric_utils.py:

• proxy1_signal_strength: $\lVert\bar{g}\rVert^2$ (squared mean gradient) — measures gradient signal strength.
• proxy2_total_power: $\mathbb{E}[\lVert\hat{g}\rVert^2]$ — total power (signal + noise).
• proxy3_pure_noise: $\frac{1}{N-1}(\text{proxy2} - \text{proxy1})$ — pure variance in gradient estimates.

If proxy3_pure_noise inflates over training, IS correction is adding unacceptable noise and you should tighten the IS threshold or switch to RS.

Training-time loss diagnostics

Standard PPO health metrics are logged even without rollout correction and interact with off-policy drift:

• actor/pg_clipfrac: fraction of tokens where PPO clipping activated. Rising clip fraction across epochs in multi-epoch PPO signals excessive divergence from the sampling policy.
• critic/vf_explained_var: $1 - \text{Var}(G - V) / \text{Var}(G)$. Low explained variance means a stale value function that is no longer tracking actual returns.

The SGA bias-variance framework

The clearest analytic framework for off-policy drift in LLM RL comes from the When Speed Kills Stability blog series (Liu, Li et al. Sep 2025). The usual question, "is this pipeline on-policy or off-policy?", turns out to be less useful than a more precise one: "what is the bias-variance profile of the gradient estimator under the actual distributional drift?"

The framework starts from the observation that policy gradient methods in LLM RL are instances of stochastic gradient ascent (SGA) on the expected reward $J(\theta) = \mathbb{E}_{\pi_\theta}[R(y)]$. When data is sampled from a behavior policy $\mu \neq \pi_\theta$, the gradient estimator becomes an importance-weighted surrogate. The key question is what happens at the token level versus the sequence level, because LLM RL computes token-level losses but assigns sequence-level rewards.

Token-level corrections optimize a surrogate. Standard practice in PPO and GRPO is to correct each token's contribution independently: the per-token ratio $\rho_t = \pi_\theta(y_t \mid x, y_{\lt t}) / \mu(y_t \mid x, y_{\lt t})$ appears in the clipped objective or as an IS weight. But this token-level correction optimizes a surrogate objective that differs from the true sequence-level expected reward. The surrogate is exact only when $\mu = \pi_\theta$; under distributional drift, it incurs a bias that grows with both the sequence length $T$ and the magnitude of the drift. Intuitively, the token-level ratio corrects each conditional $\pi_\theta(y_t \mid y_{\lt t})$ but does not account for the fact that the prefix $y_{\lt t}$ was also sampled from the wrong distribution. The accumulated prefix error is the bias.

Exact sequence-level IS is unbiased but exponentially variable. The theoretically correct correction uses the full sequence-level importance ratio:

This yields an unbiased estimator of the on-policy gradient, regardless of how different $\mu$ and $\pi_\theta$ are. The problem is variance: the product of $T$ per-token ratios can be astronomically large or small. Even with moderate per-token ratios (say, each in $[0.9, 1.1]$), a 500-token sequence can produce a sequence ratio anywhere from $\sim 10^{-23}$ to $\sim 10^{21}$. The resulting gradient estimate has variance that grows exponentially with horizon, making it useless in practice without aggressive truncation — which reintroduces bias.

Practical methods live on a bias-variance continuum. The SGA framework places every off-policy correction method on a spectrum:

• At one extreme, pure token-level correction (PPO-clip, token-level TIS) has low variance but bias that grows with drift and length.
• At the other extreme, exact sequence-level IS has zero bias but variance that can make learning impossible.
• In between, methods like geometric-mean ratios (GSPO, GMPO), prefix ratios (MinPRO), turn-level IS, and second-moment-constrained objectives (M2PO, R$^2$VPO) trade bias for variance in different ways.

The framework's concrete contribution to diagnostics is identifying the right divergence measures for each failure mode:

• Total variation (TV) distance between $\mu$ and $\pi_\theta$ bounds the bias of token-level surrogates. When $\text{TV}(\mu, \pi_\theta)$ is small, the surrogate is close to the true objective, and token-level corrections are safe.
• Chi-squared divergence $\chi^2(\pi_\theta \| \mu) = \mathbb{E}_\mu[(\pi_\theta / \mu)^2] - 1$ bounds the variance of importance-weighted estimators. When $\chi^2$ is large, even unbiased sequence-level corrections are too noisy to be useful.

A healthy off-policy pipeline needs both metrics to be moderate. High TV means the bias of cheap corrections is large. High $\chi^2$ means the variance of exact corrections is large. When both are high, no simple correction works well, and the pipeline needs either tighter synchronization or a fundamentally different objective.

For practitioners, the SGA framework suggests a concrete diagnostic protocol: track both $\text{TV}$ (or its proxy, the per-token KL) and $\chi^2$ (at both token and sequence level) across training. When TV is low and $\chi^2$ is low, you are effectively on-policy and no correction is needed. When TV is moderate and $\chi^2$ is low, token-level IS or geometric-mean RS suffices. When TV is low but $\chi^2$ is high (rare, but possible with long sequences and mild per-token drift), sequence-level truncation or masking is the right tool. When both are high, the mismatch is too severe for algorithmic correction alone — you need to reduce the system-level gap (precision alignment, weight synchronization, routing replay) before the algorithm can do its job.

Correcting mismatch — the rollout correction framework

Architecture overview

verl's rollout correction has two orthogonal tools that can be combined:

• Importance sampling (IS) weights: reweight each sample's gradient contribution by the likelihood ratio between training and rollout policies. Makes the gradient unbiased (up to truncation) at the cost of higher variance.
• Rejection sampling (RS) masks: zero out the response masks of sequences where the divergence exceeds a threshold. Removes high-divergence samples entirely, reducing variance but discarding data.

Configuration lives in RolloutCorrectionConfig (in verl/trainer/config/algorithm.py). The framework has two entry points: compute_rollout_correction_and_rejection_mask() is the atomic function that returns IS weights and the rejection mask; compute_rollout_correction_and_add_to_batch() is the batch-pipeline wrapper called from ray_trainer.py that attaches both to the batch. Both live in rollout_corr_helper.py.

Importance sampling correction

IS correction reweights the gradient so that training on off-policy data approximates the on-policy gradient. Two aggregation levels are available:

Token-level truncated IS (Token TIS). Each token gets its own weight, clipped to prevent extreme values:

where $\rho_t = \exp(\log\pi_{\text{old}} - \log\pi_{\text{rollout}})$ and $C$ is the truncation threshold (typical range: 1.5–5.0). Biased (clipping introduces bias) but low variance, so it works well as a default.

Sequence-level truncated IS (Sequence TIS). The entire sequence gets a single weight based on the product of token ratios:

with a typical threshold range of 2.0–10.0. Before truncation, this is closer to the unbiased sequence-level estimator than token TIS. In practice, however, the product of ratios is almost always truncated for long sequences (see the length trap below), and aggressive truncation reintroduces bias — potentially more than token TIS for long outputs. The main advantage of sequence TIS is that it preserves the correct correction unit (the sequence) at the cost of higher variance.

All computations happen in log-space with a safety bound of $\pm 20$ (since $e^{20} \approx 4.85 \times 10^8$) to prevent numerical overflow. After truncation, IS weights are optionally batch-normalized to mean 1.0, which prevents the effective learning rate from drifting.

To enable token-level IS correction:

# Enable token-level IS correction
rollout_corr_config = RolloutCorrectionConfig.decoupled_token_is()

Rejection sampling correction

RS correction identifies and excludes sequences where the training and rollout policies disagree too strongly. Three KL estimators are available:

K1 (ratio-based). Computes the raw log-ratio at each token:

With K1, thresholds are specified as ratio bounds (lower and upper), rejecting sequences where the ratio falls outside $[\ell, u]$.

K2 (MSE-based). A squared divergence measure:

K2 is always non-negative. The threshold is an upper bound on the divergence.

K3 (low-variance KL). Schulman's estimator:

Also always non-negative, with lower variance than K1.

These token-level divergences can be aggregated to the sequence level in four ways, yielding 11 total modes:

The geometric-mean modes (seq_mean_*) solve the length trap (detailed in the failure modes section below). With sequence-sum aggregation, long sequences accumulate higher total divergence even if per-token divergence is small, so they get preferentially rejected. For chain-of-thought (CoT) and agent tasks where response lengths vary dramatically, this creates a systematic bias against long (and often correct) reasoning traces. Geometric-mean normalization divides by sequence length, making the criterion length-invariant (Li & Liu 2025). See also kalomaze (2026) for a broader argument against length-dependent rollout exclusion.

Bypass vs decoupled mode

The framework operates in one of two modes, differing in how many distinct policies are involved.

Bypass mode (2 policies). Sets $\pi_{\text{old}} = \pi_{\text{rollout}}$, skipping the expensive actor forward pass that recomputes log-probabilities on the training engine. The IS ratio is computed directly between the current policy $\pi_\theta$ and the rollout policy. This is faster (one fewer forward pass per batch) but less accurate: it conflates training-inference mismatch with policy staleness into a single ratio. Implemented in apply_bypass_mode() in rollout_corr_helper.py.

Decoupled mode (3 policies). Recomputes $\pi_{\text{old}}$ on the training engine, giving three distinct sets of log-probabilities: $\pi_{\text{rollout}}$ (from the rollout engine), $\pi_{\text{old}}$ (from the training engine at the rollout weights), and $\pi_\theta$ (from the training engine at the current weights). This lets you measure and correct the two factors of the importance weight decomposition independently. More accurate, but costs an additional forward pass.

When to use which: Bypass when the mismatch is small (synchronized pipelines, same precision, or when speed matters more than correction accuracy). Decoupled when mismatch is large (async pipelines, precision gaps, multi-epoch training where you need precise staleness measurement).

The full set of configuration presets:

The two *_token_icepop presets implement IcePop-style double-sided masking: rollout_is_threshold is set to a lower_upper string like "0.5_5.0", which zeroes the IS weight (and therefore the token's gradient contribution) whenever the engine-mismatch ratio falls outside the band. They are a verl translation of the Ring-1T mechanism described above.

Policy loss functions as staleness mechanisms

Your choice of policy loss function determines how the training objective responds to the gap between the current policy and the sampling policy. Each loss implicitly handles (or fails to handle) off-policy drift through its trust region.

Vanilla PPO clip

The standard PPO clipped objective (Schulman et al. 2017) is the most widely used:

where $r_t(\theta) = \pi_\theta(y_t \mid x, y_{\lt t}) / \pi_{\text{old}}(y_t \mid x, y_{\lt t})$.

PPO clip works well on-policy but breaks down under distribution shift. It over-penalizes tokens where the old policy assigned low probability (small absolute changes produce large ratios) and under-penalizes tokens where the old policy assigned high probability (large absolute changes produce small ratios). Under off-policy drift, this asymmetry compounds: $r_t$ reflects both legitimate policy updates and spurious mismatch, and the clipping mechanism cannot tell them apart (Qi et al. 2026). verl's vanilla loss also implements dual-clip PPO (Ye et al. 2020) with a configurable lower bound clip_ratio_c (default 3.0) that prevents the ratio from dropping too far below 1 on negative-advantage tokens, adding a floor that standard PPO clip lacks.

DPPO-TV / DPPO-KL

DPPO (Qi et al. 2026) replaces ratio-based clipping with per-token divergence bounds:

• DPPO-TV: constrains total variation distance between $\pi_\theta$ and $\pi_{\text{old}}$ at each token position.
• DPPO-KL: constrains binary KL divergence at each token position.

These bound the actual distributional distance rather than relying on a ratio proxy, so the trust region size is invariant to the base probability level. More stable under distribution shift. Both use truncated IS (threshold 20.0) internally to stabilize the ratio computation.

GSPO — geometric sequence ratios

GSPO (Zheng et al. 2025) operates at the sequence level using the geometric mean of token ratios:

This geometric mean is then clipped and used in a PPO-style objective. Dividing by $T$ in the exponent prevents long sequences from producing extreme ratios, making GSPO naturally resistant to the length trap. The vanilla GSPO objective uses ordinary gradient flow through $\log\bar{r}$ — only the GSPO-token variant (§4.3 of the paper) introduces a stop-gradient on $\bar{r}$ so that gradients reach $\log\pi_\theta$ only through the per-token factors, which is what enables per-token credit assignment in long CoT.

SAPO — smooth objective shaping

SAPO (Gao et al. 2025) replaces PPO's hard clip with a smooth surrogate objective whose shaping function is

where $\tau$ is a temperature parameter. The function itself is monotone in $r$ and asymptotes at $4/\tau$ — it is the objective shape, not a gradient gate. The actual gradient weight on $\log\pi_\theta$ is its derivative, $g'(r,\tau) = 4\,\sigma(\tau(r-1))(1-\sigma(\tau(r-1)))$, which peaks at $r=1$ and decays smoothly on both sides. Separate temperatures $\tau_{\text{pos}}$ and $\tau_{\text{neg}}$ for positive and negative advantages give asymmetric trust regions. The smooth shape avoids the discontinuous gradients of PPO's hard clip while still bounding the effective step size near $r=1$.

GMPO — geometric mean clipping

GMPO (Zhao et al. 2025) applies sign-aware geometric-mean clipping at the sequence level. It clamps the per-token log-ratio $\log\pi_\theta - \log\pi_{\text{old}}$ within $[-\epsilon_{\text{low}}, \epsilon_{\text{high}}]$ based on the sign of the advantage, then exponentiates the sequence mean.

CISPO — stop-gradient clipping

CISPO (MiniMax 2025) applies a stop-gradient to the clipped ratio:

where $\hat{r}_{i,t}(\theta) = \text{clip}(r_{i,t}(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}})$ and $\text{sg}[\cdot]$ is the stop-gradient operator. Gradients flow only through $\log\pi_\theta$, not through the ratio. This preserves gradients for reflective reasoning tokens that PPO clip would suppress (because their ratios hit the clip boundary).

One caveat: the DPPO analysis shows CISPO lacks an explicit trust region. The stop-gradient removes the mechanism by which the clipped ratio constrains update size. Under growing mismatch, this can lead to progressively larger updates, which increase mismatch further, a positive feedback loop.

GPG (group policy gradient)

GPG (gpg) is a REINFORCE-style loss that applies the raw policy gradient $-\log\pi_\theta \cdot \hat{A}$ without ratio clipping, combined with group-normalized advantages (similar to GRPO). When rollout IS weights are provided, they are applied as explicit multipliers. GPG is useful as a simpler baseline that avoids the ratio-clipping artifacts of PPO while still benefiting from the rollout correction framework.

Covariance-based token filtering

verl also provides two covariance-based losses that take a different approach to staleness: instead of constraining the ratio or divergence, they identify and suppress individual tokens where the advantage estimate is unreliable.

Clip-cov (clip_cov) computes the covariance $\text{Cov}(\hat{A}, \log\pi)$ per token. Tokens with covariance in a specified range are randomly sampled and their loss contribution is zeroed out, reducing the influence of tokens where the advantage and log-probability are strongly coupled (a sign of noisy gradient signal under off-policy data).

KL-cov (kl_cov) takes a softer approach: instead of zeroing high-covariance tokens, it adds a KL penalty $\beta \cdot D_{\text{KL}}(\pi_{\theta_{\text{old}}} \| \pi_\theta)$ on those tokens, discouraging large policy updates where the gradient signal is unreliable. (The paper's eq. 14 uses formal KL; some implementations approximate it with $\beta|\log\pi_\theta - \log\pi_{\text{old}}|$ for efficiency.) Both clip_cov and kl_cov were introduced in Cui et al. 2025 (arXiv:2505.22617).

Bypass mode loss

The bypass mode loss (compute_policy_loss_bypass_mode() in core_algos.py) dispatches to either PPO-clip or REINFORCE while layering the full rollout correction framework:

• For REINFORCE: IS weights are applied explicitly as multipliers on the policy gradient.
• For PPO-clip: IS weights are not applied (in bypass mode, $r_t(\theta) = \pi_\theta / \pi_{\text{rollout}}$ already incorporates the full policy shift, so an additional IS multiplier would double-count). RS masks are applied in both cases.

This is verl's primary anti-staleness mechanism. It combines a policy loss with IS reweighting and RS filtering in one place, so correction applies regardless of which underlying loss you pick.

The broader algorithmic landscape

The previous sections covered the core rollout correction framework: IS and RS correction, trust-region losses, and IS-aware baselines. But these are pieces of a larger algorithmic conversation that has unfolded rapidly since mid-2025. Once the community recognized that most "on-policy" LLM RL is secretly off-policy, a wave of methods appeared that redesign how importance ratios are constructed, how trust regions are enforced, and whether off-policyness should be corrected at all or simply accepted as the default operating condition.

This section surveys those methods, organized by design philosophy rather than chronology.

An organizing principle that helps when reading the methods below: the correction unit must match the source of mismatch. Token-level clipping is the right tool when the mismatch is per-token numerical noise, but it is the wrong tool when the mismatch is prefix-state drift, a quantized rollout policy, KV-cache eviction, an async snapshot whose old logits were never saved, a routed MoE feedback path, or a speculative-decoding window. Each cell in the comparison table below picks a different unit at which behavior-policy provenance is preserved or reconstructed; the rest is implementation.

A comparison map of all methods

Before walking through the design families, here is a single dense view of every method this post discusses, aligned on the axes that distinguish them.

Reading the table. Three clusters dominate. Token-clip variants (PPO, ASPO, DISPO, R²VPO, CISPO, SAPO, KL-cov, clip-cov) keep the per-token loss structure and modify the trust region — the progression is hard-clip → soft-gate → divergence-bounded → second-moment-bounded → variance-aware → query-adaptive. Aggregation-shifters (TOPR, MinPRO, CTPO, GSPO, GMPO, VESPO, TRM, turn-level ST-PPO) carve out a bias-variance Pareto frontier for sequence-correct objectives at different aggregation units. Systems-aligned correctors (R3, IcePop, Keep Routing, Keep Sampling Mask, ESPO, TBIK, FP16/FP8/QuRL/QaRL/QeRL, A-3PO, ECHO-2, ROLL Flash, AReaL, Periodic Async, Seer, DORA) attack the engineering root cause and let standard PPO/GRPO ride on top.

A heuristic for picking a row: (i) if your pipeline is synchronous and dense, start with vanilla PPO-clip + decoupled_geo_rs and read the decision tree for any symptom you observe; (ii) if you are async, layer ECHO-2-style staleness bounds and consider VCPO's ESS-driven LR scaling; (iii) if you are MoE, routing alignment is non-negotiable — pick IcePop (lightweight masking) or Keep Routing (production-grade) before tuning the loss; (iv) if you are quantized, treat the quantization gap as a first-class behavior policy (QuRL/QaRL) rather than a defect to be patched.

Of the nearly 50 methods in the table, six are the practical defaults a 2026 practitioner should know by heart: PPO-clip, GSPO, M2PO, an MoE routing aligner (IcePop or Keep Routing), FP8-RL, and VCPO. The rest are mapped here so you can locate them when you encounter them in the literature, not so you must learn all of them.

Importance sampling design

Standard IS applies a single ratio $\rho_t = \pi_\theta(y_t \mid x, y_{\lt t}) / \pi_{\text{old}}(y_t \mid x, y_{\lt t})$ symmetrically to all tokens regardless of advantage sign, aggregation level, or causal structure. The methods below challenge each of those defaults.

Asymmetric trajectory-level treatment. The earliest departure from symmetric IS in this wave is TOPR (Tapered Off-Policy REINFORCE, Le Roux et al. 2025), which operates at the trajectory level: each complete response gets a single importance weight $\pi(\tau)/\mu(\tau)$, and the asymmetry is applied to the trajectory-level ratio based on the sign of the trajectory reward. For positive-reward trajectories, TOPR uses an SFT-style gradient (no importance weighting), since a large ratio on a good trajectory reflects desirable policy improvement. For negative-reward trajectories, it applies truncated importance weighting to prevent over-suppression: without truncation, the gradient pushes probability mass away from bad trajectories faster than the reward signal warrants. This trajectory-level asymmetric treatment foreshadows the sign-dependent clipping that reappears in several later methods.

ASPO (Wang et al. 2025) makes the asymmetry more explicit. For positive-advantage tokens, ASPO "flips" the ratio so that the correction favors exploration of beneficial actions rather than conservative clamping. For negative-advantage tokens, it retains standard clipping. The intuition is that PPO's symmetric clip suppresses both upside and downside equally, but under off-policy drift the downside suppression matters more (it prevents collapse), while the upside suppression just wastes signal. ASPO keeps the safety guardrail on the negative side while removing it on the positive side. This matters for mismatch because off-policy drift inflates ratios in both directions, and symmetric treatment discards gradient signal that is actually informative.

Prefix-level correction. A deeper question is whether the token-level ratio is even the correct correction object. MinPRO (Lei et al. 2026) argues it is not. In autoregressive generation, the probability of token $t$ depends on all preceding tokens. The causally correct importance weight for correcting the distribution at position $t$ is therefore the prefix ratio $\prod_{s=1}^{t} \rho_s$, not the token ratio $\rho_t$ alone. But cumulative products of ratios explode with sequence length (the length trap from earlier). MinPRO proposes a non-cumulative surrogate: instead of the full product, it tracks the minimum token-level ratio over the strictly preceding prefix, then reintroduces the current token's ratio as a separate factor:

(with the convention $\bar{\rho}_1 \triangleq 1$ for the empty preceding prefix.)

When all per-token ratios exceed 1 (the common case under policy improvement), $\bar{\rho}_t$ underestimates the true prefix product and is therefore conservative. When some ratios are below 1, $\bar{\rho}_t$ can overestimate the shrinking product, but the overall effect is still stabilizing: it avoids the exponential blowup of the full product while preserving the causal structure that pure token-level IS ignores. This matters most in long-horizon tasks where token-level IS accumulates systematic bias across hundreds of tokens because it fails to account for the distributional shift introduced at earlier positions. A concurrent paper, CTPO (Zhang et al. 2026), formalizes the same intuition differently: it uses the full cumulative prefix ratio $\rho_{1:t} = \prod_{s\le t}\rho_s$ but clips in log-space with bounds scaling as $\sqrt{t}$, giving a position-adaptive trust region that preserves unbiasedness in expectation.

Turn-level correction for agentic settings. When the "response" spans multiple dialogue turns with tool calls and environment feedback, even prefix-level correction operates at the wrong granularity. Li et al. 2025 argue that in multi-turn agentic RL, the natural correction unit is the turn, not the token: each turn is a coherent action whose importance weight should reflect the policy's probability of that entire action given the conversation history. They construct turn-level ratios and pair them with a clipping-triggered normalization scheme that re-normalizes weights within each turn when clipping activates, preventing the bias that standard PPO clipping introduces in long-horizon settings. This is important because agentic tasks are where staleness hits hardest: environment latency means rollouts are old by the time the learner processes them, and the trajectory length means token-level corrections compound their errors.

Clipping, masking, and trust-region methods

Standard PPO clips the ratio $r_t(\theta)$ to $[1-\epsilon, 1+\epsilon]$. This is a blunt instrument: it applies the same trust region to every token in every sequence for every prompt, and it only constrains a ratio proxy rather than the actual distributional divergence. The methods below refine each of these dimensions.

Second-moment trust regions. M2PO (Zheng et al. 2025) starts from an empirical observation: stale data can help early in training (the "prosperity" phase), because the diversity of off-policy samples provides useful exploration. But as the policy moves further from the behavior policy, the second moment of the log importance ratio grows, the chi-squared divergence between policies increases, and training collapses. M2PO replaces PPO's first-moment clip with a second-moment trust constraint on the log-ratio:

This bounds the Pearson chi-squared divergence between the current and behavior policies (the paper shows $\chi^2(\pi_\theta \| \pi_{\text{behav}}) \leq R^2 \cdot M_2$ where $M_2 = \mathbb{E}[(\log \rho)^2]$ and $R$ is the assumed sup-norm bound on $|\log \rho|$), providing a tighter control on distributional shift than bounding the ratio itself. In practice, M2PO tolerates larger mean ratio deviations (which carry gradient signal) while strictly controlling the tail (which carries noise). The "prosperity before collapse" framing is itself a diagnostic contribution: if you see improving metrics under stale data, do not assume stability. Watch the second moment of the log-ratio.

Sequence-validity screening. Trust Region Masking (TRM, Li et al. 2025) takes a different approach: rather than clipping individual token ratios, it evaluates whether a sequence is trustworthy as a whole using the maximum per-token KL divergence. A sequence is accepted only if $\max_t D_{\text{KL}}(c_t) \leq \delta$; otherwise it is masked entirely. This criterion is explicitly length-invariant (unlike ratio-based bounds, which accumulate with sequence length) and converts local divergence checks to a sequence-level admission test. TRM is most relevant for long-horizon tasks (code generation, multi-step reasoning, agentic trajectories) where a single high-divergence token can corrupt the gradient for the entire sequence, yet token-level clipping only addresses that one position while leaving the rest of the sequence's gradient intact. By masking entire sequences with trust-region violations, TRM trades data efficiency for gradient cleanliness.

Query-adaptive trust regions. QUATRO (Lee et al. 2026) observes that different prompts have different drift sensitivity. A math problem with a unique solution path is fragile: small policy changes can flip the answer. An open-ended creative prompt is robust: the policy can shift substantially without degrading output quality. A globally fixed $\epsilon$ treats both identically, which is either too conservative for robust prompts (wasting gradient signal) or too permissive for fragile ones (allowing collapse). QUATRO solves the trust-region-constrained problem per query, deriving a per-query Lagrange multiplier $\lambda_q^*$ from each prompt's reward-distribution variance. Queries with diffuse, high-variance rewards get larger $\lambda_q^*$ (more conservative updates); queries with concentrated rewards get smaller $\lambda_q^*$ (sharper updates). The trust-region radius itself remains a global hyperparameter; what is adapted per-query is the multiplier on the constraint. This is especially relevant under non-uniform staleness, where some prompts in a batch may be far more off-policy than others.

Industrial adoption: DeepSeek-V3.2's stack. The DeepSeek-V3.2 technical report (arXiv:2512.02556, §3.1) consolidates four explicit training-inference mismatch techniques into one production GRPO recipe. Keep Sampling Mask records the top-p / top-k truncation mask used by the inference engine during sampling and re-applies the same mask to the training engine's distribution during loss computation, so the two policies share an identical action subspace and importance weights are well-defined. It is not a probability-comparison or KL-divergence test. The companion off-policy sequence masking (Eq. 8–9) is the divergence-based gate: it computes per-sequence KL between the sampling distribution and the current policy using the inference engine's returned probabilities, and masks only negative-advantage sequences whose divergence exceeds a threshold. Using inference-returned probabilities is the load-bearing detail: the same gate captures both engine mismatch and minibatch staleness. The third technique, Keep Routing, is discussed below under MoE alignment; the fourth, an unbiased KL estimator that IS-corrects Schulman's k3, is discussed under KL estimator variants.

Off-policy-native algorithms

The methods above all share a premise: off-policyness is a problem to be corrected or constrained. A parallel line of work in early 2026 takes the opposite stance: off-policyness is the natural operating condition for large-scale LLM RL, and the objective should be designed for it from the start rather than patched after the fact.

Variational sequence-level correction. VESPO (Shen et al. 2026) derives a sequence-level reshaping kernel from a variational objective. The standard approach to sequence-level IS uses the raw product $\prod_t \rho_t$, which has correct expectation but catastrophic variance. VESPO instead optimizes a variational bound that allows the reshaping kernel to trade off a small amount of bias for a large reduction in variance. The resulting kernel keeps the semantics of sequence-level correction (matching the unit of reward) while reducing the variance enough for practical async training, with reported stability under staleness up to 64× the synchronous baseline. VESPO parameterizes the kernel so that the bias is bounded and controllable, giving practitioners an explicit bias-variance knob rather than truncation heuristics.

ESS-guided variance control. VCPO (Huang et al. 2026) operationalizes the effective sample size (ESS) as a runtime control signal rather than just a diagnostic. When ESS drops (indicating that a few samples dominate the gradient), VCPO scales down the learning rate proportionally, preventing the optimizer from amplifying noisy gradients. It also derives a minimum-variance off-policy baseline (OPOB) that weights each sample's contribution by both its squared importance weight and its per-sample gradient norm:

This accounts for how much each sample actually influences the parameter update, not just its importance weight. The combination of ESS-guided step-size scaling and variance-optimal baselines makes VCPO one of the most complete "accept and control" approaches. Its diagnostic contribution is equally important: VCPO shows that ESS collapse and gradient-norm spikes are reliable indicators of impending training failure, giving practitioners an actionable early warning signal.

Optimal off-policy objectives. OAPL (Ritter et al. 2026) goes further and asks: if the inference policy is always lagged, what is the optimal objective for that setting? Rather than starting from an on-policy objective and adding IS corrections, OAPL derives a regression-based objective that minimizes the squared discrepancy between scaled log-probability ratios — taken against the lagged inference policy — and optimal advantage estimates:

This is qualitatively different from standard policy gradient objectives: instead of weighting gradients by advantages, OAPL regresses the log-ratio toward the target advantage. The denominator $\pi_{\text{inf}}$ is the lagged inference/rollout policy (using log-probabilities returned by the rollout engine directly), not the previous-step PPO snapshot. Critically, OAPL does not require a learned critic — $\hat{V}^*(x)$ is estimated from groups of rollouts under $\pi_{\text{inf}}$, the same way GRPO computes group-relative baselines. The practical tradeoff is dependence on enough grouped samples per prompt and on periodic buffer clearing when sync occurs, not on a critic head whose accuracy degrades with lag. The paper reports stable training with policy lags above 400 gradient updates.

Ratio-variance regularization. R$^2$VPO (Luo et al. 2026) replaces hard ratio constraints (clipping at a fixed threshold) with a variance-aware adaptive clip range. For each GRPO group, R$^2$VPO computes the group-level variance of the importance ratio $\rho$, then sets the clip range proportionally: high-variance groups receive tighter clips and low-variance groups receive looser ones. The motivation is that a fixed clip wastes gradient signal on groups whose ratios are concentrated and over-extends on groups whose ratios are dispersed; adapting the boundary to the local distribution recovers the wasted signal while still bounding the worst case. This is closely related to M2PO's second-moment view of the trust region but applied as an adaptive boundary rather than a global constraint.

Decoupled clipping for correct vs. incorrect responses. DISPO (Karaman et al. 2026) observes that the appropriate clip range depends on two signs jointly: the sign of the advantage (positive vs. negative) and the sign of the response label (correct vs. incorrect). The four combinations produce four distinct regimes, each with a different failure mode under symmetric clipping. DISPO uses four independently-tuned clip boundaries — one for each regime — so that, for example, the upper clip on correct responses can be wider (preserving informative positive updates) while the lower clip on incorrect responses is tighter (preventing over-suppression that collapses entropy). This is complementary to ASPO's ratio-flipping: ASPO changes the form of the ratio for positive advantages, while DISPO changes the bounds of the clip across four conditioned regimes.

Deliberate off-policy mixing via logit fusion

The methods above all treat off-policyness as an accident — a gap between the intended on-policy distribution and the actual data distribution, caused by system-level mismatch or async lag. A different line of work deliberately constructs an off-policy behavior policy by fusing a teacher model's distribution into the rollout, turning off-policyness into a feature rather than a bug.

Logit fusion as a behavior policy. Zhang et al. (2026) (code) propose blending teacher and student logits at every decoding step during rollout generation. At token position $i$, the fused logits are:

# Straight-through: backward gradient from `backward_score`,
# forward value from `forward_score`
kl = backward_score - backward_score.detach() + forward_score.detach()

Step N:   [---- train(N-1) ----] [sync]
          [---- gen(N) ---------------]
Step N+1: [---- train(N) ------] [sync]
          [---- gen(N+1) -------------]

rollout:
  free_cache_engine: false
  checkpoint_engine:
    backend: "nccl"

# Python API: select bypass mode preset
rollout_corr_config = RolloutCorrectionConfig.bypass_ppo_clip()  # or .bypass_pg_geo_rs() for RS

algorithm:
  rollout_correction: { rollout_is: null, rollout_rs: null }
actor:
  model: { dtype: float16 }
rollout: { dtype: float16 }

rollout_corr_config = RolloutCorrectionConfig.decoupled_geo_rs_seq_tis(
    is_threshold=2.0, rs_threshold="0.99_1.01"
)

rollout_corr_config = RolloutCorrectionConfig.decoupled_geo_rs(rs_threshold="0.99_1.01")

rollout_corr_config = RolloutCorrectionConfig.decoupled_token_icepop(
    threshold_lower=0.5, threshold=5.0
)

if mean_response_len / len_100_steps_ago > 1.20:
    optimizer.param_groups[0]["lr"] *= 0.5
    rollout_corr_config = RolloutCorrectionConfig.bypass_pg_geo_rs()

@article{liu2026offpolicydrift,
  title   = {Off-Policy Drift in LLM RL},
  author  = {Liu, Chris Yuhao},
  year    = {2026},
  month   = {March},
  url     = {https://chrisliu298.ai/posts/off-policy-drift-in-llm-rl/},
  note    = {Blog post; v2 updated May 2026}
}