The Other Staleness Problem in Asynchronous PPO

Abstract

Asynchronous reinforcement learning decouples trajectory generation from model optimization, allowing rollout workers to generate data while the trainer continuously updates the policy. This architecture is particularly attractive for large language models and long-horizon agentic tasks, where rollout latency can vary dramatically. However, asynchronous training introduces a fundamental distribution mismatch: trajectories collected by older policies are consumed by a continuously evolving trainer.

Most recent work focuses on correcting the actor gradient, using importance sampling, clipping, masking, or score centering [1, 5, 6, 7]. We argue that an equally important problem arises in the critic. A value model trained directly on returns from stale trajectories estimates the expected return of their behavior policies, not necessarily that of the current policy. Updating the critic more frequently does not eliminate this mismatch: a perfectly optimized critic can still converge to the wrong value function if its training targets correspond to older policies. At the same time, maintaining a separate critic for every historical actor checkpoint would be prohibitively expensive at the scale of modern LLM reinforcement learning.

We investigate an alternative: Version Centered Critic Correction (VCCC), a framework for training a single target-policy critic using trajectories generated by many different behavior-policy versions. Our approach relies on three observations:

1. Score centering removes the actor-gradient drift caused by inaccurate state-dependent baselines, but it does not correct the critic’s bootstrapping target.
2. The difference between the target-policy and behavior-policy Bellman operators can be expressed exactly as a conditional covariance involving importance ratios.
3. This identity yields a single-critic training target whose conditional expectation is independent of the behavior-policy version, provided the action probabilities are correctly logged and the usual support assumptions hold.

We derive the method, explain its connection to classical importance-weighted temporal-difference learning [3] and control-variate off-policy evaluation [4], and analyze variance reduction, clipped importance weights, the duality between actor and critic centering, and multi-step credit assignment. The framework provides a principled starting point for correcting both actor staleness and critic staleness, without maintaining a history of value models.

1. Background

1.1 Why asynchronous RL?

In conventional synchronous PPO [1], training proceeds in alternating phases:

1. Generate rollouts using the current policy.
2. Compute rewards and advantages.
3. Update the policy and value model.
4. Synchronize the updated policy to the rollout workers.

This design imposes a synchronization barrier between rollout generation and optimization. For long-horizon tasks, especially coding agents and tool-using language models, rollout durations vary substantially: some trajectories terminate after a handful of actions, while others require hundreds of tool interactions. Waiting for every rollout in a batch can leave expensive training resources idle.

Asynchronous RL removes this barrier. Rollout workers continuously produce trajectories, and the trainer consumes them as they become available. The result is higher hardware utilization, but an unavoidable consequence is that a trajectory may have been generated several optimizer steps before it is used.

Let the current training policy be

and let the behavior policy that generated a particular action be

where $i$ identifies a historical policy version or rollout configuration. In general, $\mu_i\neq\pi_\theta$, and this mismatch becomes more pronounced as the trainer advances while rollout workers continue generating data.

1.2 Actor-side corrections

For a sampled action $a_t$, the basic importance ratio is

Methods such as PPO clipping [1], truncated importance sampling (TIS), and masked importance sampling (MIS) limit the influence of stale actions. Recent work on Score Centering [6] additionally identifies a systematic drift in off-policy policy gradients and introduces an additive correction to remove it, while subsequent analysis shows that once drift is removed, optimization stability is governed primarily by the policy distance accumulated under staleness [7]. In the agentic regime, SAO (Single-Rollout Asynchronous Optimization) [5] combines single-rollout training with strict double-sided token-level importance-ratio masking and practical improvements to value-model training.

These methods provide important mechanisms for stabilizing actor optimization. But actor optimization is only half of an actor-critic algorithm.

1.3 Stale critic targets

Consider the discounted return from step $t$,

and the state-value function of a policy $\pi$:

The value of a state depends on the policy used to generate future actions. For the same state $s$, two policies can have very different values:

Imagine that a mathematical reasoning model has reached an intermediate proof state. An older model might successfully complete the proof with probability $0.3$, while the current model might succeed with probability $0.7$. For binary terminal rewards and $\gamma=1$:

Now suppose we train a critic on Monte Carlo returns from the old policy:

The population minimizer is

The critic can achieve zero estimation bias relative to its training distribution while remaining systematically wrong about the current policy. This is not merely an optimization-speed issue; it is a mismatch in the statistical target being learned.

In an asynchronous replay buffer containing multiple behavior policies, the situation is even more subtle. If the critic is not conditioned on the behavior-policy version, then the optimal Monte Carlo predictor is a conditional mixture of historical returns:

This mixture depends on the distribution of policy versions represented in the replay buffer at any given moment. As the training system evolves, that version distribution shifts continuously, and there is no reason for $V_{\mathrm{replay}}$ to equal the value function of the current trainer policy.

Increasing the critic update frequency can help optimization track the replay target more closely, but cannot guarantee that the replay target is the correct one. This distinction motivates our approach.

1.4 Problem formulation

We consider a finite-horizon Markov decision process

For finite-horizon problems, time is included in the state when necessary, and terminal states have value zero.

At training step $k$, the trainer maintains:

• A target actor $\pi_k \equiv \pi_{\theta_k}$ (we write $\pi_k$, $\pi_\theta$, or simply $\pi$ interchangeably when the step index is clear from context).
• A single trainable critic $V_\phi$.
• An optional detached or slowly updated critic target $\bar V$.
• A replay buffer containing trajectories from historical behavior policies $\mu_i$.

Each recorded transition contains

The logged quantity $\log\mu_i(a_t\mid s_t)$ is the log-probability assigned by the actual sampling distribution, including temperature scaling and any other sampling transformations. The behavior-policy version $i$ can also be logged for monitoring and diagnostics. Crucially, we do not require storing the parameters of $\mu_i$ or maintaining a separate value model associated with that version.

We want to train one critic satisfying

using transitions generated by many different $\mu_i$. The desired Bellman fixed point is

where the Bellman evaluation operator is

The challenge is estimating $\mathcal T_{\pi_k}V$ using actions drawn from $\mu_i$, potentially with a different behavior policy for each recorded transition. We assume that the environment transition kernel $P(s'\mid s,a)$ is policy-independent conditional on $(s,a)$, and that the target policy has support within each behavior policy:

These assumptions will be essential in the derivation that follows.

2. Score Centering and the Critic

Before correcting the critic itself, it is useful to understand precisely what Score Centering [6] accomplishes for the actor—and what it leaves unresolved.

2.1 The baseline-drift problem

Define the score function of the current policy:

Consider a state-dependent baseline $b(s)$ and a Monte Carlo return $G_t$ obtained from the behavior-policy continuation. An uncorrected off-policy actor update has conditional expectation

Define the behavior-policy mean score

Expanding the expectation gives

For a stationary behavior policy, $\mathbb E_\mu[G_t\mid s]=V^\mu(s)$, which yields

The second term is the baseline-induced drift. Under on-policy sampling, $\mathbb E_{\pi_\theta}[u_\theta(s,a)\mid s]=0$, so any state-only baseline drops out in expectation. Under off-policy sampling, however, $m_\mu(s)\neq 0$ in general. Thus, critic error becomes a systematic gradient bias rather than merely increasing variance.

Notice a subtle consequence of Equation (1): a critic that perfectly predicts the old value function $V^\mu$ eliminates this baseline drift, whereas a critic that predicts the current value function $V^{\pi_\theta}$ does not. This demonstrates that actor baseline correction and target-policy value estimation are conceptually distinct problems.

2.2 Removing baseline dependence

Let $w(s,a)$ be an action-dependent importance weight that is conditionally independent of the sampled future return given $(s,a)$. Common choices include $w(s,a)=1$, the exact ratio $w(s,a)=\frac{\pi_\theta(a\mid s)}{\mu(a\mid s)}$, or a clipped/masked version of this ratio.

Define the weighted score

and its behavior-policy conditional mean

The centered score is

By construction, $\mathbb E_\mu[\widetilde Z_w\mid s]=0$. Therefore,

We obtain the identity

Because Equation (2) holds for any state-only baseline $b(s)$, Score Centering removes the dependence of the expected actor update on the explicit baseline. The critic still serves as a variance-reduction baseline, but its state-value prediction error no longer creates the drift term in Equation (1).

2.3 What it does not solve

Equation (2) is a powerful invariance property, but it does not imply an unbiased current-policy gradient. If we plug in a complete stale rollout return $G_t^\mu$, that return still describes what happens when future actions follow $\mu$. Even with exact current-action importance weighting, the resulting update evaluates $Q^\mu(s,a)$ rather than $Q^{\pi_\theta}(s,a)$. Score Centering removes the baseline-drift component; it does not change the policy under which the future return was generated.

Moreover, in an actor-critic method using TD or GAE [2], the critic appears not only as a subtracted state baseline $-V_\phi(s_t)$, but also inside the bootstrap target:

Because $s_{t+1}$ depends on the sampled action $a_t$, the bootstrap term $\gamma V_\phi(s_{t+1})$ is action-dependent. Its errors cannot be canceled by subtracting a state-only baseline from the actor advantage. We therefore need a separate mechanism to make the critic’s Bellman target consistent with the current policy.

3. Version Centered Critic Correction

3.1 The Bellman covariance identity

Our derivation starts with an arbitrary frozen value function $\bar V$; it does not need to be an accurate value function of either policy. Define the one-step Bellman sample

For a fixed state $s$ and behavior policy $\mu$, the behavior-policy Bellman operator is

while the desired target-policy Bellman operator is

Define the exact action importance ratio

Under the support assumption, $\mathbb E_\mu[\rho\mid s]=1$. Because the environment transition kernel $P(s'\mid s,a)$ is identical under both policies, it cancels in the joint transition ratio $\frac{\pi(a\mid s)P(s'\mid s,a)}{\mu(a\mid s)P(s'\mid s,a)} = \rho(s,a)$, giving

Expanding the right-hand side yields

Because $\mathbb E_\mu[\rho\mid s]=1$, the second term is an exact conditional covariance:

Consequently,

Equation (3) decomposes the target-policy Bellman operator into two terms:

The covariance is positive when actions receiving higher probability under the target policy tend to produce higher one-step bootstrapped returns, and negative when the target policy shifts probability toward actions with lower expected returns. This provides a direct statistical interpretation of critic staleness: the Bellman target changes when the actor changes because the action-probability shift and the bootstrapped returns are correlated.

An uncorrected critic trained on old data ignores precisely this covariance term. Importantly, Equation (3) holds for any frozen $\bar V$ and does not require a critic tailored to the old policy—allowing us to avoid maintaining one critic per historical actor version.

We now turn Equation (3) into a practical training target.

3.2 A centered Bellman estimator

For a transition generated by behavior policy $\mu_i$, let

Choose any baseline $b(s)$ that does not depend on the sampled action, and define the Version Centered Critic Correction target:

The second term is a centered correction to the ordinary Bellman sample. Taking its conditional expectation under $\mu_i$ gives

The final term vanishes because $\mathbb E_{\mu_i}[\rho_i-1\mid s]=0$. Therefore,

The conditional expectation of the corrected target is independent of both the state baseline $b(s)$ and the behavior-policy version $i$. This is the central property of VCCC.

3.3 Why one critic is sufficient

Suppose a replay buffer contains transitions generated by historical policies $\mu_1,\mu_2,\ldots,\mu_N$. Each transition uses its own logged importance ratio $\rho_i=\frac{\pi_k(a\mid s)}{\mu_i(a\mid s)}$. Because

holds for every version $i$, taking the expectation over any replay mixture of behavior versions at state $s$ also gives

provided the corresponding conditional expectations exist and the logged probabilities are exact. The behavior-policy version distribution disappears completely from the conditional mean, allowing a single critic to learn from arbitrarily many behavior versions while targeting one Bellman operator:

where $\operatorname{sg}$ denotes stop-gradient.

Under standard tabular policy-evaluation conditions, repeated application of $\mathcal T_{\pi_k}$ converges to $V^{\pi_k}$ (via contraction for $\gamma<1$ or backward induction in finite-horizon MDPs). With function approximation and a moving actor, convergence requires additional stability conditions and cannot be inferred from conditional unbiasedness alone.

3.4 Connection to importance-weighted TD

Equation (4) simplifies cleanly when we choose $b(s)=\bar V(s)$:

Writing the one-step TD residual as $\delta_t^{\bar V} = r_t+\gamma\bar V(s_{t+1})-\bar V(s_t)$, we obtain

This is the classical one-step importance-weighted TD target [3]. The contribution of the centered formulation is to clarify why importance-weighted TD resolves critic staleness, how the subtraction of $\bar V(s_t)$ acts as a zero-mean control variate $(\rho_i-1)\bar V(s_t)$, and why a single value network suffices across arbitrarily many logged behavior-policy versions.

Consider an asynchronous rollout in which the worker receives updated weights mid-trajectory:

A Monte Carlo return from the start of the trajectory depends on this entire sequence of future behavior policies and is not a sample from the value function of any single stationary policy. By contrast, the one-step target in Equation (4) depends only on the current state and action, the observed reward and next state, the behavior probability of the current action, the target probability of that action, and the bootstrap value $\bar V(s_{t+1})$. Future behavior-policy versions are irrelevant to the conditional correctness of the one-step estimator, making it naturally suited to systems where rollouts span checkpoint updates.

4. Variance and Stabilization

4.1 The variance-minimizing baseline

Although the corrected target in Equation (4) is conditionally unbiased for any state-only $b(s)$, its variance can be substantial. Rewriting Equation (4) as

we can solve for the baseline $b^*(s)$ that minimizes $\operatorname{Var}_\mu(Y_b\mid s)$, or equivalently minimizes the second moment $\mathbb E_\mu[(\rho H-(\rho-1)b)^2\mid s]$. Differentiating with respect to $b$ gives the orthogonality condition

which yields

When the denominator $\operatorname{Var}_\mu(\rho\mid s)=\mathbb E_\mu[(\rho-1)^2\mid s]$ is zero, $\rho=1$ almost surely and the baseline has no effect. Using $\mathbb E_\mu[(\rho-1)\rho\mid s] = \mathbb E_\mu[(\rho-1)^2\mid s] = \operatorname{Var}_\mu(\rho\mid s)$ and $H = \bar V(s) + \delta^{\bar V}$, we can also decompose Equation (9) around $\bar V(s)$:

This decomposition shows why $b(s)=\bar V(s)$ is a natural default—and when it is suboptimal. Whenever the weighted TD residual $\rho\delta^{\bar V}$ remains strongly correlated with $\rho-1$ (for instance, early in training or under large policy shifts), $b^*(s)$ departs from $\bar V(s)$. This opens an interesting design space: the critic can estimate target-policy value $V^{\pi_k}$, while a lightweight auxiliary baseline head $b_\psi(s)$ can be trained to minimize the empirical variance of the Bellman target.

Consider a one-step bandit with two actions and deterministic rewards $r(a_0)=0$ and $r(a_1)=1$. Suppose the behavior and target policies assign

so that $V^\mu=0.1$ and $V^\pi=0.7$, with importance ratios $\rho(a_1)=7$ and $\rho(a_0)=\frac{1}{3}$. An uncorrected critic trained on behavior rewards converges to $0.1$.

For the corrected estimator $Y_b = H + (\rho-1)(H-b)$, three baseline choices illustrate the variance tradeoff. All have the exact mean $\mathbb E_\mu[Y_b]=0.7$:

1. Stale value baseline ($b = V^\mu = 0.1$): $(Y(a_1),Y(a_0))=(6.4,0.0667)$, with $\operatorname{Var}_\mu(Y_b)=3.61$.
2. Target value baseline ($b = V^\pi = 0.7$): $(Y(a_1),Y(a_0))=(2.8,0.4667)$, reducing the variance more than sevenfold to $0.49$.
3. Variance-minimizing baseline: Equation (9) gives $b^*=1.05$, for which $Y(a_1)=Y(a_0)=0.7$ and the variance is zero.

In general multi-action MDPs with stochastic transitions, a single state-dependent scalar $b(s)$ cannot drive the variance to zero, and rare actions with large importance ratios can still produce destructive critic updates. Controlling these tails requires explicit weight truncation.

4.2 Clipped weights and critic centering

When behavior and target policies are far apart—as in long-running asynchronous rollouts—unclipped importance ratios can exhibit heavy tails.

For the canonical baseline $b(s)=\bar V(s)$, replacing $\rho_t$ in Equation (8) with a clipped or masked weight $w_t = w(s_t,a_t)$, such as $w_t=\min(\rho_t,c)$, yields

Clipping bounds the influence of any single transition, at the cost of altering the Bellman fixed point. For a fixed behavior policy $\mu$, define the expected clipped weight

and the induced effective policy

Assuming $Z(s)>0$, the expected TD update satisfies

Thus, in the tabular fixed-policy setting, Equation (10) shares the same fixed point $V^{\widetilde\pi}$ as the Bellman operator of $\widetilde\pi$ (where $\widetilde\pi(a\mid s)\propto\min(c\mu(a\mid s),\pi(a\mid s))$ for one-sided clipping, matching V-trace [3]). With multiple behavior versions in replay, each $\mu_i$ induces its own $\widetilde\pi_i$, so clipping trades exact version-invariance for bounded variance.

A subtle point arises if we combine clipping with a general baseline $b(s)\neq \bar V(s)$. Because clipping makes $Z(s)=\mathbb E_\mu[w_t\mid s]\neq 1$ in general, naively substituting $w_t-1$ for $\rho_t-1$ in Equation (4) would yield

causing the arbitrary baseline $b(s)$ to leak directly into the Bellman fixed point whenever $b(s)\neq \bar V(s)$. To preserve the fixed point $V^{\widetilde\pi}$ for an arbitrary control-variate baseline $b(s)$, the clipped weight in the baseline control variate must be centered around its true behavior-policy mean $Z(s_t)=\mathbb E_\mu[w_t\mid s_t]$:

Because $\mathbb E_\mu[w_t - Z(s_t)\mid s_t=s]=0$, the control-variate term has zero conditional mean for any state-only $b(s)$, and reduces back to Equation (4) when $w_t=\rho_t$ ($Z(s_t)=1$) or to Equation (10) when $b(s_t)=\bar V(s_t)$. Notice the exact structural duality with Actor Score Centering in Equation (2):

Centering the critic weight by $\mathbb E_\mu[w\mid s]$ is precisely the scalar ($u_\theta\equiv 1$) counterpart of centering the actor’s weighted score by $m_w(s)=\mathbb E_\mu[w u_\theta\mid s]$.

4.3 Why not Monte Carlo returns?

A full trajectory return generated by an old policy reflects the old continuation distribution across all future steps. Correcting a Monte Carlo return to the current policy requires cumulative products of importance ratios, $\prod_{j=t}^{T-1}\rho_j$, whose variance grows exponentially with horizon length. One-step TD avoids cumulative products by bootstrapping from $\bar V(s_{t+1})$, at the cost of relying on the critic’s current accuracy. When rewards are sparse and terminal, pure one-step bootstrapping can propagate reward signals slowly and accumulate bias—the classic bias-variance tradeoff of temporal-difference learning.

4.4 Multi-step correction with V-trace

To balance reward propagation speed against cumulative importance-ratio variance, we can extend the one-step target to an $n$-step V-trace target [3]:

Here $\rho_k^{\,c}=\min(\bar\rho,\rho_k)$ controls the per-step Bellman correction and $c_j=\lambda\min(\bar c,\rho_j)$ (with $\lambda\in[0,1]$) controls eligibility trace propagation across steps. When the ratios are untruncated ($\bar\rho=\bar c=\infty$) and $\lambda=1$, Equation (11) recovers the exact multi-step target-policy Bellman backup; when the data are on-policy ($\rho_k\equiv 1$), it reduces directly to the standard $\mathrm{GAE}(\gamma,\lambda)$ value target [2] widely used in synchronous LLM PPO. Crucially, every per-step ratio $\rho_k$ is still computed from the single behavior probability logged when action $a_k$ was sampled, requiring no historical critic ensemble.

5. Actor-Critic Integration

We now have two complementary mechanisms addressing distinct sources of off-policy error:

• Actor Score Centering removes baseline-induced drift in the off-policy actor update.
• Version Centered Critic Correction aligns the critic’s Bellman target with the current policy.

These mechanisms naturally compose in an asynchronous actor-critic architecture.

5.1 A one-step actor-critic update

Suppose the critic has learned an approximation to $V^\pi$, and define the one-step TD advantage

With exact action-level importance weighting, the expected actor update at state $s$ is

If $V_\phi=V^\pi$, then $\mathbb E[r_t+\gamma V_\phi(s_{t+1})\mid s_t=s,a_t=a] = Q^\pi(s,a)$, and therefore

Hence,

Equation (12) recovers the exact per-state policy-gradient contribution of $\pi$. This is strictly stronger than eliminating baseline drift alone: a target-policy critic replaces the future return collected under the old policy with a bootstrap estimate of the current policy’s continuation value, directly addressing the future-policy mismatch that Score Centering leaves behind. (As noted in Section 7, it still does not reweight the off-policy state-visitation distribution, and both PPO clipping and critic approximation error introduce additional bias.)

5.2 Duality and information requirements

Let $w(s,a)$ be a token-level clipping or masking weight, such as truncated importance sampling or the double-sided masking used in SAO [5]. Defining the behavior-policy mean of the weighted score,

we can write the score-centered actor update at state $s$ as

For any state-only baseline inside $\hat A_t$, subtracting $m_w(s_t)$ eliminates baseline-induced drift, leaving only the bias from weight clipping and approximate bootstrap values.

Comparing Equation (13) with the clipped critic target in Section 4.2 reveals both a clean mathematical duality and a precise computational distinction:

1. Duality between actor and critic centering. Under clipped weights $w(s,a)$, the actor centers the vector-valued weighted score $w(s,a)u_\theta(s,a)$ by subtracting $m_w(s) = \mathbb E_\mu[w(s,a)u_\theta(s,a)\mid s]$, whereas a general-baseline critic centers the scalar weight $w(s,a)$ by subtracting $Z(s) = \mathbb E_\mu[w(s,a)\mid s]$. The critic’s centering constant $Z(s)$ is the exact $u_\theta \equiv 1$ reduction of the actor’s centering vector $m_w(s)$.
2. When a single logged log-probability suffices—and when full behavior-distribution statistics are needed. Because both $m_w(s)$ and $Z(s)$ are expectations under the behavior distribution $\mu(\cdot\mid s)$, neither can be evaluated under clipped weights from the sampled action’s probability alone. For LLMs, estimating either quantity requires summary statistics of the behavior distribution, such as top-$k$ sampler log-probabilities combined with a modeled tail [6].
   • Lightweight regime ($b = \bar V$ or unclipped $\rho$). When the critic uses the canonical TD baseline $b(s)=\bar V(s)$ (Equations 8 and 10), the control-variate term $(w_t - Z(s_t))(b(s_t) - \bar V(s_t))$ vanishes identically; similarly, when weights are unclipped ($w_t=\rho_t$), $Z(s)=1$ is known analytically. In both cases, the critic correction requires only the single logged log-probability $\log\mu_i(a_t\mid s_t)$ of the sampled action, and can be deployed even in pipelines that do not log top-$k$ distribution statistics for Actor Score Centering.
   • General control-variate regime ($b \neq \bar V$ with clipping). Once a learned control-variate baseline $b_\psi(s)\neq \bar V(s)$ is combined with clipped weights $w_t$, proper critic centering requires estimating $Z(s)=\mathbb E_\mu[w\mid s]$, which demands the exact same class of behavior-distribution information as $m_w(s)$ in Actor Score Centering. Conveniently, in a system that already logs top-$k$ sampler log-probabilities for Actor Score Centering, $Z(s)$ can be computed directly from those same logged statistics with negligible additional overhead.

6. Training Procedure

A practical implementation of VCCC maintains one trainable critic $V_\phi$ (and optionally a detached or slowly updated target snapshot $\bar V$), without maintaining critics indexed by historical actor versions.

Rollout collection

For every generated action token $a_t$, the rollout worker records the state prefix $s_t$, action $a_t$, reward $r_t$, next state $s_{t+1}$, actual sampling log-probability $\log\mu_{i_t}(a_t\mid s_t)$, and behavior-policy version $i_t$ (plus top-$k$ sampler log-probabilities if clipped Actor Score Centering or $Z(s)$-centered control variates are enabled).

In tool-using and agentic environments, external tool observations should be treated as environment state transitions rather than policy actions: importance ratios apply only to model-generated action tokens, not to environment-supplied observation tokens.

Critic update

At trainer step $k$, freeze the current actor snapshot $\pi_k$ for computing importance ratios. For each transition $t$ in a replay minibatch of size $B$:

1. Evaluate $\log\pi_k(a_t\mid s_t)$ under the current target actor and compute the importance ratio against the logged behavior policy $\mu_{i_t}$:

   $$\rho_t= \exp\left( \log\pi_k(a_t\mid s_t) - \log\mu_{i_t}(a_t\mid s_t) \right).$$

2. Evaluate the detached target values $\bar V(s_t)$ and $\bar V(s_{t+1})$, and form the TD residual:

   $$\delta_t= r_t+\gamma\bar V(s_{t+1})-\bar V(s_t).$$

3. Construct the corrected value target. In the one-step setting with $b=\bar V$ and optional clipping weight $w_t=\min(\rho_t,c)$,

   $$y_t= \bar V(s_t)+w_t\delta_t,$$

   or, for long-horizon trajectories with sparse rewards, compute the multi-step V-trace($\lambda$) target $v_t$ from Equation (11).

4. Update the critic by minimizing the mean squared error over the minibatch:

   $$\mathcal L_V(\phi)= \frac{1}{B} \sum_{t=1}^{B} \left( V_\phi(s_t)-\operatorname{sg}[y_t] \right)^2.$$

A detached target $\bar V$ evaluated before each optimization step suffices for the mathematical formulation; a slowly updated Polyak/EMA target network can further improve stability under function approximation, without requiring historical behavior-policy critics.

Actor update

Compute token-level advantages $\hat A_t$ using the updated target-policy critic $V_\phi$. For clipped PPO-like updates, apply the chosen actor weight $w_t^{\mathrm{actor}}$ and optionally center the weighted score by $m_w(s_t)$ as in Equation (13). Note that the actor and critic need not share identical clipping thresholds, since their bias-variance tradeoffs differ.

Why the procedure scales

In the standard $b=\bar V$ configuration, the extra overhead per transition is minimal:

• One logged behavior log-probability $\log\mu_{i_t}(a_t\mid s_t)$ per sampled action token.
• One current-policy log-probability $\log\pi_k(a_t\mid s_t)$ (already computed during the actor forward pass).
• Value evaluations at current and next states.

The system never reloads historical actor checkpoints or maintains an ensemble of version-specific critics. Every transition in the replay buffer—regardless of which policy version generated it—contributes to estimating the fixed point of the same current-policy Bellman operator $\mathcal T_{\pi_k}$.

7. Limitations

While the Bellman covariance identity guarantees conditional unbiasedness of the unclipped one-step target under standard assumptions, several practical challenges govern real-world performance.

When the current policy assigns substantial probability to actions rarely sampled by a stale behavior policy, importance ratios $\rho_t$ become heavy-tailed, yielding an unbiased estimator with prohibitive variance. Truncation controls this variance at the price of targeting the surrogate policy $\widetilde\pi$ rather than $\pi_k$. Consistent with recent empirical findings that large trainer-sampler KL distance degrades asynchronous RL even after actor drift is removed [7], practical implementations should monitor importance-ratio tails, effective sample size (ESS), and version-dependent clipping rates.

Action-level importance weighting corrects the conditional action distribution at states present in the replay buffer, but cannot synthesize observations of states that $\pi_k$ would visit and historical policies $\mu_i$ never reached. VCCC therefore does not eliminate state-visitation distribution shift or replace exploration; sufficient state coverage and critic generalization remain necessary.

The importance-sampling identity requires $\pi_k(a\mid s)>0 \implies \mu_i(a\mid s)>0$. Aggressive top-$k$, top-$p$ (nucleus), hard token masks, or greedy decoding during rollout collection can violate this support condition: if $\mu_i(a\mid s)=0$ while $\pi_k(a\mid s)>0$, the exact Bellman correction is unidentifiable from behavior samples alone. Furthermore, the logged probability $\mu_i(a_t\mid s_t)$ must reflect the exact sampling distribution (including temperature and any logits post-processing); logging raw, untransformed logits invalidates the importance ratio.

The conditional Bellman identity holds for a fixed target snapshot $\pi_k$ and frozen bootstrap function $\bar V$. In deep RL, however, $\pi_k$ evolves continuously and $V_\phi$ is a neural network trained via bootstrapped semi-gradients on off-policy replay—an instance of the classic “deadly triad.” Target-network update rates, critic update frequency, replay buffer staleness bounds, and actor learning rates remain vital for stable convergence.

In long chain-of-thought reasoning or multi-turn agent trajectories where nonzero reward arrives only at termination, pure one-step TD propagates credit slowly across thousands of tokens. Multi-step V-trace($\lambda$) targets (Section 4.4) accelerate propagation, but reintroduce products of truncated ratios across steps. Tuning the horizon and truncation thresholds $(\bar\rho, \bar c, \lambda)$ is a key empirical design choice.

Both clipped Actor Score Centering (via $m_w(s)$) and clipped general-baseline Critic Centering (via $Z(s)$) rely on expectations over the full action vocabulary under $\mu$. When approximated via top-$k$ logged probabilities and a tail model [6], residual approximation error remains and should be tracked separately from Bellman target error.

8. Related Work

VCCC synthesizes ideas from several lines of reinforcement learning research:

• PPO and GAE [1, 2]. PPO establishes the clipped actor-critic framework, while GAE formalizes the bias-variance tradeoff in multi-step TD advantage estimation.
• IMPALA and V-trace [3]. IMPALA directly addresses policy lag between distributed actors and learners using importance-weighted TD residuals and truncated trace coefficients. Our one-step target with $b(s)=\bar V(s)$ recovers importance-weighted TD, and Section 4.4 adopts V-trace for multi-step credit assignment.
• Score Centering and Off-Policy Drift [6, 7]. Marek and Ryabinin [6] show that sampler-trainer mismatch induces a persistent actor-gradient drift proportional to $m_\mu(s)$ and eliminate it via conditional score centering. Liu, Fang, and Agarwal [7] further decompose asynchronous off-policy error into drift and trainer-sampler policy distance. Our analysis applies the same conditional-centering lens to the critic, showing why actor score centering does not fix stale bootstrap targets and establishing the exact duality between actor score centering ($m_w(s)=\mathbb E_\mu[wu_\theta\mid s]$) and clipped critic weight centering ($Z(s)=\mathbb E_\mu[w\mid s]$).
• SAO [5]. Single-Rollout Asynchronous Optimization highlights the central role of value models in asynchronous agentic LLM RL when group-wise GRPO baselines are unavailable, introducing double-sided token-level ratio masking and frequent critic updates. VCCC complements SAO by ensuring that the critic’s statistical target itself is aligned with the current policy rather than the historical replay mixture.
• Control Variates and Doubly Robust Evaluation [4]. Off-policy evaluation methods use value predictions as control variates to reduce importance-sampling variance. The centered estimator in Equation (4) is a state-dependent control-variate formulation of importance-weighted TD (distinct from full action-dependent doubly robust estimators that require an action-value model $\hat Q(s,a)$).

Our focus is integrating these principles into a version-agnostic training design that trains a single target-policy critic across arbitrarily many historical behavior policies.

9. Conclusion

Asynchronous reinforcement learning introduces two distinct forms of policy staleness:

1. Actor staleness: policy gradients are evaluated on actions sampled from an older behavior policy.
2. Critic staleness: value targets are constructed from trajectories whose future continuation follows an older behavior policy.

These two problems interact, but they are not interchangeable. Score Centering removes baseline-induced actor-gradient drift, yet cannot make a critic trained on stale returns estimate the current policy’s value function. Meanwhile, maintaining a separate critic for every historical actor checkpoint is neither scalable nor aligned with estimating current-policy advantages.

We therefore formulate Version Centered Critic Correction (VCCC) around a single principle: every stale transition should contribute to learning the same current-policy Bellman operator, regardless of which historical actor generated it.

In its canonical $b(s)=\bar V(s)$ form, the exact one-step target is

whose conditional expectation under any valid behavior version $\mu_i$ is

This target is invariant to the historical behavior-policy version, requires only the sampled action’s logged probability when $b=\bar V$, and exhibits an exact mathematical duality with Actor Score Centering when extended to general baselines and clipped weights. The remaining challenges are practical rather than algebraic: controlling importance-ratio variance via V-trace truncation, maintaining state coverage, and stabilizing bootstrapping as the target policy evolves. Combining Score-Centered Actor Optimization with Policy-Consistent Critic Learning provides a unified foundation for addressing both halves of the asynchronous actor-critic problem.

References

1. Schulman et al. (2017). Proximal Policy Optimization Algorithms ↗.
2. Schulman et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation ↗.
3. Espeholt et al. (2018). IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures ↗.
4. Jiang and Li (2016). Doubly Robust Off-policy Value Evaluation for Reinforcement Learning ↗.
5. Hou et al. (2026). Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning ↗.
6. Marek and Ryabinin (2026). Score Centering Stabilizes Off-policy Reinforcement Learning ↗.
7. Liu, Fang, and Agarwal (2026). Drift and Distance: When Is Score Centering Enough in Async RL? ↗.

Citation

If you cite this post, please use the following reference:

Hanning Wang. “The Other Staleness Problem in Asynchronous PPO.” TheUnknownBlog, November 11, 2026. https://theunknownth.ing/blog/vccc ↗.

@misc{theunknownthing2026staleness,
  author = {Hanning Wang},
  title = {The Other Staleness Problem in Asynchronous PPO},
  year = {2026},
  month = nov,
  url = {https://theunknownth.ing/blog/vccc},
  note = {TheUnknownBlog, blog post}
}