Ahmed Haroon

From Baselines to Critics

The policy-gradient estimator weights each action's score by its expected future return:

\nabla_\theta J(\theta) =\mathbb E_{\pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta\log\pi_\theta(a_t\mid s_t) Q^{\pi_\theta}(s_t,a_t) \right].

The true $Q^{\pi_\theta}$ is unknown. REINFORCE replaces it with the sampled return

G_t=\sum_{k=0}^{T-t-1}\gamma^k r_{t+k}.

This is unbiased, but each $G_t$ represents only one possible future. Actor-critic methods learn a value estimator that can average across many futures.

An actor represents the policy

a_t\sim\pi_\theta(\cdot\mid s_t),

while a critic estimates values such as

V_\phi(s)\approx V^\pi(s)

Q_\phi(s,a)\approx Q^\pi(s,a).

The critic supplies the return or advantage estimate used to update the actor.

The value-function baseline

We can subtract any state-dependent, action-independent baseline without changing the expected policy gradient. The natural choice is

V^\pi(s) =\mathbb E_{a\sim\pi(\cdot\mid s)}[Q^\pi(s,a)].

This centers action values around their policy-weighted average:

A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s).

Therefore,

\boxed{ \nabla_\theta J(\theta) =\mathbb E \left[ \sum_t \nabla_\theta\log\pi_\theta(a_t\mid s_t) A^\pi(s_t,a_t) \right] }.

Positive advantage means the action is better than the policy's average action at that state. Negative advantage means it is worse.

See derivation

Condition on a state $s$ :

\begin{aligned} &\mathbb E_{a\sim\pi_\theta(\cdot\mid s)} \left[ \nabla_\theta\log\pi_\theta(a\mid s)b(s) \right]\\ &=b(s)\sum_a \pi_\theta(a\mid s) \nabla_\theta\log\pi_\theta(a\mid s)\\ &=b(s)\sum_a\nabla_\theta\pi_\theta(a\mid s)\\ &=b(s)\nabla_\theta\sum_a\pi_\theta(a\mid s)\\ &=b(s)\nabla_\theta1\\ &=0. \end{aligned}

The baseline changes variance, not the expected gradient.

A baseline is not always a critic

With

\widehat A_t^{\mathrm{MC}} =G_t-V_\phi(s_t),

the value function is only a baseline. Even if $V_\phi$ is inaccurate, the actor gradient remains unbiased because the return estimate is still Monte Carlo.

With

\widehat A_t^{(1)} =r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t),

the learned value appears inside the return estimate. It is now a bootstrapped critic, and its approximation error can bias the actor update.

Suppose

V_\phi(s)=V^\pi(s)+e(s).

Then

\begin{aligned} \mathbb E[\widehat A_t^{(1)}\mid s_t=s,a_t=a] &=A^\pi(s,a) +\gamma\mathbb E_{s'}[e(s')] -e(s). \end{aligned}

The $-e(s)$ term is state-only and behaves like a baseline. The next-state error can depend on the chosen action through the transition distribution, so it is the main source of critic-induced bias.

Learning the Critic

A critic can learn from Monte Carlo returns or from bootstrapped temporal-difference targets.

Monte Carlo regression

The target is the complete sampled return:

y_t^{\mathrm{MC}}=G_t.

We minimize

\mathcal L_{\mathrm{MC}}(\phi) =\frac12\mathbb E \left[ \left(V_\phi(s_t)-G_t\right)^2 \right].

Function approximation averages different returns observed from similar states. This target does not bootstrap, but it has high variance and may not be available until the trajectory ends.

One-step TD regression

The Bellman equation suggests

y_t^{(1)} =r_t+\gamma(1-d_t)V_\phi(s_{t+1}),

where $d_t=1$ for a true terminal transition. Define the TD residual

\boxed{ \delta_t =r_t+\gamma(1-d_t)V_\phi(s_{t+1})-V_\phi(s_t) }.

The critic loss is

\boxed{ \mathcal L_V(\phi) =\frac12\mathbb E \left[ \left( V_\phi(s_t) -\operatorname{stopgrad}[y_t^{(1)}] \right)^2 \right] }.

The semi-gradient update is

\phi\leftarrow\phi+ \alpha_V\delta_t\nabla_\phi V_\phi(s_t).

We detach the target so that the critic moves its current-state prediction toward the target without simultaneously changing the next-state prediction to make that target easier to fit.

Target critics

A slowly moving target network further stabilizes bootstrapping:

y_t =r_t+\gamma(1-d_t)V_{\bar\phi}(s_{t+1}).

We can periodically copy the critic

\bar\phi\leftarrow\phi

or use Polyak averaging:

\boxed{ \bar\phi \leftarrow \tau\phi+(1-\tau)\bar\phi, \qquad \tau\ll1 }.

For a finite-horizon task, the exact value can also depend on time:

V_t^\pi(s) =\mathbb E_\pi \left[ \sum_{k=0}^{T-t-1}\gamma^k r_{t+k} \mid s_t=s \right].

A stationary critic $V_\phi(s)$ is appropriate when the task is time-homogeneous or time is included in the state.

Basic Actor-Critic

For one-step actor-critic, the critic target and advantage are

y_t=r_t+\gamma(1-d_t)V_\phi(s_{t+1}),

\widehat A_t =y_t-V_\phi(s_t) =\delta_t.

The actor minimizes

\boxed{ \mathcal L_\pi(\theta) =-\mathbb E \left[ \log\pi_\theta(a_t\mid s_t) \operatorname{stopgrad}[\widehat A_t] \right] }.

The critic and actor have separate parameters, losses, and optimizers. The advantage is detached during the actor update.

\textbf{Batch Actor-Critic}

\begin{aligned} &\text{Initialize actor parameters }\theta \text{ and critic parameters }\phi.\\[0.25em] &\text{Repeat:}\\[0.25em] &\qquad \text{collect trajectories using }\pi_\theta,\\[0.25em] &\qquad y_t\leftarrow r_t+\gamma(1-d_t)V_\phi(s_{t+1}),\\[0.25em] &\qquad \phi\leftarrow\arg\min_\phi \frac12\sum_t \left(V_\phi(s_t)-\operatorname{stopgrad}[y_t]\right)^2,\\[0.25em] &\qquad \widehat A_t\leftarrow \operatorname{stopgrad}[y_t-V_\phi(s_t)],\\[0.25em] &\qquad \theta\leftarrow\arg\min_\theta -\sum_t\log\pi_\theta(a_t\mid s_t)\widehat A_t. \end{aligned}

With a single transition, the updates reduce to

\phi\leftarrow\phi+ \alpha_V\delta_t\nabla_\phi V_\phi(s_t),

\theta\leftarrow\theta+ \alpha_\pi\delta_t \nabla_\theta\log\pi_\theta(a_t\mid s_t).

Online updates use fresh data but are noisy and temporally correlated. Batch rollouts average gradients efficiently. Parallel environment workers can decorrelate experience without replay: asynchronous A3C applies worker gradients to shared parameters, while synchronous A2C aggregates all workers before updating.

Multi-Step Returns

One-step TD delegates the entire future after $r_t$ to the critic:

\widehat G_t^{(1)} =r_t+\gamma(1-d_t)V(s_{t+1}).

An $n$ -step return uses $n$ real rewards before bootstrapping:

\boxed{ \widehat G_t^{(n)} =\sum_{k=0}^{n-1}\gamma^k r_{t+k} +\gamma^n V(s_{t+n}) }.

The bootstrap term is removed if a terminal state occurs inside the window. Its advantage estimate is

\widehat A_t^{(n)} =\widehat G_t^{(n)}-V(s_t).

At the end of a finite episode, $V(s_T)=0$ . Choosing $n=T-t$ recovers the Monte Carlo return:

\widehat G_t^{(T-t)}=G_t.

Thus one-step TD and Monte Carlo are endpoints of the same family:

\widehat G_t^{(1)} \longleftrightarrow \widehat G_t^{(2)} \longleftrightarrow \cdots \longleftrightarrow G_t.

Larger $n$ reduces dependence on critic accuracy but includes more sampled randomness. Smaller $n$ produces earlier, lower-variance updates but more bootstrap bias.

An $n$ -step advantage is a discounted sum of one-step TD residuals:

\boxed{ \widehat A_t^{(n)} =\sum_{k=0}^{n-1}\gamma^k\delta_{t+k} }.

See derivation

Substitute

\delta_{t+k} =r_{t+k}+\gamma V(s_{t+k+1})-V(s_{t+k}).

Then

\begin{aligned} \sum_{k=0}^{n-1}\gamma^k\delta_{t+k} &=\sum_{k=0}^{n-1}\gamma^k r_{t+k}\\ &\quad+\sum_{k=0}^{n-1}\gamma^{k+1}V(s_{t+k+1}) -\sum_{k=0}^{n-1}\gamma^kV(s_{t+k}). \end{aligned}

The value terms telescope:

\sum_{k=0}^{n-1}\gamma^{k+1}V(s_{t+k+1}) -\sum_{k=0}^{n-1}\gamma^kV(s_{t+k}) =\gamma^nV(s_{t+n})-V(s_t).

Therefore,

\sum_{k=0}^{n-1}\gamma^k\delta_{t+k} =\widehat G_t^{(n)}-V(s_t) =\widehat A_t^{(n)}.

This identity connects multi-step returns, generalized advantage estimation, and eligibility traces.

Generalized Advantage Estimation

Choosing one fixed $n$ commits every update to one point on the bias-variance spectrum. Generalized Advantage Estimation mixes many horizons:

\widehat A_t^{\mathrm{GAE}(\gamma,\lambda)} =(1-\lambda) \sum_{n=1}^{\infty} \lambda^{n-1}\widehat A_t^{(n)}, \qquad \lambda\in[0,1].

Using the TD-residual identity, this simplifies to

\boxed{ \widehat A_t^{\mathrm{GAE}(\gamma,\lambda)} =\sum_{l=0}^{\infty} (\gamma\lambda)^l\delta_{t+l} }.

See derivation

Substitute

\widehat A_t^{(n)} =\sum_{l=0}^{n-1}\gamma^l\delta_{t+l}:

\widehat A_t^{\mathrm{GAE}} =(1-\lambda) \sum_{n=1}^{\infty}\lambda^{n-1} \sum_{l=0}^{n-1}\gamma^l\delta_{t+l}.

A fixed $\delta_{t+l}$ occurs in every estimator with $n\geq l+1$ . Swapping the summations gives

\widehat A_t^{\mathrm{GAE}} =(1-\lambda) \sum_{l=0}^{\infty}\gamma^l\delta_{t+l} \sum_{n=l+1}^{\infty}\lambda^{n-1}.

\sum_{n=l+1}^{\infty}\lambda^{n-1} =\frac{\lambda^l}{1-\lambda}.

Therefore,

\widehat A_t^{\mathrm{GAE}} =\sum_{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}.

We compute GAE in one backward pass:

\boxed{ \widehat A_t =\delta_t +\gamma\lambda(1-d_t)\widehat A_{t+1} }.

\textbf{Backward GAE Recursion}

\begin{aligned} &\widehat A_{T}\leftarrow0.\\[0.25em] &\text{For }t=T-1,T-2,\ldots,0:\\[0.25em] &\qquad \delta_t\leftarrow r_t+\gamma(1-d_t)V(s_{t+1})-V(s_t),\\[0.25em] &\qquad \widehat A_t\leftarrow \delta_t+\gamma\lambda(1-d_t)\widehat A_{t+1}. \end{aligned}

At $\lambda=0$ ,

\widehat A_t^{\mathrm{GAE}(\gamma,0)}=\delta_t,

so we recover the one-step estimate. At $\lambda=1$ in a completed episode,

\widehat A_t^{\mathrm{GAE}(\gamma,1)} =G_t-V(s_t),

so we recover Monte Carlo reward-to-go with a value baseline.

The residual at lag $l$ is weighted by $(\gamma\lambda)^l$ . A rough effective trace horizon is

H_{\mathrm{eff}}\approx\frac1{1-\gamma\lambda}.

Larger $\lambda$ uses longer sampled horizons and usually increases variance. Smaller $\lambda$ relies more strongly on critic accuracy.

A matching value target is

\widehat R_t^\lambda =\widehat A_t+V(s_t).

We fit the critic with

\mathcal L_V(\phi) =\frac12\mathbb E \left[ \left(V_\phi(s_t)-\operatorname{stopgrad}[\widehat R_t^\lambda]\right)^2 \right].

Advantage normalization,

\widetilde A_t =\frac{\widehat A_t-\mu_A}{\sigma_A+\varepsilon},

reduces sensitivity to reward scale, but it cannot repair a systematically incorrect critic.

Off-Policy Actor-Critic

Replay buffers make training more sample-efficient, but they break the on-policy assumptions of a state-value critic. A replay action was sampled from a behavior policy $\beta$ :

a\sim\beta(\cdot\mid s), \qquad \beta\neq\pi_\theta.

The Bellman equation for the current policy is

V^\pi(s) =\mathbb E_{ a\sim\pi(\cdot\mid s),,s'\sim P } \left[r(s,a)+\gamma V^\pi(s')\right].

Regressing $V(s)$ toward a target built from arbitrary replay actions instead averages under $\beta$ . Without importance weighting or another correction, it does not cleanly learn $V^\pi$ .

A Q-function conditions on the replay action:

Q^\pi(s,a) =\mathbb E_{s'\sim P(\cdot\mid s,a)} \left[ r(s,a)+\gamma \mathbb E_{a'\sim\pi(\cdot\mid s')} Q^\pi(s',a') \right].

Even when $a$ came from an old policy, the replay tuple contains a valid environment outcome for that particular $(s,a)$ . We sample the next action from the current actor:

a'\sim\pi_\theta(\cdot\mid s'),

\boxed{ y =r+\gamma(1-d)Q_{\bar\phi}(s',a') }.

The critic minimizes

\mathcal L_Q(\phi) =\frac12\mathbb E_{\mathcal D} \left[ \left(Q_\phi(s,a)-\operatorname{stopgrad}[y]\right)^2 \right].

The actor can reuse replay states while sampling fresh current-policy actions:

s\sim\mathcal D, \qquad \widetilde a\sim\pi_\theta(\cdot\mid s).

An approximate score-function update is

\nabla_\theta J_\mathcal D \approx\mathbb E \left[ \nabla_\theta\log\pi_\theta(\widetilde a\mid s) Q_\phi(s,\widetilde a) \right].

The fresh action removes the immediate action-distribution mismatch. Replay states still follow a mixture of historical policies, and practical off-policy algorithms accept or control this remaining mismatch.

If the actor always chooses the critic's exact maximizer,

\pi(a\mid s)=\mathbb 1 \left[a=\arg\max_{\widetilde a}Q(s,\widetilde a)\right],

then the target becomes

y=r+\gamma\max_{a'}Q_{\bar\phi}(s',a'),

which is Q-learning. Actor-critic replaces an explicit maximization operator with a learned policy.

Deterministic and Reparameterized Actors

For continuous actions, we can learn a deterministic actor

\mu_\theta(s)\approx\arg\max_aQ_\phi(s,a).

It maximizes

J_\mathcal D(\theta) =\mathbb E_{s\sim\mathcal D} \left[Q_\phi(s,\mu_\theta(s))\right].

By the chain rule,

\boxed{ \nabla_\theta J_\mathcal D(\theta) =\mathbb E_{s\sim\mathcal D} \left[ \left.\nabla_aQ_\phi(s,a)\right|_{a=\mu_\theta(s)} \nabla_\theta\mu_\theta(s) \right] }.

The actor loss is simply

\mathcal L_\mu(\theta) =-\mathbb E_{s\sim\mathcal D} [Q_\phi(s,\mu_\theta(s))].

During this update, gradients must flow through the critic with respect to its action input, but the actor optimizer must not update the critic's parameters.

For a Gaussian policy, sample fixed noise

\epsilon\sim\mathcal N(0,I)

and write the action as

a=f_\theta(s,\epsilon) =\mu_\theta(s)+\sigma_\theta(s)\odot\epsilon.

Then

\mathbb E_{a\sim\pi_\theta}[Q_\phi(s,a)] =\mathbb E_\epsilon [Q_\phi(s,f_\theta(s,\epsilon))],

and the pathwise gradient is

\boxed{ \nabla_\theta \mathbb E_\epsilon[Q_\phi(s,f_\theta(s,\epsilon))] =\mathbb E_\epsilon \left[ \left.\nabla_aQ_\phi(s,a)\right|_{a=f_\theta(s,\epsilon)} \nabla_\theta f_\theta(s,\epsilon) \right] }.

This usually has lower variance than a score-function estimator because it differentiates through the sampled action. As $\sigma_\theta(s)\to0$ , it reduces to the deterministic policy gradient.

For bounded actions, we often use

u=\mu_\theta(s)+\sigma_\theta(s)\odot\epsilon, \qquad a=\tanh(u).

The transformation keeps $a$ in $[-1,1]$ while preserving a differentiable path to the actor.

DDPG

Deep Deterministic Policy Gradient combines:

a deterministic actor $\mu_\theta$ ;
a Q-critic $Q_\phi$ ;
target actor $\mu_{\bar\theta}$ and target critic $Q_{\bar\phi}$ ;
an experience-replay buffer.

The target action and critic target are

a'=\mu_{\bar\theta}(s'),

\boxed{ y=r+\gamma(1-d)Q_{\bar\phi}(s',a') }.

The critic minimizes

\mathcal L_Q(\phi) =\mathbb E_\mathcal D \left[ \left(Q_\phi(s,a)-\operatorname{stopgrad}[y]\right)^2 \right],

while the actor minimizes

\mathcal L_\mu(\theta) =-\mathbb E_{s\sim\mathcal D} [Q_\phi(s,\mu_\theta(s))].

Both target networks move slowly:

\bar\theta\leftarrow \tau\theta+(1-\tau)\bar\theta,

\bar\phi\leftarrow \tau\phi+(1-\tau)\bar\phi.

Because the actor is deterministic, behavior needs external exploration noise:

a_t=\mu_\theta(s_t)+\eta_t.

We remove $\eta_t$ during evaluation.

\textbf{DDPG}

\begin{aligned} &\text{Initialize }\mu_\theta,Q_\phi, \mu_{\bar\theta}\leftarrow\mu_\theta, Q_{\bar\phi}\leftarrow Q_\phi, \text{ and replay }\mathcal D.\\[0.25em] &\text{For each environment step:}\\[0.25em] &\qquad a_t\leftarrow\mu_\theta(s_t)+\eta_t,\\[0.25em] &\qquad \text{execute }a_t\text{ and store }(s_t,a_t,r_t,s_{t+1},d_t),\\[0.25em] &\qquad \text{sample a minibatch from }\mathcal D,\\[0.25em] &\qquad y_i\leftarrow r_i+\gamma(1-d_i) Q_{\bar\phi}(s_i',\mu_{\bar\theta}(s_i')),\\[0.25em] &\qquad \phi\leftarrow\arg\min_\phi \frac1B\sum_i(Q_\phi(s_i,a_i)-y_i)^2,\\[0.25em] &\qquad \theta\leftarrow\arg\min_\theta -\frac1B\sum_iQ_\phi(s_i,\mu_\theta(s_i)),\\[0.25em] &\qquad \text{Polyak-update }\bar\theta\text{ and }\bar\phi. \end{aligned}

DDPG is approximate continuous-action Q-learning: the actor amortizes the critic's argmax.

TD3

A learned actor can exploit positive critic errors just as a discrete max does. If the critic overestimates one narrow action region, the actor is trained to move toward it.

TD3 uses two critics

Q_{\phi_1}(s,a), \qquad Q_{\phi_2}(s,a),

and the clipped target

\boxed{ y=r+\gamma(1-d) \min_{j\in\{1,2\}} Q_{\bar\phi_j}(s',a') }.

Each critic minimizes its own Bellman loss:

\mathcal L_{Q_j}(\phi_j) =\mathbb E_\mathcal D \left[ \left(Q_{\phi_j}(s,a)-\operatorname{stopgrad}[y]\right)^2 \right].

The minimum introduces a conservative bias that prevents optimistic errors from reinforcing themselves. TD3 also delays actor and target updates, allowing the critics to improve before the actor follows their gradients.

\text{many critic updates} \longrightarrow \text{one actor and target update}.

If the actor changes faster than the critic can track, it is likely to chase transient value errors.

Soft Actor-Critic

Soft Actor-Critic optimizes reward and policy entropy:

\boxed{ J(\pi) =\mathbb E \left[ \sum_t\gamma^t \left(r(s_t,a_t)+\alpha\mathcal H(\pi(\cdot\mid s_t))\right) \right] }.

Since

\mathcal H(\pi(\cdot\mid s)) =-\mathbb E_{a\sim\pi} [\log\pi(a\mid s)],

the soft state value is

V_{\mathrm{soft}}(s) =\mathbb E_{a\sim\pi} \left[Q(s,a)-\alpha\log\pi(a\mid s)\right].

The temperature $\alpha>0$ controls the reward-entropy trade-off. Large $\alpha$ favors stochastic policies; small $\alpha$ approaches reward-greedy behavior.

Critic target

Sample a reparameterized next action

a'\sim\pi_\theta(\cdot\mid s').

With twin target critics,

\boxed{ y=r+\gamma(1-d) \left[ \min_{j\in\{1,2\}}Q_{\bar\phi_j}(s',a') -\alpha\log\pi_\theta(a'\mid s') \right] }.

Each critic minimizes

\mathcal L_{Q_j}(\phi_j) =\mathbb E_\mathcal D \left[ \left(Q_{\phi_j}(s,a)-\operatorname{stopgrad}[y]\right)^2 \right].

The replay action $a$ is used in the current critic prediction. The next action $a'$ comes from the latest actor.

Actor loss

The actor maximizes soft value, or equivalently minimizes

\boxed{ \mathcal L_\pi(\theta) =\mathbb E_{ s\sim\mathcal D,,a\sim\pi_\theta } \left[ \alpha\log\pi_\theta(a\mid s) -\min_{j\in\{1,2\}}Q_{\phi_j}(s,a) \right] }.

Reparameterization lets gradients flow through the sampled action into the critic and through the log-probability term.

Automatic temperature tuning

We can learn $\alpha$ to track a target entropy $\mathcal H_{\mathrm{tgt}}$ :

\boxed{ \mathcal L_\alpha =\mathbb E \left[ -\alpha \left( \log\pi_\theta(a\mid s) +\mathcal H_{\mathrm{tgt}} \right) \right] }.

Parameterize

\alpha=\exp(\log\alpha)

to ensure positivity. The policy terms are detached during the temperature-only update.

\textbf{Soft Actor-Critic}

\begin{aligned} &\text{Initialize actor }\pi_\theta,\text{ critics }Q_{\phi_1},Q_{\phi_2}, \text{ targets, and replay }\mathcal D.\\[0.25em] &\text{For each environment step:}\\[0.25em] &\qquad a_t\sim\pi_\theta(\cdot\mid s_t),\\[0.25em] &\qquad \text{execute }a_t\text{ and store the transition},\\[0.25em] &\qquad \text{sample a minibatch from }\mathcal D,\\[0.25em] &\qquad a_i'\sim\pi_\theta(\cdot\mid s_i'),\\[0.25em] &\qquad y_i\leftarrow r_i+\gamma(1-d_i) \left[ \min_jQ_{\bar\phi_j}(s_i',a_i') -\alpha\log\pi_\theta(a_i'\mid s_i') \right],\\[0.25em] &\qquad \text{fit both critics to }y_i,\\[0.25em] &\qquad \text{update the actor with } \alpha\log\pi_\theta(a\mid s)-\min_jQ_{\phi_j}(s,a),\\[0.25em] &\qquad \text{optionally update }\alpha,\\[0.25em] &\qquad \text{Polyak-update both target critics.} \end{aligned}

Comparing the Main Estimators

Method	Data	Actor	Critic signal	Main trade-off
One-step actor-critic	Fresh on-policy rollouts	Stochastic score function	$r+\gamma V(s')-V(s)$	Low variance, critic bias
GAE actor-critic	Fresh on-policy rollouts	Stochastic score function	Weighted TD residuals	Tunable bias and variance
DDPG	Replay	Deterministic pathwise	Target Q at target-actor action	Efficient continuous control, fragile critic
TD3	Replay	Deterministic pathwise	Minimum of twin target critics	Less overestimation, conservative bias
SAC	Replay	Stochastic reparameterized	Entropy-regularized twin-Q target	Stable exploration, more interacting components

The on-policy methods use value critics because their rollout actions follow the current policy. Replay-based methods use Q-critics because the stored action may come from an older policy.

Practical Diagnosis

Symptom	Likely cause	Useful response
Advantages are extremely large	Bad reward scale, terminal mask, or critic fit	Inspect TD residuals and reset GAE at episode boundaries
Critic loss falls but return does not improve	Distribution mismatch or actor exploiting errors	Track realized returns and use twin critics
Q-values grow far beyond possible returns	Bootstrapping or terminal-handling bug	Verify detachment, masks, discount, and target updates
Actor saturates at action bounds	Critic overestimation or excessive actor updates	Lower actor rate and delay actor updates
Gaussian entropy collapses early	Temperature too small or actor step too large	Monitor log standard deviation and entropy
DDPG evaluates worse than training	Exploration noise was left enabled	Remove action noise during evaluation
GAE behaves like noisy Monte Carlo	$\gamma\lambda$ is too large	Lower $\lambda$ or improve the critic
GAE is biased and short-sighted	$\lambda$ is too small or critic is weak	Increase $\lambda$ and critic capacity carefully

Use separate actor and critic optimizers. Detach Bellman targets, GAE advantages, value targets, and policy terms used only for temperature tuning. During a pathwise actor update, preserve gradients through the critic's action input while preventing the actor optimizer from changing critic parameters.

For true terminal transitions,

y=r.

Administrative time-limit truncations may still have future value and should not automatically be masked as terminal when the environment distinguishes them.

Finally, actor-critic results can vary substantially across seeds. Report multiple runs, separate training returns from evaluation returns, and log both critic predictions and realized discounted returns.

The central progression is

\underbrace{G_t}_{\text{Monte Carlo}} \longrightarrow \underbrace{G_t-V(s_t)}_{\text{baseline}} \longrightarrow \underbrace{r_t+\gamma V(s_{t+1})-V(s_t)}_{\text{one-step critic}} \longrightarrow \underbrace{\sum_l(\gamma\lambda)^l\delta_{t+l}}_{\text{GAE}}.

For replay-based continuous control, the corresponding progression is

\underbrace{Q(s,\mu(s))}_{\text{DDPG}} \longrightarrow \underbrace{\min_jQ_j(s,\mu(s))}_{\text{TD3}} \longrightarrow \underbrace{\min_jQ_j(s,a)-\alpha\log\pi(a\mid s)}_{\text{SAC}}.

Actor-critic methods work by making two approximators improve one another: the critic turns experience into a useful local learning signal, and the actor changes the data distribution toward actions the critic values. Their strength and instability both come from this feedback loop.