Offline Reinforcement Learning

Learning Without New Interaction

Online reinforcement learning repeatedly closes the loop:

\text{collect data} \longrightarrow \text{update values or policies} \longrightarrow \text{collect new data}.

Offline reinforcement learning removes the final arrow. Training receives only a fixed dataset

\mathcal D =\{(s_i,a_i,r_i,d_i,s_i')\}_{i=1}^{N},

generated by one or more behavior policies $\pi_\beta$ :

s_i\sim d^\beta(s), \qquad a_i\sim\pi_\beta(a\mid s_i), \qquad s_i'\sim p(s'\mid s_i,a_i).

The objective is still

\boxed{ \max_\pi J(\pi) \quad\text{using only}\quad \mathcal D }.

No new transition may be requested to test a value estimate or correct a bad action during training.

Behavioral cloning maximizes

\mathbb E_{(s,a)\sim\mathcal D} [\log\pi_\theta(a\mid s)].

It reproduces the dataset's average behavior. This is sensible when the data are consistently expert, but a fixed dataset may contain failed trajectories, policies of different quality, and locally useful fragments that never occur together.

Offline RL tries to select and stitch the useful parts. If one trajectory reliably reaches an intermediate state and another continues from there to the goal, Bellman backups can combine them even when no demonstration contains the complete solution.

This ambition requires counterfactual reasoning: the learned policy must compare actions and sequences that were not executed together. That is also where offline RL becomes dangerous.

The Counterfactual Problem

The dataset supports estimates near its own state-action distribution:

d^\beta(s,a)=d^\beta(s)\pi_\beta(a\mid s).

The learned policy induces another distribution:

d^\pi(s,a)=d^\pi(s)\pi(a\mid s).

A small regression error under $d^\beta$ says little about error under $d^\pi$ :

\mathbb E_{d^\beta}[\ell(s,a)]\text{ small} \centernot\implies \mathbb E_{d^\pi}[\ell(s,a)]\text{ small}.

Ordinary distribution shift is already difficult. Offline RL adds optimization:

a^*=\arg\max_a\widehat Q(s,a).

The policy actively searches for actions with the largest predicted value, so it tends to select positive approximation errors.

Unsupported Bellman targets

A standard value-based target is

y=r+\gamma\max_{a'}Q_{\bar\phi}(s',a').

In actor-critic form,

y =r+\gamma \mathbb E_{a'\sim\pi_\theta(\cdot\mid s')} [Q_{\bar\phi}(s',a')].

The transition $(s,a,r,s')$ came from the dataset. The target action $a'$ may not have. The critic then copies an unsupported prediction into the value of an observed action:

\mathcal L_Q(\phi) =\mathbb E_{\mathcal D} \left[ \left(Q_\phi(s,a)-y\right)^2 \right].

That error can propagate backward through every later Bellman backup.

Write

\widehat Q(s,a)=Q(s,a)+\varepsilon(s,a).

Even when a randomly selected error has zero mean, maximization preferentially chooses positive errors:

\boxed{ \mathbb E\left[ \varepsilon \left(s,\arg\max_a\widehat Q(s,a)\right) \right]>0 }.

The actor creates the same failure directly:

\max_\theta \mathbb E_{ s\sim\mathcal D, a\sim\pi_\theta(\cdot\mid s) } [Q_\phi(s,a)].

It is an optimizer over critic extrapolation errors.

Model-based RL suffers the analogous problem when a planner finds actions for which learned dynamics are optimistic. Importance-sampled policy gradients suffer when

\frac{\pi_\theta(a\mid s)}{\pi_\beta(a\mid s)}

becomes enormous because behavior probability is tiny.

Online RL can execute the suspicious action, observe its consequence, and correct the estimate. Offline RL cannot. Errors that would be temporary online can remain self-reinforcing offline.

Most offline algorithms use one or more of three ideas:

\boxed{ \begin{aligned} \text{constrain the policy} &\Rightarrow \pi\text{ stays near behavior support},\\ \text{make values pessimistic} &\Rightarrow \text{unsupported actions look unattractive},\\ \text{use in-sample backups} &\Rightarrow \text{the critic never queries actor actions}. \end{aligned} }.

These are different ways to refuse unsupported counterfactuals.

Constraining Policy Improvement

The unconstrained actor seeks

\max_\pi \mathbb E_{s\sim\mathcal D,a\sim\pi} [Q(s,a)].

A behavior-regularized version solves

\max_\pi \mathbb E_{s\sim\mathcal D,a\sim\pi} [Q(s,a)] \quad\text{s.t.}\quad D(\pi(\cdot\mid s),\pi_\beta(\cdot\mid s)) \le\epsilon.

The Lagrangian form is

\boxed{ \max_\pi \mathbb E_{s\sim\mathcal D} \left[ \mathbb E_{a\sim\pi}[Q(s,a)] -\alpha D(\pi,\pi_\beta) \right] }.

Small $\alpha$ permits aggressive improvement and more extrapolation. Large $\alpha$ approaches imitation.

The ideal constraint would only require

\operatorname{supp}\pi(\cdot\mid s) \subseteq \operatorname{supp}\pi_\beta(\cdot\mid s).

It would allow any supported action without preserving bad behavior frequencies. Exact support is difficult to estimate in continuous high-dimensional spaces, so practical methods use smooth divergences or learned behavior models.

Forward and reverse KL behave differently

Forward KL is

D_{\mathrm{KL}}(\pi_\beta\|\pi_\theta) =\mathbb E_{a\sim\pi_\beta} \left[ \log\frac{\pi_\beta(a\mid s)} {\pi_\theta(a\mid s)} \right].

Up to a constant in $\theta$ ,

\boxed{ D_{\mathrm{KL}}(\pi_\beta\|\pi_\theta) =-\mathbb E_{a\sim\pi_\beta} [\log\pi_\theta(a\mid s)] +\text{const} }.

It is directly estimated from dataset actions and pressures the policy to cover all behavior modes—including poor ones.

Reverse KL is

D_{\mathrm{KL}}(\pi_\theta\|\pi_\beta) =\mathbb E_{a\sim\pi_\theta} \left[ \log\frac{\pi_\theta(a\mid s)} {\pi_\beta(a\mid s)} \right].

Using policy entropy,

\boxed{ D_{\mathrm{KL}}(\pi_\theta\|\pi_\beta) =-\mathbb E_{a\sim\pi_\theta} [\log\pi_\beta(a\mid s)] -\mathcal H(\pi_\theta(\cdot\mid s)) }.

Reverse KL can focus on one good behavior mode, but it requires evaluating the unknown behavior density.

Actor-critic plus behavioral cloning

Using forward KL gives a particularly simple actor objective:

\boxed{ J_{\mathrm{AC+BC}}(\theta) = \mathbb E_{s\sim\mathcal D,a\sim\pi_\theta} [Q_\phi(s,a)] +\alpha \mathbb E_{(s,a)\sim\mathcal D} [\log\pi_\theta(a\mid s)] }.

See derivation

Begin with

J(\theta) =\mathbb E_{a\sim\pi_\theta}[Q(s,a)] -\alpha D_{\mathrm{KL}}(\pi_\beta\|\pi_\theta).

Expand the divergence:

-\alpha D_{\mathrm{KL}}(\pi_\beta\|\pi_\theta) =-\alpha\mathbb E_{a\sim\pi_\beta} [\log\pi_\beta(a\mid s)] +\alpha\mathbb E_{a\sim\pi_\beta} [\log\pi_\theta(a\mid s)].

The behavior log-likelihood is independent of

Dropping that constant leaves the cloning term estimated from $\mathcal D$ .

For a deterministic Gaussian actor, cloning becomes

-\|\pi_\theta(s)-a\|_2^2.

AC+BC is easy to add to an existing actor-critic. Its weakness is the coefficient: one global $\alpha$ must balance reward scale, critic scale, support quality, and the value of deviating in every state. The critic target may also still evaluate unsupported actor actions.

A BRAC-style method places an explicit behavior divergence in both policy improvement and critic evaluation:

y =r+\gamma \mathbb E_{a'\sim\pi_\theta} \left[ Q_{\bar\phi}(s',a') -\lambda D(\pi_\theta,\pi_\beta) \right].

The critic then estimates behavior-regularized return rather than unconstrained return.

Advantage-Weighted Regression

Advantage-weighted methods never optimize arbitrary actor actions. They fit only dataset actions, giving better actions larger weights.

For one state, consider the reverse-KL constrained improvement problem

\max_{\pi} \int\pi(a\mid s)Q(s,a)da

subject to

D_{\mathrm{KL}}(\pi\|\pi_\beta)\le\epsilon, \qquad \int\pi(a\mid s)da=1.

Its nonparametric solution is

\boxed{ \pi^*(a\mid s) =\frac1{Z(s)} \pi_\beta(a\mid s) \exp\left(\frac{Q(s,a)}{\lambda}\right) }.

See derivation

Introduce multipliers $\lambda>0$ and $\eta(s)$ :

\begin{aligned} \mathcal L &=\int\pi(a\mid s)Q(s,a)da\\ &\quad-\lambda \int\pi(a\mid s) \log\frac{\pi(a\mid s)}{\pi_\beta(a\mid s)}da\\ &\quad+\eta(s) \left(1-\int\pi(a\mid s)da\right). \end{aligned}

The functional derivative is

\frac{\delta\mathcal L}{\delta\pi(a\mid s)} =Q(s,a) -\lambda \left( \log\frac{\pi(a\mid s)}{\pi_\beta(a\mid s)}+1 \right) -\eta(s).

Setting it to zero and rearranging,

\log\frac{\pi(a\mid s)}{\pi_\beta(a\mid s)} =\frac{Q(s,a)}{\lambda} -1-\frac{\eta(s)}{\lambda}.

Exponentiating gives

\pi(a\mid s) =\pi_\beta(a\mid s) e^{Q(s,a)/\lambda} e^{-1-\eta(s)/\lambda}.

The final factor depends only on

Since $Q(s,a)=V(s)+A(s,a)$ , the state-only factor $e^{V(s)/\lambda}$ is absorbed into the normalizer:

\pi^*(a\mid s) \propto \pi_\beta(a\mid s) \exp\left(\frac{A(s,a)}{\lambda}\right).

Projecting this improved distribution into a parameterized policy gives advantage-weighted regression:

\boxed{ J_{\mathrm{AWR}}(\theta) =\mathbb E_{(s,a)\sim\mathcal D} \left[ \exp\left(\frac{A(s,a)}{\lambda}\right) \log\pi_\theta(a\mid s) \right] }.

See derivation

Fit $\pi_\theta$ by minimizing

\mathbb E_{s\sim\mathcal D} D_{\mathrm{KL}}(\pi^*\|\pi_\theta).

The terms independent of $\theta$ disappear, leaving

\max_\theta \mathbb E_{a\sim\pi^*} [\log\pi_\theta(a\mid s)].

Reweight this expectation through the behavior distribution:

\begin{aligned} \mathbb E_{a\sim\pi^*} [\log\pi_\theta(a\mid s)] &=\mathbb E_{a\sim\pi_\beta} \left[ \frac{\pi^*(a\mid s)}{\pi_\beta(a\mid s)} \log\pi_\theta(a\mid s) \right]\\ &=\mathbb E_{a\sim\pi_\beta} \left[ \frac{e^{A(s,a)/\lambda}}{Z(s)} \log\pi_\theta(a\mid s) \right]. \end{aligned}

Dropping

Dataset samples now estimate the remaining expectation.

The exponential is the exact KL-constrained solution, not an arbitrary heuristic. It also keeps weights positive:

w(s,a)=\exp(A(s,a)/\lambda)>0.

In practice,

w(s,a) =\min\left( w_{\max}, e^{A(s,a)/\lambda} \right)

limits variance. Large $\lambda$ approaches ordinary cloning; small $\lambda$ concentrates on the highest-advantage samples.

AWAC combines this supervised actor update with an actor-critic value estimate. Its actor remains in-sample, though its conventional critic target may still query current-policy actions.

Implicit Q-Learning

Implicit Q-learning, or IQL, removes actor actions from critic training altogether. It separates

Q(s,a)\leftarrow r+\gamma V(s')

from an in-sample estimate of the best supported value:

V(s) \approx \max_{a\in\operatorname{supp}\pi_\beta(\cdot\mid s)} Q(s,a).

The Q loss uses only dataset transitions:

\boxed{ \mathcal L_Q(\phi) =\mathbb E_{(s,a,r,s',d)\sim\mathcal D} \left[ \left( Q_\phi(s,a) -r -\gamma(1-d)V_\psi(s') \right)^2 \right] }.

In continuous state spaces, the exact same state rarely appears with many recorded actions. A literal sample maximum is therefore unusable. IQL approximates an upper supported value with expectile regression.

Define

\ell_2^\tau(x) = \begin{cases} (1-\tau)x^2,&x>0,\\ \tau x^2,&x\le0, \end{cases} \qquad \tau\in[0.5,1).

Then fit

\boxed{ \mathcal L_V(\psi) =\mathbb E_{(s,a)\sim\mathcal D} \left[ \ell_2^\tau \left(V_\psi(s)-Q_{\bar\phi}(s,a)\right) \right] }.

When $V<Q$ , the residual receives weight $\tau$ . For $\tau>0.5$ , underestimating high Q-values is penalized more than overestimating low ones, pulling $V$ toward the upper tail of supported actions.

What the expectile solves

Let $X=Q(s,A)$ for $A\sim\pi_\beta(\cdot\mid s)$ . The expectile $v$ satisfies

\boxed{ (1-\tau)\mathbb E[(v-X)_+] =\tau\mathbb E[(X-v)_+] }.

See derivation

For fixed $s$ , minimize

L(v)=\mathbb E[\ell_2^\tau(v-X)].

Differentiating the two regions and setting the result to zero gives

\mathbb E \left[ (1-\tau)(v-X)\mathbf 1\{v>X\} +\tau(v-X)\mathbf 1\{v\le X\} \right]=0.

Move the negative-residual term to the other side:

(1-\tau)\mathbb E[(v-X)_+] =\tau\mathbb E[(X-v)_+].

At $\tau=0.5$ , $v$ is the mean. As $\tau$ approaches one, high supported Q-values exert more influence and $v$ approaches a soft in-support maximum.

For two equally likely values $-10$ and $10$ , with $-10<v<10$ ,

L(v) =\frac12(1-\tau)(v+10)^2 +\frac12\tau(v-10)^2.

The stationary point is

\boxed{ v=10(2\tau-1) }.

Thus $\tau=0.985$ gives $v=9.7$ : close to the best supported value without ever maximizing over a policy-sampled action.

After learning $Q$ and $V$ , extract the policy with weighted cloning:

\boxed{ J_{\mathrm{IQL\text{-}actor}}(\theta) =\mathbb E_{(s,a)\sim\mathcal D} \left[ e^{(Q_\phi(s,a)-V_\psi(s))/\lambda} \log\pi_\theta(a\mid s) \right] }.

\textbf{Implicit Q-Learning}

\begin{aligned} &\text{Sample }(s_i,a_i,r_i,d_i,s_i')\sim\mathcal D.\\[0.25em] &\psi\leftarrow\arg\min_\psi \sum_i\ell_2^\tau \left(V_\psi(s_i)-Q_{\bar\phi}(s_i,a_i)\right).\\[0.25em] &y_i\leftarrow r_i+\gamma(1-d_i)V_\psi(s_i').\\[0.25em] &\phi\leftarrow\arg\min_\phi \sum_i\left(Q_\phi(s_i,a_i)-y_i\right)^2.\\[0.25em] &w_i\leftarrow \min\left(w_{\max}, e^{(Q_\phi(s_i,a_i)-V_\psi(s_i))/\lambda} \right).\\[0.25em] &\theta\leftarrow\arg\max_\theta \sum_i w_i\log\pi_\theta(a_i\mid s_i). \end{aligned}

No expectation under $\pi_\theta$ appears in critic learning. The actor is a final extraction step, which is why the Q-learning is implicit rather than a conventional actor-critic loop.

Conservative Q-Learning

IQL avoids evaluating unsupported actions. Conservative Q-learning, or CQL, evaluates broader actions but deliberately lowers their values.

Let $\mu(a\mid s)$ be a broad proposal, such as a mixture of uniform, behavior, and current-policy actions. CQL augments Bellman regression with

\boxed{ \mathcal L_{\mathrm{CQL}}(Q) =\mathcal L_{\mathrm{Bellman}}(Q) +\alpha \left( \mathbb E_{s\sim\mathcal D,a\sim\mu}[Q(s,a)] -\mathbb E_{(s,a)\sim\mathcal D}[Q(s,a)] \right) }.

Minimization pushes down values on broad or policy-generated actions and restores values on observed actions. Bellman regression preserves temporal consistency.

The broad proposal can itself seek high Q-values. For discrete actions, consider

\max_\mu \left{ \mathbb E_{a\sim\mu}[Q(s,a)] +\mathcal H(\mu) \right}.

The optimum is

\mu^*(a\mid s) =\frac{e^{Q(s,a)}}{\sum_b e^{Q(s,b)}},

and the optimized value is

\boxed{ \max_\mu \left( \mathbb E_\mu Q+\mathcal H(\mu) \right) =\log\sum_a e^{Q(s,a)} }.

See derivation

Add a normalization multiplier:

\mathcal J(\mu,\eta) =\sum_a\mu(a)Q(a) -\sum_a\mu(a)\log\mu(a) +\eta\left(1-\sum_a\mu(a)\right).

Stationarity gives

Q(a)-\log\mu(a)-1-\eta=0,

\mu^*(a)\propto e^{Q(a)}.

Normalize with $Z=\sum_a e^{Q(a)}$ . Since

\log\mu^*(a)=Q(a)-\log Z,

we obtain

\begin{aligned} \mathbb E_{\mu^*}Q+\mathcal H(\mu^*) &=\mathbb E_{\mu^*} [Q-\log\mu^*]\\ &=\log Z. \end{aligned}

The common discrete-action regularizer is therefore

\boxed{ \alpha\mathbb E_{s\sim\mathcal D} \left[ \log\sum_a e^{Q(s,a)} -\mathbb E_{a\sim\pi_\beta(\cdot\mid s)}Q(s,a) \right] }.

The log-sum-exp emphasizes whichever actions currently have suspiciously large values. In continuous spaces, it is approximated with importance-weighted samples from several proposals.

After pessimistic critic learning, a discrete policy can take

\pi(s)=\arg\max_a\widehat Q(s,a),

while a continuous actor maximizes $\widehat Q$ in the usual way. Unsupported actions are now less attractive because the critic was trained to undervalue them.

CQL directly attacks overestimation and can support ordinary policy optimization. Too much conservatism, however, suppresses genuinely good actions, increases sensitivity to $\alpha$ , and can slow later online adaptation.

Expressive Policies Preserve Behavior Modes

A unimodal Gaussian actor fitted to multimodal behavior may average between separated modes. The mean can lie in a region where the dataset contains no action at all.

Diffusion and flow policies can represent a richer behavior distribution:

\pi_{\mathrm{gen}}(a\mid s) \approx\pi_\beta(a\mid s).

Their main offline-RL role is support modeling: generate plausible candidates, then use values to choose among them.

Sample and rank

IDQL first trains an IQL critic, then fits a diffusion or flow policy with behavioral cloning. At deployment it samples

a_1,\ldots,a_K \sim\pi_{\mathrm{gen}}(\cdot\mid s)

and selects

\boxed{ a^*=\arg\max_{a_k}Q_\phi(s,a_k) }.

Candidate actions remain near the learned behavior manifold, while Q-ranking improves over average behavior. More candidates improve the chance of finding a good supported mode but increase generative sampling and critic-evaluation cost.

Distilling a flow into a fast actor

Flow Q-learning trains a latent-conditioned behavior generator

a^{\mathrm{flow}} \sim\pi_{\mathrm{flow}}(\cdot\mid s,z), \qquad z\sim p(z),

then learns an ordinary actor with a Q objective and a same-latent cloning term:

\boxed{ \begin{aligned} J(\theta) =\mathbb E_{s,z} \Big[ &Q_\phi(s,\pi_\theta(s,z))\\ &+\lambda \log\pi_\theta(a^{\mathrm{flow}}\mid s,z) \Big]. \end{aligned} }.

Conditioning both models on the same $z$ preserves the mapping from latent regions to behavior modes. Without it, cloning may collapse distinct modes into their average.

Another option performs online exploration in the generator's latent space:

w\sim\pi^w(\cdot\mid s), \qquad a=\pi_{\mathrm{gen}}(s,w).

Every latent action passes through the pretrained decoder, relocating exploration from raw action space to a structured action manifold.

Model-Based Offline RL

A learned dynamics model faces the same counterfactual problem. It is fitted on

(s,a)\sim d^\beta,

then queried under a new policy:

a_t\sim\pi(\cdot\mid s_t), \qquad s_{t+1}\sim\widehat p_\eta(\cdot\mid s_t,a_t).

Unsupported action errors compound into unsupported imagined states. Offline model-based methods therefore make synthetic experience pessimistic.

MOPO penalizes uncertain rewards

Model-Based Offline Policy Optimization uses

\boxed{ \widetilde r(s,a) =r(s,a)-\lambda u(s,a) },

where $u(s,a)$ estimates epistemic uncertainty, often through dynamics-ensemble disagreement. The policy optimizes

\widetilde J(\pi) =\mathbb E_{\widehat p,\pi} \left[ \sum_t\gamma^t (r(s_t,a_t)-\lambda u(s_t,a_t)) \right].

An imagined trajectory must offer enough reward to compensate for entering uncertain regions.

COMBO penalizes synthetic values

COMBO combines model rollouts with a CQL-like contrast:

\boxed{ \begin{aligned} Q^{k+1}\leftarrow\arg\min_Q &\;\beta \left( \mathbb E_{(s,a)\sim\rho_{\text{model}}}[Q(s,a)] -\mathbb E_{(s,a)\sim\mathcal D}[Q(s,a)] \right)\\ &+\frac12 \mathbb E_{(s,a,s')\sim d_f} \left[ \left( Q(s,a)-\widehat{\mathcal B}^{\pi}Q^k(s,a) \right)^2 \right]. \end{aligned} }.

The first line lowers model-generated tuples relative to real tuples. The second performs Bellman regression on a real-synthetic mixture. MOPO inserts pessimism through explicit uncertainty; COMBO inserts it through conservative value learning.

Both methods use extra synthetic data without treating the learned simulator as uniformly trustworthy.

From Offline Pretraining to Deployment

Offline safety and online exploration prefer different biases:

\begin{aligned} \text{offline:}&\quad \text{pessimism, support constraints, in-sample evaluation},\\ \text{online:}&\quad \text{exploration, rapid support expansion, fast recalibration}. \end{aligned}

A conservative critic may initially stagnate when online fine-tuning begins because it must undo deliberate pessimism before exploiting new evidence.

A simple alternative initializes an ordinary online learner from scratch but mixes old and new replay:

p_{\mathrm{batch}} =\frac12p_{\mathrm{off}} +\frac12p_{\mathrm{on}}.

Then

\nabla\mathcal L =\frac12 \mathbb E_{\mathcal D_{\mathrm{off}}}[\nabla\ell] +\frac12 \mathbb E_{\mathcal D_{\mathrm{on}}}[\nabla\ell].

This retains the coverage benefit of prior data without inheriting a badly calibrated offline critic.

Selecting an offline policy is difficult because its real return cannot be measured without violating the setting. Low training Bellman error is insufficient: the critic may fit data actions while remaining wrong on policy actions.

Useful but imperfect checks include held-out Bellman error, behavior divergence, conservative value estimates, stability across seeds, and off-policy evaluation when its coverage assumptions are credible. Limited safe deployment data is stronger evidence when it is available.

Method choice should begin with the dataset:

Narrow expert data often favors behavioral cloning or strong constraints.
Mixed-quality data with useful fragments favors AWR, AWAC, or IQL.
Severe critic overestimation motivates CQL.
Multimodal actions favor diffusion or flow behavior models.
Predictable dynamics with measurable uncertainty can justify MOPO or COMBO.

No objective can recover arbitrary behavior absent from the data. Offline RL can reweight, generalize, and stitch supported evidence, but it cannot make unsupported counterfactuals reliable.

The core design rule is therefore

\boxed{ \text{improve where the dataset supports comparison} \quad\text{and remain conservative everywhere else} }.