Inverse Reinforcement Learning

Why Learn a Reward?

Standard reinforcement learning assumes the reward is known. In many real tasks, specifying the correct reward is as difficult as solving the task itself.

A hand-written reward may measure a convenient proxy, omit rare constraints, assign the wrong relative weights, or reward an unintended shortcut. Reward learning changes the pipeline from

\text{known reward} \longrightarrow \text{learn a policy}

\text{human behavior or judgments} \longrightarrow \text{learn }r_\psi \longrightarrow \text{learn a policy}.

The supervision can be demonstrations, preferences, rankings, corrections, or success labels.

Behavioral cloning learns

\pi_\theta(a\mid s)

by copying what the expert did. Reward learning attempts to infer why that behavior was desirable.

This distinction matters when dynamics change. A copied action may no longer produce the same outcome, while a transferable reward can be optimized again under new transitions $\widetilde p$ :

\pi_{\widetilde p}^* =\arg\max_\pi \mathbb E_{\tau\sim p_{\pi,\widetilde p}} [R_\psi(\tau)].

Transfer only works if the learned reward captures the relevant intent and generalizes to the new states.

The same behavior can be explained by many rewards:

\text{one observed policy} \quad\Longleftarrow\quad \text{many possible reward functions}.

A demonstrated route may reflect destination seeking, obstacle avoidance, path following, low control effort, or an arbitrary reward defined only on visited states.

Inverse reinforcement learning therefore needs assumptions such as a restricted reward class, smoothness, sparsity, priors, feature representations, or a model of approximately optimal behavior. Its goal is usually not to identify a unique true reward, but to find one that explains the evidence and supports reliable downstream control.

Maximum-Entropy Inverse Reinforcement Learning

Let the learned reward be $r_\psi(s,a)$ and the trajectory reward be

R_\psi(\tau)=\sum_{t=1}^{T}r_\psi(s_t,a_t).

Control as inference models optimality with

p(O_t=1\mid s_t,a_t,\psi) \propto e^{r_\psi(s_t,a_t)}.

Conditioning on optimality produces the energy-based trajectory model

\boxed{ p_\psi(\tau\mid O_{1:T}=1) =\frac1{Z_\psi} p(\tau)e^{R_\psi(\tau)} },

where $p(\tau)$ assigns probability to physically feasible trajectories and

\boxed{ Z_\psi =\int p(\tau)e^{R_\psi(\tau)}d\tau }

is the partition function.

High-reward trajectories are exponentially more likely, but lower-reward trajectories retain nonzero probability. This permits variable, imperfect, or multimodal expert behavior.

Given expert demonstrations

\mathcal D_E=\{\tau_i^E\}_{i=1}^{N},

maximum likelihood gives

\begin{aligned} \mathcal L(\psi) &=\frac1N\sum_{i=1}^{N} \log p_\psi(\tau_i^E\mid O=1)\\ &=\frac1N\sum_iR_\psi(\tau_i^E)-\log Z_\psi +\text{constant}. \end{aligned}

Therefore,

\boxed{ \mathcal L(\psi) =\mathbb E_{\tau\sim\pi_E}[R_\psi(\tau)] -\log Z_\psi }.

The first term raises expert reward. The partition function prevents the trivial solution of raising every trajectory's reward equally.

The IRL Gradient

Differentiating the likelihood gives the central maximum-entropy IRL update:

\boxed{ \nabla_\psi\mathcal L =\mathbb E_{\tau\sim\pi_E} [\nabla_\psi R_\psi(\tau)] -\mathbb E_{\tau\sim p_\psi(\tau\mid O)} [\nabla_\psi R_\psi(\tau)] }.

The reward increases on expert trajectories and decreases on trajectories made likely by the current reward model.

See derivation

Start from

\mathcal L(\psi) =\mathbb E_{\pi_E}[R_\psi(\tau)]-\log Z_\psi.

Then

\nabla_\psi\mathcal L =\mathbb E_{\pi_E}[\nabla_\psi R_\psi(\tau)] -\frac1{Z_\psi}\nabla_\psi Z_\psi.

Differentiate the partition function:

\nabla_\psi Z_\psi =\int p(\tau)e^{R_\psi(\tau)} \nabla_\psi R_\psi(\tau)d\tau.

Therefore,

\begin{aligned} \nabla_\psi\log Z_\psi &=\int \frac{p(\tau)e^{R_\psi(\tau)}}{Z_\psi} \nabla_\psi R_\psi(\tau)d\tau\\ &=\mathbb E_{\tau\sim p_\psi(\tau\mid O)} [\nabla_\psi R_\psi(\tau)]. \end{aligned}

Substituting gives the result.

Feature expectation matching

Suppose the reward is linear:

r_\psi(s,a)=\psi^\top\phi(s,a).

Define trajectory features

\Phi(\tau)=\sum_{t=1}^{T}\phi(s_t,a_t).

Then

R_\psi(\tau)=\psi^\top\Phi(\tau), \qquad \nabla_\psi R_\psi(\tau)=\Phi(\tau).

The gradient becomes

\boxed{ \nabla_\psi\mathcal L =\underbrace{\mathbb E_{\pi_E}[\Phi(\tau)]}_{\text{expert features}} -\underbrace{\mathbb E_{p_\psi}[\Phi(\tau)]}_{\text{model features}} }.

At a stationary point,

\mathbb E_{\pi_E}[\Phi(\tau)] =\mathbb E_{p_\psi}[\Phi(\tau)].

Maximum-entropy IRL therefore matches the expected features of soft-optimal behavior to those of the expert.

For linear rewards, the Hessian is

\nabla_\psi^2\mathcal L =-\operatorname{Cov}_{p_\psi}[\Phi(\tau)] \preceq0,

so the likelihood is concave in $\psi$ for a fixed feature representation. Neural reward models lose this global guarantee.

Why the model expectation requires control

The expert expectation is estimated directly from demonstrations. The model expectation requires samples from

p_\psi(\tau\mid O) \propto p(\tau)e^{R_\psi(\tau)}.

We therefore need a policy that is approximately optimal under the current learned reward:

\pi_\psi \approx\arg\max_\pi \mathbb E_{\tau\sim\pi} \left[ R_\psi(\tau) +\sum_t\mathcal H(\pi(\cdot\mid s_t)) \right].

Reward learning contains a nested control problem:

\text{reward update} \longrightarrow \text{solve maximum-entropy RL} \longrightarrow \text{estimate the next reward gradient}.

Sample-Based Reward Learning

If policy trajectories were exact samples from the current energy model, we could estimate

\widehat{\nabla_\psi\mathcal L} =\frac1N\sum_i\nabla_\psi R_\psi(\tau_i^E) -\frac1M\sum_j\nabla_\psi R_\psi(\tau_j^\pi).

But fully solving a new RL problem after every reward update is impractical. The current policy is only an approximate proposal distribution $q(\tau)$ .

Importance sampling corrects its samples with unnormalized weight

w(\tau) =\frac{p(\tau)e^{R_\psi(\tau)}}{q(\tau)}.

When $q$ is induced by a policy,

q(\tau) =p_0(s_1) \prod_{t=1}^{T-1}p(s_{t+1}\mid s_t,a_t) \prod_{t=1}^{T}\pi(a_t\mid s_t).

The environment factors cancel, giving

\boxed{ w(\tau) =\frac{e^{R_\psi(\tau)}} {\prod_{t=1}^{T}\pi(a_t\mid s_t)} }.

Equivalently,

\log w(\tau) =R_\psi(\tau)-\sum_t\log\pi(a_t\mid s_t).

This cancellation lets us compute weights without knowing the transition model.

Normalize the weights:

\overline w_j =\frac{w_j}{\sum_kw_k}.

Then

\boxed{ \widehat{\nabla_\psi\mathcal L} =\frac1N\sum_i\nabla_\psi R_\psi(\tau_i^E) -\sum_j\overline w_j \nabla_\psi R_\psi(\tau_j) }.

Self-normalization removes the unknown partition function. It introduces finite-sample bias but is consistent as the sample count grows.

For numerical stability, compute

\overline w_j =\frac{\exp(\ell_j-\ell_{\max})} {\sum_k\exp(\ell_k-\ell_{\max})},

where

\ell_j =R_\psi(\tau_j)-\sum_t\log\pi(a_{j,t}\mid s_{j,t}).

Importance sampling still requires support. If the proposal rarely visits trajectories favored by the learned reward, a few samples dominate. A useful diagnostic is

\mathrm{ESS} =\frac1{\sum_j\overline w_j^2}.

Small effective sample size indicates a high-variance reward update.

Guided Cost Learning

Guided cost learning alternates a reward model and a maximum-entropy policy. A cost $c_\psi$ is equivalent to reward $r_\psi=-c_\psi$ .

The reward update is

\psi\leftarrow\psi+ \eta_r \left[ \frac1N\sum_i\nabla_\psi R_\psi(\tau_i^E) -\sum_j\overline w_j\nabla_\psi R_\psi(\tau_j^\pi) \right].

Holding the reward fixed, improve the policy with

\max_\theta \mathbb E_{\tau\sim\pi_\theta} \left[ R_\psi(\tau) +\sum_t\mathcal H(\pi_\theta(\cdot\mid s_t)) \right].

\textbf{Guided Cost Learning}

\begin{aligned} &\text{Initialize reward parameters }\psi \text{ and policy parameters }\theta.\\[0.25em] &\text{Repeat:}\\[0.25em] &\qquad \text{improve }\pi_\theta\text{ for a few max-ent RL steps under }r_\psi,\\[0.25em] &\qquad \tau_j^\pi\sim\pi_\theta,\\[0.25em] &\qquad \ell_j\leftarrow R_\psi(\tau_j^\pi)-\sum_t\log\pi_\theta(a_{j,t}\mid s_{j,t}),\\[0.25em] &\qquad \overline w_j\leftarrow \operatorname{softmax}(\ell_j),\\[0.25em] &\qquad g_\psi\leftarrow \frac1N\sum_i\nabla_\psi R_\psi(\tau_i^E) -\sum_j\overline w_j\nabla_\psi R_\psi(\tau_j^\pi),\\[0.25em] &\qquad \psi\leftarrow\psi+\eta_r g_\psi. \end{aligned}

The policy is warm-started rather than solved from scratch after every reward change. It gradually tracks the evolving reward.

Adversarial Imitation Learning

Maximum-entropy IRL already resembles a two-player game:

the reward distinguishes expert from learner trajectories;
the policy changes to obtain greater learned reward.

This can be turned into adversarial imitation. Expert demonstrations are real data, policy trajectories are generated data, and a discriminator supplies the policy's learning signal.

Occupancy matching

A policy's discounted state-action occupancy measure is

\rho_\pi(s,a) =(1-\gamma) \sum_{t=0}^{\infty}\gamma^t \Pr_\pi(s_t=s,a_t=a).

Behavioral cloning matches actions on expert states:

\pi_\theta(a\mid s) \approx\pi_E(a\mid s) \quad\text{for demonstrated }s.

Adversarial imitation instead matches the state-action distribution induced by the full rollout:

\boxed{ \rho_{\pi_\theta}(s,a) \approx\rho_E(s,a) }.

This directly accounts for the fact that actions change future states.

GAIL

Let $D(s,a)$ be the probability that a state-action pair came from the expert. Generative Adversarial Imitation Learning solves

\boxed{ \min_\pi\max_D \mathbb E_{(s,a)\sim\rho_E}[\log D(s,a)] +\mathbb E_{(s,a)\sim\rho_\pi}[\log(1-D(s,a))] -\lambda\mathcal H(\pi) }.

For a fixed policy, the optimal discriminator is

D^*(s,a) =\frac{\rho_E(s,a)} {\rho_E(s,a)+\rho_\pi(s,a)}.

Substituting $D^*$ yields

-\log4 +2D_{\mathrm{JS}}(\rho_E\|\rho_\pi).

Thus minimizing the policy side minimizes a divergence between expert and learner occupancy measures.

The policy may use discriminator-derived reward such as

\widetilde r(s,a)=-\log(1-D(s,a)).

\textbf{Adversarial Imitation}

\begin{aligned} &\text{Repeat:}\\[0.25em] &\qquad \text{collect learner rollouts with }\pi_\theta,\\[0.25em] &\qquad \text{train }D\text{ to separate expert and learner state-action pairs},\\[0.25em] &\qquad \widetilde r(s,a)\leftarrow-\log(1-D(s,a)),\\[0.25em] &\qquad \text{update }\pi_\theta\text{ with reinforcement learning under }\widetilde r. \end{aligned}

GAIL primarily learns a policy that matches expert occupancy. Its converged discriminator is relative to the learner and becomes $1/2$ when the distributions match. It should not automatically be interpreted as a reusable task reward.

Maximum-entropy IRL tries to recover an explicit energy or reward model. GAIL bypasses explicit partition-function estimation and directly optimizes imitation performance.

What Inverse RL Recovers

Maximum-entropy IRL learns an explicit reward by increasing expert likelihood:

p_\psi(\tau) =\frac1{Z_\psi}p(\tau)e^{R_\psi(\tau)}

\Downarrow

\boxed{ \nabla_\psi\mathcal L =\mathbb E_{\pi_E}[\nabla_\psi R_\psi] -\mathbb E_{p_\psi}[\nabla_\psi R_\psi] }.

Guided cost learning approximates the model expectation with adaptive trajectory samples. GAIL bypasses explicit partition-function estimation and directly matches expert occupancy.

Neither route identifies a unique numerical reward without structural assumptions. Additive constants, positive scaling in hard optimal control, potential-based shaping, and unconstrained unvisited regions can preserve the observed behavior. The goal is therefore a reward or occupancy objective that explains demonstrations and supports the intended transfer—not recovery of a uniquely determined hidden scalar function.

When supervision arrives as comparisons rather than demonstrations, the observation model changes. That setting is developed in reward learning.