Ahmed Haroon

Learning Rewards from Judgments

An expert trajectory demonstrates how to act. A preference says which of two outcomes is better:

\tau_i\succ\tau_j.

Comparisons are often easier to provide than complete demonstrations, especially when competent behavior is difficult to execute but easy to recognize. Reward learning converts those judgments into a scalar model

r_\psi(s,a)

that can be optimized by a reinforcement-learning agent.

Unlike inverse reinforcement learning, preference learning does not assume the observed behavior came from an approximately optimal policy. It assumes only that comparison probabilities depend on latent cumulative reward.

Preference-Based Reward Learning

Producing a high-quality demonstration may be difficult, while comparing two behaviors is easy. Given trajectories $\tau_i$ and $\tau_j$ , an evaluator supplies

\tau_i\succ\tau_j.

The reward model assigns scores

R_\psi(\tau_i)=\sum_t r_\psi(s_t^{(i)},a_t^{(i)}),

R_\psi(\tau_j)=\sum_t r_\psi(s_t^{(j)},a_t^{(j)}).

The Bradley-Terry model defines

\boxed{ p_\psi(\tau_i\succ\tau_j) =\frac{e^{R_\psi(\tau_i)}} {e^{R_\psi(\tau_i)}+e^{R_\psi(\tau_j)}} =\sigma\left(R_\psi(\tau_i)-R_\psi(\tau_j)\right) }.

Preference learning is logistic regression on cumulative reward differences.

For ordered comparisons, maximize

\boxed{ \mathcal L_{\mathrm{pref}}(\psi) =\sum_{(i,j)} \log\sigma \left(R_\psi(\tau_i)-R_\psi(\tau_j)\right) }.

Equivalently, minimize

\mathcal J_{\mathrm{pref}}(\psi) =\sum_{(i,j)} \operatorname{softplus} \left(-[R_\psi(\tau_i)-R_\psi(\tau_j)]\right).

For a binary label $y_{ij}$ and predicted probability $p_{ij}$ ,

\boxed{ \nabla_\psi\mathcal L =\sum_{i,j}(y_{ij}-p_{ij}) \left[ \nabla_\psi R_\psi(\tau_i) -\nabla_\psi R_\psi(\tau_j) \right] }.

The winner's reward increases relative to the loser's, with a larger update when the model was wrong or uncertain.

A temperature $\beta$ can model annotator noise:

p_\psi(\tau_i\succ\tau_j) =\sigma \left( \frac{R_\psi(\tau_i)-R_\psi(\tau_j)}{\beta} \right).

Small $\beta$ represents nearly deterministic choices. Large $\beta$ represents noisier judgments.

\textbf{Iterative Preference Learning}

\begin{aligned} &\text{Repeat:}\\[0.25em] &\qquad \text{sample trajectory pairs from the current policy},\\[0.25em] &\qquad \text{ask an evaluator which trajectory is preferable},\\[0.25em] &\qquad \mathcal D_{\mathrm{pref}} \leftarrow\mathcal D_{\mathrm{pref}}\cup\{\tau_i\succ\tau_j\},\\[0.25em] &\qquad \psi\leftarrow\arg\max_\psi \sum_{(i,j)\in\mathcal D_{\mathrm{pref}}} \log\sigma(R_\psi(\tau_i)-R_\psi(\tau_j)),\\[0.25em] &\qquad \theta\leftarrow\arg\max_\theta \mathbb E_{\tau\sim\pi_\theta}[R_\psi(\tau)]. \end{aligned}

Useful comparisons are uncertain, diverse, near the current policy, or safety-critical. Pure uncertainty sampling can overselect nearly indistinguishable pairs that humans cannot label reliably, so query selection should balance informativeness and coverage.

Reward Identifiability

Observed behavior typically identifies only reward differences relevant to that behavior.

Additive constants. Preference probabilities are unchanged by

R'(\tau)=R(\tau)+c

because the constant cancels from $R'(\tau_i)-R'(\tau_j)$ . Absolute reward offset is unidentifiable.

Positive scaling. In deterministic optimal control,

r'(s,a)=c\,r(s,a), \qquad c>0,

preserves the maximizing policies. In maximum-entropy models, scaling changes the sharpness of

p(\tau)\propto e^{cR(\tau)},

so reward scale and behavioral temperature are coupled.

Potential-based shaping. The shaped reward

r'(s,a,s') =r(s,a,s')+\gamma\Phi(s')-\Phi(s)

preserves optimal policies under standard assumptions. Even complete optimal behavior therefore need not determine a unique numerical reward.

Unvisited regions. If demonstrations and comparisons never cover a region $\mathcal U$ , they provide little constraint on

r_\psi(s,a), \qquad(s,a)\in\mathcal U.

A reward can fit all observed evidence and behave arbitrarily elsewhere.

Reward Hacking and Validation

Let the learned reward be

\widehat r(s,a)=r^*(s,a)+\epsilon(s,a),

where $\epsilon$ is model error. Policy optimization chooses

\widehat\pi =\arg\max_\pi \mathbb E_\pi[\widehat R(\tau)].

A capable optimizer searches for regions where $\epsilon$ is positive, even when the intended reward is poor. This is the optimizer's curse:

\max_\pi\mathbb E_\pi[r^*+\epsilon]

can exploit the error term. Stronger policy optimization can therefore make a fixed reward model fail more severely.

The problem is amplified by distribution shift:

p_{\pi_{\mathrm{new}}}(\tau) \neq p_{\mathrm{reward\ train}}(\tau).

New rollouts may enter regions never constrained by demonstrations or comparisons.

Useful protections include reward regularization, conservative policy updates, repeated human feedback, and uncertainty penalties. With an ensemble,

\overline r(s,a)=\frac1K\sum_{k=1}^{K}r_{\psi_k}(s,a),

\sigma_r(s,a) =\sqrt{ \frac1K\sum_k (r_{\psi_k}(s,a)-\overline r(s,a))^2 },

we can optimize the conservative reward

r_{\mathrm{cons}}(s,a) =\overline r(s,a)-\lambda\sigma_r(s,a).

This discourages exploitation of regions where reward models disagree.

Validation should include:

held-out preference accuracy and probability calibration;
expert likelihood or feature matching;
expert-versus-learner occupancy statistics;
direct evaluation of policies optimized against the reward;
counterexamples with high predicted reward but poor human judgment.

Prediction accuracy alone is insufficient. A reward model may rank familiar examples correctly while assigning extreme scores to novel behaviors found by the optimizer.

A Reward Must Survive Optimization

Preference accuracy on familiar held-out pairs is necessary but insufficient. The deployed policy searches for trajectories with maximal predicted reward:

\widehat\pi =\arg\max_\pi \mathbb E_{\tau\sim\pi}[\widehat R_\psi(\tau)].

This changes the evaluation distribution and exposes errors that ordinary validation may never see. Reliable reward learning therefore alternates among data collection, reward fitting, policy optimization, and adversarial evaluation of the resulting behavior.

The central loop is

\boxed{ \text{comparisons} \longrightarrow \text{reward model} \longrightarrow \text{optimized policy} \longrightarrow \text{new comparisons} }.

A reward model is successful when the policies it induces remain aligned with evaluator judgments, including in the novel regions found by optimization.