Ahmed Haroon

From Exploration to Reusable Behavior

Action-level exploration is inefficient when useful behavior must persist for many individually unrewarded steps. Instead of perturbing every action independently, we can learn a repertoire of temporally extended behaviors before a downstream task is known.

Let $z$ denote a latent skill and let

a_t\sim\pi_\theta(a_t\mid s_t,z).

A useful repertoire should be diverse, controllable, and reusable. Information-theoretic objectives make these requirements measurable: the chosen skill should predict the behavior that follows, while the resulting behavior should reveal which skill was chosen.

This chapter continues from exploration, replacing information-seeking actions with distinguishable skills and reachable goals.

Discovering Skills Without External Reward

Action-level exploration struggles when useful behavior must persist. Skill learning introduces a latent variable $z$ and a conditioned policy

z\sim p(z), \qquad a_t\sim\pi_\theta(a_t\mid s_t,z).

Different skills should produce distinguishable state distributions

d_\pi(s\mid z).

This can pretrain reusable behaviors before a downstream reward is known, supply options for hierarchical control, and explore the space of achievable behavior.

The ideal diversity objective is

\boxed{ I(Z;S)=\mathcal H(Z)-\mathcal H(Z\mid S) }.

With a fixed uniform skill prior, $\mathcal H(Z)$ is constant. The policy must make the selected skill identifiable from the states it visits.

The true posterior $p(z\mid s)$ is unavailable, so train a discriminator $q_\phi(z\mid s)$ . This produces the lower bound

\boxed{ I(Z;S) \ge \mathbb E_{p(z,s)} \left[ \log q_\phi(z\mid s)-\log p(z) \right] }.

See derivation

Start from

I(Z;S) =\mathbb E_{p(z,s)} [\log p(z\mid s)-\log p(z)].

Add and subtract $\log q_\phi(z\mid s)$ :

\begin{aligned} I(Z;S) &=\mathbb E_{p(z,s)} [\log q_\phi(z\mid s)-\log p(z)]\\ &\quad+ \mathbb E_{p(s)} D_{\mathrm{KL}} \left( p(z\mid s)\|q_\phi(z\mid s) \right). \end{aligned}

Dropping it gives the variational lower bound.

For a fixed uniform prior, $-\log p(z)$ is constant, so the policy can use

\boxed{ r_{\mathrm{int}}(s,z)=\log q_\phi(z\mid s) }.

If every skill visits the same states, then $p(z\mid s)\approx p(z)$ and $I(Z;S)\approx0$ . The discriminator cannot identify the skill, so the policy receives little reward. Distinguishable behaviors increase the bound.

\textbf{Discriminator-Based Skill Discovery}

\begin{aligned} &\text{Initialize skill policy }\pi_\theta \text{ and discriminator }q_\phi.\\[0.25em] &\text{Repeat:}\\[0.25em] &\qquad z\sim p(z),\\[0.25em] &\qquad \text{roll out }\pi_\theta(a\mid s,z),\\[0.25em] &\qquad \phi\leftarrow \arg\max_\phi\sum_t\log q_\phi(z\mid s_t),\\[0.25em] &\qquad r_{\mathrm{int}}(s_t,z) \leftarrow\log q_\phi(z\mid s_t),\\[0.25em] &\qquad \text{update }\pi_\theta\text{ with intrinsic return}. \end{aligned}

The discriminator may exploit irrelevant visual cues, and the environment may contain few independently controllable behaviors. Diversity in discriminator space is not automatically useful diversity.

Empowerment Measures Control Authority

Novelty rewards reaching unfamiliar states. Empowerment rewards being able to choose among reliably different futures.

One-step empowerment is

\boxed{ I(A_t;S_{t+1}) =\mathcal H(S_{t+1}) -\mathcal H(S_{t+1}\mid A_t) }.

It is high when the agent can reach many future states and each action has a predictable, distinguishable consequence. Random environmental motion may increase state entropy, but it also increases conditional entropy and therefore does not imply control.

For temporally extended control, replace $A_t$ with a skill $Z$ and $S_{t+1}$ with $S_{t+H}$ :

I(Z;S_{t+H}).

Skill discovery can thus be interpreted as learning a compact channel from chosen latent behaviors to reachable outcomes.

Goals Turn Diversity into Reachability

Replace the abstract skill with a goal $g$ :

a_t\sim\pi_\theta(a_t\mid s_t,g).

The same information objective is

I(G;S)=\mathcal H(G)-\mathcal H(G\mid S).

The first term asks for diverse proposed goals. The second asks the achieved state to reveal which goal was intended. A learned discriminator may supply

r(s,g)=\log q_\phi(g\mid s),

while explicit state goals may use

r(s,g)=\mathbf 1\{\|s-g\|\le\varepsilon\}.

Sampling goals only from common achieved states causes the agent to rehearse what it already knows. Skew-Fit reweights achieved goals by inverse density:

w(g)=p_\psi(g)^\alpha, \qquad \alpha\in[-1,0),

and fits

\psi\leftarrow\arg\max_\psi \mathbb E_{g\sim p_{\mathrm{data}}} [w(g)\log p_\psi(g)].

The implied target density is

\boxed{ q_\alpha(g) \propto p(g)^{1+\alpha} }.

See derivation

Weighted samples from $p(g)$ contribute density proportional to

p(g)w(g) =p(g)p(g)^\alpha =p(g)^{1+\alpha}.

Since $\alpha\in[-1,0)$ ,

1+\alpha\in[0,1).

At $\alpha=-1$ , the idealized target is uniform over represented support.

\textbf{Skew-Fit Goal Exploration}

\begin{aligned} &\text{Fit a goal model }p_\psi(g)\text{ to achieved states}.\\[0.25em] &\text{Repeat:}\\[0.25em] &\qquad g\sim p_\psi(g),\\[0.25em] &\qquad \text{attempt }g\text{ with }\pi_\theta(a\mid s,g),\\[0.25em] &\qquad \text{update the goal-conditioned policy},\\[0.25em] &\qquad w(g_i)\leftarrow p_\psi(g_i)^\alpha,\\[0.25em] &\qquad \psi\leftarrow \arg\max_\psi\sum_iw(g_i)\log p_\psi(g_i). \end{aligned}

Rare achieved states become more likely future goals, expanding coverage outward. The generator may still propose unreachable goals, and uniform state coverage may be irrelevant to downstream tasks. Goal quality depends on a representation whose distances and densities reflect controllable structure.

From Diversity to Downstream Control

Skill discovery, empowerment, and goal-conditioned learning emphasize different aspects of the same channel:

\boxed{ \text{chosen latent or goal} \longrightarrow \text{temporally extended behavior} \longrightarrow \text{distinguishable reachable outcome} }.

Use discriminator-based skills when broad behavioral diversity is valuable before the task is known. Use empowerment when controllability matters more than raw novelty. Use goal-conditioned policies when desired outcomes can be represented directly and practiced through relabeling or adaptive goal proposals.

These objectives remain representation dependent. A discriminator may exploit irrelevant cues, a goal model may propose unreachable states, and uniform coverage may not align with future usefulness. Learned diversity becomes valuable only when downstream control can compose, select, or fine-tune the resulting behaviors.