Goal-Conditioned RL

A single goal-conditioned policy learns to reach any goal, turning many tasks into one parameterized problem.

One policy, many goals

Goal-conditioned RL trains a policy pi(a | s, g) and value function that take a goal g as an extra input. Instead of solving each task separately, the agent learns to reach arbitrary goals from the same state space. This unifies a family of tasks and lets experience toward one goal inform the pursuit of others.

Universal value functions

The core object is the Universal Value Function Approximator (UVFA): a value function V(s, g) that generalizes across goals via a shared network. Because goals and states often live in the same space, the network can interpolate to goals never explicitly trained, giving zero-shot generalization to new targets.

Sparse reward and relabeling

Goals usually come with sparse reward: 1 (or 0) for reaching the goal, nothing otherwise. This is where hindsight relabeling is transformative. Any trajectory reaches some state, so relabeling that state as the goal yields a success example, letting the agent learn from every episode regardless of whether it hit the intended goal. Hindsight Experience Replay is the standard implementation.

# Goal-conditioned Q-learning target
y = r(s2, g) + gamma * max_a Q(s2, a, g)
# reward is typically:  r(s, g) = 0 if ||s - g|| < eps else -1

Uses and links

• Robotics: reach, push, or place objects at commanded configurations
• Hierarchy: a manager sets subgoals that a goal-conditioned worker achieves
• Exploration: automatically proposing achievable goals builds a curriculum

Goal-conditioned RL connects tightly to hierarchical RL (subgoals), successor features (reward is a goal indicator), and curriculum learning (proposing goals of the right difficulty). It reframes control as a general goal-reaching skill rather than a collection of separate policies.