An explicit goal variable

Goal-conditioned RL uses π(a∣s,g)\pi(a\mid s,g), V(s,g)V(s,g), and Q(s,a,g)Q(s,a,g) with goal-dependent reward r(s,a,s′,g)r(s,a,s',g). Goals may be coordinates, images, or language, but task completion must be defined operationally. A UVFA shares value approximation across goals; input conditioning alone does not establish generalization.

Hindsight relabeling

HER reinterprets transitions using achieved goals and recomputes their rewards and termination information. Failure under the original goal can supply success examples for another goal. The physical transition must remain valid after relabeling; goal-dependent dynamics can invalidate naive reuse.

Future-goal selection in stochastic environments can introduce hindsight bias. Relabeling rules, correction, and algorithm assumptions deserve separate attention. Compatibility with replay does not make every reward or goal encoding interchangeable.

Operation When What changes
Goal/subgoal selection During collection Behavior and visitation
Hindsight relabeling After collection Training interpretation of recorded data
Hierarchical control Across decision timescales High-/low-level interfaces

Goal conditioning and temporal abstraction can be combined but are different concepts. Contrastive formulations connect future reachability to representations under particular sampling and modeling assumptions.

Evaluation

Fix training/test goal splits and report success, time to goal, collisions or limit violations, and distance. Separate interpolation, OOD goals, and new skills: reaching a new coordinate is not evidence of an entirely new task capability.

Related reading: options, Meta-RL, and abstraction.

Selected references from the source notes