Defining an option

An option is o=(Io,πo,βo)o=(I_o,\pi_o,\beta_o): an initiation set, internal policy, and probabilistic termination rule. A primitive action is a one-step terminating special case. A high-level policy chooses options while internal policies generate physical actions.

SMDP timing

An option lasting τ\tau steps uses a duration-dependent target:

Q(s,o)←Q(s,o)+α[∑k=0τ−1γkrt+k+γτmax⁡o′Q(st+τ,o′)−Q(s,o)].Q(s,o)\leftarrow Q(s,o)+\alpha\left[\sum_{k=0}^{\tau-1}\gamma^k r_{t+k}+\gamma^\tau\max_{o'}Q(s_{t+\tau},o')-Q(s,o)\right].

Different durations cannot all use a single-step discount. Genuine task termination removes the bootstrap. Markov options depend on current state; more general semi-Markov options can use execution history.

Learning within an option

Intra-option learning uses intermediate transitions rather than waiting for completion and may update other options compatible with the observed action. A typical continuation value is

U(s′,o)=(1−βo(s′))Q(s′,o)+βo(s′)max⁡o′Q(s′,o′).U(s',o)=(1-\beta_o(s'))Q(s',o)+\beta_o(s')\max_{o'}Q(s',o').

The target is r+γU(s′,o)r+\gamma U(s',o). Reuse requires policy compatibility or appropriate correction; a transition is not automatically unbiased for every option. Option discovery, termination learning, and fixed-option value estimation are different problems.

Robot interfaces

Specify high-level interruption rules, skill success/failure, accumulated rewards, occupied action time, and memory resets. A faster skill does not necessarily imply more frequent high-level decisions.

Related reading: goal-conditioned learning, abstraction, and research design.

Selected references from the source notes