State is different from observation

The Markov property says that a sufficient current state and action screen off earlier history from the future. Physical dynamics can be Markov while a single image omits velocity. Partial observation creates a POMDP; using an RNN does not itself change the environment’s dynamics.

History, belief, and hidden state

A policy can condition on Ht=(o0,a0,…,ot)H_t=(o_0,a_0,\ldots,o_t). Under appropriate model assumptions, the belief bt(s)=P(st=s∣Ht)b_t(s)=P(s_t=s\mid H_t) is a recursively updateable sufficient statistic. An RNN compresses history through ht=f(ht−1,ot,at−1,rt−1)h_t=f(h_{t-1},o_t,a_{t-1},r_{t-1}). It is a learned approximation, not automatically an exact belief or Markov state.

The expanded system (st,ht)(s_t,h_t) may have a recursive state description, while the agent-visible hth_t can still omit predictive information. Memory capacity and restored observability are separate claims.

Reset semantics

Episode, task, and trial resets may differ. Ordinary independent episodes often reset memory; RL²-style adaptation retains information across episodes of one task and clears it when the task changes. Parallel environments require separate memories.

Evaluation

Compare memoryless, stacked-observation, and recurrent policies with matched budgets and inputs. Test occlusion, hidden velocity, delays, and task switches; report reset rules and recovery speed. Hidden-state adaptation is distinct from test-time gradient fine-tuning.

Related reading: Meta-RL, RL², and occupancy measures.