Partial Observability

The agent never sees the true state, only a sensor reading that can look identical in different situations, so a policy that reads just the current reading fails where one that remembers history succeeds.

Two hidden states emit the same observation, so a policy reading only the current observation averages 0.0 while one that remembers history scores +1.0.

Why Does This Exist?

Every RL textbook starts with the Markov decision process: the agent observes the true state $s_t$ , picks an action, and the next state depends only on $s_t$ and the action. That assumption quietly breaks in the real world. A robot with a dusty camera, a game agent that sees one screen of pixels, or a trader who sees prices but not other traders' intentions never gets $s_t$ . They get an observation $o_t$ : a noisy, cropped, sometimes misleading glimpse.

When two different situations produce the same glimpse, the optimal action differs but the agent cannot tell them apart. That is the whole problem this page is about. You will see exactly where the MDP math stops applying, work the numbers on a corridor where one reading hides two positions, and learn why the fix is memory. The full machinery built on top of this idea (POMDPs, belief states, recurrent policies) gets its own pages; here you learn to spot the break.

Think of It Like This

Two identical hotel corridors

You wake up in a hotel corridor with beige walls, beige carpet, and no room numbers in sight. Your room is either to the left or to the right, and you genuinely cannot tell which corridor you are in: the north-wing corridor and the south-wing corridor were decorated identically.

If you must guess from this one glance, left and right are equally good, and you will be wrong half the time. But if you remember how you got here (you took the stairs up from the lobby on the north side), the identical corridors stop being identical: your history disambiguates them, and you walk straight to your room.

Where the analogy stops: a hotel guest remembers effortlessly, but an RL policy only remembers what its inputs include. A policy fed just the current camera frame has no stairs memory. History has to be built into the observation the policy sees, through stacked frames, an explicit memory vector, or a recurrent connection, or the agent stays lost in the beige corridor forever.

How It Actually Works

State, observation, and the observation function

Keep three symbols apart and the topic gets easy:

$s_t$ is the true state: everything that matters about the world at time $t$ (both corridor positions, including which side you are on).
$o_t$ is the observation: what your sensors actually report (one beige-corridor photo).
$O(o \mid s)$ is the observation function: the probability of seeing reading $o$ when the world is in state $s$ .

An MDP assumes $o_t = s_t$ . Partial observability means $o_t$ is drawn from $O$ , and the trouble starts when $O$ maps two states to the same reading. That collision has a name: perceptual aliasing. When $O(o \mid s_A) = 1$ and $O(o \mid s_B) = 1$ for one shared photo $o$ , the posterior $P(s \mid o)$ splits across both states and no function of $o$ alone can pick the right action for each. The standard fix is to condition on history $h_t = (o_0, a_0, o_1, a_1, \dots, o_t)$ instead of just $o_t$ : the policy becomes $\pi(a \mid h_t)$ , and the past readings carry the missing bit.

Worked example: the aliased corridor

Make it concrete. Two positions, one photo:

Hidden states: $s_A$ (north side) and $s_B$ (south side). Both emit the same photo $o$ ("grey corridor"), so $O(o \mid s_A) = O(o \mid s_B) = 1$ .
You visit each side equally often, so on seeing $o$ the posterior is $P(s_A \mid o) = P(s_B \mid o) = 0.5$ .
Actions are left and right. Rewards: in $s_A$ , left pays $+1$ and right pays $-1$ ; in $s_B$ , it flips (left pays $-1$ , right pays $+1$ ).

Now score a memoryless policy $\pi(a \mid o)$ , which must pick the same way in both states:

Always go left: $0.5 \times (+1) + 0.5 \times (-1) = 0.0$ .
Always go right: $0.5 \times (-1) + 0.5 \times (+1) = 0.0$ .
Any coin flip between them: still $0.0$ , because the two states cancel exactly at every mixing probability.

Now score a history-conditioned policy $\pi(a \mid h_t)$ that remembers which door it entered through (north door means $s_A$ , south door means $s_B$ ), going left from the north side and right from the south side:

Expected return: $0.5 \times (+1) + 0.5 \times (+1) = +1.0$ .

One bit of memory is worth a full point of return here: $+1.0$ versus $0.0$ . That gap is the price of pretending the observation is the state.

Code

A tiny check of the corridor arithmetic: every memoryless mixing probability scores $0.0$ , while the history-conditioned policy scores $+1.0$ .

1from typing import Dict, Tuple2
3Payoffs = Dict[Tuple[str, str], float]4
5REWARDS: Payoffs = {6    ("A", "left"): 1.0,7    ("A", "right"): -1.0,8    ("B", "left"): -1.0,9    ("B", "right"): 1.0,10}11
12
13def memoryless(p_left: float, prior_a: float = 0.5) -> float:14    """Expected return of a policy that picks left with probability p_left on photo o."""15    prior_b = 1.0 - prior_a16    from_a = p_left * REWARDS[("A", "left")] + (1.0 - p_left) * REWARDS[("A", "right")]17    from_b = p_left * REWARDS[("B", "left")] + (1.0 - p_left) * REWARDS[("B", "right")]18    return prior_a * from_a + prior_b * from_b19
20
21def history_conditioned(prior_a: float = 0.5) -> float:22    """Expected return when the entry door disambiguates A (go left) from B (go right)."""23    return prior_a * REWARDS[("A", "left")] + (1.0 - prior_a) * REWARDS[("B", "right")]24
25
26for prob in (0.0, 0.5, 1.0):27    print(f"memoryless p(left)={prob:.1f} -> {memoryless(prob):+.1f}")  # -> +0.0 every time28print(f"history-conditioned  -> {history_conditioned():+.1f}")  # -> +1.0

1memoryless p(left)=0.0 -> +0.02memoryless p(left)=0.5 -> +0.03memoryless p(left)=1.0 -> +0.04history-conditioned  -> +1.0

Watch Out For

Bootstrapping Q(o, a) as if the photo were the state

Symptom: you run Q-learning or any temporal-difference update on raw observations, and the values for one observation swing up and down instead of settling. That oscillation is two states' returns ( $+1$ from $s_A$ , $-1$ from $s_B$ ) fighting inside a single $Q(o, a)$ entry, which converges toward their $0.0$ average and teaches the agent that nothing matters.

Fix: check the diagnosis first by grouping the returns that followed the same observation. If identical observations precede sharply different returns, you have aliasing, not noise. Then give the value function memory: stack recent frames, append the last actions to the input, or switch to a recurrent policy, so $Q$ is a function of history instead of one photo.

Stacking more frames when the sensor never saw the answer

Symptom: you lengthen the history window from 4 frames to 16 and the agent is still stuck, while training gets slower. Longer memory only helps if the missing information appeared in some past observation. If the key fact (which door you entered, what is behind you) never entered any frame, no window length recovers it.

Fix: fix the observation, not the window. Add the missing sensor (a rear camera, an inventory flag, the previous action as input), or maintain an explicit memory state the agent can write to. When in doubt, ask: could a human looking at exactly these stacked frames tell the states apart? If not, neither can the agent.

The Quick Version

Partial observability means the agent sees an observation $o_t$ drawn from $O(o \mid s)$ , not the true state $s_t$ , so the MDP assumption $o_t = s_t$ stops holding.
Perceptual aliasing is when two states emit the same reading; no policy reading only the current observation can act optimally in both.
In the worked corridor, every memoryless policy averages $0.0$ while remembering the entry door scores $+1.0$ , so one bit of history is worth a full point of return.
The standard repair is conditioning on history, $\pi(a \mid h_t)$ , via stacked frames, history features, or recurrent policies.
Same observation followed by sharply different returns is the signature of aliasing; treat it as a sensor and memory problem, not as noise to explore away.