Value Functions

A value function is a single number per situation: how much future reward you can expect from here on. It turns far-off consequences into a choice you can make right now.

A reward of 100 backs up a three-state chain with discount 0.9, settling state values at 81.0, 90.0 and 100.0

Why Does This Exist?

An agent that chases immediate reward walks into traps: the tasty move now leads to disaster later. The goal is the total future reward, but totals arrive too late to steer by. You need a number, available now, that already contains the future.

Value functions are that number. Each state, or each state action pair, maps to the expected return from there on, so choosing well means picking the neighbour with the highest value. Planning far ahead collapses into comparing a few numbers.

This page defines the two value functions and the discount, with one hand-computed backup chain. The recursion they satisfy is Bellman equations, the setup they live in is basic concepts, and the networks that learn them are deep Q-networks.

The one idea you need first

The return $G_t$ is the sum of discounted future rewards from step $t$ : $R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3}$ onward. The discount $\gamma$ in $[0, 1]$ shrinks later rewards, so the sum stays finite and sooner rewards count more. Values are expectations of this return.

Think of It Like This

A chess player who prices positions, not pieces

Capturing a pawn feels good now. A strong player ignores that feeling and prices the position: this board is worth a likely win, that one a likely loss. They will sacrifice a whole knight for a position that forces checkmate in five moves, because the position's price already includes those five moves.

State values are position prices. Action values price each candidate move from the current position. Once every reachable position has a price tag, the player stops calculating variations and simply steps toward the highest tag.

Where the analogy stops: chess ends, so prices can be settled by working backward from checkmate. Many agent tasks never end, which is why the discount exists: without it, infinite futures would have infinite prices.

How It Actually Works

The state value $V^\pi(s)$ is the expected return from state $s$ while following policy $\pi$ thereafter. Here $\pi$ is the agent's rule for picking actions, and the expectation averages over the randomness of the world and the policy:

$V^\pi(s) = \mathbb{E}_\pi\left[\left. \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} \right| S_t = s \right]$

The action value $Q^\pi(s, a)$ fixes the first move to $a$ and follows $\pi$ after that. The difference matters for control: with $Q$ you act by picking the highest number, no world model needed. With only $V$ you must predict which action leads to the best next state, which requires knowing how the world responds.

The optimal values $V^*$ and $Q^*$ belong to the best achievable policy. Most of reinforcement learning is a hunt for one of these two objects, and values are never absolute: change the policy and every number moves, except at the optimum.

Worked example

Three states in a chain, $S_0 \to S_1 \to S_2 \to \mathrm{End}$ , with rewards $0$ , $0$ , and $100$ on the last step, and $\gamma = 0.9$ . Start all values at zero and apply $V(s) = R + \gamma V(s')$ three times.

Sweep 1: only $S_2$ sees reward, $V(S_2) = 100.0 + 0.9 \times 0.0 = 100.0000$ . Others stay zero.

Sweep 2: $V(S_1) = 0.0 + 0.9 \times 100.0 = 90.0000$ . $S_0$ still sees only zeros.

Sweep 3: $V(S_0) = 0.0 + 0.9 \times 90.0 = 81.0000$ .

Final values: $81.0$ , $90.0$ , $100.0$ . The reward needed one sweep per step to travel backward, which is exactly why one-step methods learn slowly over long delays.

Code

The backup chain in plain Python, with type hints, matching the numbers above:

1values: dict[str, float] = {"S0": 0.0, "S1": 0.0, "S2": 0.0, "End": 0.0}2transitions: dict[str, tuple[str, float]] = {3    "S0": ("S1", 0.0),4    "S1": ("S2", 0.0),5    "S2": ("End", 100.0),6}7gamma = 0.98
9for _ in range(3):10    values = {11        s: 0.0 if s not in transitions12        else transitions[s][1] + gamma * values[transitions[s][0]]13        for s in values14    }15
16print(f"V(S0) {values['S0']:.1f}")  # -> V(S0) 81.017print(f"V(S1) {values['S1']:.1f}")  # -> V(S1) 90.018print(f"V(S2) {values['S2']:.1f}")  # -> V(S2) 100.0

Watch Out For

Reading V as if it tells you what to do

$V(s)$ prices the situation under the current policy, but it never names a move. Agents wired to "take the action with the best V" are stuck, because V has no action slot. The symptom is a planner that evaluates positions beautifully and then picks moves at random.

Fix it by matching the object to the job: learn $Q(s, a)$ for model-free control, or pair $V(s)$ with a model that predicts next states so you can compare them.

Forgetting values belong to a policy

There is no true value of a state on its own, only the value under some way of behaving. Improve the policy and yesterday's values are stale; evaluate a new policy with old numbers and the agent defends positions its new behavior has abandoned. The symptom is evaluation scores that disagree with actual rollouts after every policy change.

Fix it by re-estimating values whenever the policy moves, and by treating $V^*$ as the only policy-free object on the page.

The Quick Version

A value function turns the whole future into one number per state or state action pair: the expected discounted return from there.
$V^\pi(s)$ prices the situation under a policy; $Q^\pi(s, a)$ prices one move first, which is what model-free control needs.
The discount $\gamma$ keeps infinite futures finite and makes sooner rewards count more.
In the worked chain, reward 100 with discount 0.9 backs up to values 81.0, 90.0 and 100.0, one sweep per step.
Values belong to a policy, so they go stale the moment behavior changes; only the optimal values are policy-free.