Evaluation & RL

Policy Gradient

Watch an agent learn a grid world by directly updating its action probabilities. See the probabilities shift after each episode via REINFORCE, and add a baseline to reduce variance.

Agent Trajectory
Action Probability

Episode0

Current Return (G)0.00

A grid world with walls, a goal, and a penalty tile. The agent interacts with states and receives rewards.

Challenge

Can you reach a deterministic policy (one action with near 100% probability) for the shortest path?

Let the agent run enough episodes with REINFORCE and see if a baseline helps it converge faster.

Check your understanding

1 questions in the bank. Each attempt draws a fresh set in a fresh order, so a second go is a real second go.

In many Reinforcement Learning problems, it's easier to directly learn which action to take rather than estimating the exact value of taking it. Policy Gradient methods do exactly this: they parameterize the policy directly and adjust probabilities to maximize expected reward.

REINFORCE

The simplest policy gradient algorithm is REINFORCE. The agent runs a full episode, collects the sequence of states, actions, and rewards, and then looks at the total return (cumulative reward) from each step.

If the return was high, it increases the probability of the actions it took. If the return was low or negative, it decreases them.

\theta \leftarrow \theta + \alpha \gamma^t G_t \nabla \ln \pi(a|s)

The key part is $G_t$ , the actual return obtained from time $t$ onwards.

Adding a Baseline (Actor-Critic)

A major problem with REINFORCE is high variance. Because returns depend on the entire random trajectory, they can fluctuate wildly, causing the policy to take huge, erratic steps.

To fix this, we subtract a baseline from the return. The most common baseline is the state-value function $V(s)$ (the Critic). Instead of asking "was the return good?", we ask "was the return better than expected?"

This difference, $G_t - V(s_t)$ , is called the Advantage. By using it to scale our updates, the policy (the Actor) learns much more stably.

Reference

REINFORCE: θ ← θ + α γ^t G_t ∇ln π(a|s)
Baseline: Subtracting V(s) from G_t reduces variance without biasing the gradient.
Softmax Policy: π(a|s) = exp(θ_a) / Σ exp(θ_b)

Break it on purpose

A high learning rate without a baseline causes probabilities to saturate prematurely and collapse into a suboptimal policy.