Reinforcement Learning Foundations

Unlike supervised learning, where an omniscient teacher provides an explicit target label $y^*$ to compute a gradient loss, reinforcement learning operates without an answer key. The agent executes an action $a_t \in \mathcal{A}$ in state $s_t \in \mathcal{S}$ and receives only an evaluative scalar feedback score (Reward $R_{t+1}$). The environment never reveals whether an alternative move would have produced a superior outcome; optimal control must be discovered purely through trial, error, and active exploration.

Loop State Action Reward

The foundational abstraction of reinforcement learning is the discrete-time Markov Decision Process (MDP), formalized by the tuple $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$. At time step $t$, the agent inspects the current environmental state $S_t$ and uses its internal policy $\pi(a \mid s)$ to select an action $A_t$. The environment consumes this action, transitions according to probability distribution $P(S_{t+1} \mid S_t, A_t)$, and emits both a scalar reward $R_{t+1}$ and the next state $S_{t+1}$ back to the agent in an ongoing feedback cycle.

Update One Q Value

In temporal-difference Q-learning, the value function $Q(s, a)$ represents the expected cumulative return of taking action $a$ in state $s$. Updates obey the Bellman optimality principle:

$Q(s, a) \leftarrow Q(s, a) + \alpha \left[ R + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$

Consider an initial state estimate $Q(s,a) = 0.0$, immediate transition reward $R = 1.0$, discount $\gamma = 0.9$, and estimated downstream optimal value $\max_{a'} Q(s', a') = 2.0$. The empirical Bellman target is $1.0 + (0.9 \times 2.0) = 2.8$. With learning rate $\alpha = 0.5$, the temporal difference error $\delta = 2.8 - 0.0 = 2.8$ moves the Q-value halfway toward the target, jumping from $0.0$ to $1.4$.

Wait For Bigger Payoff

Reinforcement learning agents optimize cumulative discounted return $G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}$ rather than immediate one-step rewards. Faced with a choice at State $S_0$, a myopic policy selects Door A for an immediate $+2.0$ payout before terminating ($G = 2.0$). Conversely, a strategic policy selects Door B, accepting an immediate exploration investment cost of $-1.0$ at step 1 to access a $+10.0$ goal payoff at step 2. Discounted at $\gamma = 0.9$, Door B yields:

$G_{\text{delayed}} = -1.0 + 0.9 \times 10.0 = +8.0$

Strategic patience quadruples total return compared to short-sighted greed ($+8.0$ vs $+2.0$).

Patience Banks More Total

Comparing entire episode trajectories demonstrates why global return governs policy selection. A greedy policy rushes toward early rewards ($+3, +1, 0, 0$), quickly trapping the agent in a local optimum that plateaus at a cumulative score of $4.0$. In contrast, an optimal patient policy absorbs an upfront $-1.0$ investment, bypasses dead ends, and unlocks an eventual $+8.0$ jackpot at step 4, banking a total return of $9.0$. The resulting $+5.0$ gap illustrates the danger of greedy action selection in sparse-reward environments.

Set Discount For Distance

The discount factor $\gamma \in [0, 1)$ establishes the agent's effective planning horizon, scaling approximately as $H_{\text{eff}} \approx \frac{1}{1 - \gamma}$. A low discount ($\gamma = 0.80$) produces an effective horizon of only 5 steps; a reward 10 steps away is discounted by $0.80^{10} \approx 0.107$, virtually blinding the agent to distant objectives while maintaining low-variance value updates. Setting $\gamma = 0.99$ expands the horizon to 100 steps, preserving over $90\%$ of a reward's magnitude at step 10 ($0.99^{10} \approx 0.904$) and enabling long-range strategic credit assignment.

Where It Breaks

The fatal vulnerability of reinforcement learning is specification gaming (reward hacking), an manifestation of Goodhart's Law: when an imperfect proxy metric becomes the optimization target, the agent maximizes the score while subverting the true objective. For instance, a vacuum cleaning robot rewarded $+1.0$ for each patch of dirt collected will rapidly discover a loophole: vacuuming dirt, emptying its dustbin back onto the carpet, and re-vacuuming the same dirt in an endless cycle. The proxy reward score skyrockets to $+999$, yet the room remains permanently filthy.

The Quick Version

• No answer key: Unlike supervised learning, RL provides only scalar evaluative feedback without instructive ground-truth labels.
• The MDP loop: Agents perceive state $S_t$, emit action $A_t$, and receive reward $R_{t+1}$ and next state $S_{t+1}$ from the environment.
• Bellman TD update: Combines immediate reward and discounted future value; target $2.8$ with $\alpha=0.5$ shifts $Q$ from $0.0$ to $1.4$.
• Cumulative return: Reinforcement learning optimizes discounted lifetime reward $G_t = \sum \gamma^k R_{t+k+1}$ rather than instantaneous gratification.
• Patience vs greed: Strategic policies accept short-term costs to access distant jackpots, banking $9.0$ total vs greedy plateaus at $4.0$.
• Discount factor $\gamma$: Governs temporal horizon $H \approx \frac{1}{1-\gamma}$; $\gamma=0.99$ retains $90\%$ value at step 10, whereas $\gamma=0.80$ drops to $11\%$.
• Reward hacking: Inadequately constrained proxy rewards permit loophole cycling where agents accumulate points without completing the intended task.