Core ML

Reinforcement Learning Foundations

There's no answer key, just a number that lands after you act. The framework is a Markov decision process, and the reward never names a correct action.

The agent acts, the environment answers with a reward and a next state, and that loop is the whole setup — the reward is the only teacher and it names no correct action

Why Does This Exist?

A refrigerated warehouse holds every zone between minus 20 and minus 18 degrees, and its chillers are the largest line on the electricity bill. You want a controller that spends less without letting a pallet warm up. Train it the usual way and you'd need a table of situations paired with the correct fan speed for each. Nobody can write that table, because whether today's fan speed was right depends on what the temperature does over the next two hours.

Supervised learning wants inputs paired with correct answers, drawn from a distribution that stays put. You have neither. What you have is a meter and a thermometer — a number arriving after you act, telling you how it went rather than what you should have done.

One framework underneath, then three consequences.

Think of It Like This

The thermostat you can only judge by the monthly bill

You move into a flat with an unlabelled dial on the wall. You pick a setting, live with it a month, and the only feedback is one number on the bill plus your memory of whether you were cold.

That number covers roughly 720 hours of decisions and won't say which hour was the mistake. It never tells you the right setting either, only what yours cost. And you learn nothing about the half of the dial you never touched, because the flat only shows you consequences of your own choices.

Delayed feedback, no answer key, and data shaped entirely by your own behaviour.

How It Actually Works

The five moving parts

A Markov decision process has five pieces.

The state is what the controller sees at one moment: eight zone temperatures, the bay doors, outside air temperature, the hour. The action is what it can do — fan speed, compressor setpoint, damper positions. The transition function is what the world does in response, usually stochastically: a forklift opens a bay door and warm air pours in whatever you chose. The reward is one number arriving after each action; here, minus the kilowatt-hours, minus a penalty per degree-minute out of band. The discount factor $\gamma$ , between zero and one, says what a reward one step later is worth now. Near one, the controller plans hours ahead; at 0.9 it barely sees past a few minutes.

The Markov property binds them: given the current state, the next state and reward don't depend on how you arrived. Held against a real warehouse, that's false. Pallets have thermal mass, so a zone at minus 19 after an hour of falling temperature behaves nothing like one at minus 19 on the way up. The fix is a better state — recent deltas, a rolling mean, minutes since the door closed. State design is most of the real work.

Three ways this isn't supervised learning

No answer key. A reward scores the action you took and says nothing about the eleven you didn't. Minus 47 kilowatt-hours is a verdict, not a direction.

Credit assignment over time. Pre-cooling before a delivery costs energy now and prevents an excursion ninety minutes later. The payoff arrives long after the action that earned it, so credit has to be walked backward through a sequence.

The data depends on the policy. Your actions choose what you see next. A controller that never pre-cools never observes what pre-cooling does, and as it improves, the mix of states it visits moves underneath it — a distribution shift you caused yourself. That breaks the fixed-distribution assumption everything else here relies on.

Which forces a choice every step: the best action you know, or a worse one to learn something. Exploitation and exploration. In a warehouse full of food, exploring means deliberately doing something you expect to be worse, which is why these systems train against a simulator first.

What you actually estimate

A policy $\pi$ maps a state to an action, or to a distribution over actions. The return from any moment is the discounted sum of every reward after it. The state value is the return you expect from a state if you keep following $\pi$ :

V^{\pi}(s) = \mathbb{E}\left[\sum_{k=0}^{\infty} \gamma^{k} r_{t+k} \mid s_{t} = s\right]

$\mathbb{E}$ averages over randomness in the world and the policy, $r_{t+k}$ is the reward $k$ steps later, and $\gamma^{k}$ shrinks it. The action value $Q^{\pi}(s, a)$ is the same, except you commit to action $a$ first.

$Q$ is the one worth wanting: know it and the policy costs nothing. The next two pages are two answers to how you get it.

Show Me the Code

Discounting is what lets a delayed reward reach the action that caused it.

1import numpy as np2
3
4def discounted_returns(rewards: np.ndarray, gamma: float) -> np.ndarray:5    """Credit runs backward: what each step is worth includes everything after it."""6    out = np.zeros_like(rewards, dtype=float)7    running = 0.08    for t in range(len(rewards) - 1, -1, -1):  # backward, so each step reuses the next9        running = float(rewards[t]) + gamma * running10        out[t] = running11    return out12
13
14# Twelve minutes of cooling. Only the last minute pays: the zone came back in band.15rewards = np.zeros(12)16rewards[-1] = 10.017
18for gamma in (0.99, 0.90, 0.70):19    g = discounted_returns(rewards, gamma)20    print(f"gamma {gamma:.2f}: the first action is worth {g[0]:5.2f}, the seventh {g[6]:5.2f}")21    # -> gamma 0.99: the first action is worth  8.95, the seventh  9.5122    # -> gamma 0.90: the first action is worth  3.14, the seventh  5.9023    # -> gamma 0.70: the first action is worth  0.20, the seventh  1.68

At $\gamma = 0.7$ the first action sees 0.20 of a payoff worth 10 — effectively nothing, so pre-cooling can never be learned. The discount factor isn't a tuning detail. It sets how far back credit reaches.

Watch Out For

A reward that's satisfiable in a way you didn't mean

You penalise degree-minutes out of band at the one sensor per zone, and reward every kilowatt-hour saved. Reward climbs, then keeps climbing. Then someone walks the aisles and finds the far end of zone four at minus 15: the controller learned to aim airflow at the sensor and starve everything behind it.

The tell is a healthy reward curve next to obviously wrong behaviour, and the fix is never more training: the agent already maximises exactly what you asked for. Add the term you left out, then expect a new loophole.

A state that quietly isn't Markov

The policy runs well for three weeks and then falls apart on Fridays. No code changed, and nothing in the logs looks wrong.

Fridays are delivery days, bay doors stand open, and outside air temperature never made it into the state. Two situations the controller reads as identical have different dynamics, so its estimates average over both and are right for neither. Works in some conditions, inexplicably fails in others — that pattern is nearly always a missing state variable. Group transitions from near-identical states and test whether their outcomes split along a column you left out.

The Quick Version

Five pieces: states, actions, a transition function, a reward, a discount factor $\gamma$ .
The Markov property assumes the state holds everything relevant about the past. Real problems violate it, so state design is the work.
No answer key — the reward scores what you did, not what you should have done.
Rewards arrive late, so credit gets pushed backward to the actions that earned it.
Your policy generates your data, so the training distribution moves as you learn. And you have to explore, which is expensive on real equipment.
Get $Q$ and the policy is free. That's what the next two pages chase.