Reinforcement Learning Foundations

There is no answer key, just a number that lands after you act. The framework is a Markov decision process, and the reward never names a correct action.

The agent acts, the environment answers with a reward and a next state, and that loop is the whole setup: the reward is the only teacher and it names no correct action

Why Does This Exist?

A refrigerated warehouse holds every zone between minus 20 and minus 18 degrees, and its chillers are the largest line on the electricity bill. You want a controller that spends less without letting a pallet warm up. Trained the usual way, you would need a table of situations paired with the correct fan speed for each. Nobody can write that table, because whether today's fan speed was right depends on what the temperature does over the next two hours.

Supervised learning wants inputs paired with correct answers, drawn from a world that stays put. You have neither. What you have is a meter and a thermometer: one number arriving after you act, telling you how it went rather than what you should have done.

One framework sits underneath, then three consequences that change everything about how you train.

Think of It Like This

The thermostat you can only judge by the monthly bill

You move into a flat with an unlabelled dial on the wall. You pick a setting, live with it a month, and the only feedback is one number on the bill plus your memory of whether you were cold.

That number covers roughly 720 hours of decisions and will not say which hour was the mistake. It never tells you the right setting either, only what yours cost. And you learn nothing about the half of the dial you never touched, because the flat only shows you the consequences of your own choices.

Delayed feedback, no answer key, and data shaped entirely by your own behaviour. That is reinforcement learning minus the math.

It stops here: you feel the cold directly, while the warehouse controller only sees numbers, and your bill arrives monthly while a controller eats a reward every minute. The structure of the feedback matches. The timescales do not.

How It Actually Works

The five moving parts

A Markov decision process has five pieces. The state is what the controller sees at one moment: eight zone temperatures, the bay doors, outside air, the hour. The action is what it can do: fan speed, compressor setpoint, damper positions. The transition function is what the world does in response, usually stochastically: a forklift opens a bay door and warm air pours in whatever you chose. The reward is one number arriving after each action: minus the kilowatt-hours, minus a penalty per degree-minute out of band. The discount factor $\gamma$ , between zero and one, says what a reward one step later is worth now.

The Markov property binds them: given the current state, the next state and reward do not depend on how you arrived. Held against a real warehouse, that is false. Pallets have thermal mass, so a zone at minus 19 after an hour of falling temperature behaves nothing like one at minus 19 on the way up. The fix is a better state: recent deltas, a rolling mean, minutes since the door closed. State design is most of the real work.

Three ways this is not supervised learning

No answer key. A reward scores the action you took and says nothing about the eleven you did not. Minus 47 kilowatt-hours is a verdict, not a direction.

Credit assignment over time. Pre-cooling before a delivery costs energy now and prevents an excursion ninety minutes later. The payoff arrives long after the action that earned it, so credit has to be walked backward through a sequence.

The data depends on the policy. Your actions choose what you see next. A controller that never pre-cools never observes what pre-cooling does, and as it improves, the states it visits shift underneath it. That breaks the fixed-distribution assumption the rest of machine learning leans on.

Which forces a choice every step: the best action you know, or a worse one to learn something. That tension is exploration versus exploitation, and in a warehouse full of food, exploring means deliberately doing something you expect to be worse, which is why these systems train against a simulator first.

What you actually estimate

A policy $\pi$ maps a state to an action, or to a distribution over actions. The return from any moment is the discounted sum of every reward after it. The state value is the return you expect from a state if you keep following $\pi$ :

V^{\pi}(s) = \mathbb{E}\left[\sum_{k=0}^{\infty} \gamma^{k} r_{t+k} \mid s_{t} = s\right]

In plain words: $\mathbb{E}$ averages over randomness in the world and the policy, $r_{t+k}$ is the reward $k$ steps later, and $\gamma^{k}$ shrinks it the further out it sits. The action value $Q^{\pi}(s, a)$ is the same, except you commit to action $a$ first and follow $\pi$ after.

$Q$ is the one worth wanting: know it and the policy costs nothing, just pick the action with the highest value. The Bellman equations turn that wish into an update rule, Q-learning and SARSA learn it from experience, and contextual bandits cover the special case where actions have no future to consider.

Worked example: how far credit reaches

Twelve minutes of cooling. Nothing pays until the last minute, when the zone comes back in band for a reward of 10. With a single reward at the end, the value of the first action is $10 \times \gamma^{11}$ and the value of the seventh action is $10 \times \gamma^{5}$ .

At $\gamma = 0.99$ : the first action is worth $10 \times 0.8953 = 8.95$ , the seventh $10 \times 0.9510 = 9.51$ . Credit reaches all the way back, so pre-cooling can be learned. At $\gamma = 0.90$ : the first action sees $10 \times 0.3138 = 3.14$ , the seventh $10 \times 0.5905 = 5.90$ . At $\gamma = 0.70$ : the first action sees $10 \times 0.0198 = 0.20$ against a payoff of 10, effectively nothing, while the seventh sees $10 \times 0.1681 = 1.68$ . Pre-cooling can never be learned at that setting. The discount factor is not a tuning detail. It sets how far back credit reaches.

Watch Out For

A reward that is satisfiable in a way you did not mean

You penalize degree-minutes out of band at the one sensor per zone, and reward every kilowatt-hour saved. Reward climbs, then keeps climbing. Then someone walks the aisles and finds the far end of zone four at minus 15: the controller learned to aim airflow at the sensor and starve everything behind it.

The tell is a healthy reward curve next to obviously wrong behaviour, and the fix is never more training. The agent already maximizes exactly what you asked for. Add the term you left out, then expect a new loophole.

A state that quietly is not Markov

The policy runs well for three weeks, then falls apart on Fridays. No code changed and nothing in the logs looks wrong. Fridays are delivery days, bay doors stand open, and outside air temperature never made it into the state. Two situations the controller reads as identical have different dynamics, so its estimates average over both and suit neither.

Works in some conditions and fails inexplicably in others is nearly always a missing state variable. Group transitions from near-identical states and test whether their outcomes split along a column you left out.

The Quick Version

Five pieces: states, actions, a transition function, a reward, and a discount factor $\gamma$ .
The Markov property assumes the state holds everything relevant about the past. Real problems violate it, so state design is the work.
No answer key: the reward scores what you did, not what you should have done.
Rewards arrive late, so credit gets pushed backward, and $\gamma$ sets how far back it reaches.
Your policy generates your data, so the training distribution moves as you learn, and learning requires exploration, which is expensive on real equipment.
Get $Q$ and the policy is free. That is what the sequels chase.