Reinforcement Learning
MDPs, policy vs value methods, exploration strategies, and RL algorithms
Foundations
- Reinforcement Learning
Fundamentals of Reinforcement Learning
An agent tries actions, banks rewards, and learns a policy that maximizes long term return. Delayed rewards make credit assignment the core problem.
- Reinforcement Learning
Basic Concepts in Reinforcement Learning
An agent tries actions, gets rewards, and learns a policy that maximizes total future reward. This page names the five moving parts and shows how discounting weighs the future.
- Reinforcement Learning
Markov Decision Processes
An MDP is the rulebook for learning by trial and error: state, action, reward, next state, repeated so the agent can chase long term return.
- Reinforcement Learning
Transition Dynamics
Transition dynamics are the environment's physics written as probabilities: for each state and action, they say how likely every possible next state is.
- Reinforcement Learning
Reward Function
The reward function turns each step into one number, and the agent climbs the total. Get that number wrong and the agent learns the wrong job perfectly.
- Reinforcement Learning
Policy in Reinforcement Learning
A policy is the agent rule for picking actions, either one fixed move per state or a probability mix that keeps exploring.
- Reinforcement Learning
Dynamic Programming in RL
You know the rules of the game, so you work backward from the end instead of playing thousands of practice rounds. Each state's value comes from its successors.
- Reinforcement Learning
Policy Evaluation
Given a fixed policy, policy evaluation scores every state by the return that policy earns from there, sweeping Bellman backups until the numbers stop moving.
- Reinforcement Learning
Policy Improvement
Policy improvement upgrades a policy by switching each state to the action with the best action value, with a theorem guaranteeing no state gets worse.
- Reinforcement Learning
Policy Iteration
Policy iteration repeats two steps, fully score the policy then greedily fix it, and stops with the optimal policy once no action changes anymore.
- Reinforcement Learning
Generalized Policy Iteration (GPI)
GPI is the shared loop behind most RL: estimate how good the current policy is, then make the policy greedier, and repeat until neither can improve.
- Reinforcement Learning
Asynchronous Dynamic Programming
Update one state at a time in place, reusing fresh values at once. The good news spreads in the same pass instead of waiting for the next full sweep.
- Reinforcement Learning
Advantages of TD over MC and DP
TD updates from every step by bootstrapping off its own estimates, while Monte Carlo waits for the final score. That makes TD online and low variance at the price of early bias.
- Reinforcement Learning
First-Visit vs Every-Visit MC Prediction
When a state shows up twice in one episode, first-visit counts only the return after the first hello, while every-visit counts every hello. One is strictly unbiased, the other squeezes more data from each episode.
- Reinforcement Learning
Advanced Tabular Methods
TD and Monte Carlo are two ends of one dial, not two rival families. Turning the backup depth from 1 step toward the full episode trades bootstrapping bias for sampling variance.
- Reinforcement Learning
Eligibility Traces
One reward should credit the whole trail that led to it, with recent steps getting more credit than old ones. Eligibility traces are that fading memory.
- Reinforcement Learning
n-Step TD Prediction in RL
n-step TD waits a few real rewards before guessing the rest, so you tune how far to look ahead between fast one step updates and full episode returns.
- Reinforcement Learning
Forward View of TD(lambda)
Instead of fixing one lookahead, mix every lookahead at once: mostly the one-step return, a little less of the two-step, fading geometrically out to the full episode. One knob slides from TD to Monte Carlo.
- Reinforcement Learning
Backward View of TD(lambda)
Keep a fading memory of states you just visited. When surprise arrives, share it backward at once instead of waiting for the episode to end.
- Reinforcement Learning
Equivalence of Forward and Backward Views
Looking ahead to future rewards and looking back with a memory trace sound different, but offline they add up to the same total. The proof swaps two sums.
- Reinforcement Learning
SARSA Lambda
SARSA lambda gives every recent action a fading trace, so one reward updates the whole trail at once instead of one step at a time.
- Reinforcement Learning
Watkins's Q(lambda) Algorithm
Watkins's Q lambda lets one reward teach a whole trail of past moves at once with eligibility traces. The moment the agent tries something exploratory, the trail wipes clean so detours never rewrite the optimal plan.
- Reinforcement Learning
Tree-Backup Algorithm
Tree backup learns from other policies' experience without unstable correction ratios: untaken actions contribute their expected value, and only the taken action carries the multi-step return forward.
- Reinforcement Learning
Planning and Learning in RL
Real steps teach you and train your mental model, then the model replays fake steps so values travel back in one episode instead of many.
- Reinforcement Learning
Dyna Architecture
Learn from each real step once, then replay it many times inside a small learned model, so one trip to the goal teaches the whole path.
- Reinforcement Learning
Prioritized Sweeping
Spend planning backups where the Bellman error is largest, then chase the surprise backward through predecessors instead of sampling at random.
- Reinforcement Learning
Trajectory Sampling in RL
Plan only where you will actually go. Simulate rollouts from the start state under your current policy and spend every planning update there.
- Reinforcement Learning
Real-Time Dynamic Programming
Sweeping every state is wasteful when most states never matter. RTDP only updates the states its simulated trials actually walk through, and still converges on those.
- Reinforcement Learning
Monte Carlo Tree Search (MCTS)
A planning method that grows a lopsided lookahead tree by playing thousands of fast pretend games, digging deeper where wins pile up and dropping lines that lose.
- Reinforcement Learning
Theoretical RL: Sample, Regret, Proof
Theory in RL answers one question: how many tries does an agent need before you can trust it. It turns "it worked in my runs" into a counted cost.
Value Based Methods
- Reinforcement Learning
Bellman Equations
A state's value is its immediate reward plus the discounted value of what comes next. That recursion turns an infinite future into a one step backup you can repeat.
- Reinforcement Learning
Value Functions
A value function is a single number per situation: how much future reward you can expect from here on. It turns far-off consequences into a choice you can make right now.
- Reinforcement Learning
Tabular Solution Methods
Tabular methods store every value in its own table cell, so updating one state-action pair touches nothing else and the classic convergence proofs apply exactly.
- Reinforcement Learning
Value Iteration in RL
Value iteration finds the best value of every state by repeating one rule: assume the next values are right, then take the best action under that assumption. Repeat until nothing moves.
- Reinforcement Learning
Q-Learning
Learn the value of each action from raw experience alone, then act greedily on those values with no model of the world.
- Reinforcement Learning
Deep Q-Networks
Deep Q-Networks swap the Q table for a neural network, so one forward pass from pixels or sensors yields a value for every action.
- Reinforcement Learning
Multi-Armed Bandits
One state, several levers, unknown payouts: the smallest possible setup that still forces you to choose between cashing in and finding out.
- Reinforcement Learning
Optimistic Initial Values
Start every action value absurdly high, and a purely greedy agent will still tour each option once, because every real reward feels like a letdown that sinks the tried below the untried.
- Reinforcement Learning
Upper Confidence Bound (UCB)
Rank every action by its best plausible payoff, not its average, so under-tested options get their trial and proven losers fade on their own.
- Reinforcement Learning
Thompson Sampling in Bandits
Keep a belief curve over each option's true payout, draw one sample from each curve, and play the winner. Uncertain options win draws just by having wide curves.
- Reinforcement Learning
Gradient Bandits
Gradient bandits skip estimating payouts and instead tune a preference score per action, raising the odds of whatever beats the recent average reward.
- Reinforcement Learning
Reward Shaping
When the goal pays only at the end, shaping adds small step hints that point toward it. Done right the hints speed learning without changing the best path.
- Reinforcement Learning
Monte Carlo Methods
Monte Carlo methods price a state by playing whole episodes from it and averaging what actually came back, with no model and no bootstrapping.
- Reinforcement Learning
Monte Carlo Action Values
When the rules of the world are unknown, the way to price an action is to try it, finish the episode, and average the totals it earned.
- Reinforcement Learning
Monte Carlo Control
Monte Carlo control finds good actions without any model by finishing full episodes, averaging the returns each action earned, and switching to the winners.
- Reinforcement Learning
Monte Carlo Exploring Starts
A control method that starts every practice run from a random situation with a random first move, so even a stubborn greedy policy ends up testing everything.
- Reinforcement Learning
On-Policy First-Visit MC Control
Learn the best way to act straight from full episodes, with no model and no staged starts, by keeping every action slightly possible and improving the same policy you play.
- Reinforcement Learning
Off-Policy Prediction via Importance Sampling
Judge a policy you never run by watching a different policy act, then reweighting each episode by how much more or less likely your policy would have acted the same way.
- Reinforcement Learning
Importance Sampling: Ordinary vs Weighted
Both correct for sampling from the wrong policy with probability ratios. Ordinary divides by the episode count and can explode, while weighted divides by the total ratio weight and stays inside the observed payoffs.
- Reinforcement Learning
Inverse Reinforcement Learning
Work backward from expert moves to the hidden reward they imply. You match feature counts, then run normal RL on the recovered goal.
- Reinforcement Learning
Value-Based Methods in RL
Score every move by its predicted future payoff, then take the max. The policy falls out for free, no separate actor required.
- Reinforcement Learning
Q-Learning vs SARSA in RL
Q-learning learns the value of the best action you could take next, while SARSA learns the value of the action you actually take, exploration mistakes included. That one-word difference decides whether your agent hugs the cliff edge or walks around it.
- Reinforcement Learning
Expected SARSA
Instead of gambling on a single randomly sampled next action, Expected SARSA takes the exact average value over all next actions, weighted by how likely each one is.
- Reinforcement Learning
Double Q-Learning
One table picks the winner, a second table prices it. Splitting pick and price stops noise from looking like value.
- Reinforcement Learning
Value Function Approximation in RL
A table stores one value per state and runs out of rows the moment states turn continuous. Function approximation stores a short list of weights instead, so one update teaches every similar state at once.
- Reinforcement Learning
Linear Methods in RL
Linear methods score a state as a weighted sum of hand picked features, so the error surface is one clean bowl with a single best setting.
- Reinforcement Learning
Feature Construction in RL
Linear methods can only draw flat planes over their inputs. Feature construction rebuilds raw state numbers into richer inputs first, so the same flat learner can fit curved value surfaces.
- Reinforcement Learning
Polynomial and Fourier Basis in RL
Cosine waves give linear TD methods a bounded orthogonal feature set, so coarse shape and fine detail learn without the blowups polynomials cause.
- Reinforcement Learning
Coarse Coding in RL
Coarse coding covers continuous states with overlapping shapes, and each state is named by the set of shapes that contain it. The shape of those regions decides how far each lesson spreads.
- Reinforcement Learning
Tile Coding (CMAC) in RL
Lay several slightly shifted grids over continuous space. Each point lands in exactly one tile per grid, and the tiles it lands in explain its value.
- Reinforcement Learning
Radial Basis Functions in RL
Radial basis functions tile a continuous state space with smooth overlapping bells instead of sharp boxes, so the value surface stays differentiable and a small step in any direction gives a small sensible change.
- Reinforcement Learning
Stochastic Gradient Descent in RL
In RL, stochastic gradient descent chases a target that moves with the very weights being updated, so practical methods use semi-gradients that pretend the target stands still for one step.
- Reinforcement Learning
Gradient Monte Carlo Algorithm
Gradient Monte Carlo waits for each episode to finish, then nudges value weights toward the full returns it actually saw, which is honest descent with no borrowed guesses.
- Reinforcement Learning
Semi-Gradient TD(0) in RL
Learn a value guess after every step by aiming at your own next guess, while refusing to chase the target as it moves.
- Reinforcement Learning
Semi-Gradient SARSA in RL
SARSA for continuous states: learn action values with a weight vector, updating from the action you actually take next instead of the best imaginable one.
- Reinforcement Learning
Off-Policy Learning with Approximation
Training on data from one policy while chasing the values of another sounds efficient, until bootstrapping plus function approximation turns small errors into runaway weights.
- Reinforcement Learning
Deadly Triad: Why RL Diverges
Three good ideas, function approximation, bootstrapping, and off-policy data, turn toxic together. The updates stop shrinking errors and start growing them.
- Reinforcement Learning
Baird's Counterexample
Seven states, zero rewards, linear values, yet off policy TD blows up. Three ingredients together break the contraction that keeps learning stable.
- Reinforcement Learning
Gradient-TD Methods (TDC & GTD2)
Gradient TD methods pair the usual value weights with a fast helper vector, so off policy updates with function approximation descend a real error instead of spiraling out of control.
- Reinforcement Learning
Emphatic-TD Methods in RL
Off-policy updates visit the wrong states too often, so some corrections need extra weight. Emphatic TD adds that weight with a followon trace.
- Reinforcement Learning
Deep Reinforcement Learning
Deep reinforcement learning pairs neural perception with trial and error control, so agents learn features and decisions together from raw input.
- Reinforcement Learning
Experience Replay
Experience replay stores past transitions in a circular memory buffer and trains on random mini-batches, breaking time correlations for stable deep network training.
- Reinforcement Learning
Target Networks in Deep RL
A Q-network that learns from targets it generates itself chases its own tail, so deep Q-learning grades against a frozen copy that only syncs up every few thousand steps.
- Reinforcement Learning
Double DQN (DDQN)
Let one network point at the best action and a second, slower network say what it is worth, so lucky noise cannot vote for itself twice.
- Reinforcement Learning
Prioritized Experience Replay
Replay the transitions that still surprise the network first, using a sum tree to draw them fast and weights that undo the sampling bias.
- Reinforcement Learning
Dueling Network Architectures
Judge how good the situation is separately from how much each move matters, so one visit teaches you about every move at once.
- Reinforcement Learning
Noisy Nets for Exploration
Noisy Nets add learnable noise to network weights so one steady perturbation drives a full stretch of action, then gradients quiet the noise where values turn certain.
- Reinforcement Learning
Categorical DQN (C51)
C51 predicts a full histogram over future returns instead of one average Q-value, using 51 fixed return buckets. You keep the spread and the bad tail instead of averaging them away.
- Reinforcement Learning
Quantile Regression DQN (QR-DQN)
QR-DQN learns a return distribution by fixing evenly spaced quantile levels and learning where each one sits on the return axis, so there is no fixed grid to clip tails and no projection step to smear them.
- Reinforcement Learning
Rainbow DQN: Six Upgrades in One Agent
Rainbow bundles six separate DQN fixes into one agent and the bundle beats every part: prioritized replay and 3-step returns carry most of the gain, and the rest stack on top instead of interfering.
- Reinforcement Learning
Regret Bounds: Why Smart Agents Lose Early
Regret is the gap between the reward you got and the reward perfect play would have earned. Good learners make that gap grow slower than time itself.
- Reinforcement Learning
Information-Theoretic Regret Bounds
Regret scales with the number of true surprises your model class allows. Count those independent probes and you can predict learning cost without counting states.
- Reinforcement Learning
Risk-Sensitive RL
Risk-sensitive RL picks policies by the shape of their return distribution, not just the average, so a steady policy can beat a flashy gamble with the same mean.
- Reinforcement Learning
Conditional Value at Risk (CVaR) Optimization
CVaR optimization trains a policy for its bad days, not its average day. It maximizes the average return of the worst alpha fraction of episodes, so rare crashes count more than lucky wins.
Policy Based Methods
- Reinforcement Learning
Policy Gradient Methods
Policy gradients tune the policy itself by climbing expected return, so they handle continuous and stochastic actions where picking a max Q value fails.
- Reinforcement Learning
Actor Critic Methods
Actor-critic methods grade each action the moment it lands instead of waiting for the episode score. A learned critic supplies the grade, which cuts the noise that makes plain policy gradients so slow.
- Reinforcement Learning
Imitation Learning
Learn by copying an expert instead of exploring from scratch. You turn demos into labeled pairs and train a policy the way you train a classifier.
- Reinforcement Learning
RL from Human Feedback
Learn a scorer from human picks between two behaviors, then optimize the policy against that scorer while a leash holds it near where it started.
- Reinforcement Learning
The REINFORCE Algorithm
REINFORCE waits for full episodes, then raises the probability of actions that led to high returns and lowers the probability of actions that led to low returns.
- Reinforcement Learning
Policy Objective Functions
An objective function says what a good policy means: total return from the start, average value over visits, or reward per step forever.
- Reinforcement Learning
Advantage Actor-Critic (A2C)
A2C judges each action against what the critic expected, not against zero. Actions that beat the baseline get reinforced and actions that miss it get dialed back.
- Reinforcement Learning
Asynchronous Advantage Actor-Critic (A3C)
Many CPU workers play their own copies of the game at once and push gradients to one shared brain without locks. The mixed updates decorrelate the data, so no replay buffer is needed.
- Reinforcement Learning
Continuous Action Spaces in RL
Discrete buttons cannot steer a robot arm, so the policy outputs smooth real-valued controls instead. It predicts a Gaussian per joint, samples with a trick that keeps gradients alive, and squashes into motor limits.
- Reinforcement Learning
Deterministic Policy Gradients
Deterministic policy gradients steer one exact action per state by chaining the critic value slope into the actor weights, with no sampling over actions.
- Reinforcement Learning
Deep Deterministic Policy Gradient (DDPG)
DDPG outputs one exact continuous action per state instead of searching over action values, and keeps learning stable with slowly moving target networks.
- Reinforcement Learning
Twin Delayed DDPG (TD3)
TD3 fixes DDPG optimism by asking two critics and believing the gloomier one, blurring target actions with noise, and holding the actor back until the critics settle.
- Reinforcement Learning
Trust Region and Proximal Methods
These methods cap how far a policy may move in one update, measured in distribution space rather than weight space, so one bad gradient step cannot wreck the policy.
- Reinforcement Learning
Trust Region Policy Optimization (TRPO)
TRPO improves a policy only inside a small KL divergence budget around the current behavior, so each update is guaranteed safe instead of risking a collapse.
- Reinforcement Learning
Actor Critic using Kronecker-Factored Trust Region (ACKTR)
ACKTR gives actor-critic networks second-order updates without ever building the giant Fisher matrix. It splits each layer curvature into two small Kronecker factors that are cheap to invert.
- Reinforcement Learning
Phasic Policy Gradient (PPG)
Phasic Policy Gradient splits training into a PPO policy phase and a value distilling auxiliary phase, so shared features help without wrecking the policy.
- Reinforcement Learning
Maximum Entropy RL
Score the goal and the spread. The policy keeps every good path alive in proportion to its value instead of collapsing to one narrow line.
- Reinforcement Learning
Soft Q-Learning
Soft Q-learning adds an entropy bonus to every backup, so the agent keeps several good actions alive instead of collapsing onto one greedy choice.
- Reinforcement Learning
Soft Actor-Critic (SAC)
An off-policy actor-critic that stays curious on purpose: it maximizes reward plus randomness, so one policy covers many good motions instead of one brittle trick.
- Reinforcement Learning
Policy Distillation: Compressing Policies
Policy distillation compresses big teacher policies into one small fast student by copying soft action chances instead of learning from rewards alone.
- Reinforcement Learning
Evolutionary Methods and Neuroevolution
Instead of computing gradients, keep a population of policies, score each one, and breed the winners with small random tweaks until behavior improves.
- Reinforcement Learning
Evolution Strategies (ES)
Evolution strategies tune a policy without backpropagation: they shake the weights with Gaussian noise, score each shaky copy, and step toward the noise that scored best.
Core Concepts
- Reinforcement Learning
Exploration vs. Exploitation
An agent that only repeats its best-known action never finds better ones, while one that only tries new things never cashes in. The fix is a rule that mixes both, tilted by uncertainty.
- Reinforcement Learning
Epsilon-Greedy
Most of the time you pick the best action you know, and sometimes you try something random just to check. Epsilon is the chance you wander.
- Reinforcement Learning
Temporal Difference Learning
Temporal difference learning updates each value from the next state's estimate instead of waiting for an episode to end, so it learns at every step from guesses built on guesses.
- Reinforcement Learning
Hierarchical RL
Hierarchical RL splits a long task in two: a manager picks the next milestone every few steps, and a worker takes the small actions that reach it.
- Reinforcement Learning
Partial Observability
The agent never sees the true state, only a sensor reading that can look identical in different situations, so a policy that reads just the current reading fails where one that remembers history succeeds.
- Reinforcement Learning
Partially Observable MDPs (POMDPs)
A POMDP is an MDP where you cannot see the true state, so you plan from a belief: a probability distribution over where you might be. Every observation reshapes that belief with Bayes rule, and the policy maps beliefs to actions.
- Reinforcement Learning
Belief States
When you cannot see the true state, keep odds over every possibility instead. A belief state is that probability distribution, updated with Bayes rule after each action and observation.
- Reinforcement Learning
PAC-MDP Framework
PAC-MDP is a promise about efficient exploration: except for a limited number of learning steps, the agent acts nearly as well as if it already knew the environment.
Reasoning And Planning
Multi Agent Systems
- Reinforcement Learning
Multi-Agent RL
Reinforcement learning with several learners in one world, where each agent is part of everyone else's environment and the ground keeps shifting.
- Reinforcement Learning
Value Decomposition Networks (VDN)
Split the team score into per-agent utilities that add up to the total, so each agent can pick its own best move with no radio contact at game time.
Safety And Alignment
- Reinforcement Learning
Safe Reinforcement Learning
Safe RL chases reward while respecting a hard cost limit. A constraint says what is allowed, a penalty teaches it, and a shield enforces it.
- Reinforcement Learning
Lyapunov-based Safe RL
Lyapunov safe RL gives the agent an energy meter that must drop on every step, so only actions that move toward the goal count as safe.
- Reinforcement Learning
Shielding
A shield is a safety filter between your policy and the environment: it checks every proposed action against a written safety rule and swaps out the ones that would cause harm.
- Reinforcement Learning
Robust Reinforcement Learning
Robust RL plans for the hardest plausible world, not the likeliest one. It picks the policy whose worst case still pays.
- Reinforcement Learning
Adversarial Attacks on RL Policies
A sticker-sized change to what the agent sees can flip what it does. Tiny crafted observation shifts push a trained policy into the wrong action.
Reinforcement Learning
- Reinforcement Learning
Actor-Critic Architecture
One network picks actions and a second network grades them step by step. The actor learns from the critic instant feedback instead of waiting for the final score.
- Reinforcement Learning
Advanced and PhD-Level Topics in RL
Frontier RL stretches past single-agent trial and error with world models, offline archives, multi-agent games, and temporal abstraction. Each fix answers one way the textbook setup breaks in the real world.
- Reinforcement Learning
Model-Based Deep RL
The agent learns a small simulator of the world from real tries, then practices thousands of imagined moves inside it before each real move.
- Reinforcement Learning
Learning Latent Dynamics
The agent squeezes camera frames into a small code and imagines futures in that code instead of predicting pixels, so planning runs thousands of rollouts per second.
- Reinforcement Learning
Model-Based Policy Optimization (MBPO)
MBPO trains a policy mostly on short imagined rollouts that start from real states, so model errors stay small while real environment steps stay few.
- Reinforcement Learning
Dreamer: World Models for RL
Learn a small simulator of the world from past play, then train the player inside that simulator where mistakes cost nothing and steps run in bulk.
- Reinforcement Learning
MuZero and AlphaZero
AlphaZero plans with a perfect rulebook and deep search; MuZero learns its own compact rulebook from play and searches inside it instead.
- Reinforcement Learning
Planning with Learned Models
Learn a world model, roll out candidate action plans inside it, run the best first step, then replan from the real next state.
- Reinforcement Learning
Advanced RL Paradigms
Flat trial and error stalls on long horizons, moving peers, empty rewards, and new tasks. Four structural fixes answer each one: options, team training, curiosity, and learning to learn.
- Reinforcement Learning
Offline Reinforcement Learning
Offline RL trains from logged transitions with no live steps, by pushing down unseen action values and by generating actions from high return examples.
- Reinforcement Learning
Offline / Batch Reinforcement Learning
Offline RL learns a policy from fixed logs with no live steps, by treating unseen actions as guilty until proven safe and stitching good segments together.
- Reinforcement Learning
Distributional Shift in Offline RL
You train from a fixed log with no chance to try a new move, so one flattering guess about an action you never saw can poison every value behind it.
- Reinforcement Learning
Batch Constrained Q-Learning (BCQ)
Instead of letting an agent dream up risky actions no human ever demonstrated, BCQ confines its choices to tracks already in the dataset, tuning them only with a short mathematical leash.
- Reinforcement Learning
Conservative Q-Learning (CQL)
CQL trains offline values that refuse to brag about actions the dataset never tried. It pushes unseen actions down and dataset actions up, so the learned values sit safely below the truth.
- Reinforcement Learning
Implicit Q-Learning
Learn strong values from logged data only. You fit the top of what you saw with expectile regression and clone the good moves with weights.
- Reinforcement Learning
Decision Transformers and Trajectory Transformer
Decision Transformer treats control as next token prediction. You state the return you want and it generates the actions that earned that return in past data.
- Reinforcement Learning
Cooperative vs. Competitive Environments
When several agents share one world, the reward rules decide everything. Shared rewards need teamwork, opposite rewards need unexploitable play, and mixed rewards tempt everyone into mutual ruin.
- Reinforcement Learning
Markov Games
One MDP with many deciders. Next state and each reward depend on the full joint action, so agents must settle on mutual best responses.
- Reinforcement Learning
Independent Q-Learning
Let each agent run its own Q-learning and pretend teammates are weather. It scales with zero messaging but the ground shifts as everyone learns at once.
- Reinforcement Learning
Multi-Agent DDPG (MADDPG)
Each agent acts alone from its own view, but training gives each critic the full scene plus all actions, which steadies learning in continuous team play.
- Reinforcement Learning
Multi-Agent PPO (MAPPO)
One shared policy controls every agent from local views, while one central critic judges the team from full state. Clipped updates keep the team moving together.
- Reinforcement Learning
QMIX and QTRAN in Multi-Agent RL
QMIX lets each agent act alone at runtime while a state-made mixing network combines their values during training, with weights locked non-negative so a better local choice can never lower the team score.
- Reinforcement Learning
Mean Field Multi-Agent RL
When hundreds of agents share one space, each one stops tracking every neighbor and steers by the average move of the few agents next to it.
- Reinforcement Learning
Self-Play and Fictitious Play
Self-play trains an agent against itself so the opponent improves automatically. Fictitious play answers the average of past opponents, which breaks cycles.
- Reinforcement Learning
The Options Framework
One call can run a whole skill instead of one muscle twitch. An option says where it can start, what it does while it runs, and when it hands control back.
- Reinforcement Learning
Feudal Networks (FuN)
A slow Manager points where the situation should drift over the next few steps, and a fast Worker earns pay for moving that way. Direction only, no micromanagement.
- Reinforcement Learning
MAXQ Value Function Decomposition
Instead of solving one huge task all at once, MAXQ splits it into a tree of smaller jobs where each job only watches the details it needs.
- Reinforcement Learning
Goal-Conditioned Reinforcement Learning
One policy learns many goals at once by taking the target as input. The same network walks to any destination instead of retraining per address.
- Reinforcement Learning
Hindsight Experience Replay (HER)
Hindsight replay files every miss under the goal it actually reached, so a failed run still teaches the agent how to get somewhere on purpose.
- Reinforcement Learning
Option-Critic Architecture
Give an agent reusable skills plus a learned sense of when to drop them, and both the skills and the quitting rules improve straight from reward, with no hand-placed subgoals.
- Reinforcement Learning
Behavioral Cloning
Instead of designing rewards, record an expert and train a policy to copy each move with ordinary supervised learning. It works until one small mistake lands somewhere the expert never visited.
- Reinforcement Learning
Apprenticeship Learning via Inverse RL
You do not need the expert's hidden reward to match them. Match how often their behavior visits each feature, and the return gap stays small no matter the reward weights.
- Reinforcement Learning
Generative Adversarial Imitation Learning (GAIL)
GAIL copies an expert without writing a reward function. A discriminator learns to spot learner moves, and its suspicion becomes the training reward.
- Reinforcement Learning
Adversarial Inverse RL (AIRL)
AIRL copies the goal instead of the motions. It splits what the expert wanted from how the old world moved, so the aim still works when the world changes.
- Reinforcement Learning
Maximum Entropy Inverse Reinforcement Learning
Watch an expert, count what they visit, then pick the reward that makes those counts likely while assuming nothing else. Higher return means higher chance.
- Reinforcement Learning
Dataset Aggregation (DAgger)
Let the learner drive, watch where it wobbles, and ask the expert what it should have done there. Collect those corrections and the wobbles disappear.
- Reinforcement Learning
Random Network Distillation (RND)
A student network tries to copy a frozen random teacher on every state it visits. Large copy errors mean the state is new, so the error itself becomes the curiosity reward.
- Reinforcement Learning
Exploration and Intrinsic Motivation
When task rewards are rare or missing, intrinsic motivation pays the agent self-made curiosity bonuses for visiting unfamiliar states until the real goal shows up.
- Reinforcement Learning
Count-Based Exploration
Keep a tally of where you have been and pay yourself to go where the tally is low. Rare states earn a big bonus, visited ones earn almost nothing.
- Reinforcement Learning
Curiosity-Driven Exploration (ICM)
Reward the agent for surprise. When its model of the world predicts badly, pay it to go back and learn that corner until the surprise runs out.
- Reinforcement Learning
Go-Explore
Go-Explore splits hard search in two: first map new states by returning to frontiers without noise, then train a sturdy policy from the best path.
- Reinforcement Learning
Meta-Reinforcement Learning
Meta RL trains one starting policy across many small worlds so a new world takes a few practice tries instead of months of retraining.
- Reinforcement Learning
Model-Agnostic Meta-Learning (MAML) in RL
MAML learns a starting policy that sits close to many tasks at once, so one or two gradient steps on a new task land near good behavior.
- Reinforcement Learning
Recurrent Policies with Memory in RL
A recurrent policy gives your agent a memory: it carries a hidden state from step to step so today's action can depend on what it saw a minute ago, not just the current frame.
- Reinforcement Learning
Transformers in RL
A transformer policy treats its own past as a memory it can reread. At each step it looks back over recent states, actions and goals, and puts the most weight on the moments that matter now.
- Reinforcement Learning
Sample Complexity of Exploration
Sample complexity counts how many environment steps an agent must pay before it can promise a near-optimal policy with high confidence.
- Reinforcement Learning
Transfer Learning and Generalization
A policy trained on one task can jump-start learning on a related task, but only when the two tasks share structure. Otherwise the old habits hurt.
- Reinforcement Learning
Sim-to-Real Transfer
Train a robot brain across thousands of fake physics settings in simulation, then ship that same brain to the real robot with no extra training, as long as reality falls inside the range you trained on.
- Reinforcement Learning
Zero-Shot and Few-Shot RL
Zero-shot RL runs on a new task with no practice episodes, while few-shot RL gets K quick tries to figure it out. Both test whether training on many tasks taught a policy that actually generalizes.
- Reinforcement Learning
RL Squared (Fast RL via Slow RL)
RL squared trains one recurrent policy slowly across many tasks so that, on a new task, it adapts fast inside its own memory with no weight updates.
- Reinforcement Learning
Probabilistic Embeddings for Actor-Critic RL
Infer which task you face from a few recent steps, then run one task aware policy from that belief with no fresh gradients.
- Reinforcement Learning
Covariance Matrix Adaptation Evolution Strategy (CMA-ES)
A gradient-free optimizer that keeps reshaping a Gaussian cloud of tries. It slides the cloud toward the winners and stretches it along directions that keep paying off.
- Reinforcement Learning
NEAT: Evolving Neural Topologies
NEAT grows neural nets instead of fixing their size first: it starts from the smallest wiring, stamps each change with a number, and breeds topologies by task score.