Zero-Shot and Few-Shot RL

Why Does This Exist?

A standard RL policy is a one-task specialist. Train a cheetah to run at 1.0 m/s for 10 million steps and it looks great. Ask for 2.0 m/s in a new track with slightly different friction and it stumbles. You then pay millions of fresh steps to retrain from zero. That bill repeats for every new goal, friction, or layout.

This page covers the evaluation side of that problem: what happens on a held-out task when you allow zero practice episodes versus K practice episodes. It does not re-teach how to train across tasks. That training story lives in meta-reinforcement learning and the gradient version lives in MAML in RL. Goal wording as a second way to name tasks lives in goal-conditioned RL.

You need one prerequisite idea first. A task is a Markov decision process with states, actions, transition dynamics, and rewards. A policy $\pi(a \mid s)$ is your rule for picking action $a$ in state $s$. Meta-RL trains one policy across a distribution of such tasks $p(\mathcal{T})$ so it can face a new draw. If those terms feel shaky, read the meta-RL page first, then come back.

Think of It Like This

How It Actually Works

Write $T$ for one task, $p(\mathcal{T})$ for the distribution you trained on, and $T_{test}$ for a held-out task you never trained on. Write $s$ for state, $a$ for action, $G$ for episode return, and $\pi$ for policy. A context $c$ is a small vector that summarizes what the agent has seen of the new task. $D_K$ is the set of $K$ adaptation episodes on $T_{test}$. An encoder $f$ maps that set to a context, so $c = f(D_K)$.

Zero-shot means empty context and frozen weights

Zero-shot evaluation freezes everything. The agent sees $T_{test}$, builds an empty context $c_0$, and acts with $\pi(a \mid s, c_0)$. No gradient step runs. No weight changes. The return $J_0$ is the average $G$ under that frozen policy.

This works only when training diversity already covered the test variation. Broad training plus steady sensors plus a fixed action meaning can carry a policy surprisingly far. If the test task needs a brand-new skill, zero-shot fails plainly. That failure is honest signal, not a bug.

Few-shot means K episodes become a context

Few-shot evaluation allows exactly $K$ episodes before scoring. The agent rolls out $K$ tries, collects $D_K$, computes $c = f(D_K)$, then acts with $\pi(a \mid s, c)$ on fresh scored episodes. The weights may stay frozen. Only the context changes.

Two common builds do this. A recurrent build carries a hidden state across the K episodes. An encoder build reads the K episodes into $c$ directly, in the style of probabilistic task embeddings. Either way the policy signature tells the story: the action depends on both the current state and the inferred task summary. MAML-style methods swap the encoder for $K$ gradient steps, which you will find on the MAML page rather than here.

Generalization benchmarks make the split fair. They fix a train set and a disjoint test set, then report both numbers. Train on slow target speeds, test on a fast one. Train on ten manipulation tasks, test on five new ones. Same agent, same weights, two scores: $J_0$ with no practice and $J_K$ after K tries.

Worked example: 0-shot 42.5 to 5-shot 78.3

Take one held-out task with a small illustrative fixture you can check by hand. The numbers below are a teaching fixture, not a benchmark claim. They trace the same shape real context-conditioned agents show: a big early jump, then shrinking gains.

• K equal to 0 gives 42.5. Empty context, frozen policy.
• K equal to 1 gives 58.1, a gain of 15.6. One episode names the task coarsely.
• K equal to 2 gives 67.4, a gain of 9.3. The context sharpens.
• K equal to 3 gives 73.0, a gain of 5.6.
• K equal to 4 gives 76.2, a gain of 3.2.
• K equal to 5 gives 78.3, a gain of 2.1.

Total adaptation gain is $78.3 - 42.5 = 35.8$. A from-scratch oracle trained only on this task scores 82.0, so five shots reach $78.3 / 82.0$, which is 95.5 percent of oracle with a handful of episodes instead of millions of steps. The diagram draws these six bars side by side so you can see the curve flatten.

The trade is simple. Zero-shot costs nothing at test time and tells you what truly transferred. Few-shot costs K episodes plus an encoder, and buys fast calibration when tasks share structure the context can name. When the test task sits far outside $p(\mathcal{T})$, neither saves you. Broaden training or retrain instead of adding more shots.

Code

The snippet below recomputes the fixture: per-shot gains, total gap, and oracle coverage.

1from typing import Dict2
3
4def summarize(returns: Dict[int, float], oracle: float) -> None:5    gains = [returns[k] - returns[k - 1] for k in range(1, 6)]6    gap = returns[5] - returns[0]7    coverage = returns[5] / oracle * 100.08    print(f"gains: {[round(g, 1) for g in gains]}")9    print(f"gap: {gap:.1f}")10    print(f"oracle coverage: {coverage:.1f}%")11
12
13if __name__ == "__main__":14    fixture: Dict[int, float] = {15        0: 42.5,16        1: 58.1,17        2: 67.4,18        3: 73.0,19        4: 76.2,20        5: 78.3,21    }22    summarize(fixture, 82.0)

1# -> expected output:2gains: [15.6, 9.3, 5.6, 3.2, 2.1]3gap: 35.84oracle coverage: 95.5%

Watch Out For

The Quick Version

• Zero-shot RL scores a frozen policy on a new task with no practice episodes and an empty context.
• Few-shot RL allows K episodes, encodes them into a context vector, and conditions the same policy on that context.
• Context-conditioned policies adapt without weight updates by changing only the task summary they read.
• In the worked fixture, return climbs from 42.5 at 0-shot to 78.3 at 5-shot, a 35.8 point gain that reaches 95.5 percent of an 82.0 oracle.
• Generalization benchmarks report both scores on disjoint train and test tasks so transfer reads as two numbers, not a feeling.

What to Read Next

• Meta-Reinforcement Learning
• Model-Agnostic Meta-Learning (MAML) in RL
• Goal-Conditioned Reinforcement Learning