Epsilon-Greedy

Most of the time you pick the best action you know, and sometimes you try something random just to check. Epsilon is the chance you wander.

With epsilon 0.1 the agent exploits the best arm valued 3.1 with chance 0.925 and explores at random with chance 0.1

Why Does This Exist?

An agent that always picks its current best gets stuck. Early estimates are noisy, so the first action that looks good keeps winning even when a better action sits untried. An agent that always picks at random never cashes in what it learns. That tension is the exploration versus exploitation tradeoff every RL method must handle.

Epsilon-greedy is the simplest fix that works. With probability $\varepsilon$ (epsilon) you sample uniformly from all actions, and with probability $1 - \varepsilon$ you pick the highest estimated value. One number controls the whole balance, and decaying it over time shifts you from checking to cashing in.

Think of It Like This

Your lunch street with one new stall

You work on a street with ten lunch stalls. One rice stall is reliably good. A new stall just opened with no reviews.

Nine days out of ten you eat at the rice stall. One day out of ten you roll a die and try whatever stall it picks, good or bad. Most lunches are safe, but you still sample the street. If the new stall turns out great, it becomes your new default.

Where it stops: you might use reviews to pick which new stall to try. Epsilon-greedy does not. Its random days are uniform across all stalls, including ones you already know are bad.

How It Actually Works

Let $Q(a)$ be the estimated value of action $a$ and $\varepsilon$ be a number in $[0, 1]$ . Draw $p$ uniformly from $[0, 1)$ each step:

If $p < \varepsilon$ , explore: pick an action uniformly at random.
If $p \ge \varepsilon$ , exploit: pick $a = \arg\max_a Q(a)$ , breaking ties at random.

Start $\varepsilon$ high when you know nothing, often $1.0$ , then decay it. A common schedule multiplies by $0.99$ each episode down to a floor like $0.05$ . You explore early and trust your values late.

Worked example

Take four actions with $Q = [1.2, 0.5, 3.1, 2.0]$ and $\varepsilon = 0.1$ . The greedy action is index 2 with value $3.1$ .

Chance of exploring: $0.1$ .
Chance of exploiting: $0.9$ .
During exploration each arm gets $0.1 / 4 = 0.025$ .
Chance of picking the best arm: $0.9 + 0.025 = 0.925$ .
Chance of picking a specific bad arm, say index 1: $0.025$ .

Decay works the same way every schedule does. From $1.0$ with factor $0.99$ per episode, after 100 episodes $\varepsilon = 0.99^{100} \approx 0.366$ . You still wander about one step in three, which is why most runs keep decaying toward $0.05$ . The bandit setting is where this arithmetic matters most.

Code

The whole rule is one coin flip plus an argmax. With $\varepsilon = 0.0$ the answer is always the greedy arm.

1import random2
3def epsilon_greedy(q: list, eps: float) -> int:4    if random.random() < eps:  # wander with chance eps5        return random.randint(0, len(q) - 1)6    return max(range(len(q)), key=lambda i: q[i])7
8
9print(epsilon_greedy([1.2, 0.5, 3.1, 2.0], 0.0))10# -> 211print(f"{0.9 + 0.1 / 4:.3f} {0.99 ** 100:.3f}")12# -> 0.925 0.366

The first print is the greedy index for the worked example. The second print is the chance of picking the best arm at $\varepsilon = 0.1$ and the decayed epsilon after 100 episodes.

Watch Out For

Spending exploration on actions you know are bad

Uniform means uniform. With 100 actions where 95 are clearly bad, epsilon-greedy still gives each bad action the same chance as each promising one. In large spaces most random picks teach you nothing you did not already know.

When the action count grows or bad actions cost a lot, switch to methods that aim exploration at uncertainty, or at least decay epsilon faster once values separate.

Decaying epsilon before values settle

If epsilon hits near zero after a few episodes, the agent locks onto whatever looked best early. Those early values are mostly noise, so the lock-in is often wrong and never revisits the true best arm.

Match the schedule to the task size. Keep epsilon high until each action has several samples, floor it above zero instead of zero, and watch visit counts before trusting the greedy pick.

The Quick Version

Epsilon-greedy explores with chance epsilon and exploits the best estimate otherwise.
Exploration is uniform over actions, which is simple but wasteful in large spaces.
The best arm at epsilon 0.1 with four arms is picked with chance 0.925 in the example.
Start epsilon near 1.0 and decay toward a small floor like 0.05.
Decaying too fast locks in early noise, so pace the schedule to the task.