Skip to content
AI360XpertBeta

Repetition Penalties

When a model starts echoing itself, repetition penalties tax the scores of words it has already used so a fresh word can win.

A frequency penalty of 1.0 applied to a token seen twice drops its logit from 5.0 to 3.0, so the untouched token at 4.5 wins instead
A frequency penalty of 1.0 applied to a token seen twice drops its logit from 5.0 to 3.0, so the untouched token at 4.5 wins instead

Why Does This Exist?

You have watched it happen. A model writes a decent sentence, likes one phrase a little too much, then repeats it with growing confidence until the whole answer is one loop. It is not a bug in the weights. It is what picking the favourite every step does: each repeat makes the next repeat likelier, so the loop feeds itself.

The obvious fixes do not quite solve it. Always taking the top token writes flat copy that gets stuck. Adding randomness with temperature sampling helps variety but never forbids a repeat, and cutoffs like top-p sampling only trim the tail. None of them remembers what was already said.

That memory is this page. A logit is the raw score the model gives one vocabulary token before probabilities are computed, and repetition penalties subtract a small tax from the logits of tokens that already appeared. You will see the two taxes, a worked example where a taxed 5.0 loses to a fresh 4.5, and when to turn both knobs to zero. The full picking tour lives in decoding strategies, and beam search is where loops hurt most.

Think of It Like This

A DJ with two house rules

Picture a DJ who always reaches for the current chart topper. Left alone, you would hear the same song every ten minutes, because the most played song looks like the safest pick, and each play makes it look safer.

The station imposes two rules. The presence penalty is a flat fee: played it at all today, pay once. The frequency penalty is a per play fee: each replay costs again, so the tenth replay costs ten times the first. Eventually some forgotten track becomes the rational pick, and the loop breaks.

It stops here: the DJ still chooses, and so does the model. A penalty only lowers the odds of a used word. It never bans it, and unlike house rules it can wreck grammar when repetition is the whole job, as in code.

How It Actually Works

Everything happens to the logits, after the forward pass and before temperature scaling, cutoffs, and softmax turn scores into probabilities that sum to 1. Call the raw score for token ii by ziz_i, the count of how often token ii already appears in the text so far by CiC_i, the flat fee by pp, and the per repeat fee by ff. The adjusted score zi′z'_i is what actually gets sampled:

zi′=zi−p⋅1[Ci>0]−f⋅Ciz'_i = z_i - p \cdot \mathbf{1}[C_i > 0] - f \cdot C_i

In plain words: start from the raw score, subtract pp once if the token showed up at all, then subtract ff once per appearance. Tokens with Ci=0C_i = 0 pay nothing. That is the whole mechanism, and both knobs default to doing nothing when set to zero.

Presence penalty: one flat fee

Presence asks a single yes or no question. Did token ii appear at least once, in the generated text and, depending on the API, the prompt? If yes, subtract pp from ziz_i. That is it.

Say p=0.5p = 0.5, fox has logit 5.05.0 and appeared once, cat has 4.54.5 and is new. Fox drops to 4.54.5 and ties cat. One line, one effect: reusing any word gets slightly harder, so the model reaches for fresh vocabulary. It does not care whether fox appeared once or fifty times.

Frequency penalty: a fee per repeat

Frequency counts. It multiplies the per repeat fee ff by the exact count CiC_i and subtracts that. Say it once, pay a little. Say it twenty times, pay twenty times over, and the logit sinks far enough that sampling something else becomes near certain.

This is the loop breaker. A flat fee slows a loop down. An escalating fee guarantees the loop cannot pay its way forever, because every lap raises the price of staying on it.

Worked example: fox falls from 5.0 to 3.0

Take three tokens with raw logits fox 5.05.0, cat 4.54.5, dog 4.04.0. Fox already appeared twice, so C=[2,0,0]C = [2, 0, 0]. Set f=1.0f = 1.0 and p=0p = 0 to isolate frequency. The adjusted logits are:

[5.0−1.0×2, 4.5−0, 4.0−0]=[3.0, 4.5, 4.0][5.0 - 1.0 \times 2,\ 4.5 - 0,\ 4.0 - 0] = [3.0,\ 4.5,\ 4.0]

Fox led by half a point and now trails by one and a half. Softmax makes the flip concrete. Before the tax, exponentiate to about [148.413, 90.017, 54.598][148.413,\ 90.017,\ 54.598], sum to about 293.028293.028, and divide to get about [0.506, 0.307, 0.186][0.506,\ 0.307,\ 0.186]. Fox held half the probability. After the tax the exponentials are about [20.086, 90.017, 54.598][20.086,\ 90.017,\ 54.598], the sum is about 164.701164.701, and the probabilities are about [0.122, 0.547, 0.331][0.122,\ 0.547,\ 0.331]. Fox collapses from 0.5060.506 to 0.1220.122 while cat takes the lead at 0.5470.547. Same model, same scores, different winner, because one token had history.

Why subtraction won and division lost

Early penalties divided instead of subtracting, as in zi/1.2z_i / 1.2 for used tokens. That breaks on negative logits. Dividing a negative score by 1.21.2 moves it toward zero, which raises it, so the penalty accidentally rewards repeating a token the model already disliked. Subtraction always lowers, whatever the sign. That is why every modern API subtracts.

Name the trade straight. You buy variety with control: higher taxes mean fresher words and a higher chance the model dodges a word you actually needed. The method assumes a repeated token is a worse choice, which fails whenever repetition is required by syntax or facts. Set both penalties to zero for code, JSON, or any output that must repeat, and reach for structured outputs or constrained decoding instead.

Code

The point is the flip, not the formula. This snippet applies both taxes to the worked example and shows the lead changing hands.

def apply_penalties(logits: list[float], counts: list[int], freq: float, pres: float) -> list[float]:    adjusted: list[float] = []    for z, c in zip(logits, counts):        step = freq * c  # escalating fee per repeat        if c > 0:            step += pres  # flat fee for having appeared at all        adjusted.append(z - step)    return adjusted

raw = [5.0, 4.5, 4.0]  # fox, cat, dogcounts = [2, 0, 0]  # fox has appeared twiceprint(apply_penalties(raw, counts, freq=1.0, pres=0.0))  # -> [3.0, 4.5, 4.0]print(apply_penalties([5.0, 4.5], [1, 0], freq=0.0, pres=0.5))  # -> [4.5, 4.5]

The first line is the worked example: fox taxed from 5.05.0 to 3.03.0, cat untouched at 4.54.5 and now winning. The second line isolates presence: a 0.50.5 flat fee pulls a seen 5.05.0 into a tie with a fresh 4.54.5.

Watch Out For

Penalties on, JSON broken

The symptom is unmistakable: you enable a small presence penalty for variety and your JSON comes back with missing quotes, renamed keys, or exotic symbols where brackets belonged. Structured syntax needs the same tokens dozens of times, and the tax punishes exactly that.

Turn both penalties to zero for code, JSON, schemas, and markdown tables. If you still want guarantees about shape, use structured outputs or constrained decoding, which restrict what can be sampled instead of taxing it.

The tax is so high the model dodges the word you need

The symptom is subtler: the answer stays fluent but never says the key term, the character name, or the product you asked about. Instead you get a parade of synonyms drifting off topic. That is a penalty doing its job too well, treating a necessary repeat as a loop.

Lower both knobs toward zero in small steps and retest on a prompt that must repeat a name. If you only wanted livelier prose, get that from temperature sampling or top-p sampling and keep the repeat taxes near zero.

The Quick Version

  • Repetition penalties subtract a tax from the logits of tokens that already appeared, before temperature and softmax turn scores into probabilities.
  • Presence subtracts a flat fee once per used token, while frequency subtracts per repeat, so a 5.0 logit with two repeats at 1.0 falls to 3.0.
  • The tax breaks greedy loops without banning any word: a taxed 3.0 loses to a fresh 4.5, and the probabilities shift from about 0.506 to 0.122 for the repeated token.
  • Turn both penalties to zero for code, JSON, and anything where repetition is required, and use structured outputs when shape must hold.
  • If the model starts dodging the word you asked for, the tax is too high, so lower it before touching anything else.