Repetition Penalties
When a model starts echoing itself, repetition penalties tax the scores of words it has already used so a fresh word can win.
Why Does This Exist?
You have watched it happen. A model writes a decent sentence, likes one phrase a little too much, then repeats it with growing confidence until the whole answer is one loop. It is not a bug in the weights. It is what picking the favourite every step does: each repeat makes the next repeat likelier, so the loop feeds itself.
The obvious fixes do not quite solve it. Always taking the top token writes flat copy that gets stuck. Adding randomness with temperature sampling helps variety but never forbids a repeat, and cutoffs like top-p sampling only trim the tail. None of them remembers what was already said.
That memory is this page. A logit is the raw score the model gives one vocabulary token before probabilities are computed, and repetition penalties subtract a small tax from the logits of tokens that already appeared. You will see the two taxes, a worked example where a taxed 5.0 loses to a fresh 4.5, and when to turn both knobs to zero. The full picking tour lives in decoding strategies, and beam search is where loops hurt most.
Think of It Like This
A DJ with two house rules
Picture a DJ who always reaches for the current chart topper. Left alone, you would hear the same song every ten minutes, because the most played song looks like the safest pick, and each play makes it look safer.
The station imposes two rules. The presence penalty is a flat fee: played it at all today, pay once. The frequency penalty is a per play fee: each replay costs again, so the tenth replay costs ten times the first. Eventually some forgotten track becomes the rational pick, and the loop breaks.
It stops here: the DJ still chooses, and so does the model. A penalty only lowers the odds of a used word. It never bans it, and unlike house rules it can wreck grammar when repetition is the whole job, as in code.
How It Actually Works
Everything happens to the logits, after the forward pass and before temperature scaling, cutoffs, and softmax turn scores into probabilities that sum to 1. Call the raw score for token by , the count of how often token already appears in the text so far by , the flat fee by , and the per repeat fee by . The adjusted score is what actually gets sampled:
In plain words: start from the raw score, subtract once if the token showed up at all, then subtract once per appearance. Tokens with pay nothing. That is the whole mechanism, and both knobs default to doing nothing when set to zero.
Presence penalty: one flat fee
Presence asks a single yes or no question. Did token appear at least once, in the generated text and, depending on the API, the prompt? If yes, subtract from . That is it.
Say , fox has logit and appeared once, cat has and is new. Fox drops to and ties cat. One line, one effect: reusing any word gets slightly harder, so the model reaches for fresh vocabulary. It does not care whether fox appeared once or fifty times.
Frequency penalty: a fee per repeat
Frequency counts. It multiplies the per repeat fee by the exact count and subtracts that. Say it once, pay a little. Say it twenty times, pay twenty times over, and the logit sinks far enough that sampling something else becomes near certain.
This is the loop breaker. A flat fee slows a loop down. An escalating fee guarantees the loop cannot pay its way forever, because every lap raises the price of staying on it.
Worked example: fox falls from 5.0 to 3.0
Take three tokens with raw logits fox , cat , dog . Fox already appeared twice, so . Set and to isolate frequency. The adjusted logits are:
Fox led by half a point and now trails by one and a half. Softmax makes the flip concrete. Before the tax, exponentiate to about , sum to about , and divide to get about . Fox held half the probability. After the tax the exponentials are about , the sum is about , and the probabilities are about . Fox collapses from to while cat takes the lead at . Same model, same scores, different winner, because one token had history.
Why subtraction won and division lost
Early penalties divided instead of subtracting, as in for used tokens. That breaks on negative logits. Dividing a negative score by moves it toward zero, which raises it, so the penalty accidentally rewards repeating a token the model already disliked. Subtraction always lowers, whatever the sign. That is why every modern API subtracts.
Name the trade straight. You buy variety with control: higher taxes mean fresher words and a higher chance the model dodges a word you actually needed. The method assumes a repeated token is a worse choice, which fails whenever repetition is required by syntax or facts. Set both penalties to zero for code, JSON, or any output that must repeat, and reach for structured outputs or constrained decoding instead.
Code
The point is the flip, not the formula. This snippet applies both taxes to the worked example and shows the lead changing hands.
def apply_penalties(logits: list[float], counts: list[int], freq: float, pres: float) -> list[float]: adjusted: list[float] = [] for z, c in zip(logits, counts): step = freq * c # escalating fee per repeat if c > 0: step += pres # flat fee for having appeared at all adjusted.append(z - step) return adjusted
raw = [5.0, 4.5, 4.0] # fox, cat, dogcounts = [2, 0, 0] # fox has appeared twiceprint(apply_penalties(raw, counts, freq=1.0, pres=0.0)) # -> [3.0, 4.5, 4.0]print(apply_penalties([5.0, 4.5], [1, 0], freq=0.0, pres=0.5)) # -> [4.5, 4.5]The first line is the worked example: fox taxed from to , cat untouched at and now winning. The second line isolates presence: a flat fee pulls a seen into a tie with a fresh .
Watch Out For
Penalties on, JSON broken
The symptom is unmistakable: you enable a small presence penalty for variety and your JSON comes back with missing quotes, renamed keys, or exotic symbols where brackets belonged. Structured syntax needs the same tokens dozens of times, and the tax punishes exactly that.
Turn both penalties to zero for code, JSON, schemas, and markdown tables. If you still want guarantees about shape, use structured outputs or constrained decoding, which restrict what can be sampled instead of taxing it.
The tax is so high the model dodges the word you need
The symptom is subtler: the answer stays fluent but never says the key term, the character name, or the product you asked about. Instead you get a parade of synonyms drifting off topic. That is a penalty doing its job too well, treating a necessary repeat as a loop.
Lower both knobs toward zero in small steps and retest on a prompt that must repeat a name. If you only wanted livelier prose, get that from temperature sampling or top-p sampling and keep the repeat taxes near zero.
The Quick Version
- Repetition penalties subtract a tax from the logits of tokens that already appeared, before temperature and softmax turn scores into probabilities.
- Presence subtracts a flat fee once per used token, while frequency subtracts per repeat, so a 5.0 logit with two repeats at 1.0 falls to 3.0.
- The tax breaks greedy loops without banning any word: a taxed 3.0 loses to a fresh 4.5, and the probabilities shift from about 0.506 to 0.122 for the repeated token.
- Turn both penalties to zero for code, JSON, and anything where repetition is required, and use structured outputs when shape must hold.
- If the model starts dodging the word you asked for, the tax is too high, so lower it before touching anything else.