Skip to content
AI360XpertBeta

Mid-Training

Mid-training resumes next-token prediction on dense domain text after pretraining ends, before instruction tuning teaches the model how to answer. Same loss, narrower bookshelf.

Mid-training keeps next-token prediction but feeds 18 domain sequences and 2 replay sequences in every 20, so new facts stick and old skills hold
Mid-training keeps next-token prediction but feeds 18 domain sequences and 2 replay sequences in every 20, so new facts stick and old skills hold

Why Does This Exist?

Pretraining gives you a fluent generalist. It has read so much text that it can continue almost anything, yet it knows no single field deeply. And it still writes like an autocompleter, not an assistant.

Say you want a model that reasons inside one domain: medicine, law, code, or a language the base run barely saw. A small instruction set won't get you there. It teaches shape, not substance. Show a base model a pile of ideal answers and it learns how replies should sound while the facts underneath stay thin. Retrieval helps at ask time, but it can't put reasoning habits into the weights.

Mid-training fills that gap. You resume the same next-token training on dense domain text, mixed with a little general text so old skills hold, before you teach manners. This page covers that middle run: the unchanged objective, the mix, and one batch worked by hand. The behavior step that follows lives in instruction tuning. The general run it continues lives in pretraining.

The one idea you need first: next-token prediction trains a model to guess each token from the ones before it. That guess is the whole supervisor. Mid-training keeps the supervisor and swaps what the model reads.

Think of It Like This

A line cook stages in a regional kitchen

You trained as a general line cook. You can chop, season, and plate anything, because your first kitchen served every cuisine at once.

Now a regional restaurant hires you. For two months you cook almost nothing but their pantry: their grains, their spices, their cuts. The dishes change. The knife work doesn't. And once a week you still cook a home meal from your old kitchen, so your hands don't forget the rest.

That weekly home meal is replay. The two-month staging is mid-training. The smile and the script you use with guests come later, in a short course on service. That course is instruction tuning, and it can't replace the two months at the stove.

Where the picture stops: you tasted and understood. The model didn't. It only adjusted its weights so the next token surprised it less, and every new dish rewrote the same shared hands.

How It Actually Works

You keep training the same weights with the same loss. Nothing about the optimizer's job changes. What changes is the reading list: most of every batch comes from the target domain, and a small share comes from the old general data. That small share is called replay, and it is the whole reason old skills survive the move.

The objective stays the same

For one sequence, the loss is the average surprise across its tokens:

L(θ)=−1T∑t=1Tlog⁡pθ(xt∣x<t)L(\theta) = -\frac{1}{T}\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{\lt t})

In plain words: LL is the loss, lower when guesses are better. θ\theta holds the model weights, the numbers training updates. TT counts the tokens in the sequence. xtx_t is the token at position tt, and x<tx_{\lt t} means every token before it. pθp_\theta is the probability your model assigns to the true next token. Mid-training never touches this formula. It only decides which sequences get fed into it.

The mix does the teaching

Write LdL_d for the average loss on domain sequences and LrL_r for the average loss on replay sequences. Pick two weights, wdw_d for the domain share and wrw_r for the replay share, with wd+wr=1w_d + w_r = 1. The batch loss is their weighted sum:

Lmix=wdLd+wrLrL_{mix} = w_d L_d + w_r L_r

So wd=0.9w_d = 0.9 and wr=0.1w_r = 0.1 means each batch is, on average, nine parts new domain to one part old general. Turn wdw_d up and the model absorbs the domain faster while general skills slide. Turn wrw_r up and general skills hold while the domain seeps in slowly. That dial is the trade-off of the whole phase, and setting the shares well matters more than any training trick. The full treatment of shares and schedules lives in data mixing.

Worked example

Take a tiny fixture batch of 2020 sequences with wd=0.9w_d = 0.9 and wr=0.1w_r = 0.1. Count first: 20×0.9=1820 \times 0.9 = 18 domain sequences and 20×0.1=220 \times 0.1 = 2 replay sequences, and 18+2=2018 + 2 = 20. Over 5050 such batches you would show 900900 domain sequences and 100100 replay ones out of 1,0001{,}000.

Now suppose the model measures Ld=2.30L_d = 2.30 on the domain slice and Lr=2.10L_r = 2.10 on the replay slice. Then:

Lmix=0.9×2.30+0.1×2.10=2.07+0.21=2.28L_{mix} = 0.9 \times 2.30 + 0.1 \times 2.10 = 2.07 + 0.21 = 2.28

Read it straight: the batch loss 2.282.28 sits near the domain loss because the batch is mostly domain, yet the replay term still pulls the gradient. Delete replay (wr=0w_r = 0) and the second term vanishes. The model then improves only at domain guesses, and general behavior quietly slides. That slide has its own name and its own page: forgetting.

Why long context lives here too

A model only trusts positions it trained on. Show it a much longer document cold and the late positions land outside everything it learned, so quality drops past the old edge. The fix is another mid-training run: keep the weights, remap or relearn the positional signal on long documents, and let the same loss adjust to the new lengths. Don't ask this page to teach the remap. It belongs to long-context extension.

When should you skip the phase entirely? When the need is behavior, not knowledge. If the model already knows the facts and only answers in the wrong shape, go straight to instruction tuning or supervised fine-tuning. Mid-training assumes your domain pile is large, clean, and worth the compute. Feed it thin or messy text and you pay full training cost to teach noise.

Code

The snippet below rebuilds the worked example: split one batch, then combine the two losses exactly as the training step does.

def split_batch(n: int, domain_share: float) -> tuple[int, int]:    n_domain = round(n * domain_share)    return n_domain, n - n_domain

def mixed_loss(l_domain: float, l_replay: float, domain_share: float) -> float:    return domain_share * l_domain + (1.0 - domain_share) * l_replay

print(split_batch(20, 0.9))  # -> (18, 2)print(f"{mixed_loss(2.30, 2.10, 0.9):.2f}")  # -> 2.28

The pair (18, 2) is the batch the diagram draws, and 2.28 matches the hand calculation above. Change domain_share to 1.0 and watch the replay term drop out. That is the zero-replay failure in one line.

Watch Out For

Setting replay to zero and watching general skills slide

The symptom shows up away from your domain. Your field's answers keep improving while coding help, math, or another language get worse week by week. Nothing errors. The domain curve looks great, so you keep going.

The cause is the missing anchor. With no replay, every gradient step rewards domain guesses only, and the shared weights drift. Fix it by putting the replay share back and watching both curves, not just the domain one. If general scores still sink, raise wrw_r before you touch anything else.

Expecting mid-training to teach manners

The symptom is a knowledgeable model with bad behavior. It knows the domain cold yet still continues your prompt instead of answering it, or ignores the format you asked for. Teams then run more mid-training, and the manners never arrive.

That's because mid-training changes what the model knows, not how it responds. The fix is the next phase, not more of this one: follow with instruction tuning on prompt plus ideal answer pairs, where the loss falls on answers only. If behavior is your only gap, skip mid-training completely.

The Quick Version

  • Mid-training resumes next-token prediction on dense domain text between pretraining and instruction tuning.
  • The objective never changes. Only the batch changes: mostly domain, with a small replay share of general text.
  • A 90/10 fixture batch of 20 holds 18 domain and 2 replay sequences, and losses of 2.30 and 2.10 combine to 2.28.
  • Replay is what keeps old skills alive. Zero replay teaches the domain while general ability slides.
  • It builds knowledge, not manners. Answering behavior still needs instruction tuning afterward.
  • Skip it when the facts are already there and only the response shape is wrong.