Gen AI

Mid-Training

Mid-training is the bridge between raw pretraining and specialized fine-tuning, used to inject deep domain knowledge or extend context windows at scale before narrowing the model's focus to instruction following.

Why Does This Exist?

Pretraining builds a general world model, and instruction tuning teaches it to be an assistant. But what if you need an assistant that is a world-class expert in analyzing medical records?

If you just take a general base model and instruction-tune it on 10,000 medical Q&A pairs, it will learn the format of answering medical questions, but it won't actually learn deep medical knowledge—there simply aren't enough tokens in an instruction dataset to fundamentally alter its understanding of biology.

Mid-training (often called continued pretraining) solves this. Before you instruction-tune the model, you take the base model and resume the unsupervised next-token prediction process, but this time exclusively on billions of tokens of medical textbooks, PubMed papers, and clinical notes. You are extending its fundamental knowledge base before you teach it how to talk.

Think of It Like This

Undergraduate, Med School, and Bedside Manner

Pretraining is an undergraduate liberal arts degree. You learn to read, write, and think generally. Mid-training is Medical School. You spend four years reading dense textbooks and acquiring deep, domain-specific knowledge. Instruction Tuning is bedside manner training. It is the short residency where you are taught exactly how to talk to a patient using the knowledge you already acquired.

You cannot teach bedside manner to someone who hasn't been to medical school and expect them to cure a disease.

How It Actually Works

The Mechanics

Technically, mid-training is identical to pretraining. The objective function is exactly the same: unsupervised next-token prediction. The difference is entirely in the dataset and the scale.

Scale: While pretraining might use 15 trillion tokens over 3 months, mid-training might use 50 billion tokens over 2 weeks.
Dataset: The dataset is heavily curated toward the target domain (e.g., law, medicine, finance, or a specific non-English language).

Context Window Extension

Mid-training is also the primary phase where context windows are extended. If a base model was trained on 4K context sequences, it cannot suddenly process a 128K document. To extend it, engineers perform a mid-training phase where the sequence length is gradually increased to 128K tokens. The model is forced to predict tokens at the very end of massive documents, adjusting its positional embeddings (like RoPE) to handle extreme lengths.

Replay and Forgetting

Just like in data mixing, if you train a model entirely on medical text for billions of tokens, it will forget how to write code or answer casual questions. To prevent this catastrophic forgetting, mid-training datasets always include a small "replay" percentage of the original general pretraining data (usually 5–10%). This anchors the general capabilities while the model absorbs the new domain.

Watch Out For

Skipping mid-training for domain expertise

Many teams try to inject complex domain knowledge purely through Supervised Fine-Tuning (SFT) or RAG. SFT is for behavior, not knowledge. If the base model doesn't already know the intricacies of your proprietary coding language, 500 examples in SFT won't teach it; it will just hallucinate in the correct format. It needs mid-training.

The Quick Version

Mid-training sits between pretraining and instruction tuning.
It uses the same unsupervised next-token prediction objective as pretraining.
It is used to inject massive amounts of domain-specific knowledge (law, medicine, code) or to extend the model's context window.
It requires a small mix of general data to prevent the model from forgetting its foundational knowledge.