Gen AI

Data Mixing and Curriculum

You don't just dump terabytes of data into a model; you carefully balance the proportions of code, math, and prose (mixing) and sometimes order them from simple to complex (curriculum) so the model learns grammar before calculus.

Data mixing assigns strict probability weights to different domains, ensuring a balanced diet of code, math, and general text during pretraining.

Why Does This Exist?

If you scrape the internet to train an LLM, your dataset will be 90% repetitive SEO blogs and Reddit arguments, and 1% highly structured code and mathematics. If you just train on this raw distribution, the model will become excellent at arguing on forums and terrible at logic and reasoning.

To build a generally capable model, you have to engineer the data diet. Data mixing is the act of deciding exactly what percentage of each training batch should come from Wikipedia, GitHub, scientific papers, or web crawls. Curriculum learning takes it a step further by changing that mixture over time—starting with simple foundational text (like a syllabus) and gradually introducing complex math and domain-specific knowledge as the model matures.

Think of It Like This

A balanced diet and a school curriculum

If you let a child eat whatever they want based on what is most abundant in the house, they will eat 90% carbohydrates and sugar. To build a healthy adult, you enforce a strict mix: 30% protein, 40% greens, 30% carbs. That is data mixing.

Furthermore, you don't teach a child calculus in kindergarten. You start with counting, then algebra, then calculus. That is curriculum learning. You sequence the complex information only after the foundational capabilities are in place.

How It Actually Works

The Mixing Weights

When a pretraining run is configured, engineers define a strict sampling distribution. For example, even if code only makes up 5% of the total available tokens on disk, the training script might be told to pull 20% of its data from the code bucket in every batch.

This upsampling is critical. Training on code has been empirically shown to improve a model's general reasoning and logic capabilities across the board, even in natural language tasks. Conversely, low-quality web text is aggressively downsampled.

The Law of Catastrophic Forgetting

Why not just train on all the text first, and then train entirely on code later? Neural networks suffer from "catastrophic forgetting"—if they stop seeing a type of data, their weights will overwrite those pathways to optimize for the new data. If you train on French for a month, and then switch entirely to Python for a month, the model will forget how to speak French.

Therefore, data mixing must be continuous. The domains must be interleaved constantly, ensuring the gradients pull the weights toward a state that accommodates everything simultaneously.

Curriculum Learning Stages

While the mixture is continuous, the ratios can shift over the training run. A common curriculum involves:

Early Training: Heavy emphasis on massive web crawls and general prose to learn the basic statistical structure of language (grammar, syntax, facts).
Late Training (or Mid-Training): The mixture shifts. Web text is downsampled, and high-quality, dense data (STEM papers, code, mathematics) is heavily upsampled. This refines the model's reasoning abilities after it already knows how to "read."

Watch Out For

Over-sampling a domain

If you upsample a high-quality dataset too aggressively (meaning the model sees the exact same documents multiple times per epoch), the model will rapidly memorize those documents rather than generalizing. This leads to overfitting, where the model can recite Wikipedia verbatim but fails at novel reasoning.

The Quick Version

Data Mixing is the process of defining specific sampling probabilities for different data domains (code, math, web text) to ensure a balanced capability set, regardless of how much of that data actually exists on disk.
Curriculum Learning involves shifting those mixture weights over time, usually starting with broad knowledge and ending with dense, complex reasoning data.
Code and mathematics are frequently upsampled because they dramatically improve general reasoning.
Interleaving data types is required to prevent catastrophic forgetting.