DeepSeek-R1

The Problem

By late 2024, the AI industry recognized that standard post-training (Supervised Fine-Tuning + basic RLHF) had hit a wall. To make models significantly smarter, they needed to "think" before they spoke—generating long, hidden chains of thought to solve complex math and logic problems.

OpenAI released the o1 family of reasoning models, but their exact training methods were a closely guarded secret. The prevailing assumption in the industry was that to train a model like o1, you needed millions of examples of highly complex, human-written reasoning chains to use for Supervised Fine-Tuning, followed by expensive reinforcement learning. Acquiring millions of flawless, step-by-step mathematical proofs written by humans is incredibly expensive and slow, creating a massive data moat.

The Idea

DeepSeek proved the industry wrong. In their R1 paper, they demonstrated that you do not need a massive supervised dataset of human reasoning to teach a model how to think.

Instead, you can take a raw base model (which only knows how to predict the next word) and immediately hit it with large-scale Reinforcement Learning (using the GRPO algorithm). As long as you provide the model with a reward signal (e.g., "Did you get the right answer to this math problem?"), the model will figure out on its own that the best way to maximize its reward is to think step-by-step, verify its own work, and backtrack when it makes a mistake.

How It Works

The DeepSeek-R1 training pipeline was a masterclass in efficiency and emergent behavior:

1. Pure RL (DeepSeek-R1-Zero): First, they took a base model and applied GRPO without any supervised reasoning data. The rewards were purely rule-based (e.g., accuracy of a math answer, format compliance). During this massive RL phase, the model spontaneously learned to write <think> tags and developed complex reasoning behaviors. It experienced measurable "aha" moments where it realized a mistake mid-sentence and corrected itself. However, R1-Zero's outputs were often ugly, mixing languages and ignoring formatting.
2. Cold-Start SFT + RL (DeepSeek-R1): To fix the readability issues, they collected a tiny "cold-start" dataset of a few thousand high-quality reasoning examples. They fine-tuned the base model on this tiny dataset to teach it how to format its thoughts neatly, and then applied the massive GRPO reinforcement learning phase. This produced the final DeepSeek-R1 model, which matched or exceeded OpenAI's o1 on major reasoning benchmarks.
3. Distillation: Finally, DeepSeek proved that once you have a massive reasoning model like R1, you don't need to run the expensive RL process again for smaller models. You can just have R1 generate thousands of reasoning trajectories, and use standard Supervised Fine-Tuning to teach smaller models (like LLaMA 8B or Qwen 32B) to mimic R1's thought process. This "distillation" process successfully transferred R1's reasoning capabilities into tiny, open-weight models that can run on a laptop.

Why It Mattered

DeepSeek-R1 was a tectonic shift in the AI landscape. It proved that the "intelligence" of reasoning models doesn't come from human data; it emerges mathematically from the reinforcement learning process itself. This destroyed the concept of a data moat for reasoning capabilities.

Furthermore, DeepSeek accomplished this using their own highly efficient architecture (MoE with Multi-Head Latent Attention) and the GRPO algorithm, spending only a few million dollars on compute—a fraction of what American frontier labs were spending. The open-sourcing of both the R1 weights and the distilled smaller models completely democratized access to state-of-the-art reasoning capabilities.

What Came After

The release of DeepSeek-R1 essentially open-sourced the o1 paradigm. Immediately, the entire AI community pivoted to replicating the R1 pipeline. The distilled versions of R1 became the default models for local developers. It forced a massive re-evaluation of compute scaling laws, shifting the industry's focus away from acquiring more human data and toward finding more efficient ways to run large-scale reinforcement learning.