Orca

The Problem

Following the success of "Self-Instruct" models like Alpaca and Vicuna, researchers realized a major flaw: while these smaller open-source models sounded like GPT-4 (mimicking its tone and style), they failed miserably on tasks requiring actual reasoning. They were simply copying the "surface level" patterns of the teacher model's outputs without inheriting any of the underlying logic or knowledge.

The Idea

Microsoft researchers introduced Orca. They hypothesized that you couldn't teach a small model to reason just by showing it the prompt and the final answer. You had to show it the process. They heavily utilized System Instructions to force GPT-4 (the teacher) to "show its work." By prompting GPT-4 to "think step-by-step," "explain like I'm five," or "justify your answer," they generated a dataset of 5 million complex explanation traces. Training a 13B parameter model on these detailed explanations allowed it to internalize the reasoning process.

How It Works

The Orca training process involves two key innovations:

1. Explanation Tuning: Instead of standard prompt-response pairs, the training data consists of prompt-explanation-response triples. The small model is supervised not just to generate the correct final answer, but to generate the same logical steps the teacher took to get there.
2. Progressive Learning: They first trained the model on 5 million simpler examples generated by ChatGPT (GPT-3.5). Then, they further trained it on 1 million highly complex examples generated by GPT-4. This curriculum learning approach helped the smaller model build foundational skills before tackling the hardest problems.

Why It Mattered

Orca 13B vastly outperformed other open-source models of its era, particularly on zero-shot reasoning benchmarks like AGIEval and BigBench Hard. It proved that "imitation learning" (distillation) works, but only if you distill the reasoning process, not just the final output. It effectively established Chain-of-Thought (CoT) data as mandatory for high-quality fine-tuning.

What Came After

Microsoft continued the series with Orca 2, which taught small models to choose which reasoning strategy to use based on the task. The principle of "explanation tuning" became universal; today, almost all synthetic datasets (like OpenHermes) require the teacher model to generate Chain-of-Thought reasoning traces.