InstructGPT
The 2022 paper that detailed the RLHF methodology used to align GPT-3 into a helpful assistant, directly laying the groundwork for ChatGPT.
Paper: Training language models to follow instructions with human feedback
Authors: Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe · 2022
Read the paper
The Problem
Base models like GPT-3 (released in 2020) were incredibly capable next-token predictors, but they were notoriously difficult to use. If you prompted it with "Write a story about a frog," it might just output "Write a story about a dog." It was mimicking internet text, not acting as a helpful assistant. To make LLMs useful to consumers, they needed to be aligned to user intent.
The Idea
OpenAI applied Reinforcement Learning from Human Feedback (RLHF) to language models. While the concept of RLHF existed earlier, InstructGPT was the first massive, successful application of it to a frontier model. By having humans demonstrate how to answer prompts, and then having humans vote on which AI-generated answers were better, OpenAI trained a Reward Model. They then used an RL algorithm (PPO) to train the LLM to maximize this reward, transforming an unpredictable text generator into a polite, helpful assistant.
How It Works
The InstructGPT recipe (which became the industry standard):
Step 1: Supervised Fine-Tuning (SFT). Human labelers write high-quality prompts and answers. The base GPT-3 model is fine-tuned on this dataset to learn the basic format of an assistant.
Step 2: Reward Modeling (RM). The SFT model generates several different answers to a single prompt. A human ranks the answers from best to worst. A smaller "Reward Model" is trained to predict these human rankings (outputting a scalar score for any given text).
Step 3: Reinforcement Learning (PPO). The SFT model is exposed to new prompts. It generates an answer, the Reward Model scores it, and the Proximal Policy Optimization (PPO) algorithm updates the SFT model's weights to generate higher-scoring answers in the future, while penalizing it for straying too far from the original SFT model.
Why It Mattered
InstructGPT was a massive success: human labelers significantly preferred the outputs of a 1.3B parameter InstructGPT model over the massive 175B parameter un-aligned GPT-3 model. This paper was the exact technical precursor to ChatGPT, which was launched later that same year.
What Came After
RLHF became the mandatory final step for almost all LLMs. It sparked a massive industry of human data labeling (companies like Scale AI). However, because RLHF is expensive and complex, newer techniques like DPO (Direct Preference Optimization) were later developed to bypass the need for a separate Reward Model and PPO phase.