LLaMA 1
The 2023 paper from Meta that introduced the first highly capable, open-weights foundation model, sparking the open-source LLM revolution.
Paper: LLaMA: Open and Efficient Foundation Language Models
Authors: Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample · 2023
Read the paper
The Problem
By early 2023, the most capable LLMs (GPT-3, PaLM, Chinchilla) were closed, proprietary, and accessible only via API. The open-source community relied on older, weaker models (like OPT or BLOOM) which were often poorly trained. Researchers outside of big tech lacked access to a state-of-the-art foundation model to study, fine-tune, or build upon. Furthermore, the "Chinchilla" scaling laws proved that models were under-trained, but nobody had released an open model trained according to those laws.
The Idea
Meta AI released LLaMA, a family of foundation models ranging from 7B to 65B parameters. The core philosophy was to strictly follow the Chinchilla scaling laws, and then go further: they "over-trained" the models on vastly more data (up to 1.4 Trillion tokens) than was strictly optimal for the training compute budget. Their goal was to produce models with the best possible inference efficiency, meaning they would run fast and cheap on consumer hardware while matching the performance of much larger models.
How It Works
LLaMA made several key architectural tweaks to the standard Transformer to improve stability and performance:
- Pre-normalization (RMSNorm): To improve training stability, the input to each transformer sub-layer is normalized (instead of the output), using the simpler RMSNorm instead of standard LayerNorm.
- SwiGLU Activation: They replaced the standard ReLU non-linearity with the SwiGLU activation function, which yielded better performance.
- Rotary Positional Embeddings (RoPE): Instead of absolute positional embeddings, they injected absolute position at every layer using RoPE, which better captures the relative distance between tokens.
Crucially, the entire training dataset (1.4T tokens) was composed entirely of publicly available data, proving you didn't need proprietary datasets to reach state-of-the-art performance.
Why It Mattered
LLaMA leaked to the public via a torrent shortly after release and single-handedly ignited the open-source AI boom. Because the 7B and 13B models could be quantized to run on consumer GPUs and even MacBooks, thousands of developers began fine-tuning them (creating Alpaca, Vicuna, etc.), rapidly democratizing AI research.
What Came After
LLaMA established the baseline architecture for almost all subsequent open-weights models. The community quickly developed techniques like LoRA (for cheap fine-tuning) and GGML/GGUF (for CPU/Edge inference) specifically to run LLaMA models. It paved the way for Llama 2 (with a commercial license) and the modern open-source ecosystem.