Flamingo: a Visual Language Model for Few-Shot Learning
Pioneered few-shot multimodal learning by injecting visual tokens directly into a frozen LLM using gated cross-attention, allowing it to adapt to new tasks rapidly.
Paper: Flamingo: a Visual Language Model for Few-Shot Learning
Authors: Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Mariante Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Cobo, Simon Osindero, Karen Simonyan, Oriol Vinyals · 2022
Read the paper
The Problem
By 2022, Large Language Models (LLMs) like GPT-3 had demonstrated incredible "few-shot" capabilities—you could show them 3 examples of a task in text, and they could accurately solve the 4th. However, vision-language models did not have this capability. If you wanted a model to classify a specific rare dog breed or analyze a new type of medical scan, you had to collect a large dataset and completely fine-tune the model. Training multimodal models was also incredibly expensive because the language backbone had to be retrained from scratch to understand the visual tokens.
The Idea
DeepMind's Flamingo solved both problems: it achieved state-of-the-art few-shot visual reasoning, and it did so by keeping the underlying LLM (Chinchilla) completely frozen. Instead of retraining the LLM, they trained a "bridge" between a frozen vision encoder and the frozen LLM. This bridge consisted of a Perceiver Resampler (to compress the image data) and Gated Cross-Attention layers (inserted between the frozen LLM layers).
How It Works
1. The Frozen Backbones Flamingo uses a pre-trained Normalizer-Free ResNet (NFNet) for image extraction, and a pre-trained LLM (like Chinchilla) for reasoning. Both are kept frozen to retain their massive pre-trained knowledge.
2. Perceiver Resampler An image might have thousands of visual features, and a video might have tens of thousands. Feeding all of these into an LLM would blow up the context window. The Perceiver Resampler uses cross-attention to compress any number of spatiotemporal visual features into a fixed, small number of visual tokens (e.g., 64 tokens per image/video).
3. Gated Cross-Attention To inject these 64 visual tokens into the frozen LLM, the authors inserted new cross-attention layers between the existing, frozen self-attention and feed-forward layers of the LLM. Crucially, these new layers are initialized with a tanh gating mechanism set to zero at the start of training. This means at step 0, the model acts exactly like a normal text-only LLM (the visual layers have 0 impact). Over training, the gate slowly opens, allowing the LLM to cleanly learn how to incorporate the visual information without destroying its pre-trained text abilities.
4. Interleaved Training Data The model was trained on massive datasets of freely interleaved image and text data from the web (e.g., a blog post with an image, some text, another image). This allowed it to naturally learn in-context few-shot prompting.
Why It Mattered
Flamingo was a landmark paper that defined the standard recipe for creating Vision-Language Models (VLMs) for years to come. The technique of freezing the LLM and training a small visual "adapter" (like the Perceiver and cross-attention gates) became the de facto standard, enabling the rapid explosion of open-source VLMs (like LLaVA and IDEFICS) that could be trained cheaply on top of existing models like LLaMA.
What Came After
Flamingo directly inspired open-source replications (IDEFICS) and heavily influenced the design of subsequent multimodal models that rely on adapter architectures or visual token injection (e.g., LLaVA, BLIP-2).