Gemini

The Problem

Before Gemini, multimodal models were typically "stitched" together. You would take a pre-trained text model (like LLaMA) and graft a pre-trained vision model (like CLIP) onto it (e.g., LLaVA). If you wanted audio, you added a speech-to-text model (like Whisper). This stitching caused information loss: a speech-to-text model loses the emotion, tone, and background noise of the audio. The models couldn't truly "reason" across modalities simultaneously because they weren't trained together.

The Idea

Google DeepMind built Gemini from scratch as a natively multimodal model. The architecture was designed to accept interleaved text, images, audio, and video natively as input tokens, and output interleaved text and images. By training on all these modalities simultaneously from the very first step of pre-training, Gemini developed a deep, unified understanding of the world, rather than just translating everything into text.

How It Works

Gemini processes multiple inputs natively:

1. Text: Tokenized via SentencePiece and fed as embeddings.
2. Vision (Images/Video): Video is treated as a sequence of frames. Images and frames are passed through a custom vision encoder to extract continuous visual embeddings, which are fed directly into the core Transformer alongside text.
3. Audio: Audio is passed through a Universal Speech Model (USM) encoder to capture raw audio features (including tone and non-speech sounds) without converting it to text first.

The models were released in three sizes:

• Ultra: The flagship model, designed for complex reasoning (matching or beating GPT-4).
• Pro: The workhorse model, optimized for cost and speed.
• Nano: A heavily distilled, 4-bit quantized version designed to run entirely locally on mobile phones (Pixel 8).

Why It Mattered

Gemini proved that native multimodality was the correct path forward for frontier models. By ingesting raw audio and video, the model could perform tasks impossible for text-bottlenecked models, like listening to a song and explaining the emotional tone, or watching a physics experiment video and calculating the velocity of an object.

What Came After

Google rapidly iterated, releasing Gemini 1.5. This version introduced a massive breakthrough in context windows (utilizing Ring Attention), allowing the model to process up to 2 million tokens simultaneously—enough to ingest entire codebases, hour-long videos, or entire series of books in a single prompt.