Image Captioning
Image captioning models act as a bridge between vision and language, taking an image as input and generating a coherent textual description of its contents.
Why Does This Exist?
While image classification can tell you that a picture contains a "dog" and a "frisbee", it cannot articulate that "a golden retriever is catching a red frisbee in mid-air." Real-world understanding requires composing recognized objects, their attributes, and their relationships into natural language. Image captioning exists to cross this modality gap, powering accessibility tools (like screen readers for the visually impaired), automated image indexing, and multimodal conversational agents.
Think of It Like This
The reporter and the typist
Imagine a two-person team writing a news article based on a photograph.
The first person is a visual analyst (the Image Encoder). They look at the photo and write down a dense, highly structured list of visual features: "Object A: Dog, Action: Jumping, Object B: Frisbee, Setting: Park."
They hand this list to the second person, a writer (the Text Decoder). The writer has never seen the photo, but they know how grammar works. They read the list and construct a fluent English sentence: "A dog is jumping to catch a frisbee in the park." Image captioning combines these two experts into one continuous pipeline.
How It Actually Works
The standard architecture for image captioning is an Encoder-Decoder model, often enhanced with visual attention.
- The Image Encoder (Vision): The input image is passed through a pre-trained Convolutional Neural Network (CNN) like ResNet, or a Vision Transformer (ViT). Instead of using the final classification layer, the network extracts the rich, high-dimensional feature maps from an intermediate layer. This vector representation encodes the spatial and semantic contents of the image.
- The Text Decoder (Language): The visual feature vector is fed into a language model, historically a Recurrent Neural Network (RNN/LSTM) but nowadays almost always a Transformer decoder. The language model uses the visual context as its starting state or cross-attention context.
- Autoregressive Generation: The decoder generates the caption one token (word or subword) at a time. To predict the next word, it looks at the visual features and all the words it has generated so far. It stops when it outputs a special
<EOS>(End of Sequence) token. - Visual Attention: Modern models don't just look at a single, static image vector. Using cross-attention, the language decoder dynamically "looks at" different parts of the image as it generates specific words. When generating the word "dog", the model's attention weights spike on the pixels containing the dog.
Code
# -> Simulating a single step of cross-attention in an image captionerimport torchimport torch.nn.functional as F
def visual_attention_step(): # Sequence of 16 image patches, each with a 256-dim feature vector image_features = torch.randn(1, 16, 256) # The decoder's current hidden state (e.g., trying to generate the next word) decoder_state = torch.randn(1, 1, 256) # Calculate attention scores (dot product between decoder state and image patches) attention_scores = torch.bmm(decoder_state, image_features.transpose(1, 2)) # -> Shape: [1, 1, 16] # Normalize scores into probabilities summing to 1 attention_weights = F.softmax(attention_scores, dim=-1) # Compute the context vector: a weighted sum of the image patches context_vector = torch.bmm(attention_weights, image_features) # -> Shape: [1, 1, 256] return context_vector.shapeWatch Out For
Language Prior Hallucination
Because the text decoder is a strong language model, it can sometimes "hallucinate" details based on language priors rather than visual evidence. If the model sees a kitchen counter, it might aggressively predict a "microwave" simply because microwave commonly appears with kitchens in its training text, even if none is actually in the image.
Exposure Bias
During training, the decoder is fed the true previous words (teacher forcing). During inference, it must rely on its own generated words. If it makes a mistake early in the sentence, the error cascades because the model has never been trained to recover from its own bad predictions.
The Quick Version
- Image captioning translates visual data into natural language sentences.
- It typically uses an Encoder-Decoder architecture: a Vision model (CNN/ViT) encodes the image, and a Language model (Transformer/LSTM) decodes the text.
- Text is generated autoregressively, predicting one word at a time based on the image and previous words.
- Visual attention mechanisms allow the model to focus on specific regions of the image relevant to the word currently being generated.
- Hallucination is a common issue, where the model outputs plausible-sounding objects that aren't actually present.