Visual Question Answering (VQA)
You show the model a photo and ask a written question about it. It looks at the right patches, mixes what it sees with what you asked, and picks one answer.
Why Does This Exist?
You can already buy a detector that draws neat boxes around every chair, table and lamp in a kitchen photo. Ask that same system which object sits left of the chair and you get silence. Boxes say where things are. They do not read your question, compare two objects, or pick a phrase as the answer.
That silence is why visual question answering exists. You hand the model pixels plus words and it must hand back words. Is the worker wearing goggles? Which bar is tallest? What does the sign say? One input shape, many kinds of reasoning. Detection alone cannot do any of them because the answer lives in the link between what you asked and what the camera caught. You do not get reasoning from coordinates. You get coordinates.
This page stays on that link: how regions get scored against your question, mixed by weight, and decoded into one pick. The eyes underneath live in vision-language models. The trick that makes image and text numbers comparable lives in contrastive vision-language pretraining. The mixing step itself gets its own treatment in multimodal fusion.
The one idea you need first: a vision-language model turns image patches into vectors the language model can read, much like word vectors turn words into numbers. If that sentence feels fuzzy, read the vision-language page first, then come back. You will need it.
Think of It Like This
Asking a friend about a fridge photo
You text a friend a photo of your open fridge and ask, is there enough milk for pancakes tomorrow?
She does three things you can watch. She scans the whole shelf to get oriented. She rereads your question and lets the word milk pull her eyes to the cartons, ignoring the magnets and leftovers. She compares what she sees against what she knows about pancakes and answers, yes, or no, nearly empty.
That maps cleanly. The photo is the image features. The words milk and pancakes are the question vector. Her focus on the cartons is attention. Her spoken answer is the decode.
It stops here. Your friend knows your breakfast habits and will ask what you mean when the photo is blurry. The model knows only pixels plus training patterns, and when the evidence is missing it still answers. That last habit is the failure to watch for.
How It Actually Works
You are asking the model to point with numbers, then speak. The pointing is attention. The speaking is a short decode from the pointed view plus your question.
From patches and words to one answer
The model cuts your photo into regions. Call region 's visual vector , plain words: what patch looks like, as a list of numbers. It encodes your question into one vector , plain words: what you asked, as a list of numbers in the same space. Both come from a vision-language backbone, which is why the VQA page assumes you have met that idea.
Then it scores every region against your question with , plain words: the relevance score for region , computed as a dot product, the overlap between what you asked and what region shows. A region that matches the question gets a high score. One that does not gets a low score.
Scores are not shares yet, so it normalizes them with a softmax. Call the result , plain words: the attention weight for region , its share of focus. The rule is , plain words: exponentiate each score with , the exponential function, then divide by the sum over all regions so the weights add to . The symbol just means add up over every region.
Next it mixes with , plain words: the pooled visual vector, a focus-weighted average where each region contributes in proportion to its weight. High weight, loud voice. Low weight, whisper. Finally it decodes to , plain words: the predicted answer, the model's pick, chosen from the pooled view plus the question, either by a classifier over a fixed answer set or by a short language generate.
The trade is plain. You buy open-ended questions with one model instead of many narrow ones. You pay in compute, data hunger and a new way to be wrong with confidence. When your labels are fixed, use a detector. When you must count exact instances or read tiny print, use a detector plus an OCR path like document understanding. When the fact is fresh or external, retrieve first instead of asking the pixels.
The assumption underneath: the answer is visible or can be inferred from the pixels plus training knowledge. Break it and VQA fails the same way every time. Blur the sign, shrink the text below what the encoder can resolve, or ask about something that left the frame, and the weights still sum to and the model still picks. Certainty stays. Evidence does not.
Worked example
Take a kitchen corner with three regions. R1 holds the table, R2 the chair, R3 the lamp. Your question is which is left of the chair. Suppose the scorer returns , plain words: R1 matches best, R2 weakly, R3 not at all.
Exponentiate to since , and . Add them to . Divide each by the sum:
So and . R1 takes more than half the focus, the pooled view is mostly table, and comes out as table. Ask a new question about the lamp and the same three boxes get new scores, new weights and a new pick. That is the whole trick the diagram draws: same regions, question-dependent focus.
Code
This snippet is the check from the worked example. It turns the three scores into weights and names the winner, which is what the fusion step does before any words are generated.
import numpy as np
def attention_weights(scores: list[float]) -> list[float]: s = np.array(scores, dtype=float) e = np.exp(s - np.max(s)) # subtract max for stability, same result return [float(x) for x in e / e.sum()]
w = attention_weights([1.0, 0.2, 0.0])print([f"{x:.2f}" for x in w]) # -> ['0.55', '0.25', '0.20']print(int(np.argmax(w))) # -> 0The first line prints the three shares that sum to . The second line prints , the index of R1, so the answer path starts from the table region. Change the scores and the winner changes, which is exactly what a new question does.
Watch Out For
Answering from the question alone
The model learns your language habits faster than it learns to look. Ask what color is the banana and it says yellow even when your photo shows a green one, because yellow was the common answer in training. The symptom is easy to miss: accuracy looks fine until the photo contradicts the prior, and blanking the image barely drops the score because the text path carried the pick.
Fix it with pairs. Take one question and two photos that demand different answers, then require the model to flip. If it does not flip, it is not reading. Prefer evaluations built from such pairs and distrust a VQA score from questions the language prior can guess. For rare cases in production, add a grounding check before you act on the answer.
Inventing detail the pixels cannot support
Small print and occluded objects invite fluent fiction. The model reads a blurry door sign as closes at 9 pm with total confidence, then gives the same time after you crop the sign out entirely. Symptom: a specific answer with almost no weight on the region that should prove it. The cause is the same assumption from above: the decoder must produce words whether or not the evidence resolved.
Give it a way out. Allow a not visible answer, run a zoomed second pass before trusting tiny text, and route document-heavy questions through a tiling OCR path like document understanding. For high-stakes calls, require the model to name the region it used and refuse to act when that region is empty. If you want the mechanism behind the fluency, read hallucination mechanisms.
The Quick Version
- VQA takes an image plus a written question and returns one answer that depends on both inputs at once.
- Each region is scored against your question, softmax turns those scores into shares like 0.55, 0.25 and 0.20, and the weighted mix decides the pick.
- It buys open-ended reasoning at the price of compute, data hunger and confident wrong answers when evidence is thin.
- Do not use it for fixed labels, exact counts or tiny print; a detector or OCR path is cheaper and more exact there.
- If blanking the image barely changes the answer, the model is reading the question, not the photo. Test it with paired images that force a flip.