Skip to content
AI360XpertBeta

Visual Question Answering (VQA)

You show the model a photo and ask a written question about it. It looks at the right patches, mixes what it sees with what you asked, and picks one answer.

VQA fuses image regions and question words with attention weights 0.55, 0.25 and 0.20 to pick one answer
VQA fuses image regions and question words with attention weights 0.55, 0.25 and 0.20 to pick one answer

Why Does This Exist?

You can already buy a detector that draws neat boxes around every chair, table and lamp in a kitchen photo. Ask that same system which object sits left of the chair and you get silence. Boxes say where things are. They do not read your question, compare two objects, or pick a phrase as the answer.

That silence is why visual question answering exists. You hand the model pixels plus words and it must hand back words. Is the worker wearing goggles? Which bar is tallest? What does the sign say? One input shape, many kinds of reasoning. Detection alone cannot do any of them because the answer lives in the link between what you asked and what the camera caught. You do not get reasoning from coordinates. You get coordinates.

This page stays on that link: how regions get scored against your question, mixed by weight, and decoded into one pick. The eyes underneath live in vision-language models. The trick that makes image and text numbers comparable lives in contrastive vision-language pretraining. The mixing step itself gets its own treatment in multimodal fusion.

The one idea you need first: a vision-language model turns image patches into vectors the language model can read, much like word vectors turn words into numbers. If that sentence feels fuzzy, read the vision-language page first, then come back. You will need it.

Think of It Like This

Asking a friend about a fridge photo

You text a friend a photo of your open fridge and ask, is there enough milk for pancakes tomorrow?

She does three things you can watch. She scans the whole shelf to get oriented. She rereads your question and lets the word milk pull her eyes to the cartons, ignoring the magnets and leftovers. She compares what she sees against what she knows about pancakes and answers, yes, or no, nearly empty.

That maps cleanly. The photo is the image features. The words milk and pancakes are the question vector. Her focus on the cartons is attention. Her spoken answer is the decode.

It stops here. Your friend knows your breakfast habits and will ask what you mean when the photo is blurry. The model knows only pixels plus training patterns, and when the evidence is missing it still answers. That last habit is the failure to watch for.

How It Actually Works

You are asking the model to point with numbers, then speak. The pointing is attention. The speaking is a short decode from the pointed view plus your question.

From patches and words to one answer

The model cuts your photo into regions. Call region ii's visual vector viv_i, plain words: what patch ii looks like, as a list of numbers. It encodes your question into one vector qq, plain words: what you asked, as a list of numbers in the same space. Both come from a vision-language backbone, which is why the VQA page assumes you have met that idea.

Then it scores every region against your question with si=q⋅vis_i = q \cdot v_i, plain words: the relevance score for region ii, computed as a dot product, the overlap between what you asked and what region ii shows. A region that matches the question gets a high score. One that does not gets a low score.

Scores are not shares yet, so it normalizes them with a softmax. Call the result aia_i, plain words: the attention weight for region ii, its share of focus. The rule is ai=exp⁡(si)/∑jexp⁡(sj)a_i = \exp(s_i) / \sum_j \exp(s_j), plain words: exponentiate each score with exp⁡\exp, the exponential function, then divide by the sum over all regions jj so the weights add to 11. The symbol ∑j\sum_j just means add up over every region.

Next it mixes with vˉ=∑iaivi\bar{v} = \sum_i a_i v_i, plain words: the pooled visual vector, a focus-weighted average where each region contributes in proportion to its weight. High weight, loud voice. Low weight, whisper. Finally it decodes to y^\hat{y}, plain words: the predicted answer, the model's pick, chosen from the pooled view plus the question, either by a classifier over a fixed answer set or by a short language generate.

The trade is plain. You buy open-ended questions with one model instead of many narrow ones. You pay in compute, data hunger and a new way to be wrong with confidence. When your labels are fixed, use a detector. When you must count exact instances or read tiny print, use a detector plus an OCR path like document understanding. When the fact is fresh or external, retrieve first instead of asking the pixels.

The assumption underneath: the answer is visible or can be inferred from the pixels plus training knowledge. Break it and VQA fails the same way every time. Blur the sign, shrink the text below what the encoder can resolve, or ask about something that left the frame, and the weights still sum to 11 and the model still picks. Certainty stays. Evidence does not.

Worked example

Take a kitchen corner with three regions. R1 holds the table, R2 the chair, R3 the lamp. Your question is which is left of the chair. Suppose the scorer returns s=[1.0,0.2,0.0]s = [1.0, 0.2, 0.0], plain words: R1 matches best, R2 weakly, R3 not at all.

Exponentiate to [2.72,1.22,1.00][2.72, 1.22, 1.00] since exp⁡(1.0)≈2.72\exp(1.0) \approx 2.72, exp⁡(0.2)≈1.22\exp(0.2) \approx 1.22 and exp⁡(0.0)=1.00\exp(0.0) = 1.00. Add them to 4.944.94. Divide each by the sum:

a1=2.72/4.94≈0.55,a2=1.22/4.94≈0.25,a3=1.00/4.94≈0.20a_1 = 2.72 / 4.94 \approx 0.55,\quad a_2 = 1.22 / 4.94 \approx 0.25,\quad a_3 = 1.00 / 4.94 \approx 0.20

So a=[0.55,0.25,0.20]a = [0.55, 0.25, 0.20] and 0.55+0.25+0.20=1.000.55 + 0.25 + 0.20 = 1.00. R1 takes more than half the focus, the pooled view vˉ\bar{v} is mostly table, and y^\hat{y} comes out as table. Ask a new question about the lamp and the same three boxes get new scores, new weights and a new pick. That is the whole trick the diagram draws: same regions, question-dependent focus.

Code

This snippet is the check from the worked example. It turns the three scores into weights and names the winner, which is what the fusion step does before any words are generated.

import numpy as np

def attention_weights(scores: list[float]) -> list[float]:    s = np.array(scores, dtype=float)    e = np.exp(s - np.max(s))  # subtract max for stability, same result    return [float(x) for x in e / e.sum()]

w = attention_weights([1.0, 0.2, 0.0])print([f"{x:.2f}" for x in w])  # -> ['0.55', '0.25', '0.20']print(int(np.argmax(w)))  # -> 0

The first line prints the three shares that sum to 11. The second line prints 00, the index of R1, so the answer path starts from the table region. Change the scores and the winner changes, which is exactly what a new question does.

Watch Out For

Answering from the question alone

The model learns your language habits faster than it learns to look. Ask what color is the banana and it says yellow even when your photo shows a green one, because yellow was the common answer in training. The symptom is easy to miss: accuracy looks fine until the photo contradicts the prior, and blanking the image barely drops the score because the text path carried the pick.

Fix it with pairs. Take one question and two photos that demand different answers, then require the model to flip. If it does not flip, it is not reading. Prefer evaluations built from such pairs and distrust a VQA score from questions the language prior can guess. For rare cases in production, add a grounding check before you act on the answer.

Inventing detail the pixels cannot support

Small print and occluded objects invite fluent fiction. The model reads a blurry door sign as closes at 9 pm with total confidence, then gives the same time after you crop the sign out entirely. Symptom: a specific answer with almost no weight on the region that should prove it. The cause is the same assumption from above: the decoder must produce words whether or not the evidence resolved.

Give it a way out. Allow a not visible answer, run a zoomed second pass before trusting tiny text, and route document-heavy questions through a tiling OCR path like document understanding. For high-stakes calls, require the model to name the region it used and refuse to act when that region is empty. If you want the mechanism behind the fluency, read hallucination mechanisms.

The Quick Version

  • VQA takes an image plus a written question and returns one answer that depends on both inputs at once.
  • Each region is scored against your question, softmax turns those scores into shares like 0.55, 0.25 and 0.20, and the weighted mix decides the pick.
  • It buys open-ended reasoning at the price of compute, data hunger and confident wrong answers when evidence is thin.
  • Do not use it for fixed labels, exact counts or tiny print; a detector or OCR path is cheaper and more exact there.
  • If blanking the image barely changes the answer, the model is reading the question, not the photo. Test it with paired images that force a flip.