Multimodal RAG
Text-only search cannot read charts. Multimodal RAG files images next to text, so a question about a bar chart pulls the chart itself into the answer.
Why Does This Exist?
Your report holds the answer on page 4: a bar chart of 2024 revenue growth by quarter. Your RAG system never saw it. The parser kept the paragraphs, dropped the figure, and your question about growth gets a shrug. Or worse, a confident answer built from the wrong paragraph.
That blindness scales badly. Slide decks, equipment manuals, scanned forms, pathology slides: in all of them the picture carries the fact and the prose around it is commentary. A text-only pipeline reads the commentary and bins the fact. Multimodal RAG closes that gap by making pictures retrievable, so the evidence behind an answer can be a chart instead of only a paragraph.
The one idea you need first. Standard RAG chops documents into text chunks, ranks them against your question, and hands the winners to a model that writes the answer. The full loop is retrieval-augmented generation. Everything here keeps that loop and widens the index: chunks can now be images too.
This page covers the widened index and the two ways to build it. How a model actually reads pixels belongs to vision-language models, and pulling tables out of documents continues in document understanding.
Think of It Like This
A photo library whose librarian can actually look
Your family photos live in boxes, each with a sticky note: "beach, 2019." You ask your cousin for the sunset photo where Dad holds the big fish. If someone wrote it on the note, you win. If nobody wrote it down, the photo might as well not exist. That is text-only RAG. It searches the notes, never the pictures.
Now the library hires a librarian with eyes. She studies every photo herself and files each one by what it shows. Your question goes to her, she walks to the shelf, and she hands you the photo itself. The sticky notes stay for anything she has not studied yet, but the sunset picture is found because she looked at it, not because someone described it.
Where it stops. The librarian understands any photo at a glance, and your model does not. It matches patterns it trained on, it misreads small print, and it cannot reason about anything legible but unfamiliar. Treat its ranking as a suspect lineup, not a verdict. The reader still has to look.
How It Actually Works
Two builds, one ranking rule. Both end with your question scored against every chunk, whatever the chunk is made of.
Write your question as a vector , a list of numbers an encoder produced from your words. Write chunk as , its own list of numbers from the same or a paired encoder. The index just numbers the chunks: 1, 2, 3, and so on. The score is the cosine of the angle between the two lists:
In plain words: is the dot product, pairwise multiply then add, which is large when the lists point the same way. and are the lengths of each list, the norms, and dividing by both removes size so only direction counts. lands between and , and bigger wins. Rank every chunk, keep the top , and hand the winners with their ids to a reader model that can see images. is just how many winners you keep; the example below uses 2.
The two builds
Joint embeddings. One contrastive model trains a text encoder and an image encoder side by side, so the sentence "a rising bar chart" and a picture of one land near each other. You index both vector kinds in one store and score directly. This is the short path, and the one the diagram draws. Its weakness is fine print: these encoders match broad shapes and scenes, not axis labels. The training trick behind them is contrastive vision-language pretraining.
Captions at ingest. A vision-language model reads each figure once, up front, and writes a dense description: "bar chart, 2024 revenue growth by quarter, Q4 tallest." You index that description as ordinary text, so retrieval stays text-only and every mature text trick keeps working. The price is ingest cost on every image plus a new failure mode: whatever the caption writer misreads becomes your ground truth.
Worked example
Three chunks, one question. The question is "2024 revenue growth" and its toy vector is . Toy means two dimensions so you can check the arithmetic; production vectors run hundreds of dimensions under the same cosine rule.
- , the bar-chart image: .
- , a text paragraph about 2023 results: .
- , an unrelated hallway photo: .
Lengths first: . , , .
Dots next: . . .
Scores:
So the ranking is at , at , at . With the store keeps the chart and the paragraph, drops the hallway, and the answer cites []. Notice mentions revenue too, but its 2023 direction points elsewhere in the space, which is exactly what the 0.60 says.
From winners to an answer
The reader receives the top chunks with their ids: the chart pixels for , the paragraph text for . It reads the Q4 bar off the image, writes the sentence, and attaches the citation [] so you can check the figure yourself. And know when to skip all of this: a born-digital PDF with a real text layer parses exactly and cheaply, and a single image with no archive behind it needs no index. Just ask the model.
Code
The snippet below is a check on the hand computation above, not a second lesson. Same vectors, same ranking, in plain Python with no extra packages:
import math
def cosine(a: list[float], b: list[float]) -> float: """Cosine of the angle between two vectors; 1.0 means same direction.""" dot = sum(x * y for x, y in zip(a, b)) norm_a = math.sqrt(sum(x * x for x in a)) norm_b = math.sqrt(sum(y * y for y in b)) return dot / (norm_a * norm_b)
q: list[float] = [3.0, 1.0]chunks: dict[str, list[float]] = { "C1": [3.0, 0.0], # bar-chart image "C2": [1.0, 3.0], # 2023 text paragraph "C3": [0.0, 2.0], # unrelated photo}
scores = {cid: round(cosine(q, vec), 2) for cid, vec in chunks.items()}print(scores) # -> {'C1': 0.95, 'C2': 0.6, 'C3': 0.32}
ranked = sorted(scores, key=lambda cid: scores[cid], reverse=True)print(ranked[:2]) # -> ['C1', 'C2']Same three numbers as the worked example: 0.95, 0.60 (printed as 0.6), 0.32, with the chart and the paragraph kept.
Watch Out For
The right chart, cited, with the wrong number
The answer points at the correct figure and still misreads it. A Q4 bar becomes last year's value, or two close lines swap names. Joint embeddings match the chart's overall shape, not its digits, so retrieval succeeding says nothing about reading succeeding.
Fix it at ingest. Run a vision-language pass over figures and store the extracted numbers as text next to the image vector. Then the reader gets both the pixels and the digits, and your spot check compares the answer against the stored table instead of your memory of the picture.
Every retrieved image goes into the prompt
Five figures ride along to the reader, latency climbs, and answers get vaguer as the prompt fills with pixels. More evidence starts reading like less.
Send only the top image as pixels and the rest as captions. Shrink images to the smallest size where the axes stay legible, and keep each chunk id attached so the citation names the figure the words came from.
The Quick Version
- Multimodal RAG makes pictures retrievable: images and text share one ranking, so the evidence behind an answer can be a chart.
- Each chunk scores by cosine similarity between the question vector and the chunk vector, and only the top few reach the reader.
- In the worked example the chart scores 0.95, the text paragraph 0.60, and the stray photo 0.32, so the answer cites the chart.
- Joint embeddings match images directly and miss fine print, while captioning at ingest reads the digits but charges per image.
- Skip the machinery when a PDF text layer parses exactly or when one image needs no index.
- A retrieved chart is not a read chart: store the extracted numbers as text or the digits will drift.