Document Understanding with Vision Models
Photograph a receipt and get back a checked database row instead of a pile of words. The model reads the photo itself, so every number keeps the row and column it came from.
Why Does This Exist?
Every shop, clinic, and tax office sits on paper that should have been a database row years ago. Receipts, invoices, lab reports, customs forms. Someone still retypes them, or an optical character recognition (OCR) pipeline does: OCR finds the letters and hands you one long string of text.
That string is where the trouble starts. A receipt is not a sentence. Its meaning lives in positions. The number 42.50 means nothing until you see it sitting in the Total row, under the amount column, beneath the coffee, sandwich, and cake you actually bought. Flatten the page into a one dimensional string and that placement is gone. Multi-column invoices come out scrambled, table cells attach to the wrong headers, and the total becomes just another number floating in the pile. Teams patch this with rules per vendor layout, and it works until the next format arrives and breaks the rules again.
Document understanding skips the flattening. The model looks at the page photo itself, reads the words and their positions together, and answers with fields: store, items, subtotal, tax, total. The till arithmetic on this page's receipt is the running thread. Coffee at 12.00 plus a sandwich at 8.50 plus cake at 17.50 makes a subtotal of 38.00. Tax adds 4.50. The Total row reads 42.50. The diagram above shows that 42.50 staying glued to its row all the way into the JSON.
This page covers reading one page image into trusted fields. The surrounding pipeline, getting thousands of such pages cleaned, split, embedded, and indexed, lives in document ingestion pipelines. The guarantee that the answer parses as JSON with the right shape lives in structured outputs.
The one idea you need first. A vision-language model is a language model with sight: a vision encoder turns image patches into vectors the language model can read. Document understanding is that architecture pointed at pages, with two upgrades this page explains: tiling so small print survives, and a position attached to every word.
Think of It Like This
A seating chart, not a guest list
You are ushering at a wedding. The guest list names every guest, alphabetically, and it cannot answer the one question anyone asks you: who sits next to whom. The seating chart can. Same names, plus positions, and suddenly next to means something.
A scanned receipt works the same way. OCR hands you the guest list: every word in reading order, with the seating torn off. Document understanding keeps the chart. Total is a name, the cell to its right is its seat, and the row they share is what makes 42.50 the total instead of a rumor. When the chart says two boxes share a row, you trust the pairing the way you trust two cards on the same table.
Where the chart stops: it tells you where everyone sits, not whether the caterer spelled a name right. Positions pair the label with the number, but reading small digits correctly is separate work, and that is exactly where these models still slip.
How It Actually Works
Tile the page so small print survives
A vision encoder chops the image into square patches of side pixels and turns each patch into one vector. For an image pixels tall and pixels wide, the patch count is:
In plain words: is the image height, is the image width, is the patch side length, and is the number of patch vectors the language model must read.
Work it with small numbers. A 1024 by 1024 scan with gives patch vectors. Shrink that page down to the small square older encoders expect and each patch covers print too tiny to read. The digits smear. So document models tile instead. A 2048 by 1536 page cut into 512 pixel tiles makes tiles, each read at full sharpness. The price is context: 12 tiles means roughly 12 times the patch vectors, which costs memory and attention. That is the trade-off, stated plainly. Fine print or cheap inference. Pick one.
Pin every word to its box
Alongside the patches, the model gets each word's bounding box . Read it as: and mark the box's top-left corner, and mark its bottom-right, measured in pixels from the page's top-left. Two boxes share a row when their vertical centers sit within a small tolerance, and share a column when their horizontal centers do.
Back to the receipt. The TOTAL label sits in a box centered near y = 620, and 42.50 sits in a box centered near y = 622. Two pixels apart, inside any sane tolerance, so the model pairs them before it ever asks what the digits say. Then the till arithmetic confirms the pairing. The line items sum exactly: 12.00 + 8.50 + 17.50 = 38.00. Add the tax: 38.00 + 4.50 = 42.50. The Total row's number matches the sum, which is how you know the pairing is right and not merely confident.
Emit fields, then check them before you trust them
The model answers against a fixed schema, store, items, subtotal, tax, total, and the output is validated like any structured output: required keys present, numbers parse, and subtotal + tax == total. Anything that fails gets retried or routed to a person. No silent saves.
Say the assumption out loud. This works when the print is legible at tile resolution and the layout follows the grid the model trained on: rows, columns, labeled totals. Handwriting, heavy blur, or a truly exotic form breaks the assumption, and the failure looks like confidence, a clean JSON row with a wrong digit. When your source is a born-digital PDF with an embedded text layer, skip all of this and parse that layer. It is cheaper and exact, no model needed. And when the fields you extract feed question answering over whole archives, the natural next step is multimodal RAG, which indexes pages as images instead of flattening them first.
Code
This snippet is the check, not the model. It pairs a label with its value by row and replays the till arithmetic from the receipt above.
from dataclasses import dataclass
@dataclassclass Field: text: str y_center: float value: float
def same_row(a: Field, b: Field, tol: float = 10.0) -> bool: """Two boxes share a row when their vertical centers nearly agree.""" return abs(a.y_center - b.y_center) <= tol
items = [ Field("coffee", 500.0, 12.00), Field("sandwich", 502.0, 8.50), Field("cake", 504.0, 17.50),]label = Field("TOTAL", 620.0, 0.0)total = Field("42.50", 622.0, 42.50)tax = 4.50
subtotal = round(sum(f.value for f in items), 2)print(subtotal) # -> 38.0print(same_row(label, total)) # -> Trueprint(round(subtotal + tax, 2) == total.value) # -> TrueThe label and the number sit 2 pixels apart vertically, inside the 10 pixel tolerance, so they pair. The items sum to 38.0, plus 4.5 of tax gives 42.5, which equals the Total row's value. A real pipeline runs this kind of replay on every row it saves and rejects the ones that do not add up.
Watch Out For
A clean row with one wrong digit
Generative models read. They do not scan. On a slightly blurred photo, 42.50 comes back as 42.58, wrapped in the same tidy JSON with the same confident row. The symptom is reconciliation failures downstream: books that miss by pennies on random invoices, never the same field twice.
Replay the arithmetic on every saved row, the way the snippet above does, and hold money fields to a second opinion when the image is soft: a classic OCR read of the same box, or a human pass. Trust the layout pairing. Verify the digits.
Shrinking the page into a smudge
The classic self-inflicted failure is resizing the whole page down to whatever small square the encoder accepts, then wondering why the model invents totals. The symptom is specific: big headers read fine while small amounts and line items go wrong, because a full-height digit became a rumor of one.
Tile at native resolution so digits stay sharp, and refuse thumbnails at the door. If a page needs more tiles than your budget allows, route it to a cheaper text-layer parse or a human, not to a downscaled guess.
The Quick Version
- Document understanding reads the page photo itself, so each number keeps the row and column that give it meaning.
- Tiling splits a big page into full-resolution pieces, because shrinking small print until it smears is how totals get invented.
- Every word carries a bounding box, and boxes whose centers share a band pair up before anyone reads the digits.
- Output is schema-first JSON that must replay the page's own arithmetic, like 38.00 plus 4.50 equaling 42.50, or it does not ship.
- Born-digital PDFs with a text layer need a parser, not a model. Recheck every money digit before it reaches the books.