GPT-4V Style High-Res Chat

Cut a big photo into tiles plus one small thumbnail and feed all pieces to a chat model. Tiles keep text sharp while the thumbnail keeps layout.

GPT-4V style models read tiled high-res crops plus a thumbnail

Why Does This Exist?

Downscaling a document photo to $336$ pixels blurs its words, while full $2048$ pixels explode token counts. Tiling keeps $512$ pixel detail per piece plus one thumbnail for layout. Chart and document QA then work in one chat call, building on text detection skill.

This page covers tiling prompts only. Projector training is the LLaVA page.

The one idea you need first.

Tiles are non-overlapping $512$ crops encoded separately; the thumbnail is the whole photo downscaled. Tokens join as thumbnail plus tiles in reading order. Cost grows with tile count.

Think of It Like This

Reading a broadsheet with a magnifier plus a photo

You photograph a full newspaper page, then magnify each column in turn. The small photo tells you where stories sit; the magnifier reads the print. Neither alone suffices.

Thumbnail is the photo, tiles are magnifier views, and the prompt order is the reading path. Where it stops: paper stays still, while token budgets force tile limits.

How It Actually Works

Resize photo to fit tile grid, cut $2 \times 3$ tiles at $512$ each, encode thumbnail plus tiles with ViT $f$ , project to LLM tokens, and ask. Text tokens attend to all pieces. A $1024 \times 1536$ photo gives $6$ tiles plus thumb, about $6 \times 576 + 576 = 4032$ visual tokens before compression.

In plain words: more tiles buy detail at token cost, and the thumb buys layout cheap.

Worked example

Thumbnail $336$ pixels makes $8$ -point text about $3$ pixels tall and unreadable. Tiles at $512$ from a $1536$ wide photo keep that text near $9$ pixels tall. OCR error on a test receipt falls from $41$ percent to $12$ percent when tiles turn on. The gap is the detail the thumb alone missed.

Watch Out For

Tile seams splitting words in half

Cuts through text lines break characters across pieces and answers garble. The symptom is half words plus wrong numbers on dense pages. Overlap tiles by $32$ pixels or ask for cell positions in optical character recognition checks.

The Quick Version

High-res assistants send tiles plus one thumbnail in one prompt.
Tiles near 512 pixels keep small text readable.
The thumbnail preserves page layout.
Token cost grows with tile count.
Overlap cuts to avoid splitting characters.