InternVL Scaled Pairing
Grow the vision encoder to billions of weights beside a strong language model. Big both sides keep fine detail through chat.
Why Does This Exist?
Small vision encoders blur text and parts before the LLM ever sees them. InternVL grows ViT toward billion weights beside the LLM and keeps high-res tiles, so optical character recognition and grounding survive chat tuning.
This page covers two-sided scaling only. Bridge training is the BLIP page.
The one idea you need first.
Params split as ViT B plus LLM B plus projector. Tiles multiply tokens per photo. Contrastive warmup aligns the pair before instruction tuning.
Think of It Like This
Strong eyes plus strong brain, not one weak link
A spotter with weak eyes misreads plates no matter how smart the analyst. Give both strong eyes and a strong brain and the team reads plates and reasons about routes.
ViT is eyes, LLM is brain, and tiles are binoculars. Where it stops: eyes and brain train jointly here, while humans grow them separately.
How It Actually Works
Pretrain ViT plus LLM jointly on noisy pairs, then instruction-tune on chat plus grounding data. Keep tiles on for detail batches. Loss sums contrastive warmup plus next-token chat terms.
In plain words: scale both, tile the input, and tune manners last.
Worked example
ViT M reads -point text at percent accuracy on a test form. ViT B with tiles reads percent. Error falls from to points, a percent relative cut. The point rise traces to encoder size plus tiles, since the LLM stays fixed in the test.
Watch Out For
Paying 6B cost for thumbnail tasks
Full tiles on easy photos waste compute with no gain. Skimming in image retrieval slows threefold. Route easy batches to thumbnails and save tiles for detail probes.
The Quick Version
- InternVL scales vision toward 6B beside a strong LLM.
- High-res tiles preserve text and small parts.
- Warmup aligns, instruction tuning teaches manners.
- OCR and grounding gain most from vision scale.
- Use tiles selectively to control cost.