Prompting Vision Models Well

Tell the model the job, the exact output shape, and what to do at edges. Clear prompts beat clever ones.

Vision prompts state the task plus output shape plus edge cases

Why Does This Exist?

Vague orders like describe this lose to exact ones that fix JSON boxes plus no-guess rules. Prompt skill moves zero-shot classification and image captioning points with no training.

This page covers wording only. Tuned vectors are the next two pages.

The one idea you need first.

A prompt has task $T$ , format $F$ , and edge rules $E$ . Few-shot adds $2$ to $4$ input-output pairs. Grade with fixed harnesses, not vibes.

Think of It Like This

Work orders with drawing size plus tolerance

A shop order saying make a bracket fails; one giving drawing A4, holes in mm table, and burr limits ships right. Paper shape plus limits beat adjectives.

$T$ is the part, $F$ is the drawing, and $E$ is tolerance. Where it stops: shops ask back, while models guess silently.

How It Actually Works

Write $T$ in one line, $F$ as schema with units $0$ to $1000$ , and $E$ for occlusion plus empty scenes. Add short pairs for style. Test wording variants on $200$ held-out photos and keep the winner. Version prompts like code.

In plain words: name it, shape it, and bound it, then measure.

Worked example

Vague detect cars scores AP $41.2$ . Shaped prompt with JSON boxes plus ignore under $20$ pixels scores $47.3$ , a $6.1$ gain. Adding $3$ pairs lifts to $48.9$ . The $7.7$ total comes from format plus edges, not model size.

Watch Out For

Flowery adjectives replacing shape

Words like carefully add nothing and scores vary randomly. Outputs drift format and parsers break. Fix schema first, then tune nouns in visual grounding tests.

The Quick Version

Name the task, fix output shape, set edge rules.
JSON boxes with units beat prose lists.
Two to four pairs teach style fast.
Test on held-out photos and version prompts.
Shape plus limits give the six-point lift.