DPT Transformer for Dense Depth

Convolutions see locally; depth needs the whole room. DPT reads the image with a vision transformer so far walls inform near chairs.

Transformer tokens gather global context before a convolutional decoder rebuilds depth

Why Does This Exist?

CNN depth backbones grow context slowly through stacked $3 \times 3$ kernels, so thin structures and far layout stay ambiguous. DPT exists to give dense prediction a global view from the start: a vision-transformer encoder attends across the whole image at every layer, then a lightweight convolutional decoder reassembles tokens from several scales into a sharp high-resolution map. It powers MiDaS $v3$ and the Depth Anything family.

Basics are monocular depth estimation; the mixed-dataset teacher is MiDaS.

Think of It Like This

A survey team with radios

A CNN crew passes notes only to neighbors, so news crosses the field slowly. A DPT crew carries radios: every member hears the far corner instantly, then draws the local map with full context.

Attention is the radio and the decoder is the local drawing. The analogy stops at cost: radios need batteries while global attention needs quadratic memory, tamed here by patch tokens.

How It Actually Works

The image splits into $16 \times 16$ patches embedded as tokens with position codes. Transformer blocks apply $\mathrm{softmax}(QK^T / \sqrt{d})V$ globally, so each token reads all others. Tokens from four stages reassemble into image-like maps at $1/4$ to $1/32$ scale, then RefineNet-style fusion blocks upsample and merge coarse-to-fine into depth at half input resolution. The same head swaps to segmentation or normals by changing the loss, which is why DPT is a dense-task backbone rather than a depth-only trick.

Worked numbers

A $384 \times 384$ image gives $(384/16)^2 = 576$ tokens. Attention over $576$ tokens costs $576^2 \approx 331000$ pairwise scores per head, trivial next to per-pixel CNN stacks. On the demo edge split, boundary F1 rises from $0.62$ CNN to $0.71$ DPT, about a $15$ percent relative gain.

Code

1def token_count(size: int, patch: int) -> int:2    return (size // patch) ** 2  # patches become transformer tokens3
4
5print(token_count(384, 16))  # -> 576

Watch Out For

Running DPT at full megapixels naively

Token count grows with the square of resolution, so $1024$ px images explode memory sixteenfold over $256$ px. The symptom is sudden out-of-memory on phone photos. Tile with overlap or cap the long side near $768$ px, then blend.

The Quick Version

DPT encodes patches with a vision transformer for global context.
A convolutional decoder fuses four scales into sharp depth.
Global attention fixes thin structures that local CNNs smear.
Token count is $(H/16) \times (W/16)$ ; watch memory at high resolution.
The same backbone serves depth, segmentation, and normals.
MiDaS $v3$ and Depth Anything build directly on DPT.