TransUNet Transformer U-Net Hybrid

TransUNet puts transformer layers in the U-Net bottleneck so global context guides exact pixel boundaries.

TransUNet routes U-Net features through transformer layers for global context

Why Does This Exist?

Pure U-Nets see locally; organs need far context like liver versus spleen sides. Pure transformers lose pixel detail. TransUNet (2021) encodes with CNNs, runs $12$ transformer layers on $16 \times 16$ patches in the bottleneck, then decodes with skips. Input is CT slices; output is multi-organ masks with global coherence.

Think of It Like This

A quilt circle with a town meeting in the middle

Quilters sew local patches, pause for a town meeting to agree which patch goes where globally, then resume stitching edges precisely. Skipping the meeting misplaces whole districts; skipping stitching frays edges.

CNNs quilt and transformers meet. The analogy stops at order: meetings sit mid-network, not mid-day.

How It Actually Works

A ResNet stem downsamples $224 \times 224$ to $14 \times 14$ tokens of dim $768$ . Twelve self-attention layers mix globally. The decoder upsamples with three skips from early CNN stages, restoring $224$ resolution. Loss is Dice plus cross-entropy per organ.

Worked example

An abdominal slice shows liver left and stomach mid. CNN-only gives liver Dice $0.93$ but flips a $3$ cm stomach patch to liver at $0.6$ score. Transformer context sees spleen on the far side and corrects the patch to stomach $0.82$ . Boundary error falls from $4$ to $2$ mm after skips restore edges.

Watch Out For

Small data overfitting the transformer

Twelve layers on $30$ scans memorize scanner texture. The symptom is high train Dice with $0.15$ test drops. Freeze early CNN stages and augment heavily before blaming architecture.

The Quick Version

TransUNet mixes CNN detail with bottleneck transformer context.
Twelve layers on $16$ -pixel patches see whole-slice layout.
Skips restore pixel edges after global mixing.
Do not train transformers on tiny sets without freezing and augmentation.
Global context fixes side flips that local U-Nets miss.