Point Transformer Attention Model

Let each point ask its neighbors what matters. Point Transformer weights neighborhoods with self-attention plus 3D position codes.

Local attention weights neighbors by features plus relative positions

Why Does This Exist?

EdgeConv in DGCNN treats all $20$ neighbors equally after max-pooling, which wastes varying relevance: a table edge matters more than a nearby floor patch. Point Transformer exists to weight neighbors with vector self-attention conditioned on relative positions, giving fine indoor segmentation where uniform pooling smears. It is the attention counterpart to graph and kernel points.

Kernel alternative is KPConv; global baseline is PointNet.

Think of It Like This

A committee that weights opinions

A chair asks neighbors for advice but weights carpenters over passersby on wood questions, using seating distance as extra context. Votes count differently by expertise and position.

Attention is the weighting and relative encoding is the seating chart. The analogy stops at locality: committees hear everyone while point attention stays inside small balls for memory.

How It Actually Works

For point $x_i$ with neighbors $\mathcal{N}(i)$ , queries, keys, values form as $q_i, k_j, v_j$ with relative encoding $\delta = \theta(p_i - p_j)$ . Vector attention gives $y_i = \sum_j \mathrm{softmax}(\gamma(q_i - k_j + \delta)) \odot (v_j + \delta)$ , so each channel gets its own weight. Blocks stack with farthest-point downsampling and trilinear-style upsampling for UNet segmentation. Neighborhoods hold $16$ points inside radius $0.15$ m indoors.

Worked numbers

Center with two neighbors scoring $2.0$ and $1.0$ gives softmax weights $0.73$ and $0.27$ . Values $5.0$ and $1.0$ blend to $0.73 \times 5 + 0.27 \times 1 = 3.92$ instead of max $5.0$ , preserving mixture. Reference S3DIS area-five mIoU moves past $70$ percent versus $65$ percent for uniform pooling baselines.

Code

1import math2
3
4def softmax(ws: list[float]) -> list[float]:5    exps = [math.exp(w) for w in ws]  # vector attention weights6    return [e / sum(exps) for e in exps]7
8
9print([round(v, 2) for v in softmax([2.0, 1.0])])  # -> [0.73, 0.27]

Watch Out For

Running global attention on full scans

Full-room attention over $100000$ points explodes quadratically and out-of-memory kills the run. The symptom is instant OOM on S3DIS rooms. Keep attention local inside sampled neighborhoods and downsample between stages.

The Quick Version

Point Transformer weights local neighbors with vector self-attention.
Relative position codes inject 3D geometry into every weight.
Each channel attends separately instead of sharing one scalar.
Local balls of $16$ points keep memory bounded.
Indoor mIoU passes $70$ percent on S3DIS area five.
Global attention on full scans is infeasible; stay local and hierarchical.