Point Cloud Processing

A point cloud processor reads every 3D point on its own with one shared network, then pools the results into a single shape summary that never depends on point order.

One shared network lifts every raw 3D point into feature space, then max-pooling collapses the whole set into a single order-free global vector.

Why Does This Exist?

LiDAR and depth cameras output unordered sets of $(x, y, z)$ points, and classic image networks cannot consume a set: shuffling the input rows must not change the answer. The early workaround forced points into structures networks already accepted. Voxel grids work, but a $64^3$ grid holds $262{,}144$ cells while a $10{,}000$ -point scan fills under 4 percent of them, so most memory stores empty space. Rendering points into 2D views is cheaper, but view choice discards geometry the sensor actually measured.

Direct processing skips both conversions and learns from the raw set. This page covers the three-stage pipeline: sample the cloud, lift each point with one shared function, then collapse the set with a symmetric pool into a task head. The representation trade itself lives in point clouds, the grid alternative in voxels, and the canonical architecture in PointNet.

Think of It Like This

Tallying parcels in any order

A mailroom tallies a pile of parcels by type. Each parcel is read alone against the same checklist, then the per-parcel notes merge onto one summary sheet. Shuffling the pile before starting changes nothing on the sheet, because no step ever uses pile position.

The checklist is the shared network, the summary sheet is the pooled global vector, and shuffle-proof totals are permutation invariance. The analogy stops at relationships: a tally never asks which parcels sat together, but a chair leg is a cluster of nearby points, not a count. Real pipelines therefore sample spread-out centroids and pool local neighborhoods before the global step, so shape survives the summary.

How It Actually Works

One shared function plus a symmetric pool

Write the cloud as a set $P = \{p_1, \dots, p_N\}$ where each $p_i$ holds 3 coordinates plus optional extras such as intensity or normals, giving vectors in $\mathbb{R}^d$ . A shared function $h$ maps $\mathbb{R}^d$ to $\mathbb{R}^K$ with one weight set applied identically to every point, so $K$ is the feature width and $N$ never enters the weights. Then a symmetric aggregator $A$ collapses the $N$ rows: with max-pooling, $g_k = \max_i h(p_i)_k$ for each channel $k$ , producing one global vector $g$ in $\mathbb{R}^K$ . Max, sum and mean all ignore input order, which is exactly why shuffling points leaves $g$ unchanged.

Worked three-point example

Take $h(p) = [x + y, y + z]$ on three points: $p_1 = (1, 0, 2)$ gives $[1, 2]$ , $p_2 = (0, 3, 1)$ gives $[3, 4]$ , and $p_3 = (2, 1, 0)$ gives $[3, 1]$ . Max-pooling takes $\max(1, 3, 3) = 3$ on channel one and $\max(2, 4, 1) = 4$ on channel two, so $g = [3, 4]$ . Reorder the input to $p_3, p_1, p_2$ and the same two maxima return $[3, 4]$ again.

Sampling and task heads

Networks budget a fixed $N$ such as 1024 or 2048 points per cloud, and farthest-point sampling picks spread-out points so sparse far regions survive; uniform random sampling instead erases distant objects first, since density falls roughly as $1/r^2$ with range $r$ . For classification an MLP reads $g$ alone. For segmentation each point keeps its own $h(p_i)$ concatenated with a copy of $g$ , so per-point logits see both local detail and whole-shape context.

Code

1pts = [(1, 0, 2), (0, 3, 1), (2, 1, 0)]2
3def h(p):4    x, y, z = p5    return [x + y, y + z]6
7feats = [h(p) for p in pts]8g = [max(f[k] for f in feats) for k in range(2)]9print(g)10# -> [3, 4]11
12shuffled = [pts[2], pts[0], pts[1]]13gs = [max(h(p)[k] for p in shuffled) for k in range(2)]14print(gs)15# -> [3, 4]16print(g == gs)17# -> True

Watch Out For

Feeding raw sensor coordinates with no canonical frame

The same chair scanned at different positions gives different raw numbers, and the network reads coordinates literally. The symptom is a model that labels the training-room chair correctly but fails on the identical chair two meters away. Subtract the centroid and scale by the maximum norm so every cloud sits centered in a unit sphere, and keep rotation augmentation in training.

Uniform random sampling on range-varying LiDAR

Capping a scan at a fixed $N$ with uniform sampling spends the budget where points are densest. The symptom is distant pedestrians vanishing from the sampled cloud while nearby road surface fills most rows. Use farthest-point sampling, or stratify the budget across range bands, so far objects keep representation.

The Quick Version

Point cloud processing learns directly from unordered $(x, y, z)$ sets with no voxel grid.
One shared function lifts every point to $\mathbb{R}^K$ , then a symmetric pool collapses the set.
Example: $h(p) = [x+y, y+z]$ on three points max-pools to $[3, 4]$ in any input order.
Shuffled inputs give identical global vectors because max ignores order.
Farthest-point sampling protects sparse far regions that uniform sampling erases.
Classification reads the global vector; segmentation concatenates it back onto each point.