Point Cloud Processing

Why Does This Exist?

When LiDAR sensors or depth cameras capture the world in 3D, they output a point cloud: a raw, unordered set of $(x, y, z)$ coordinates. Historically, computer vision models designed for 2D images couldn't handle this unordered data. The workaround was to convert these points into 3D voxel grids or render them into multiple 2D views.

However, voxel grids are mostly empty space, making them incredibly inefficient for memory and computation. We needed a way to learn representations directly from the raw, unstructured point sets without enforcing a bulky grid over them.

Think of It Like This

How It Actually Works

Processing a point cloud effectively requires handling its two most important mathematical properties: permutation invariance (the order of points doesn't matter) and transformation invariance (rotating or translating the points shouldn't change the object's identity).

Here is how modern architectures like PointNet solve this:

1. Independent Point Feature Extraction

Instead of convolving across neighboring points right away, the network applies a Shared Multi-Layer Perceptron (MLP) to every single point independently. If you have $N$ points, each with $3$ coordinates $(x, y, z)$, the MLP transforms each point into a higher-dimensional feature vector, say of size $1024$.

2. Symmetric Pooling

Because a point cloud is just a set, the network must output the same result no matter how the points are ordered in the array. To achieve this permutation invariance, the network applies a symmetric function across all points. The most common choice is Max Pooling. By taking the maximum value across all $N$ points for each of the $1024$ feature dimensions, the network collapses the $N \times 1024$ matrix into a single $1 \times 1024$ global feature vector.

3. Global and Local Feature Fusion

For tasks like classification (e.g., "is this a chair?"), the global feature vector is passed to another MLP to get the final prediction. For tasks like segmentation (e.g., "which points belong to the chair's legs?"), the global feature vector is concatenated back onto the individual point features, allowing the network to make per-point decisions with global context.

Code

Here is a simplified PyTorch snippet demonstrating how to achieve permutation invariance on a set of points.

1import torch2import torch.nn as nn3
4class MiniPointNet(nn.Module):5    def __init__(self):6        super().__init__()7        # Shared MLP applied to each point independently (3D -> 64D)8        self.shared_mlp = nn.Sequential(9            nn.Linear(3, 64),10            nn.ReLU(),11            nn.Linear(64, 1024),12            nn.ReLU()13        )14        15    def forward(self, x):16        # x shape: (batch_size, num_points, 3)17        18        # 1. Independent Point Features19        point_features = self.shared_mlp(x) # (batch, num_points, 1024)20        21        # 2. Symmetric Pooling (Max Pooling over the points dimension)22        # This makes the output invariant to point order23        global_feature = torch.max(point_features, dim=1)[0] # (batch, 1024)24        25        return global_feature26
27# -> MiniPointNet()(torch.randn(1, 1000, 3)).shape == (1, 1024)

Watch Out For

The Quick Version

• Point clouds are raw sets of $(x, y, z)$ coordinates, typically from LiDAR or depth sensors.
• Voxelization is too memory-intensive because most of 3D space is empty.
• To process points directly, networks must be permutation invariant (order doesn't matter).
• Shared MLPs extract features per point independently.
• Max Pooling aggregates all point features into one global representation, ensuring order independence.