PointNet and PointNet++

Point clouds have no inherent order, so PointNet processes each 3D coordinate independently before combining them with a symmetric max function that yields the exact same answer regardless of permutation.

PointNet uses symmetric max pooling for permutation invariance, while PointNet++ adds hierarchical set abstraction across metric scales.

Why Does This Exist?

Standard computer vision networks rely on dense, regular 2D pixel grids or 3D voxel grids. When applying convolutional neural networks to 3D sensor data such as LiDAR scans or depth cameras, practitioners historically converted irregular point sets into 3D voxel grids. However, volumetric voxelization is plagued by cubic computational complexity $\mathcal{O}(N^3)$ , consuming massive GPU memory while leaving over $90\%$ of voxels empty.

Furthermore, raw 3D point clouds are fundamentally unstructured sets of coordinates $\{x_1, x_2, \dots, x_N\} \subset \mathbb{R}^3$ . There are $N!$ possible permutations of an $N$ -point cloud representing the exact same geometric object. Standard recurrent networks or sequence models are sensitive to feed order, producing wildly differing predictions if the point list is shuffled.

PointNet introduced a direct, permutation-invariant neural architecture operating on raw point coordinates without rasterization or voxelization. PointNet++ subsequently solved PointNet's primary limitation: its inability to capture local geometric structures and neighborhood metric relationships at varying point densities.

Think of It Like This

A panel of inspectors summarizing a mosaic

Imagine a committee evaluating a pile of loose, numbered mosaic tiles scattered on a table. If each inspector reads the tiles in numerical order, shuffling the numbers would change their reading order and confuse their summary.

Instead, every inspector examines each individual tile in isolation and scores it across 1,024 different criteria (such as curvature, color warmth, or sharpness). Once all tiles are individually scored, the committee simply records the highest score achieved by any tile across each of the 1,024 criteria. Shuffling the tiles produces the exact same maximum scores.

PointNet uses this symmetric maximum pooling across all points. PointNet++ takes this further: it divides the table into small neighborhood clusters, inspects each cluster locally, and builds a hierarchical summary from fine details to global shape.

How It Actually Works

Symmetric Function and Hierarchical Set Abstraction

PointNet satisfies permutation invariance mathematically through a symmetric function:

f(\{x_1, x_2, \dots, x_n\}) \approx g\left(h(x_1), h(x_2), \dots, h(x_n)\right)

where $h: \mathbb{R}^3 \to \mathbb{R}^K$ is approximated by a multi-layer perceptron (MLP) shared across each individual point, and $g: \mathbb{R}^K \times \dots \times \mathbb{R}^K \to \mathbb{R}^K$ is a symmetric pooling operator, specifically element-wise maximum pooling:

g(v_1, \dots, v_n) = \max_{i=1 \dots n} v_i

To achieve geometric transformation invariance (rigid rotations and translations), PointNet introduces mini-networks called T-Nets that predict $3 \times 3$ affine transformation matrices applied directly to raw input coordinates, and $64 \times 64$ transformations applied to intermediate feature spaces. The $64 \times 64$ transformation is constrained to orthogonality by an explicit regularization loss:

L_{\text{reg}} = \left\| I - A A^T \right\|_F^2

While PointNet aggregates all points into a single global vector $1 \times 1024$ , it cannot learn local neighborhood context (like surface normals or edge bevels). PointNet++ introduces hierarchical Set Abstraction (SA) levels composed of three layers:

Sampling Layer: Selects a subset of $N'$ centroid points using Furthest Point Sampling (FPS) to ensure uniform geometric coverage across the surface.
Grouping Layer: Constructs local regions around each centroid using Ball Query with radius $r$ , finding all points within Euclidean distance $\|x - x_{\text{centroid}}\| \le r$ .
PointNet Layer: Applies a mini-PointNet with local coordinates $x_i - x_{\text{centroid}}$ to encode each local patch into a feature vector.

To overcome non-uniform point densities common in physical LiDAR scans, PointNet++ utilizes Multi-Scale Grouping (MSG) and Multi-Resolution Grouping (MRG), combining features extracted across varying neighborhood radii.

Worked Example

Trace a simplified PointNet max-pooling operation on 3 points in $\mathbb{R}^3$ mapped to 4-dimensional features:

Input Coordinates:
$P = \begin{bmatrix} p_1 \\ p_2 \\ p_3 \end{bmatrix} = \begin{bmatrix} 1.0 & 0.0 & 2.0 \\ -0.5 & 1.5 & 0.0 \\ 0.2 & -1.0 & 0.5 \end{bmatrix} \in \mathbb{R}^{3 \times 3}$
Shared Linear Layer $W \in \mathbb{R}^{3 \times 4}$ with weights:
$W = \begin{bmatrix} 1.0 & 0.0 & -1.0 & 2.0 \\ 0.0 & 2.0 & 1.0 & -1.0 \\ 1.0 & 1.0 & 0.0 & 0.5 \end{bmatrix}, \quad b = [0, 0, 0, 0]$
Point Feature Projection $h(p_i) = \text{ReLU}(p_i W)$ :
- For $p_1 = [1.0, 0.0, 2.0]$ : $p_1 W = [1 \cdot 1 + 2 \cdot 1, 2 \cdot 1, 1 \cdot (-1), 1 \cdot 2 + 2 \cdot 0.5] = [3.0, 2.0, -1.0, 3.0]$ After ReLU: $h(p_1) = [3.0, 2.0, 0.0, 3.0]$
- For $p_2 = [-0.5, 1.5, 0.0]$ : $p_2 W = [-0.5 \cdot 1, 1.5 \cdot 2, -0.5 \cdot (-1) + 1.5 \cdot 1, -0.5 \cdot 2 + 1.5 \cdot (-1)] = [-0.5, 3.0, 2.0, -2.5]$ After ReLU: $h(p_2) = [0.0, 3.0, 2.0, 0.0]$
- For $p_3 = [0.2, -1.0, 0.5]$ : $p_3 W = [0.2 \cdot 1 + 0.5 \cdot 1, -1.0 \cdot 2 + 0.5 \cdot 1, 0.2 \cdot (-1) - 1.0 \cdot 1, 0.2 \cdot 2 - 1.0 \cdot (-1) + 0.5 \cdot 0.5]$ $p_3 W = [0.7, -1.5, -1.2, 0.4 + 1.0 + 0.25] = [0.7, -1.5, -1.2, 1.65]$ After ReLU: $h(p_3) = [0.7, 0.0, 0.0, 1.65]$
Symmetric Element-Wise Max-Pooling $g$ :
$g(h(p_1), h(p_2), h(p_3)) = \left[ \max(3.0, 0.0, 0.7), \max(2.0, 3.0, 0.0), \max(0.0, 2.0, 0.0), \max(3.0, 0.0, 1.65) \right]$ $\mathbf{v}_{\text{global}} = [3.0, 3.0, 2.0, 3.0]$
If the points arrive in reverse order $(p_3, p_2, p_1)$ , the resulting vector $\mathbf{v}_{\text{global}}$ is identical.

Code

1import torch2import torch.nn as nn3import torch.nn.functional as F4
5class PointNetClassification(nn.Module):6    """Permutation-invariant point cloud classification backbone."""7    def __init__(self, num_classes: int = 10) -> None:8        super().__init__()9        # Shared MLP 1: 3 -> 6410        self.conv1 = nn.Conv1d(3, 64, kernel_size=1)11        self.bn1 = nn.BatchNorm1d(64)12        13        # Shared MLP 2: 64 -> 128 -> 102414        self.conv2 = nn.Conv1d(64, 128, kernel_size=1)15        self.bn2 = nn.BatchNorm1d(128)16        self.conv3 = nn.Conv1d(128, 1024, kernel_size=1)17        self.bn3 = nn.BatchNorm1d(1024)18        19        # Classification Head20        self.fc1 = nn.Linear(1024, 512)21        self.bn4 = nn.BatchNorm1d(512)22        self.fc2 = nn.Linear(512, 256)23        self.bn5 = nn.BatchNorm1d(256)24        self.fc3 = nn.Linear(256, num_classes)25
26    def forward(self, x: torch.Tensor) -> torch.Tensor:27        # x shape: (B, 3, N) where N is number of points28        batch_size = x.size(0)29        30        # Point-wise feature extraction31        h = F.relu(self.bn1(self.conv1(x)))32        h = F.relu(self.bn2(self.conv2(h)))33        h = F.relu(self.bn3(self.conv3(h)))  # (B, 1024, N)34        35        # Symmetric Function: Global Max Pooling over points dimension (dim=2)36        global_features = torch.max(h, dim=2)[0]  # (B, 1024)37        38        # Multi-layer perceptron classification head39        out = F.relu(self.bn4(self.fc1(global_features)))40        out = F.relu(self.bn5(self.fc2(out)))41        logits = self.fc3(out)42        return logits43
44# Verify permutation invariance45model = PointNetClassification(num_classes=5).eval()46# 1 batch, 3 coords, 100 points47sample_points = torch.randn(1, 3, 100)48# Shuffle points along spatial dimension49perm = torch.randperm(100)50shuffled_points = sample_points[:, :, perm]51
52with torch.no_grad():53    out_orig = model(sample_points)54    out_shuf = model(shuffled_points)55
56max_diff = (out_orig - out_shuf).abs().max().item()57print(f"Max prediction difference after shuffle: {max_diff:.8f}")58# -> Max prediction difference after shuffle: 0.00000000

Watch Out For

Density imbalance causing empty ball query neighborhoods

In physical sensor data (such as automotive LiDAR), point density drops dramatically with distance $1/d^2$ . When PointNet++ executes Ball Query with a fixed search radius $r$ , distant centroids may capture only 1 or 2 points (or zero points), causing degenerate zero-padding and vanishing gradient updates.

To prevent starvation in sparse regions, implement Multi-Scale Grouping (MSG) where features from small, medium, and large radii are concatenated. If a query sphere contains fewer than $K_{\min}$ points, duplicate the centroid coordinates rather than zero-padding, ensuring the local MLP normalizes over valid relative offsets.

The Quick Version

Overcomes $\mathcal{O}(N^3)$ voxel memory costs by feeding raw point coordinates directly into neural networks.
Enforces permutation invariance across unordered sets using shared point-wise MLPs followed by symmetric max pooling.
PointNet++ builds hierarchical multi-scale representations through Furthest Point Sampling, Ball Query grouping, and local mini-PointNets.