SECOND 3D Detection from Voxels

SECOND speeds up LiDAR detection by voxelizing the point cloud and convolving only where points exist.

SECOND turns LiDAR points into sparse voxels and detects 3D boxes only where points exist

Why Does This Exist?

LiDAR gives exact 3D positions but as an unordered scatter of points. Dense 3D convolutions over the whole driving volume waste most work on empty air. Object detection in 3D needed the accuracy of voxels without paying for empty space, and VoxelNet proved voxels work but ran slowly because it convolved everywhere.

SECOND (Sparsely Embedded Convolutional Detection, 2018) keeps the voxel idea and replaces dense 3D convolution with sparse convolution that skips empty cells. This page covers the sparse voxel encoder and the 3D box head. Camera-only methods such as BEVFormer solve a different input problem and are linked separately.

Think of It Like This

Counting occupied seats in a stadium

You must count who sits where in a 50,000-seat stadium with only 2,000 people inside. Walking every row and checking every seat wastes the day. Instead you take the ticket stubs, visit only the sections with stubs, and record those seats. Empty sections cost you nothing.

SECOND works the same way. Voxels are seats, LiDAR points are stubs, and sparse convolution visits only occupied voxels. The analogy stops at learning: you still train the features, you do not just count.

How It Actually Works

A LiDAR sweep is a set of points $P = \{(x_i, y_i, z_i)\}$ with $N$ around 100,000 per frame. SECOND voxelizes with size $v = 0.05$ m over a driving range of about $70$ m. A dense grid would hold $(70 / 0.05)^3$ cells, which is impossible, so it stores only the $K$ occupied voxels with $K$ near 5,000. Each voxel pools its few points into one feature vector.

Sparse middle extractor

Sparse convolution applies a $3 \times 3 \times 3$ kernel only at occupied positions and their occupied neighbors. Empty positions emit nothing and cost nothing. Three sparse stages downsample the height while a dense bird's-eye head predicts a 3D box $(x, y, z, l, w, h, \theta)$ plus class per anchor.

Worked example

Take a car voxel column at $x = 12.0$ m, $y = 3.0$ m with 8 points. Voxel indices are $(\lfloor 12.0/0.05 \rfloor, \lfloor 3.0/0.05 \rfloor) = (240, 60)$ . The voxel feature is the mean of the 8 point offsets, for example $[0.02, 0.01, 0.10]$ . After sparse stages the head predicts center $(12.1, 3.0, 0.9)$ , size $(4.4, 1.9, 1.6)$ m, yaw $0.03$ rad, car score $0.91$ . Residuals under $0.2$ m mean the box already overlaps the car at IoU above $0.7$ .

Watch Out For

Voxel size set without checking small objects

A $0.10$ m voxel merges a pedestrian leg with the curb and the box drifts half a meter. The symptom is good car scores but missing pedestrians past $20$ m. Halve $v$ to $0.05$ m for street scenes, and accept the higher voxel count.

Empty-voxel padding blowing up memory

Padding every frame to a fixed $K_{max} = 16,000$ voxels with most rows empty wastes GPU memory and slows the batch. The fix is to batch by actual occupancy and mask the loss on real voxels only.

The Quick Version

SECOND voxelizes LiDAR and convolves only on occupied voxels, so empty air costs nothing.
A $0.05$ m voxel keeps pedestrians separable while the sparse stages keep runtime low.
The head predicts full 3D boxes with yaw, not 2D boxes lifted to 3D.
Do not use it on images alone; it needs LiDAR points to fill voxels.
When points vanish past range, scores collapse; that is a sensor limit, not a training bug.