Robot Perception: Depth to Usable Geometry

Raw depth sensors measure distance along camera rays, but robots require spatial physical geometry: point clouds, voxel grids, or signed distance fields with an explicit distinction between observed free space and occluded unknown space.

Robot perception projects depth rays into voxel grids, distinguishing free space from occluded unknown space

Why Does This Exist?

In computer vision, a depth image is an array of scalar numbers representing distance. For generating 3D models or graphics, visualizing that surface is often enough. But an autonomous mobile robot or robotic arm cannot plan motion solely with a 2D depth map: it must know which regions of 3D space are guaranteed empty, which contain solid obstacles, and which are completely unobserved.

When an RGB-D camera looks at a coffee mug sitting on a table, the infrared projector sees the front curved surface of the ceramic. The space between the camera and the mug is empty air through which the optical rays traveled without hitting anything. The surface of the mug is occupied. But the space directly behind the mug is in shadow: the camera has gathered zero information about what lies behind it.

Robot perception turns raw optical depth rays into actionable spatial models (such as voxel occupancy grids, octrees, and truncated signed distance fields). Crucially, it enforces the cardinal rule of physical navigation: unknown space is not free space.

Basics first: the projective mathematics of camera rays live in depth-maps. Transforming 3D points from camera coordinates into the robot base frame lives in robot-coordinate-frames. Fusing multiple noisy depth frames into a consistent state estimate lives in robot-state-estimation.

Think of It Like This

Sweeping a dark warehouse with a narrow beam flashlight

Imagine entering a pitch-black warehouse holding a flashlight. Everywhere the beam of light illuminates a crate, you verify a solid obstacle. The clear air through which the beam shines is verified free space: you know you can safely step there.

Behind the crate, the flashlight casts a dark, sharp shadow. You cannot see what is inside that shadow: it might be empty floor, or it might be an open elevator shaft. If you assume that because you see no obstacle in the dark, the space is safe to step into, you fall.

Where it stops: human eyes adapt to ambient scatter and subtle reflections in the shadows. Active depth sensors (like structured light or time-of-flight cameras) receive zero returned light from shadows, specular metallic surfaces, or black velvet, returning "no depth" (NaN). A robot cannot guess: its spatial map must explicitly label those regions as UNKNOWN and route trajectories around them until a new viewpoint clears the shadow.

How It Actually Works

The pinhole camera back-projection model

An ideal pinhole camera model parameterizes the optical geometry via intrinsic parameters:

$f_x, f_y$ : Focal lengths in pixel units along the horizontal and vertical sensor axes.
$c_x, c_y$ : Principal point coordinates (the optical center of the sensor, in pixels).

A depth sensor reports a measured depth $Z$ (in meters) for pixel coordinate $(u, v)$ . To recover the true 3D spatial coordinate $\mathbf{P}_C = [X, Y, Z]^T$ in the optical camera frame, we back-project along the ray:

X = \frac{u - c_x}{f_x} Z

Y = \frac{v - c_y}{f_y} Z

Z = Z

Ray casting and 3-state voxel occupancy

Robots discretize the 3D workspace into a grid of volumetric pixels (voxels), each of side length $\delta$ (typically $1\text{ cm}$ to $5\text{ cm}$ ). When a depth measurement returns point $\mathbf{P}_C$ , ray casting (using algorithms like 3D Bresenham or Amanatides-Woo) traverses the line segment from camera center $(0,0,0)$ to $\mathbf{P}_C$ :

FREE cells: Every voxel pierced by the ray before reaching $\mathbf{P}_C$ is updated toward FREE (log-odds occupancy decreases).
OCCUPIED cells: The voxel containing $\mathbf{P}_C$ is updated toward OCCUPIED (log-odds occupancy increases).
UNKNOWN cells: All voxels beyond $\mathbf{P}_C$ along the projected ray retain their default prior UNKNOWN (log-odds $= 0$ ).

Motion planners evaluate collision risk against both OCCUPIED and UNKNOWN voxels: a robot is only allowed to command high-speed motion through verified FREE space.

Worked example: back-projecting a detected obstacle

Consider an RGB-D camera monitoring a robotic tabletop workspace:

Image resolution: $640 \times 640\text{ pixels}$ .
Principal point: $c_x = 320.0\text{ px}, c_y = 320.0\text{ px}$ .
Focal lengths: $f_x = 500.0\text{ px}, f_y = 500.0\text{ px}$ .
Measured depth at pixel $(u, v) = (420, 260)$ : $Z = 1.50\text{ m}$ .

Compute the horizontal coordinate $X$ :

X = \frac{420 - 320}{500} \times 1.50 = \frac{100}{500} \times 1.50 = 0.20 \times 1.50 = 0.30\text{ m}

Compute the vertical coordinate $Y$ :

Y = \frac{260 - 320}{500} \times 1.50 = \frac{-60}{500} \times 1.50 = -0.12 \times 1.50 = -0.18\text{ m}

The surface point in camera coordinates is:

\mathbf{P}_C = [0.30, -0.18, 1.50]^T\text{ m}

Ray casting from $(0, 0, 0)$ to $(0.30, -0.18, 1.50)$ marks the $1.5\text{ m}$ beam as traversable free space. The voxel at $(0.30, -0.18, 1.50)$ is tagged as a solid collision obstacle. All space behind $Z > 1.50\text{ m}$ within the shadow cone remains UNKNOWN.

Code

This Python snippet defines camera intrinsics, back-projects a batch of depth pixels into a 3D point cloud, and classifies ray voxels into Free, Occupied, and Unknown states.

1import numpy as np2
3
4def backproject_depth(5    u: float,6    v: float,7    z_meters: float,8    fx: float = 500.0,9    fy: float = 500.0,10    cx: float = 320.0,11    cy: float = 320.0,12) -> tuple[float, float, float]:13    """Back-project a single (u, v, z) measurement into camera frame 3D coordinates."""14    x: float = ((u - cx) / fx) * z_meters15    y: float = ((v - cy) / fy) * z_meters16    return round(x, 3), round(y, 3), round(z_meters, 3)17
18
19# Baseline test: pixel (420, 260) with 1.50 m depth20pt3d = backproject_depth(420.0, 260.0, 1.50)21print(pt3d)  # -> (0.3, -0.18, 1.5)22
23
24def classify_ray_distance(query_z: float, surface_z: float) -> str:25    """Classify space along an optical ray relative to the detected surface."""26    if query_z < surface_z - 0.02:27        return "FREE"28    elif abs(query_z - surface_z) <= 0.02:29        return "OCCUPIED"30    else:31        return "UNKNOWN"32
33
34print(classify_ray_distance(0.8, 1.50))   # -> FREE35print(classify_ray_distance(1.50, 1.50))  # -> OCCUPIED36print(classify_ray_distance(1.8, 1.50))   # -> UNKNOWN

Watch Out For

Treating unobserved occluded space as empty free space

A catastrophic failure mode occurs when occupancy algorithms initialize all voxel grids to zero (empty) and only mark voxels where depth returns hit. Because the space behind an obstacle returns no depth points, the algorithm concludes that the space behind the obstacle is empty.

The symptom is a mobile robot or manipulator attempting to swing directly through a blind occluded region, smashing into an unobserved wall or table leg. Fix it by using tri-state occupancy (Free, Occupied, Unknown) and forcing trajectory planners to treat Unknown space as impassable until actively inspected by camera rays.

Assuming depth cameras work reliably on transparent or black surfaces

Active infrared depth cameras rely on projected speckle patterns bouncing back into an IR receiver. Clear glass, transparent acrylic shields, mirror-finished chrome, and deep matte-black fabric either absorb the infrared pattern or bounce it away entirely.

The symptom is depth images filled with missing pixels (zeros or NaNs), causing the robot to treat a solid glass window as open empty space and drive right through it. Fix it by fusing multiple sensor modalities: combine depth sensors with ultrasonic rangefinders, contact bumper switches, or tactile fingertip sensors that do not depend on optical reflectivity.

The Quick Version

Robot perception transforms raw 2D depth pixel arrays into metric 3D geometric representations in physical space.
The pinhole camera model back-projects pixel $(u, v)$ with depth $Z$ using focal length $(f_x, f_y)$ and principal point $(c_x, c_y)$ .
Pixel $(420, 260)$ at $1.50\text{ m}$ depth with $f=500, c=320$ back-projects to $[0.30, -0.18, 1.50]^T\text{ m}$ .
Ray casting updates 3D voxel grids with three distinct spatial semantics: verified FREE space along the beam, an OCCUPIED surface voxel at the return, and UNKNOWN occluded shadow behind it.
Unknown space is never assumed to be free: robust motion planners refuse to route high-speed trajectories through unobserved regions.