Image Formation and Camera Model

Why Does This Exist?

In computer vision, the fundamental problem is that images are flat, but the world is not. Every photograph represents a massive loss of information—an entire three-dimensional dimension is crushed down to a two-dimensional grid of light intensities.

To build autonomous vehicles, 3D reconstructors, or augmented reality systems, we must understand exactly how that loss happens. The camera model is the mathematical bridge that allows algorithms to relate the coordinates of a pixel on a screen to the physical location of an object in the real world. Without a rigorous model of image formation, a self-driving car would see a pedestrian in an image but have no geometric basis for determining whether that pedestrian is ten feet away or a hundred.

Think of It Like This

How It Actually Works

The projection of the 3D world into 2D pixels relies on a sequence of coordinate transformations.

1. World Coordinates to Camera Coordinates (Extrinsics) An object exists in a global 3D space, $P_{world} = (X_w, Y_w, Z_w)$. The camera is located somewhere in that world, pointing in a specific direction. The extrinsic parameters define this relationship via a rotation matrix $R$ and a translation vector $T$: $P_{cam} = R \cdot P_{world} + T$ $P_{cam} = (X, Y, Z)$ represents the 3D point relative to the camera center, with the Z-axis pointing straight out from the lens.

2. Perspective Projection Once the point is relative to the camera, light travels in straight lines through the camera center to the image plane. Under the pinhole camera model, similar triangles dictate the projection. A 3D point $(X, Y, Z)$ projects to a 2D continuous point $(x, y)$ on the image plane located at a distance $f$ (the focal length) from the camera center: $x = f \frac{X}{Z}, \quad y = f \frac{Y}{Z}$ This division by $Z$ creates perspective: objects further away (larger $Z$) project to smaller coordinates on the image plane.

3. Image Plane to Pixels (Intrinsics) The continuous coordinates $(x, y)$ are in millimeters or meters. Digital images use discrete pixels. The intrinsic parameters matrix $K$ converts these metric coordinates into pixel indices $(u, v)$, accounting for the sensor's pixel width and height (often wrapped into $f_x, f_y$) and the principal point $(c_x, c_y)$, where the optical axis intersects the image plane: $u = f_x \frac{X}{Z} + c_x$ $v = f_y \frac{Y}{Z} + c_y$

4. The Unified Matrix Equation Using homogeneous coordinates (appending a 1 to vectors to allow matrix multiplication for translation), the entire pipeline from 3D world to 2D pixel becomes a single matrix multiplication: $\begin{bmatrix} u \cdot Z \\ v \cdot Z \\ Z \end{bmatrix} = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} R & T \end{bmatrix} \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}$ Or more compactly: $x = K [R | T] X$.

Code

1import numpy as np2
3def project_3d_to_pixel(point_3d_world, K, R, T):4    """5    Projects a 3D world coordinate into a 2D pixel coordinate.6    """7    # 1. Transform world coordinates to camera coordinates (Extrinsics)8    # P_cam = R * P_world + T9    point_3d_cam = R @ point_3d_world + T10    11    X, Y, Z = point_3d_cam[0], point_3d_cam[1], point_3d_cam[2]12    13    # Avoid division by zero if point is directly at the camera center14    if Z == 0:15        Z = 1e-616        17    # 2 & 3. Project to image plane and convert to pixels (Intrinsics)18    # We can do this directly with the intrinsic matrix K19    # [u*Z, v*Z, Z]^T = K * [X, Y, Z]^T20    homogeneous_pixel = K @ point_3d_cam21    22    # 4. Normalize by Z to get actual u, v coordinates23    u = homogeneous_pixel[0] / homogeneous_pixel[2]24    v = homogeneous_pixel[1] / homogeneous_pixel[2]25    26    return np.array([u, v])27
28# Example usage29K = np.array([[800.0, 0.0, 320.0],  # fx, 0, cx30              [0.0, 800.0, 240.0],  # 0, fy, cy31              [0.0, 0.0, 1.0]])32R = np.eye(3)                       # No rotation33T = np.array([0.0, 0.0, 5.0])       # Camera is 5 units away along Z34point_world = np.array([1.0, 2.0, 10.0])35
36pixel_coord = project_3d_to_pixel(point_world, K, R, T)37# -> [ 373.33333333, 346.66666667]

Watch Out For

The Quick Version

• Image formation collapses a 3D physical world into a 2D pixel array, losing depth information.
• The Extrinsic matrix ($R, T$) maps the 3D world to the camera's local 3D perspective.
• The Intrinsic matrix ($K$) projects that local 3D perspective onto a 2D image sensor and translates metric distances into pixel indices.
• The pinhole camera divides $(X,Y)$ by depth $(Z)$, meaning objects shrink in the image as they move further away (perspective projection).
• By calibrating a camera, you determine its $K$ matrix, which allows you to run computer vision algorithms that understand real-world geometry.