Hand Pose Joint Tracking

Hand pose tracks twenty-one finger joints through self-occlusion and fast motion, giving AR and robotics a grasp-ready skeleton.

Hand pose estimation finds the palm first then unfolds twenty-one finger joints from the cropped region.

Why Does This Exist?

Bodies are large and slow; hands are small, fast, and constantly hide their own fingers behind the palm. Generic 2D pose heads waste resolution on the torso and miss fingertips. Hand pipelines (MediaPipe Hands; MANO model, Romero et al., 2017) crop palms first, then regress $21$ joints (wrist plus four per finger) or MANO articulation plus shape parameters for a full mesh. The output drives pinch gestures, sign input, and robot grasping.

Think of It Like This

Unfolding a pocketknife by feel

A closed knife hides four blades in one handle silhouette. You find the handle first, then flip each blade out along its hinge and note its angle. Hand tracking works the same: palm crop first, then each finger unfolds along kinematic hinges. The topology is fixed; only the fold angles change.

Where it stops: a fist hides all blades inside the handle, and angles become pure prior.

How It Actually Works

A palm detector proposes tight boxes robust to articulation. The landmark head regresses $21$ $(x, y, depth)$ points directly (regression beats heatmaps at this scale) with bone-length and collision losses. MANO variants output pose plus shape vectors that skin a $778$ -vertex mesh. Temporal smoothing and handedness classification stabilize video. FreiHAND-style datasets mix real and synthetic grasps to cover occlusion.

Worked example

Index finger joints at $(50, 60)$ , $(52, 50)$ , $(53, 42)$ , $(54, 35)$ . Segment lengths $10.2$ , $8.1$ , $7.1$ pixels in crop space. Total finger length $25.4$ . A pinch detector fires when thumb tip $(40, 38)$ sits within $8$ pixels of index tip $(54, 35)$ : distance $\sqrt{196 + 9} \approx 14.3$ , no pinch; at $(48, 36)$ distance $\approx 6.1$ , pinch.

Code

1import math2
3# Pinch test between thumb and index tips.4def dist(a: tuple, b: tuple) -> float:5    return math.dist(a, b)6
7print(round(dist((40, 38), (54, 35)), 1), round(dist((48, 36), (54, 35)), 1))8# -> (14.3, 6.1)

Watch Out For

Motion blur on fast fingers

Rolling fingers smear tips across frames. Symptom: jittery pinch states. Fix: raise shutter speed or frame rate first; smooth trajectories rather than trusting single-frame tips.

Left-right hand flips

Mirrored hands swap labels under weak lighting. Symptom: gestures fire on the wrong hand. Fix: keep explicit handedness classification and verify per-hand accuracy.

The Quick Version

Palm detection first, then $21$ -joint regression inside the crop.
MANO parameters give a full mesh when a skeleton is not enough.
Bone and collision losses keep fingers plausible under occlusion.
Pinch and grasp logic reads fingertip distances, not raw pixels.
Fast motion needs frame rate before model tuning.