Hand Tracking

Hand tracking finds 21 finger joints from a camera so pinch, grab, and point work without gloves or controllers.

A pinch gap of 8 mm reads stable when fingertip error stays near 4 mm but flickers once error passes 9 mm

Why Does This Exist?

Controllers add friction and gloves add cost. Hands already point, pinch, and grab with millimeter intent. Headsets and kiosks need that intent from cameras alone, through motion blur and self occlusion.

Hand tracking exists for bare hand input. Input is an RGB crop with a hand. Output is $21$ joints plus a mesh and a pinch state. The body joint base is 2D pose and 3D pose. The headset home is AR and VR.

Use it when interaction must stay controller free. Use controllers or gloves when force feedback or sub millimeter truth matters.

Think of It Like This

Reading a folded map through a keyhole

You see one folded corner at a time through a keyhole and rebuild the whole map from crease angles you know by heart. A thumb hides two fingers, but the crease pattern says where they must fold.

Palm detection is the keyhole. The skeleton prior is map memory. The analogy stops at touch: you never feel the paper, so a visually perfect pinch can still miss by the depth error you cannot see.

How It Actually Works

Detect the palm, crop, then regress $21$ joints $(x, y, z)$ plus a low poly mesh. Contract is crop in, joints plus pinch flag out. Smooth over time to kill jitter, then decide pinch when thumb to index gap $g$ stays under $10$ mm for $3$ frames. A true $8$ mm pinch with $4$ mm joint error reads $4$ to $12$ mm and holds. The same pinch with $9$ mm error reads $-1$ to $17$ mm and flickers.

Worked example

Thumb tip at $[10, 20, 50]$ mm and index tip at $[12, 26, 54]$ mm give gap $\sqrt{4 + 36 + 16} = \sqrt{56} \approx 7.5$ mm, a pinch. Next frame blur shifts index by $9$ mm to gap $14$ mm, a release. Temporal smoothing averages three frames to $9.5$ mm and holds pinch through the blurred frame. Without smoothing the button drops mid press.

Watch Out For

Self occlusion that invents fingers

A fist hides four fingertips behind the palm. The model hallucinates straight fingers at $0.7$ confidence. The symptom is ghost pinches during grabs. Fix it with occlusion labels in training and by gating actions on visibility scores.

Egocentric blur that eats depth

Head mounted views swing fast, smearing fingertips by $8$ pixels. Depth error doubles while 2D looks fine. The symptom is stable overlays with broken grabs. Fix it with faster exposure, inertial fusion, and by widening the pinch window during motion.

The Quick Version

Hand tracking outputs $21$ joints plus mesh and pinch state.
Contract is hand crop in, joints plus pinch out, smoothed over frames.
The fixture holds an $8$ mm pinch at $4$ mm error and flickers past $9$ mm.
Pinch needs $10$ mm for $3$ frames, not one lucky frame.
Occlusion and egocentric blur are the first failures to test.