OpenPose Real-Time Skeletons

OpenPose finds all joints and limb directions in one pass, then assembles people with part affinity fields that point along each limb.

OpenPose draws arrows along each limb so the assembler knows which elbow connects to which wrist.

Why Does This Exist?

Multi-person 2D pose faces the grouping problem: all elbows are found, but whose elbow is whose? OpenPose (Cao et al., 2017) solved it at real-time speed with Part Affinity Fields (PAFs): alongside each joint heatmap it predicts a 2D vector field per limb pointing from one joint to the next. Linking becomes scoring candidate connections by how well they align with the field, assembled greedily person by person in one image pass.

Think of It Like This

Wind arrows between airport beacons

Every airport beacon flashes (joint heatmap peak) and wind arrows paint the sky along flight corridors (PAFs). Controllers connect beacons whose straight flight path rides the wind instead of crossing it. Linking elbows to wrists works the same: the pair whose segment best follows the painted arrows belongs together.

Where it stops: crossing flight paths in crowds tangle the arrows, and greedy controllers connect the wrong pair.

How It Actually Works

A VGG-style backbone feeds iterative refinement stages outputting $19$ heatmaps (18 joints plus background in the original MPII-era build; COCO builds use $18$ keypoints) and $38$ PAF channels ( $19$ limbs $\times$ $2$ directions). Candidate joints come from heatmap NMS. Each candidate limb pair scores the line integral of PAF vectors along the segment: high when arrows point along the segment. Greedy bipartite matching per limb type assembles skeletons, and low-score people are pruned.

Worked example

Elbow candidates $E_1, E_2$ and wrist candidates $W_1, W_2$ . Segment $E_1W_1$ samples $10$ points with mean PAF alignment $0.9$ ; $E_1W_2$ scores $0.2$ . The matcher links $E_1W_1$ first, removes those joints, then links $E_2W_2$ at $0.85$ . Total assembly is milliseconds after the single network pass, which is why crowds stay real-time.

Code

1# Greedy limb matching by PAF alignment score.2pairs = [(("E1", "W1"), 0.9), (("E1", "W2"), 0.2), (("E2", "W2"), 0.85)]3print(sorted(pairs, key=lambda p: p[1], reverse=True)[0][0])4# -> ('E1', 'W1')

Watch Out For

Greedy links in tangled crowds

Overlapping arms produce plausible wrong pairings. Symptom: swapped forearms in hugs and scrums. Fix: raise input resolution for crowds, or switch to top-down methods when accuracy outranks speed.

Running the 2016-era model file today

Old checkpoints use outdated joint sets and VGG weights. Symptom: missing ears and weak small-person recall. Fix: use maintained builds with COCO-18 or COCO-17 outputs and modern backbones.

The Quick Version

OpenPose predicts joint heatmaps plus limb direction fields in one pass.
PAF line integrals score which joints belong to the same limb.
Greedy matching assembles multi-person skeletons at real-time speed.
Crowds and tangled limbs are the accuracy ceiling.
The classic child of bottom-up pose that made crowded scenes practical.