Grasp Detection

Grasp detection proposes where a gripper should close, with what orientation and width, ranked by predicted success on the visible depth.

Candidate gripper rectangles are scored by width fit and clearance, and the best collision-free one wins

Why Does This Exist?

Knowing a mug is present does not say where fingers fit without slipping or hitting the table. Grasping needs contact geometry: two opposing edges, a width the gripper spans, an approach that clears neighbors. Grasp detection exists to propose exactly that from an image or depth map: oriented rectangles or 6D poses ranked by predicted holding success, filtered for collisions before the arm moves.

For boxes see object detection and for full poses see the 6D pose page in this wave.

Think of It Like This

Picking up a slippery bar of soap

You do not grab the middle blindly. You look for two flat sides your fingers can oppose, check the width fits your hand, and lift where neighbors will not block you. The model scores the same three facts per candidate. The analogy stops at soap: the model also predicts a success number from thousands of prior lifts.

How It Actually Works

Inputs are RGB plus depth $D$ where each pixel carries range. Common outputs are grasp rectangles $(x, y, \theta, w)$ with center, angle, and opening width. Here $\theta$ is in-plane rotation and $w$ is gripper width in pixels or millimeters. Networks score each candidate for force closure likelihood from local depth edges and normals. Non-maximum suppression keeps distinct grasps, and a collision check against the depth map rejects approaches blocked by neighbors or the table.

Worked example

A mug rim offers a candidate at angle 20 degrees with width 65 mm against a gripper spanning 85 mm: 20 mm clearance, score 0.88. A second candidate across the body needs 90 mm width, exceeding the span, score 0.20 and rejected. A third scores 0.91 but its approach ray crosses a nearby bowl, so collision filtering drops it. The first wins despite a lower raw score, which is why ranking happens after filtering rather than before.

Code

This snippet rejects grasps wider than the gripper.

1def choosegrasps(cands: list[tuple[float, float]], max_w: float = 85.0) -> tuple[float, float] | None:2    feasible = [(w, s) for w, s in cands if w <= max_w]3    if not feasible:4        return None5    return max(feasible, key=lambda t: t[1])6
7
8print(choosegrasps([(65.0, 0.88), (90.0, 0.95), (70.0, 0.80)]))  # -> (65.0, 0.88)

The 0.95 grasp loses on width. Hardware limits gate learning outputs.

Watch Out For

Depth glare on shiny metal

Specular cans and cutlery return missing or spiked depth, so edge-based scoring hallucinates grasps in midair. The symptom is high scores with empty closes on reflective objects. The fix fuses RGB edges, filters depth outliers, and lowers confidence on low-return regions instead of trusting raw depth.

The Quick Version

Grasps are oriented rectangles with width plus success scores.
Depth edges and normals decide hold quality more than RGB.
Width limits and collisions filter before ranking.
Score after filtering, not before.
Shiny objects need RGB fusion and depth skepticism.