Corner and Keypoint Detection

Corners and keypoints are distinct image regions that can be tracked across frames or matched between views, acting as stable anchors for vision tasks.

A corner scores R equal to 9.04 with two large eigenvalues, an edge scores minus 0.54, and flat ground scores near zero, so one threshold separates all three

Why Does This Exist?

Stitching a panorama, tracking a car across frames, or rebuilding a scene in 3D all need the same thing: reference points that can be found again in another image. Comparing every pixel of image A against every pixel of image B is hopeless — slow, and shattered by small shifts in light or viewpoint.

The fix is sparse, distinctive anchors called keypoints. Corners are the best simple keypoint: unlike a flat patch (nothing changes anywhere) or an edge (nothing changes along its length), a corner changes sharply in every direction, so it localizes precisely in both axes. Harris made them cheap to find; SIFT later made them survive zoom. This page covers what makes a good keypoint and how Harris scores one. The detector's own page is Harris corner detector, zoom-proof features are in scale-invariant feature transform, and joining the matches is feature matching for image stitching.

The one idea you need first

An image gradient $(I_x, I_y)$ measures how fast brightness changes horizontally and vertically at a pixel. Flat ground has near-zero gradients, an edge has a strong gradient across it but none along it, and a corner has strong gradients in both — the whole detector below is bookkeeping on these two numbers.

Think of It Like This

Matching jigsaw pieces of empty sky

Two puzzle pieces of flat blue sky cannot be aligned — no feature pins them. Pieces crossed by one straight contrail slide along it: one direction fixed, one free. But pieces sharing the sharp corner where a cloud cuts the sky snap together instantly in both directions.

Flat regions shift invisibly, edges shift along themselves, corners shift nowhere without showing it. That is the entire ranking Harris computes. The analogy stops at uniqueness: a real corner is only useful if its neighbourhood looks different from every other corner's, which is why repetitive brickwork defeats matching despite firing thousands of detections.

How It Actually Works

Harris slides a conceptual window and asks how much the patch would change if shifted. Instead of shifting, it builds the $2 \times 2$ structure tensor $M$ from summed gradient products over the window, weighted toward the center:

M = \sum w(x,y) \begin{bmatrix} I_x^2 & I_x I_y \\ I_x I_y & I_y^2 \end{bmatrix}

Here $I_x$ and $I_y$ are the horizontal and vertical gradients, and $w$ is typically a Gaussian weight. The two eigenvalues $\lambda_1, \lambda_2$ of $M$ tell the story: both small means flat, one large means edge, both large means corner. To skip the eigenvalue computation, Harris scores the response directly from determinant and trace:

R = \det(M) - k \cdot (\text{trace}(M))^2

with $\det(M) = \lambda_1 \lambda_2$ , $\text{trace}(M) = \lambda_1 + \lambda_2$ , and $k \approx 0.04$ . Corners score large and positive, edges negative, flat ground near zero — one threshold separates all three.

Worked example

Take $M = [[4, 1], [1, 3]]$ with $k = 0.04$ . Determinant $= 4 \times 3 - 1 = 11$ , trace $= 7$ , so $R = 11 - 0.04 \times 49 = 11 - 1.96 = 9.04$ : strongly positive, a corner. Its eigenvalues are $(7 \pm \sqrt{5}) / 2$ , i.e. $4.618$ and $2.382$ — both large, as claimed.

Contrast an edge, $M = [[5, 0], [0, 0.1]]$ : determinant $0.5$ , trace $5.1$ , $R = 0.5 - 0.04 \times 26.01 = -0.54$ , negative. And flat ground, $M = [[0.1, 0], [0, 0.12]]$ : $R = 0.012 - 0.04 \times 0.0484 \approx 0.01$ , near zero. Harris himself is rotation-invariant (spinning the image spins both eigenvectors together) but not scale-invariant: zoom in and the corner becomes a curve. SIFT fixes that with an image pyramid.

Code

The worked example in NumPy — response scores and the eigenvalues they stand in for:

1import numpy as np2
3k = 0.044
5def harris_R(M: np.ndarray) -> float:6    det = float(np.linalg.det(M))7    trace = float(np.trace(M))8    return det - k * trace**29
10corner = np.array([[4.0, 1.0], [1.0, 3.0]])11print(round(harris_R(corner), 2))          # -> 9.0412print(np.round(np.linalg.eigvalsh(corner), 3))  # -> [2.382 4.618]13
14edge = np.array([[5.0, 0.0], [0.0, 0.1]])15print(round(harris_R(edge), 2))            # -> -0.54

Positive and large means corner, negative means edge — the sign does the classifying before any threshold is even applied.

Watch Out For

Scale change silently deleting every match

Harris windows have one fixed size, so a corner in a wide shot becomes a smooth curve in a close-up and the detector goes blind. The symptom is two overlapping photos with hundreds of detections each and zero accepted matches.

Switch to a scale-invariant detector such as SIFT or ORB whenever zoom or distance varies between views; reserve Harris for fixed-scale video tracking.

Repetitive texture drowning the matcher

Brick walls, chessboards and fences fire thousands of strong corners whose neighbourhoods look identical. The symptom is a matcher that returns confident correspondences that are geometrically nonsense.

Detect densely but match strictly: keep Lowe's ratio test tight, demand geometric consensus with RANSAC, and treat match count without consensus as no matches at all.

The Quick Version

Keypoints are sparse, distinctive anchors used to match views, track motion and rebuild 3D.
Corners beat edges and flat patches because brightness changes sharply in every direction.
Harris builds a structure tensor from gradients and scores R from determinant minus k times trace squared.
R is large positive for corners (9.04), negative for edges (-0.54), near zero for flat ground.
Harris survives rotation but not zoom; SIFT adds scale invariance with an image pyramid.
Repetitive texture needs strict ratio tests and RANSAC, not more detections.