Corner Detection with Harris and Shi-Tomasi

Look For Two Directions

Harris and Shi-Tomasi find locations whose nearby intensities change strongly in two independent directions. Such patches constrain local displacement better than a flat area or a straight edge. They detect useful image structure, not necessarily corners of physical objects.

The solid arrows show directions across visible boundaries; image x increases rightward and y downward. The dashed arrow follows the straight edge, where sliding the window may barely change its contents. At a corner, horizontal and vertical shifts both change the observed patch. These schematic windows explain the idea; they are not the raw pixel input for the numerical fixture below.

Pool The Local Gradients

Use a synthetic two-sample window to make the arithmetic inspectable. Its gradients are g1=(3,0) and g2=(0,2), with unit weights. Each arrow uses 30 drawing units per gradient unit. Sum their outer products to form the structure tensor:

$$M=\sum_i w_i g_i g_i^\top =\begin{bmatrix}9&0\\0&0\end{bmatrix} +\begin{bmatrix}0&0\\0&4\end{bmatrix} =\begin{bmatrix}9&0\\0&4\end{bmatrix}.$$

In a real image, derivative filters provide the gradients and a box or Gaussian window pools many samples. OpenCV's Harris derivation relates this matrix to the change under a small displacement d=(u,v): E(d) ≈ dᵀMd. Here that is 9u² + 4v². Both axes incur a cost; an edge-only window would leave one displacement direction weakly constrained.

Score The Weakest Direction

The tensor's eigenvalues measure change along its principal directions. Our diagonal matrix has eigenvalues 9 and 4. Compare an edge-only tensor diag(9,0) on the identical bar scale: it has one equally strong direction but no evidence in the other.

Harris computes R = det(M) − k trace(M)². With the illustrative choice k=0.04, our corner gives 36 − 0.04×169 = 29.24. The edge gives 0 − 0.04×81 = −3.24. A large positive response indicates a corner candidate; a positive threshold rejects weak responses as well as edges.

Shi-Tomasi uses min(λ1,λ2): 4 for the corner, 0 for the edge. It directly tests whether even the weakest direction has enough evidence, as specified in OpenCV's Shi-Tomasi reference. Harris and Shi-Tomasi share the tensor but use different scoring rules. Their score values have different scales and cannot share an arbitrary numeric cutoff.

Read Displacement Cost

Plot horizontal and vertical displacements, with the origin at each crosshair and y downward. For the edge tensor, E=9u²: the contour E=36 consists of the two lines u=±2, extending indefinitely along v. For our corner, 9u²+4v²=36 gives an ellipse with horizontal radius 2 and vertical radius 3. At 30 drawing units per displacement unit, the radii are 60 and 90. Every direction eventually crosses the closed contour. The ellipse is a cost contour, not the shape of an object or an uncertainty estimate; the quadratic model approximates small shifts.

Rotate The Same Evidence

Rotate diag(9,4) by 45 degrees using an orthogonal rotation Q: M′=QMQᵀ=[[6.5,2.5],[2.5,6.5]]. Its trace stays 13 and determinant stays 36, so its eigenvalues remain 9 and 4. Both drawn cost contours have the same size; only orientation changes. Harris remains 29.24 and Shi-Tomasi remains 4. This is ideal rotational covariance of the local model. Resampling, discrete derivatives and finite windows mean a rotated raster image need not reproduce exactly the same numeric responses.

Compare Detector Rankings

Consider two new positive-semidefinite tensors with eigenvalues A=(100,10) and B=(20,20). Bars share one scale. With k=0.04, Harris gives RA=1000−0.04×110²=516 and RB=400−0.04×40²=336, preferring A. Shi-Tomasi gives minimum eigenvalues 10 and 20, preferring B. Harris rewards the combined determinant/trace response; Shi-Tomasi directly ranks the weakest direction. Neither score is simply a differently scaled version of the other across arbitrary patches.

Account For Contrast

Return to the original (9,4) tensor and multiply image intensities by 0.5 without clipping. Gradients halve, their outer products quarter, and the eigenvalues become (2.25,1), shown on the same scale. Shi-Tomasi falls from 4 to 1. Harris falls by a factor of sixteen, from 29.24 to 1.8275, because determinant and squared trace are both fourth-order in the gradient scale. Fixed absolute thresholds therefore depend on image scaling and derivative normalization. Relative thresholds compensate for uniform scaling, but local lighting changes can still alter relative rankings.

Keep Local Response Peaks

A response image can have several high-scoring pixels around one feature. In this synthetic three-by-three patch, the center score 9 exceeds all eight neighbors. Local peak selection keeps that location rather than reporting every strong response nearby. A score cutoff alone would not enforce this: a cutoff of 5 would also admit 5 and 6. This drawing isolates one neighborhood; real local maximum checks use the full response map, and optional subpixel refinement can adjust the selected position afterward.

Select Separated Candidates

Now consider four synthetic, already locally maximal Shi-Tomasi candidates, with scores A=4, B=3, C=1 and D=2.5. A relative quality level of 0.5 gives a cutoff of 0.5×4 = 2, rejecting C. Process the remaining candidates strongest first and require a minimum separation of ten image pixels.

A is at (20,20), B at (25,20), and D at (65,60). B is only five pixels from A, so discard B; D is farther away and survives. Both drawn axes use four drawing units per image pixel. The rings have radius ten image pixels; crosses mark discarded candidates. C lies at (60,20). Local peak selection, relative score filtering, spacing and an optional maximum count are distinct steps, not properties automatically delivered by a positive Harris score.

Where It Breaks

The source corner has two indistinguishable local candidates in the next image. A high corner score says that nearby motion is constrained; it does not identify the same physical point uniquely across a repeated pattern. Matching needs additional appearance or geometric evidence, and tracking also has to handle occlusion and motion size.

Scores depend on contrast and derivative normalization. Multiply image intensities by a, without clipping: gradients scale by a, the tensor and Shi-Tomasi score by a², and the Harris response by a⁴. Relative thresholds adapt to uniform score scaling, but cannot repair blur, missing texture or ambiguous repeated motifs.

The Quick Version

• Corners constrain displacement in two independent image directions.
• The structure tensor pools outer products of local gradients.
• Harris combines determinant and trace; Shi-Tomasi uses the smaller eigenvalue.
• Closed displacement-cost contours indicate evidence in every direction.
• Ideal tensor rotation preserves eigenvalues and both corner scores.
• Harris and Shi-Tomasi can rank candidates in opposite orders.
• Contrast scaling changes score magnitudes without changing ideal geometry.
• Local peak selection avoids reporting every neighboring strong response.
• Thresholding and spacing turn response candidates into separated feature points.
• Strong corners can still have ambiguous matches in repeated texture.