Eigenvalues and Eigenvectors

Why Does This Exist?

"Just lower the learning rate" is advice you have probably given and received. Eigenvalues explain why it works, why it is unsatisfying, and what the actual problem is.

The curvature of your loss surface in each direction is an eigenvalue of the Hessian. When the largest is 1000x the smallest, any single learning rate is simultaneously too large for the steep directions — where it overshoots and oscillates — and far too small for the shallow ones, where progress crawls. That ratio is the condition number, and it is the real reason training is slow. Adaptive per-parameter step sizes are an approximation to fixing it.

The same numbers show up in three more places you use. PCA finds the eigenvectors of the covariance matrix, and their eigenvalues are exactly how much variance each direction carries. The spectral norm of a weight matrix is its largest singular value, which for symmetric matrices is an eigenvalue. And PageRank is one long eigenvector computation.

Think of It Like This

How It Actually Works

An eigenvector of a square matrix $A$ is a nonzero vector whose direction survives the transformation:

In plain words: applying the matrix to $v$ gives back $v$ scaled by a number. That number $\lambda$ is the eigenvalue. The whole set of eigenvalues is the matrix spectrum, which is where "spectral" in spectral norm and spectral clustering comes from.

Eigenvectors come in families: if $v$ works, so does any multiple of it, so implementations return unit-length representatives and the sign is arbitrary.

Finding them

Rearrange to $(A - \lambda I)v = 0$. For a nonzero $v$ to satisfy this, the matrix $A - \lambda I$ must destroy a direction — it must be singular. So:

In plain words: the eigenvalues are exactly the values that make $A - \lambda I$ collapse. Expanding that determinant gives the characteristic polynomial, of degree $n$, so an $n \times n$ matrix has $n$ eigenvalues counted with multiplicity.

Nobody computes it this way past $2 \times 2$ — root-finding on a degree-1000 polynomial is numerically hopeless. Real implementations use iterative algorithms, and the simplest is worth knowing: power iteration repeatedly multiplies a random vector by $A$ and renormalises. It converges to the dominant eigenvector, because that component grows fastest. PageRank is power iteration, and so is the spectral-norm estimate in spectral normalisation.

Two free checks

These catch a large fraction of implementation errors, and both are one line:

In plain words: the eigenvalues sum to the diagonal total and multiply to the determinant. Always use them when you compute eigenvalues by hand.

Symmetric matrices are the good case

For a symmetric matrix ($A = A^{T}$) three things all become true at once: every eigenvalue is real, eigenvectors of distinct eigenvalues are orthogonal, and the matrix decomposes as

with $Q$ orthogonal and $\Lambda$ diagonal. This is the spectral theorem, and it is why symmetric matrices are so much nicer to work with. The symmetric upgrade is worked fully in eigendecomposition of symmetric matrices.

It matters because the matrices you actually care about are symmetric. Covariance matrices are, which is what makes PCA well behaved. Hessians are, which is what makes curvature analysis meaningful. Graph Laplacians and Gram matrices are.

Non-symmetric matrices can have complex eigenvalues — a rotation matrix has no real eigenvector, because no direction survives a rotation. This is the source of a common code surprise, covered below.

What the eigenvalues tell you

• Sign — for a symmetric matrix, all positive means positive definite, so a bowl and a minimum. All negative, a maximum. Mixed signs, a saddle. This is how the Hessian classifies a stationary point.
• Magnitude — the largest, $\lvert\lambda\rvert_{\max}$, is the spectral radius, and it governs stability: repeated multiplication grows if it exceeds $1$ and decays if below. This is the vanishing and exploding gradient story in recurrent networks, stated precisely.
• Ratio — the condition number $\kappa = \lvert\lambda\rvert_{\max} / \lvert\lambda\rvert_{\min}$ measures how badly stretched the geometry is. Large $\kappa$ means ill-conditioned: slow optimisation and sensitivity to small input changes.

Worked example

Take $A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$, which is symmetric, so expect real eigenvalues and orthogonal eigenvectors.

Hand-computed characteristic polynomial:

Factoring gives $(\lambda - 3)(\lambda - 1) = 0$, so $\lambda_1 = 3$ and $\lambda_2 = 1$.

Check both free identities: trace is $2 + 2 = 4$ and $3 + 1 = 4$. Determinant is $4 - 1 = 3$ and $3 \times 1 = 3$.

For $\lambda = 3$, solve $(A - 3I)v = 0$, which is $\begin{bmatrix} -1 & 1 \\ 1 & -1 \end{bmatrix} v = 0$, giving $-v_1 + v_2 = 0$, so $v = [1, 1]$. Verify directly: $A[1, 1]^{T} = [2+1, 1+2]^{T} = [3, 3]^{T} = 3[1, 1]^{T}$.

For $\lambda = 1$: $\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} v = 0$ gives $v_1 + v_2 = 0$, so $v = [1, -1]$. Verify: $A[1, -1]^{T} = [2-1, 1-2]^{T} = [1, -1]^{T} = 1 \cdot [1, -1]^{T}$.

The two eigenvectors are orthogonal: $[1, 1] \cdot [1, -1] = 1 - 1 = 0$.

Condition number: $3 / 1 = 3$. Mild. And a non-eigenvector confirms the contrast — $A[2, 0]^{T} = [4, 2]^{T}$, which points in a new direction, exactly as the diagram shows.

Code

Eigenvalues $1$ and $3$, trace $4$, determinant $3$, orthogonal eigenvectors — every number matches the hand calculation.

1import numpy as np2
3A = np.array([[2.0, 1.0], [1.0, 2.0]])  # symmetric4
5vals, vecs = np.linalg.eigh(A)  # eigh: symmetric, real, ascending6print(vals)  # -> [1. 3.]7print(round(vals.sum(), 10), round(np.trace(A), 10))  # -> 4.0 4.08print(round(float(np.prod(vals)), 10), round(np.linalg.det(A), 10))  # -> 3.0 3.09
10v = vecs[:, 1]  # eigenvector for lambda = 311print(np.allclose(A @ v, 3 * v))  # -> True12print(round(float(vecs[:, 0] @ vecs[:, 1]), 10))  # -> 0.0  orthogonal13
14# A rotation has NO real eigenvector, so eig() returns complex values.15R = np.array([[0.0, -1.0], [1.0, 0.0]])16print(np.linalg.eigvals(R))  # -> [0.+1.j 0.-1.j]17print(round(vals[1] / vals[0], 4))  # -> 3.0  condition number

The rotation matrix is the complex case: no real direction survives a 90-degree turn.

Watch Out For

The Quick Version

• $Av = \lambda v$: an eigenvector keeps its direction, the eigenvalue is the stretch factor. The set of them is the spectrum.
• Here $[[2, 1], [1, 2]]$ gives $\lambda = 3$ on $[1, 1]$ and $\lambda = 1$ on $[1, -1]$, with trace $4$ and determinant $3$ as free checks.
• They solve $\det(A - \lambda I) = 0$, but nobody computes it that way past $2 \times 2$. Power iteration finds the dominant one.
• Two free checks: eigenvalues sum to the trace and multiply to the determinant.
• Symmetric matrices give real eigenvalues, orthogonal eigenvectors, and $A = Q \Lambda Q^{T}$.
• Non-symmetric matrices can have complex eigenvalues — a rotation has no real eigenvector.
• Signs classify a stationary point: all positive is a minimum, mixed is a saddle.
• The condition number $\lvert\lambda\rvert_{\max}/\lvert\lambda\rvert_{\min}$ is why one global learning rate struggles.
• Use eigh for symmetric matrices, and symmetrise first if floating point made it slightly asymmetric.

What to Read Next

• Einsum and Tensor Shapes
• Entropy
• Symmetric Eigendecomposition