Math for Machine Learning

Linear algebra, calculus, optimization, probability, information theory, and the numerics that keep models stable.

Objects and Shapes

Object	Notation	Shape	Example
Scalar	a	()	learning rate
Vector	x	(n,)	one feature row
Matrix	X	(n, d)	dataset of n rows
Tensor	T	(b, c, h, w)	image batch

Check shapes first

Most linear-algebra bugs are shape bugs. Write the shape next to every symbol while deriving.

Vector Operations

a · b = Σ aᵢbᵢ = ‖a‖ ‖b‖ cos θ

‖x‖₂ = √(Σ xᵢ²), ‖x‖₁ = Σ |xᵢ|

Dot product: Measures alignment; zero means orthogonal.
Projection: proj_b(a) = (a · b / ‖b‖²) · b.
Unit vector: x / ‖x‖; cosine similarity compares unit vectors.

Matrix Operations

Product: (n×k)(k×m) → (n×m); the inner dimensions must match.
Transpose: (AB)^T = B^T A^T.
Inverse: A⁻¹A = I; exists only for square, full-rank matrices.
Rank: Number of linearly independent rows or columns.
Trace: Sum of the diagonal; equals the sum of eigenvalues.
Hadamard product: Element-wise multiply, written A ⊙ B.

Linear Systems

Ax = b

x̂ = (A^T A)⁻¹ A^T b

Least-squares solution when the system is overdetermined

Solve, don't invert: Use a solver (LU, QR, Cholesky) instead of computing A⁻¹.
Condition number: Large values mean small input changes cause large output changes.
Pseudo-inverse: A⁺ handles rank-deficient systems via SVD.

Eigen and Singular Value Decomposition

Av = λv

A = U Σ V^T

Σ holds singular values in decreasing order

Eigenvector: A direction that A only scales, by the eigenvalue λ.
Symmetric matrices: Real eigenvalues and orthogonal eigenvectors.
SVD: Works for any matrix; truncating to k values gives the best rank-k approximation.
PCA link: Principal components are the top right-singular vectors of centered data.

Positive Definiteness

Definition: A is positive definite if x^T A x > 0 for every nonzero x.
Eigenvalue test: All eigenvalues are positive (≥ 0 for semidefinite).
Why it matters: A PD Hessian means a strict local minimum; covariance matrices are PSD.
Cholesky: A = L L^T exists exactly when A is positive definite.

Derivatives

Function	Derivative
xⁿ	n xⁿ⁻¹
e^x	e^x
ln x	1 / x
σ(x)	σ(x)(1 − σ(x))
tanh x	1 − tanh² x
ReLU(x)	1 if x > 0 else 0

Gradients and the Chain Rule

dz/dx = (dz/dy) · (dy/dx)

∇f(x) = [∂f/∂x₁, …, ∂f/∂xₙ]

Gradient: Points in the direction of steepest increase.
Jacobian: Matrix of first derivatives of a vector-valued function.
Hessian: Matrix of second derivatives; describes curvature.
Backpropagation: The chain rule applied from the loss backwards through the graph.

Optimization

Compute loss
Compute gradient
Update θ ← θ − η∇L
Repeat until converged

One gradient-descent step; η is the learning rate

θₜ₊₁ = θₜ − η ∇L(θₜ)

SGD: Gradient from a mini-batch; noisy but cheap and generalizes well.
Momentum: Accumulates a velocity to smooth noisy gradients.
Adam: Per-parameter step sizes from running gradient moments.
Learning-rate schedule: Warmup then decay usually beats a constant rate.

Convexity

Convex function: Any chord lies above the curve; every local minimum is global.
Test: Twice-differentiable f is convex if its Hessian is PSD everywhere.
Convex losses: MSE with linear models, logistic loss, hinge loss.
Non-convex: Neural networks; saddle points are more common than bad local minima.

Probability Basics

P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

P(A | B) = P(A ∩ B) / P(B)

Independence: P(A ∩ B) = P(A) · P(B).
Law of total probability: P(A) = Σ P(A | Bᵢ) P(Bᵢ).
Expectation: E[X] = Σ x · P(x); linear even for dependent variables.

Bayes' Rule

P(θ | x) = P(x | θ) · P(θ) / P(x)

posterior = likelihood × prior / evidence

Prior: Belief about θ before seeing data.
Likelihood: How probable the observed data is under each θ.
Evidence: Normalizer P(x) = Σ P(x | θ) P(θ); often skipped when comparing θ values.

Base rates matter

With a 1% prior, a 99%-accurate test that fires is right only about half the time. Always include the prior.

Distributions

Distribution	Models	Mean	Variance
Bernoulli(p)	Single yes/no	p	p(1 − p)
Binomial(n, p)	Count of successes	np	np(1 − p)
Poisson(λ)	Events per interval	λ	λ
Normal(μ, σ²)	Sums, noise	μ	σ²
Exponential(λ)	Waiting time	1/λ	1/λ²

Covariance and Correlation

Cov(X, Y) = E[(X − E[X]) · (Y − E[Y])]

ρ(X, Y) = Cov(X, Y) / (σ(X) · σ(Y))

Correlation range: −1 to 1; measures linear association only.
Covariance matrix: Σ = (1/n) X_c^T X_c for centered data X_c.
Correlation ≠ causation: Confounders can create correlation without a causal link.

Information Theory

H(p) = −Σ p(x) log p(x)

H(p, q) = −Σ p(x) log q(x)

KL(p ‖ q) = Σ p(x) log (p(x) / q(x))

Entropy: Average surprise; maximal for a uniform distribution.
Cross-entropy: The classification loss; equals entropy plus KL divergence.
KL divergence: Non-negative and not symmetric.

Likelihood and Estimation

Likelihood: L(θ) = Π p(xᵢ | θ); work with the log-likelihood to avoid underflow.
MLE: Pick θ that maximizes the likelihood; MSE is MLE under Gaussian noise.
MAP: MLE plus a log-prior; an L2 penalty equals a Gaussian prior.
Bias–variance of estimators: Unbiased is not always best; shrinkage can lower total error.

Numerical Stability

Log-sum-exp: log Σ e^(xᵢ) = m + log Σ e^(xᵢ − m), with m = max x.
Stable softmax: Subtract the max logit before exponentiating.
Use fused losses: Cross-entropy from logits, not from already-softmaxed probabilities.
Feature scaling: Similar scales improve conditioning and speed up gradient descent.
Epsilons: Add a small ε inside log and division to avoid log(0) and divide-by-zero.

If the loss becomes NaN, check the learning rate, unscaled inputs, and log of zero first.

Formula Memory List

MSE = (1/n) Σ (yᵢ − ŷᵢ)²

softmax(z)ᵢ = e^(zᵢ) / Σₖ e^(zₖ)

σ'(z) = σ(z)(1 − σ(z))

Normal equation: w = (X^T X)⁻¹ X^T y.
Gradient step: θ ← θ − η∇L.
Bayes: posterior = likelihood × prior / evidence.