Tensor Algebra

The algebra of multi-axis arrays: outer products that build them, mode products that transform one axis at a time, and unfoldings that flatten them for ordinary matrices.

The outer product of 1 2 with 3 4 tiles rows into a rank one matrix, and a mode product then transforms one axis while leaving the others untouched

Why Does This Exist?

Vectors and matrices handle one or two axes. Real data has more: video adds time to height, width and channels; attention batches heads with sequences; recommendation logs cross users, items and contexts. Tensor algebra is the set of operations on these multi-axis arrays: building them (outer products), transforming single axes (mode- $n$ products), contracting axes away (summation), and flattening them for matrix tools (unfoldings).

Containers and shapes are taught in vectors, matrices and tensors, index notation in einsum and tensor shapes, and factorisation into tensor decomposition. This page is the operations manual between them: what you are allowed to do to a tensor and what each operation costs.

The one idea you need first

The order of a tensor is its number of axes (a matrix has order 2) and its shape lists each axis length. Do not confuse order with rank: tensor rank means the smallest number of rank-one pieces summing to it, a much wilder quantity than matrix rank, as the pitfalls show.

Think of It Like This

A shipping warehouse with aisles, shelves and bins

A warehouse organises stock along three axes: aisle, shelf and bin. The outer product stocks the whole warehouse from three one-dimensional lists: put quantity $a_i b_j c_k$ in slot $(i, j, k)$ and the full 3D inventory is determined by three short manifests.

A mode product relabels one axis: renumber every aisle according to a conversion chart while shelves and bins stay put. An unfolding picks one axis as the new "rows" and lays all the rest out as one long "columns" corridor so a forklift (your matrix routine) can drive through. Each operation touches the structure along exactly the axes it names and leaves the rest alone.

How It Actually Works

The four operations:

Outer product: $x \circ y \circ z$ builds the tensor with entries $x_i y_j z_k$ . Rank-one tensors are exactly these; everything else is a sum of them.
Mode- $n$ product: $\mathcal{X} \times_n M$ multiplies every mode- $n$ fibre (fixed all-other-indices line) by $M$ . Transforming one axis costs only that axis's share.
Contraction: sum over a shared axis of two tensors, generalising matrix multiplication. Einstein summation writes any contraction in one line.
Unfolding (matricization): $X_{(n)}$ lays mode $n$ out as rows and merges the rest into columns, so SVD and QR apply to tensor data.

Worked example

Outer-multiply $u = [1, 2]$ with $v = [3, 4]$ . Entry $(i, j)$ is $u_i v_j$ :

u \circ v = \begin{bmatrix} 1 \cdot 3 & 1 \cdot 4 \\ 2 \cdot 3 & 2 \cdot 4 \end{bmatrix} = \begin{bmatrix} 3 & 4 \\ 6 & 8 \end{bmatrix}

Rows are multiples of each other (row two is twice row one), so the result has matrix rank 1: the signature of a pure outer product. Now hit mode 1 with $M = [[1, 1], [0, 1]]$ : each column $[a, b]$ becomes $[a + b, b]$ , giving $[[7, 12], [6, 8]]$ ... check column one $[3, 6] \to [9, 6]$ , column two $[4, 8] \to [12, 8]$ . One axis transformed, the other untouched.

Code

The point is the check: build the outer product two ways and confirm the rank-one signature.

1import numpy as np2
3
4u = np.array([1, 2])5v = np.array([3, 4])6T = np.einsum("i,j->ij", u, v)  # outer product via index notation7print(T.tolist())  # -> [[3, 4], [6, 8]]8print(np.linalg.matrix_rank(T))  # -> 1

Watch Out For

Treating tensor rank like matrix rank

Matrix rank computes in polynomial time via SVD. Tensor rank (for order 3 and up) is NP-hard to compute, and a best rank- $k$ approximation may not even exist (border-rank weirdness: a sequence of rank-2 tensors converging to a rank-3 one). Never port matrix-rank instincts to tensors; use CP with a chosen rank and validate reconstruction error instead of "computing the rank".

Unfolding away the structure you needed

Flattening a tensor to a matrix so standard tools apply destroys multi-way correlations: a video unfolded to frames-by-pixels forgets that pixels neighbour each other in two spatial axes plus time. Unfold for computation, but interpret and regularise in tensor form (tensor decompositions, mode-wise penalties) wherever the axes mean something.

The Quick Version

Outer products build tensors from vectors; mode- $n$ products transform one axis at a time.
The worked outer product gives rows 3 4 and 6 8, visibly rank one.
Contraction generalises matrix multiplication; unfolding flattens tensors for matrix tools.
Tensor rank is NP-hard and badly behaved, unlike matrix rank: fix a rank and check error.
Unfold to compute, but keep the multi-axis structure for interpretation.