Visual explainer
Tensor Processing Units
Follow a small weight-stationary array through multiply-accumulate operations, operand reuse, staggered timing and memory limits.
Tensor Processing Units (TPUs) accelerate tensor operations with specialized hardware, including matrix-multiply units. In a systolic array, neighboring multiply-accumulate cells exchange operands on local paths, allowing reuse without fetching every intermediate from a large memory. The mint wire carries the same activation, 1, onward to a second cell.
High Bandwidth Memory (HBM) and on-chip buffering still supply data. This is a conceptual path, not a claim that CPUs fetch DRAM for every arithmetic operation. CPUs and GPUs also reuse registers, caches and local storage. Google's architecture documentation distinguishes matrix, vector and scalar units; this page focuses on a small model of the matrix datapath, not every TPU component or generation.
Multiply and Accumulate
MAC means multiply-accumulate. With activation x = 2, stored weight w = 3 and incoming partial sum p = 4, the new sum is p + xw = 4 + 2 × 3 = 10. The horizontal activation remains 2; the changed sum exits downward. These are separate payloads.
All following timing uses an explicitly simplified machine: preloaded weights, one MAC and one neighbor hop per tick, zero starting sums, sufficient buffering and exact small-integer arithmetic. These ticks are not measurements of a real TPU's latency.
Arrange the Weight Tile
Use this synthetic matrix product:
The four cell labels are stationary weights. Physical array rows represent the reduction index k; physical columns represent output columns j. The upper left input stream is 1 then 3, while the lower stream is 2 then 4 with a one-tick initial delay. They are columns of X arriving over time, not its two original rows presented simultaneously. Activations move right; partial sums move down.
The right-hand matrix is the assembled final result, not a single clock snapshot. Check its entries: 1 × 5 + 2 × 7 = 19; 1 × 6 + 2 × 8 = 22; 3 × 5 + 4 × 7 = 43; 3 × 6 + 4 × 8 = 50.
The original TPU paper, section 2 and its Figure 4 discussion, describes preloaded weights, streamed data and a diagonal wavefront. Our two-by-two model illustrates that dataflow with its own stated timing assumptions.
Reuse a Loaded Weight
Read time downward in this drawing. The tall outline represents one physical cell reused across two ticks, not two cells with simultaneous inputs. Its stored weight 5 acts first on activation 1, producing 5, then on activation 3, producing 15. Both incoming partial sums are zero at the top array row.
Weight-stationary means retaining a weight tile while processing an activation stream. Different tiles still need loading, and training eventually updates weights. Reuse does not imply that memory traffic disappears or that weights remain fixed throughout a training run.
Forward One Activation
At tick 0, activation 1 multiplies weight 5. One hop later it multiplies weight 6 in the neighboring output column. With zero incoming sums at the top row, those products are 5 and 6. The two downward paths represent contributions to different output columns; the operations occur at different ticks.
For this tiny loaded tile, four weights and four activation values enter the modeled computation, and the four outputs are eventually collected. Activations serve both columns and weights serve both input rows. This accounting excludes instructions, physical memory transactions, spills and reloads. It establishes no universal CPU-versus-TPU fetch count or energy ratio.
Align the Arrivals
Trace just the first output column for the first row of X. Tick 0 produces 1 × 5 = 5 in the upper cell. At tick 1, that partial sum reaches the lower cell together with the deliberately delayed activation 2. The lower MAC adds 2 × 7 = 14, yielding 19. The delay is part of scheduling, not evidence of a memory stall.
More generally, let i identify an input row, k the reduction row and j the output column, all starting at zero. Inject X[i, k] at tick i + k. It reaches cell (k, j) at tick i + k + j, aligned with the matching partial sum from the row above. Feeding all reduction rows at the same instant would misalign this model's computation.
Zeros Still Use Slots
Change only the weights to rows [5, 0] and [0, 8]. The hatched zero-weight cells remain part of the dense array. With the same X, the new Y has rows [5, 16] and [15, 32]. Eight MAC slots still execute in this model; four products contain a zero weight. A cell multiplying zero is computing a zero contribution, not necessarily waiting or skipping a cycle.
This scope matters: dense execution does not automatically exploit sparsity. Actual sparse support depends on the hardware and software path; separate sparse hardware is outside the little dense array shown here. Do not infer a utilization percentage from zero count alone.
See the Wavefront Gaps
Return to the original dense W. Each row names a cell by its weight; each column is a tick. A dot marks one valid MAC, while an empty slot has no valid work in this model. Tick 0 uses only weight 5. Tick 1 uses 5, 6 and 7. Tick 2 uses 6, 7 and 8. Tick 3 uses only 8.
Output 19 finishes at tick 1, outputs 22 and 43 at tick 2, and output 50 at tick 3. There are eight valid MACs across four cells times four ticks: 8 / 16 = 50% occupancy for this toy schedule. Weight loading and memory stalls are excluded. This is not a real TPU utilization figure. Longer streams can amortize these initial and final gaps, subject to other resource constraints.
Account for Padded Columns
Now request only W's first column, [5, 7], giving output column [19, 43]. In the drawing's fixed two-column tile, a padded second column of zero weights produces results the requested operation does not need. Dashed outlines distinguish padding from requested data; the cross marks discarded output.
This is a spatial fit issue, distinct from the temporal fill/drain gaps. Real compiler layouts depend on the target. Google's performance guide explains that padding can consume extra compute and storage. The pictured half-width tile is illustrative, not a prediction for a particular TPU model.
Where It Breaks
Return to the first-column calculation 5 + 2 × 7 = 19. If activation 2 is not ready, partial sum 5 cannot complete that matched MAC. The empty input buffer and stop bar block progress; no output arrow reaches the pending 19. A system must hold the matching state or pause the relevant pipeline until its dependencies are available. This picture does not specify a real chip's stall-control implementation.
Local reuse reduces repeated transfers inside the operation, but data still must reach it. Missing activations, unloaded weights or insufficient memory bandwidth can limit throughput. The original paper explicitly notes operand-readiness stalls and memory-bound workloads. There is no universally correct batch size or guaranteed utilization; inspect the actual workload, layout and memory path.
The Quick Version
- Systolic cells exchange operands over local connections.
- A MAC adds a product to a partial sum.
- Array rows reduce; columns produce different output columns.
- Loaded weights serve successive input rows.
- Forwarded activations serve neighboring columns.
- Staggered inputs keep matching terms aligned.
- Dense zero multiplies can still occupy compute slots.
- Short streams leave fill and drain gaps.
- Padding can consume unneeded compute and storage.
- Unavailable data can still block the computation.