Graph Neural Networks
Each node collects information from its neighbors, combines it with its own, and repeats, so a network can learn from relationships, not just isolated rows.
Why Does This Exist?
A social platform scores accounts for bot behavior. Each account has its own features: post rate, account age, profile completeness. A row-by-row classifier gets most accounts right, then misses one coordinated cluster: a hundred accounts that each look plausible alone but follow each other, share the same links, and joined within the same hour.
That pattern lives in the connections, not in any single row. A model that scores rows independently never sees it. What is missing is the graph itself, who links to whom, plus a rule that lets each account's score depend on its neighbors.
A graph neural network folds the links into the model. Each node builds its state from its own features plus a summary of its neighbors, and stacking that step carries information two or three hops out. The static version of this idea, walks turned into vectors, is graph representation learning. The step-by-step picture is graph neural networks.
Think of It Like This
Neighbors describing one house
An appraiser prices one house. The house itself matters: size, age, condition. But the three nearest sales matter just as much, because buyers pay for the street, not only the bricks.
So the appraiser averages the three neighbors, then blends that average with the house itself. Next round, each neighbor's price already reflects its own neighbors, so the house quietly absorbs two streets away. That is message passing: average around, blend with self, repeat.
Where it stops: houses keep their address no matter how many rounds pass. Node vectors do not. Average too many rounds and every house price converges toward the town average, which is exactly the over-smoothing failure below.
How It Actually Works
One layer is one round of messages. Every node gathers its neighbors' vectors, summarizes them into one vector, and merges that summary with its own state:
Here is node 's vector after layers, is its neighbor set, handles any neighbor count with a sum, mean, or max, and merges the neighbor summary with the node's own state. Layer zero is the raw input features.
Worked example: center 1.0 and 0.0 with three neighbors
Center . Neighbors , , . Mean aggregation sums each coordinate and divides by three:
So the neighbor summary is . Merging with self by averaging all four vectors gives:
The new center is . The coordinate stays 1.0 because the neighbors balance, while the coordinate rises toward the one neighbor carrying signal. The diagram shows these exact four dots, the mean arrow, and the updated center.
Why the three famous variants differ
A Graph Convolutional Network averages with degree normalization so high-degree hubs do not dominate. GraphSAGE samples a fixed neighbor count per node, which caps cost on graphs where one node has thousands of links. A Graph Attention Network learns a weight per neighbor instead of fixed averaging, so each node decides whom to listen to. All three use the same worked update above and differ only in the step. The contrastive way to train such vectors without labels is contrastive learning, and the per-pair weighting idea comes from attention mechanism.
Code
The point is the check: rebuild the worked aggregation and confirm the summary reads 1.000 and 0.667 with the update at 1.000 and 0.500.
center: list[float] = [1.0, 0.0]neighbors: list[list[float]] = [[3.0, 0.0], [1.0, 2.0], [-1.0, 0.0]]
agg: list[float] = [sum(n[d] for n in neighbors) / len(neighbors) for d in range(2)]new: list[float] = [(center[d] + sum(n[d] for n in neighbors)) / (len(neighbors) + 1) for d in range(2)]
print([f"{v:.3f}" for v in agg]) # -> ['1.000', '0.667']print([f"{v:.3f}" for v in new]) # -> ['1.000', '0.500']Change from to and both outputs collapse to . The update only carries what the neighbors actually contain.
Watch Out For
Stacking layers the way a vision net stacks them
Symptom: depth goes from two layers to six and accuracy falls instead of rising, with node vectors looking more alike each layer.
Fix: sweep depth as its own small search around two to three layers and stop there. Each layer is an averaging round, and past that radius every node blends toward the same value. Depth here buys reach, not expressiveness.
Aggregating full neighborhoods on a hub-heavy graph
Symptom: training time swings wildly by batch because one sampled node has thousands of neighbors and its full aggregation dominates the step.
Fix: sample a fixed neighbor count per node, the GraphSAGE rule, so per-node cost stays capped no matter the true degree. Keep full aggregation for small graphs only.
The Quick Version
- A graph neural network layer averages neighbor vectors and merges the result with the node's own vector.
- Our center at 1.0 and 0.0 with neighbors 3.0 and 0.0, 1.0 and 2.0, minus 1.0 and 0.0 gives neighbor mean 1.0 and 0.667.
- Including the center itself, the updated vector is 1.0 and 0.5.
- GCN normalizes by degree, GraphSAGE samples neighbors for scale, and GAT learns per-neighbor weights.
- Past three or four layers, repeated averaging erases the distinctions the model needs, so practical nets stay shallow.