Skip to content
AI360XpertBeta

Memory-Augmented Architectures

A small controller network that keeps its facts in a searchable outside table instead of baking them into its weights, reading and rewriting that table as it works.

A controller scores every memory row by content similarity and reads back a weighted blend, so facts live outside the weights
A controller scores every memory row by content similarity and reads back a weighted blend, so facts live outside the weights

Why Does This Exist?

A standard language model keeps everything it knows baked into its weights. That works until you ask it to change one fact. There is no slot labeled "CEO" to rewrite, so you retrain, you fine-tune and hope, or you watch it state the old answer with total confidence. Researchers call the weights parametric memory, and it carries three honest costs: capacity, because facts compete with skills for the same parameters; staleness, because the world moves on while the weights sit still; and silent guessing whenever a fact never made it in.

Memory-augmented architectures split the job in two. A small controller network learns how to think, and an outside memory matrix holds what to think about. The controller reads the rows it needs and rewrites rows as the task runs, so updating knowledge can mean overwriting a row instead of retraining a model.

Keep two neighbours straight. Retrieval-augmented generation works at the macro level: it pastes text passages into the prompt. This page works at the micro level: the read and write happen inside the forward pass, in vectors the network invents for itself. The KV cache is the narrow, inference-time cousin: it files away keys and values so decoding skips recomputation, but it never learns what to keep. And when you want the practical agent version with sessions, stores and forgetting rules, that is memory systems. Here you learn the mechanism those systems borrow.

One grounding idea, briefly, since the whole page rests on it. Attention blends a set of vectors using weights that sum to 1, so its output is a weighted average, never a copy of one slot. Memory addressing reuses exactly that trick: similarity scores go in, blending weights come out, and the read is their weighted sum. Louder matches get louder votes.

Think of It Like This

A repair bench with labeled parts bins

You are the repair tech at a workbench, and the wall behind you holds rows of small parts bins. You know how to fix things, and you don't carry every screw and fuse in your pockets. A customer describes a broken hinge, you scan the labels, you pull mostly from the hinge bin with a pinch from the lookalike bin, and you fix the thing. When a supplier changes a part, you dump one bin and refill it. Your hands never had to relearn anything.

The tech is the controller, the bins are the memory rows, the description is the key, and the handful you pull is the weighted read.

It stops here: your bins have readable labels and a human who tidies them. The network has rows of numbers with no labels at all, similarity is vector arithmetic instead of reading, and nobody tidies up. Rows blur together when keys are vague, and that blur is where most failures start.

How It Actually Works

The idea started with the Neural Turing Machine and its successor, the Differentiable Neural Computer: a controller wired to an addressable table, trained end to end because every step stays differentiable. Three moves make it run: address, read, write.

The two parts

The controller is an ordinary network, small on purpose. At each step it sees the current input plus whatever it read last, and it emits the vectors that steer the memory: a key kk that says what it wants, an erase vector ee that says what to fade, and an add vector aa that says what to store. The memory matrix MM is a table of numbers with NN rows (slots) and DD columns (numbers per slot), so row MiM_i means slot number ii. Nothing in MM understands language. It only has to keep rows distinct enough that similarity can tell them apart.

Read by content, not by address

A normal computer reads slot 7 because you asked for slot 7. This machine has no addresses, only resemblance. For each slot it scores how much the slot looks like the key with cosine similarity: sis_i is the score for slot ii, the dot product k⋅Mik \cdot M_i divided by both lengths, ∥k∥∥Mi∥\lVert k \rVert \lVert M_i \rVert. So si=1s_i = 1 means same direction and si=0s_i = 0 means unrelated.

Scores become weights through a softmax: wi=exp⁡(si)/∑jexp⁡(sj)w_i = \exp(s_i) / \sum_j \exp(s_j). Each wiw_i is the weight on slot ii, a number between 0 and 1, and all NN weights sum to 1. The read is the weighted blend r=∑iwiMir = \sum_i w_i M_i. The symbol rr is the read vector, the one thing the controller actually gets back. Because softmax is smooth, a tiny change in the key shifts the weights a tiny amount, and gradients flow straight through the operation. That smoothness is the whole reason the machine can learn.

Write means erase then add

Reads alone make a lookup table. Writes make it memory. Each slot updates in place:

Mi←Mi⊙(1−wie)+wiaM_i \leftarrow M_i \odot (1 - w_i e) + w_i a

Read it left to right. The symbol ⊙\odot means multiply element by element. The term wiew_i e scales the erase vector by this slot's weight, so slots the read ignored (wiw_i near 0) barely change, while the attended slot fades wherever ee sits near 1. Then wiaw_i a adds the new content in proportion to the same weight. Set ee to all ones and the slot wipes before the write; set ee near zero and the write blends with what was there. Both ee and aa come from the controller, so the network learns its own storage format instead of writing words.

Worked example

Three slots, two numbers each. The table holds M1=[1,0]M_1 = [1, 0], M2=[0,1]M_2 = [0, 1], M3=[1,1]M_3 = [1, 1], and the controller asks with key k=[1,0]k = [1, 0].

First the lengths: ∥k∥=1\lVert k \rVert = 1, ∥M1∥=1\lVert M_1 \rVert = 1, ∥M2∥=1\lVert M_2 \rVert = 1, ∥M3∥=2≈1.414\lVert M_3 \rVert = \sqrt{2} \approx 1.414. The dot products are k⋅M1=1k \cdot M_1 = 1, k⋅M2=0k \cdot M_2 = 0, k⋅M3=1k \cdot M_3 = 1. Divide each by both lengths:

s=[1.0, 0.0, 1/1.414]≈[1.0, 0.0, 0.707]s = [1.0,\ 0.0,\ 1/1.414] \approx [1.0,\ 0.0,\ 0.707]

Softmax turns the scores into weights. Exponentiate: [exp⁡(1.0),exp⁡(0.0),exp⁡(0.707)]≈[2.718, 1.000, 2.028][\exp(1.0), \exp(0.0), \exp(0.707)] \approx [2.718,\ 1.000,\ 2.028]. The sum is about 5.7465.746, so:

w≈[2.718/5.746, 1.000/5.746, 2.028/5.746]≈[0.473, 0.174, 0.353]w \approx [2.718/5.746,\ 1.000/5.746,\ 2.028/5.746] \approx [0.473,\ 0.174,\ 0.353]

Slot 1 wins without taking everything, which is the point: the read blends, it never just picks. The blend is:

r=0.473[1,0]+0.174[0,1]+0.353[1,1]=[0.826, 0.527]≈[0.83, 0.53]r = 0.473[1, 0] + 0.174[0, 1] + 0.353[1, 1] = [0.826,\ 0.527] \approx [0.83,\ 0.53]

One write step with the same weights. Suppose the controller emits erase vector e=[1,0]e = [1, 0] and add vector a=[0,0]a = [0, 0], meaning fade the first coordinate and add nothing. Slot 1 updates as M1⊙(1−0.473[1,0])=[1,0]⊙[0.527,1]=[0.527,0]M_1 \odot (1 - 0.473[1, 0]) = [1, 0] \odot [0.527, 1] = [0.527, 0], about [0.53,0][0.53, 0]. The memory just dimmed one number in one row, differentiably, mid-task. That is the whole trick: knowledge as numbers you can fade and rewrite while the thinking stays in the controller.

The trade-off is lookup cost. Every read scores every slot, so a step costs on the order of N×DN \times D operations: double the rows and you double the lookup bill. That prices dense external memory out of internet-scale stores, so for those you want retrieval-augmented generation or an approximate index, and you accept text instead of learned vectors. The mechanism also assumes rows stay distinguishable: when keys blur across slots, weights flatten toward uniform and the read returns mush, which is also why weights that memorize facts can leak them, as memorization and extraction shows. Skip this architecture when your store is huge, cold and text-shaped. Use it when a small working store must change during the task itself.

Code

The point of this snippet is the check: it replays the worked example above and prints the similarities, the weights and the read, so you can confirm the hand arithmetic.

import numpy as np

def softmax(x: np.ndarray) -> np.ndarray:    e = np.exp(x - np.max(x))    return e / e.sum()

def cosine_similarities(M: np.ndarray, k: np.ndarray) -> np.ndarray:    return (M @ k) / (np.linalg.norm(M, axis=1) * np.linalg.norm(k))

M: np.ndarray = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])k: np.ndarray = np.array([1.0, 0.0])
sims: np.ndarray = cosine_similarities(M, k)w: np.ndarray = softmax(sims)r: np.ndarray = w @ M
print([round(float(v), 3) for v in sims])  # -> [1.0, 0.0, 0.707]print([round(float(v), 3) for v in w])  # -> [0.473, 0.174, 0.353]print([round(float(v), 2) for v in r])  # -> [0.83, 0.53]

Same three numbers as the hand computation, which is the point: the blend favors slot 1 without copying it.

Watch Out For

The controller learns to ignore the memory

Symptom: training loss falls, yet the read weights sit near uniform (about 1/N1/N per slot) or freeze onto one slot whatever the input, and zeroing the read changes nothing. Early in training the table holds noise, so noise comes back, so the controller routes around it, and then the table never earns a useful gradient. It is a loop: ignored memory stays useless because it is ignored.

Fix it by forcing reliance. Train first on toy tasks that are unsolvable without carry-over (copy, repeat, associative recall), watch the entropy of ww during training instead of only the loss, and add a small auxiliary loss that reconstructs the written content. If the entropy never moves, your memory is decoration. Check the weights before you scale anything else.

Lookups get slow as the table grows

Symptom: step time climbs with every thousand rows you add, because each read scores all NN slots at DD multiplies each, which is O(N×D)O(N \times D) per step, paid twice when you read and write. A demo with 128 rows feels instant; the same code with a million rows never finishes a step.

Don't scale the dense table. Keep learned external memory small and hot (the working set the task rewrites), and move the cold internet-scale store to retrieval-augmented generation or an approximate index. If you must grow NN, reach for sparse or top-k addressing first, and measure milliseconds per step against row count before you commit.

The Quick Version

  • A memory-augmented network splits thinking from storage: a small controller learns how, and an outside matrix of NN rows holds what.
  • It addresses by resemblance, never by slot number: cosine scores pass through a softmax, and the read is the weighted blend of every row.
  • Writing is erase-then-add in place, so one fact can change without retraining, and every step stays differentiable.
  • In the three-slot example the key [1,0][1, 0] gives weights about [0.47,0.17,0.35][0.47, 0.17, 0.35] and reads back about [0.83,0.53][0.83, 0.53].
  • Each step costs on the order of slots times width, so keep the dense table small and hot, and use retrieval for the cold store.
  • If the read weights sit near uniform and zeroing the read changes nothing, the controller has learned to ignore the memory: force it with recall-first toy tasks.