Visual explainer

Long Short-Term Memory

LSTMs add a cell highway with three gates so old context survives the trip forward.

Plain recurrence shrinks gradients to a millionth over ten steps.

You run a vanilla RNN and each step multiplies the signal by 0.25 at most. After ten steps you keep a millionth of it. Early words vanish, so long sentences lose their start.

Straight Memory Highway

A straight cell highway carries memory unchanged by default.

You lay a second track beside the hidden state. Left at 1.0, it arrives at 1.0 unless gates touch it. Gradients ride it backward untouched, so old context survives.

Three Gates Edit

Gates edit the cell from 1.0 to 1.38 in one step.

Forget keeps 0.9 of the old 1.0, giving 0.9. Input takes 0.6 of candidate 0.8, adding 0.48. Your cell moves 1.0 to 1.38 with a paper trail you can check.

Read Out Loud

The output gate reads 0.62 from the updated cell.

You squash 1.38 with tanh to 0.88. Output opens at 0.7, so hidden becomes 0.62. That is the only part the next step sees.

Old Words Steer

A word from twenty steps back still steers today.

Give both nets a name from twenty steps ago. The plain net guesses blind while your gated net uses it. Same sentence, same scale, but the highway keeps the clue alive.

Bias The Forget

Start forget bias at 1.0 for long memory tasks.

You start forget bias at 1.0 when passages run long, so cells default to keeping. Drop it toward 0 when streams reset often. Flip it when your net clings to stale clues.

Where It Breaks

Every step waits for the last, so GPUs idle.

Step 3 needs step 2 done, so one hundred tokens cost one hundred serial waits. You cannot widen this like attention. For long texts you switch architectures.

The Quick Version

Plain nets keep a millionth after ten steps.
A cell track carries 1.0 untouched.
Gates move 1.0 to 1.38.
Output gate reads out 0.62.
Old names still steer new words.
Bias forget to 1.0 for length.
Serial steps stall your GPUs.