Long Short-Term Memory

The Problem

Before Long Short-Term Memory (LSTM) networks, standard Recurrent Neural Networks (RNNs) struggled severely to learn connections between events separated by significant time lags. When training standard RNNs with backpropagation through time, the error signals either decayed exponentially to zero (the vanishing gradient problem) or grew uncontrollably (the exploding gradient problem). This meant standard RNNs simply forgot earlier inputs when processing long sequences, capping their practical use to tasks with only short-term context.

The Idea

Instead of using a simple neural layer that updates its hidden state at each time step, Hochreiter and Schmidhuber proposed a more complex "memory cell" designed to maintain its state over time. The core idea is to introduce a constant error carousel (CEC) backed by gating units that learn when to let new information in, when to forget the old information, and when to output the stored memory.

How It Works

An LSTM layer relies on memory blocks that contain one or more memory cells, heavily regulated by multiplicative gate units:

The Memory Cell: At the heart of the block is a self-connected linear unit (the Constant Error Carousel) whose weight is precisely 1.0. This allows a gradient to flow backward across many time steps without vanishing or exploding.

The Gates:

• Input Gate: Protects the memory contents from irrelevant incoming signals. It decides what new information should be written into the memory cell.
• Output Gate: Protects other units from irrelevant memory contents. It decides which parts of the cell state should be output to the rest of the network.
• Forget Gate: (Added in a later revision by Gers et al., though conceptually linked to modern LSTMs) Learns to reset the cell state when the stored information is no longer needed.

By orchestrating these gates via learned weights, the network can explicitly decide to store an important feature indefinitely and read it back hundreds of steps later.

Why It Mattered

LSTM brought recurrent neural networks out of the theoretical realm and into practical dominance for sequence tasks. It successfully bypassed the vanishing gradient bottleneck, achieving state-of-the-art results in problems that required memory over thousands of discrete time steps. For over two decades, LSTMs were the default choice for speech recognition, language modeling, machine translation, and time-series forecasting.

What Came After

LSTMs became the foundational building block for seq2seq models and modern neural machine translation systems (such as Google Neural Machine Translation in 2016). They inspired subsequent gated architectures like the Gated Recurrent Unit (GRU). Their dominance in NLP was eventually challenged and largely superseded by the Transformer architecture ("Attention Is All You Need"), which abandoned recurrence entirely in favor of parallelized attention mechanisms.