Hidden State
An internal vector representation in a neural network that captures and retains information from previous inputs in a sequential data stream over time.
Think of It Like This
Like a runner's momentum in a marathon, carrying the cumulative effect of all the steps taken so far.
In recurrent architectures, the hidden state is updated continuously as new tokens are processed, acting as the model's working memory. It allows the network to understand context by summarizing the sequence history. Modern transformer architectures have largely replaced recurrent hidden states with attention mechanisms for better parallelization.