ELMo

Contextualized word vectors computed from stacked bidirectional language models. Downstream tasks learn a weighted combination of character, syntactic, and semantic layers.

Why Does This Exist?

Prior to 2018, the standard foundation of natural language systems relied on static word embeddings such as Word2Vec and GloVe. These models assigned a single immutable vector to each string in the vocabulary.

This static assumption broke down on polysemy. In the sentence "The river bank overflowed after heavy rain" and the sentence "The investment bank approved the commercial acquisition", the word "bank" carries entirely different syntactic roles and semantic definitions. Static embeddings collapsed both senses into a single compromise centroid vector, degrading downstream performance on question answering, named entity recognition, and sentiment classification.

Introduced by Peters et al. in 2018, ELMo (Embeddings from Language Models) broke this bottleneck by computing deep contextualized representations. Instead of assigning a fixed vector lookup, ELMo processes the complete sentence through a multi-layer bidirectional Language Model (biLM). The resulting representation for each word is a function of the entire sentence context. Furthermore, ELMo exposed a key mechanistic insight: lower recurrent layers encode syntax and part-of-speech roles, while higher layers capture complex semantic senses.

Think of It Like This

A specialist contractor interviewed across hierarchical management tiers

Imagine an independent contractor entering a large corporation to solve an operational problem. Depending on who you ask about the contractor's role, you receive different descriptions:

At the front desk (Layer 0, Subword/Character CNN), security checks their physical badge and appearance—verifying spelling, morphological prefixes, and base credentials regardless of context.

On the engineering floor (Layer 1, Syntactic biLSTM), the immediate team lead describes their operational grammar: how their role interfaces with adjacent team members, whether they act as an operative or a reviewer, and their procedural placement in daily standups.

In the executive boardroom (Layer 2, Semantic biLSTM), the vice president describes their high-level business objective: what market problem they solve and how their presence changes quarterly strategy.

If the company assigns the contractor to write code (a syntactic downstream task like POS tagging), the team heavily relies on the engineering lead's evaluation ( $s_1$ receives high weight). If they assign the contractor to renegotiate a corporate merger (a semantic task like sentiment or coreference), the executive's assessment dominates ( $s_2$ receives high weight). ELMo lets each downstream task tune its own mixture of these management perspectives.

How It Actually Works

Bidirectional Language Models and Layer-Wise Pooling

ELMo generates representations by pre-training a multi-layer bidirectional Language Model (biLM) on large unlabeled text corpora, then freezing the model and extracting linear layer combinations for downstream architectures.

1. The Bidirectional Language Model (biLM)

Given a sequence of $N$ tokens $(t_1, t_2, \dots, t_N)$ , ELMo first converts each token into a context-independent vector $x_k \in \mathbb{R}^{d_{\text{embed}}}$ using a character-level Convolutional Neural Network (CNN). This character foundation prevents out-of-vocabulary failures by extracting robust subword morphological features.

Next, the token passes through $L$ stacked layers of bidirectional LSTMs:

Forward LSTM: Computes directional hidden states $\vec{h}_{k, j}^{\text{LM}}$ at step $k$ and layer $j \in \{1, \dots, L\}$ , optimizing the probability of predicting token $t_k$ given past context $(t_1, \dots, t_{k-1})$ :
$p(t_1, t_2, \dots, t_N) = \prod_{k=1}^N p(t_k \mid t_1, \dots, t_{k-1})$
Backward LSTM: Computes directional hidden states $\overleftarrow{h}_{k, j}^{\text{LM}}$ at step $k$ and layer $j$ , optimizing the probability of predicting $t_k$ given future context $(t_{k+1}, \dots, t_N)$ :
$p(t_1, t_2, \dots, t_N) = \prod_{k=1}^N p(t_k \mid t_{k+1}, \dots, t_N)$

At each step $k$ , layer $j$ outputs the concatenated representation:

h_{k, j}^{\text{LM}} = \left[ \vec{h}_{k, j}^{\text{LM}} \,;\, \overleftarrow{h}_{k, j}^{\text{LM}} \right] \in \mathbb{R}^{2 d_{\text{lstm}}}

Defining $h_{k, 0}^{\text{LM}} = [x_k \,;\, x_k]$ , the biLM produces a set of $2L + 1$ representations for each token:

R_k = \left\{ h_{k, j}^{\text{LM}} \mid j = 0, \dots, L \right\}

2. Downstream Task-Specific Aggregation

Rather than selecting only the topmost LSTM layer, ELMo allows downstream task models (such as an NER classifier or a relation extractor) to learn a task-specific linear combination across all internal layers:

\text{ELMo}_k^{\text{task}} = \gamma^{\text{task}} \sum_{j=0}^L s_j^{\text{task}} h_{k, j}^{\text{LM}}

where:

$s^{\text{task}} = [s_0^{\text{task}}, \dots, s_L^{\text{task}}]^\top$ are softmax-normalized layer weights:
$s_j^{\text{task}} = \frac{\exp(w_j^{\text{task}})}{\sum_{i=0}^L \exp(w_i^{\text{task}})}$
$\gamma^{\text{task}}$ is a learnable scalar scale parameter allowing the downstream model to scale ELMo's overall vector norm to match its internal representations.

Empirical probes demonstrate that $s_1$ receives highest weight for syntactic tasks like part-of-speech tagging, whereas $s_2$ dominates for semantic tasks like word sense disambiguation.

Worked Example

Consider an ELMo model with $L = 2$ biLSTM layers (total $L+1 = 3$ representations: layer 0, layer 1, and layer 2) with feature dimension $d = 4$ .

Let the extracted vectors for the polysemous word "bank" in the sentence "The river bank overflowed" be:

h_{k, 0} = \begin{bmatrix} 1.00 \\ 0.50 \\ 0.20 \\ -0.40 \end{bmatrix} \quad (\text{Character CNN})

h_{k, 1} = \begin{bmatrix} 0.80 \\ 1.20 \\ -0.60 \\ 0.50 \end{bmatrix} \quad (\text{Layer 1: Syntactic Noun})

h_{k, 2} = \begin{bmatrix} 1.50 \\ 2.00 \\ -1.00 \\ 1.20 \end{bmatrix} \quad (\text{Layer 2: Semantic River/Geology})

Let a downstream word sense disambiguation classifier learn unnormalized softmax logits $w^{\text{task}} = [0.20, 0.80, 1.80]^\top$ .

Step 1: Compute Softmax Layer Weights

Compute exponentials:

e^{0.20} \approx 1.2214, \quad e^{0.80} \approx 2.2255, \quad e^{1.80} \approx 6.0496

Sum of exponentials:

\sum_{j=0}^2 e^{w_j} = 1.2214 + 2.2255 + 6.0496 = 9.4965

Normalized weights:

s_0 = \frac{1.2214}{9.4965} \approx 0.1286, \quad s_1 = \frac{2.2255}{9.4965} \approx 0.2343, \quad s_2 = \frac{6.0496}{9.4965} \approx 0.6371

Notice that the semantic layer ( $s_2$ ) receives $63.71\%$ of the total attention.

Step 2: Compute Weighted Representation Sum

Multiply each vector by its scalar weight:

s_0 h_{k, 0} = [0.1286, \, 0.0643, \, 0.0257, \, -0.0514]^\top

s_1 h_{k, 1} = [0.1874, \, 0.2812, \, -0.1406, \, 0.1172]^\top

s_2 h_{k, 2} = [0.9557, \, 1.2742, \, -0.6371, \, 0.7645]^\top

Sum across dimensions:

h_{\text{combo}} = \begin{bmatrix} 0.1286 + 0.1874 + 0.9557 \\ 0.0643 + 0.2812 + 1.2742 \\ 0.0257 - 0.1406 - 0.6371 \\ -0.0514 + 0.1172 + 0.7645 \end{bmatrix} = \begin{bmatrix} 1.2717 \\ 1.6197 \\ -0.7520 \\ 0.8303 \end{bmatrix}

Step 3: Apply Downstream Scale $\gamma = 1.50$

\text{ELMo}_k^{\text{task}} = 1.50 \times \begin{bmatrix} 1.2717 \\ 1.6197 \\ -0.7520 \\ 0.8303 \end{bmatrix} = \begin{bmatrix} 1.9076 \\ 2.4296 \\ -1.1280 \\ 1.2455 \end{bmatrix}

Code

1import numpy as np2
3def compute_elmo_representation(4    layer_vectors: list[np.ndarray],5    task_logits: np.ndarray,6    gamma: float = 1.07) -> tuple[np.ndarray, np.ndarray]:8    """Compute task-specific ELMo contextualized representation."""9    # Numerically stable softmax layer weights10    shift = task_logits - np.max(task_logits)11    exp_weights = np.exp(shift)12    s_weights = exp_weights / np.sum(exp_weights)13
14    # Weighted linear combination: sum_j s_j * h_j15    stacked = np.stack(layer_vectors, axis=0)  # Shape: (L+1, d)16    weighted_sum = np.sum(s_weights[:, np.newaxis] * stacked, axis=0)17
18    # Apply task scalar scale parameter gamma19    elmo_out = gamma * weighted_sum20    return np.round(elmo_out, 4), np.round(s_weights, 4)21
22# Three layer vectors for token 'bank' (Char CNN, Layer 1 biLSTM, Layer 2 biLSTM)23h_0 = np.array([1.00, 0.50, 0.20, -0.40], dtype=np.float64)24h_1 = np.array([0.80, 1.20, -0.60, 0.50], dtype=np.float64)25h_2 = np.array([1.50, 2.00, -1.00, 1.20], dtype=np.float64)26
27# Learned downstream logits favoring semantic layer28logits = np.array([0.20, 0.80, 1.80], dtype=np.float64)29scale_gamma = 1.5030
31elmo_vector, weights = compute_elmo_representation(32    [h_0, h_1, h_2],33    logits,34    gamma=scale_gamma35)36
37print(f"Layer Softmax Weights: {weights.tolist()}")38# -> Layer Softmax Weights: [0.1286, 0.2343, 0.6371]39
40print(f"Contextualized Vector: {elmo_vector.tolist()}")41# -> Contextualized Vector: [1.9076, 2.4296, -1.128, 1.2455]

Watch Out For

ELMo concatenates independent forward and backward passes rather than true joint cross-conditioning

A frequent misconception is assuming ELMo's bidirectional LSTMs attend jointly to left and right context simultaneously.

In reality, ELMo trains two strictly independent unidirectional models: the forward LSTM sees only preceding tokens, while the backward LSTM sees only succeeding tokens. Their hidden vectors are concatenated post-hoc: $[\vec{h} \,;\, \overleftarrow{h}]$ . True bidirectional interaction—where each token's representation attends to all other tokens jointly across intermediate layers—was only realized later with masked language modeling in BERT and self-attention transformers.

The Quick Version

ELMo replaces fixed word embeddings with dynamic, sentence-dependent vectors extracted from a deep bidirectional Language Model.
A character CNN provides out-of-vocabulary resilience, while stacked biLSTMs capture directional linguistic context.
Downstream tasks learn a softmax-weighted sum across all model layers, prioritizing syntax or semantics depending on the application.