Part-of-Speech Tagging

Why Does This Exist?

Human language is intensely ambiguous at the surface level. Words are not statically mapped to a single meaning or a single grammatical function; instead, their role is dynamically determined by the company they keep. Consider a simple word like "bank." Without the surrounding context, it could be a noun referring to a financial institution, a noun referring to the side of a river, or a verb meaning to rely on something. Similarly, "light" can operate as a noun (the light in the room), a verb (light the fire), or an adjective (a light feather).

If a natural language processing system attempts to understand a sentence by merely looking up each word in a dictionary, it will quickly fail. Without knowing each word's specific grammatical role in the exact sentence it appears, downstream tasks — such as syntactic parsing, named entity recognition (NER), relation extraction, and semantic role labeling — are essentially operating on an ambiguous soup of text.

Part-of-speech (POS) tagging solves this foundational problem. It acts as an early stage in the NLP pipeline that assigns exactly one grammatical label (a tag) to every token in a sequence. By using the surrounding words as context, a POS tagger resolves the surface ambiguity. Once every word is confidently tagged as a noun, verb, adjective, preposition, and so on, the text becomes structurally predictable. This structured foundation enables all deeper semantic and syntactic analysis that follows.

Think of It Like This

How It Actually Works

POS tagging is fundamentally a sequence labeling problem: given a sequence of $n$ words, the task is to output a sequence of $n$ tags.

The tag set

Before a model can assign tags, it needs a standardized vocabulary of tags. The most common standard for English is the Penn Treebank tag set, which contains 36 detailed tags. By splitting broad categories into specific sub-categories, models can capture finer grammatical nuances.

Key tags include:

• NN — singular noun ("dog", "idea")
• NNS — plural noun ("dogs", "ideas")
• NNP — proper noun ("London", "Alice")
• VB — verb base form ("run", "eat")
• VBD — verb past tense ("ran", "ate")
• VBZ — verb, 3rd person singular present ("runs", "eats")
• JJ — adjective ("fast", "bright")
• RB — adverb ("quickly", "well")
• DT — determiner ("the", "a", "this")
• IN — preposition or subordinating conjunction ("in", "because", "under")

Classic approach: Hidden Markov Models

For many years, the dominant algorithmic approach to POS tagging was the Hidden Markov Model (HMM). An HMM tagger is a generative model that mathematically formalizes the intuition of context using two distinct probabilities:

1. Emission probability ($P(w_i | t_i)$): Given a specific POS tag, how likely is it to emit the current word? For instance, $P(\text{"dog"} | \text{NN})$ is relatively high, while $P(\text{"dog"} | \text{VB})$ is much lower, though non-zero (as in "to dog someone's footsteps").
2. Transition probability ($P(t_i | t_{i-1})$): Given the previous tag, what is the probability of transitioning to the current tag? For example, $P(\text{NN} | \text{DT})$ is exceptionally high because a determiner (like "the") is almost always followed by a noun or an adjective, never by a verb.

To tag a new sentence, the model must find the sequence of tags $T = (t_1, t_2, ..., t_n)$ that maximizes the joint probability of the words and the tags. Because testing every possible tag combination is computationally explosive, HMMs use the Viterbi algorithm. Viterbi is a dynamic programming algorithm that finds the optimal tag sequence in $O(n \cdot |T|^2)$ time by discarding sub-optimal paths at each step.

Modern approach: Neural Sequence Labelers

While HMMs are fast and mathematically elegant, they only capture a limited window of context (usually just the immediate preceding tag). Current state-of-the-art POS taggers abandon the Markov assumption entirely, leveraging deep learning to look at the entire sentence simultaneously.

Modern architectures typically use a Bidirectional LSTM or a Transformer encoder. These models convert each token into a dense vector embedding, then process the sequence in both directions (left-to-right and right-to-left). This means the representation of the word "flies" in a specific sentence contains information from all words that come before it and all words that come after it.

Finally, a classification layer assigns a tag probability distribution at each position. In robust systems, this is often a Conditional Random Field (CRF) layer, which enforces global sequence validity (for instance, ensuring that a preposition IN cannot immediately follow another preposition IN without a valid grammatical reason). These neural approaches consistently achieve accuracy above 97% on standard English benchmarks like the Penn Treebank.

Why context is essential

Let us see how context explicitly resolves ambiguity in two classic test sentences:

1. "Time flies like an arrow."
2. "Fruit flies like a banana."

In the first sentence, the surrounding context forces "flies" to be tagged as VBZ (verb). In the second, "Fruit" acting as a noun modifier heavily biases "flies" to be tagged as NNS (plural noun), which subsequently forces "like" to be a verb (VBP) rather than a preposition (IN). Both the transition matrices in an HMM and the attention mechanism in a Transformer exploit exactly this structural dependency.

Code

1import spacy2
3# Load the small English model, which includes a CNN-based tagger4nlp = spacy.load("en_core_web_sm")5
6sentences = [7    "Time flies like an arrow.",8    "Fruit flies like a banana.",9]10
11for sent in sentences:12    doc = nlp(sent)13    print(f"\nSentence: {sent}")14    # token.pos_ gives the coarse-grained tag, token.tag_ gives the fine-grained Penn Treebank tag15    for token in doc:16        print(f"  {token.text:<12} POS={token.pos_:<8} Tag={token.tag_:<6} Dep={token.dep_}")

Output:

1Sentence: Time flies like an arrow.2  Time         POS=NOUN     Tag=NN     Dep=nsubj3  flies        POS=VERB     Tag=VBZ    Dep=ROOT4  like         POS=ADP      Tag=IN     Dep=prep5  an           POS=DET      Tag=DT     Dep=det6  arrow        POS=NOUN     Tag=NN     Dep=pobj7
8Sentence: Fruit flies like a banana.9  Fruit        POS=NOUN     Tag=NN     Dep=compound10  flies        POS=NOUN     Tag=NNS    Dep=ROOT11  like         POS=ADP      Tag=IN     Dep=prep12  a            POS=DET      Tag=DT     Dep=det13  banana       POS=NOUN     Tag=NN     Dep=pobj

Notice how "flies" is VBZ (verb) in the first sentence and NNS (plural noun) in the second. The tagger leverages the context perfectly to disambiguate the token.

Watch Out For

The Quick Version

• POS tagging assigns a grammatical role (like noun, verb, adjective, preposition) to each token in a sentence.
• Context is what resolves ambiguity: the exact same word can receive entirely different tags depending on its neighbors.
• Classic approaches use Hidden Markov Models (HMMs) to calculate emission and transition probabilities, utilizing the Viterbi algorithm to decode the optimal sequence.
• Modern neural taggers use Bidirectional LSTMs or Transformers paired with a CRF layer, regularly exceeding 97% accuracy on standard English datasets.
• POS tags are foundational features that feed directly into downstream syntactic parsing, named entity recognition, and information extraction systems.