Part-of-Speech Tagging

Why Does This Exist?

Words refuse to sit still. Bank can be a financial house, a river edge or a verb meaning to rely. Light can be a noun, a verb or an adjective. Flies can be a verb or a plural noun. A dictionary lookup alone leaves every sentence as a pile of maybe-meanings.

Downstream steps need one decided structure. Parsing, name finding and relation extraction all read grammar roles, not raw words. Without decided tags they guess on soup.

This page covers the labelling step itself: the tag set, how context picks the tag, and how the best whole-sentence path is found. For the chain this sits in, see the NLP pipeline. For the name-span step that reads these tags, see named entity recognition. For the arrow links that need tags first, see dependency parsing.

The one idea you need first

Text must be split into tokens before any token can carry a label. A token is usually a word or subword. The full treatment is in tokenization in NLP.

Think of It Like This

How It Actually Works

Tagging is sequence labelling. Given $n$ words $w_1, \ldots, w_n$, output $n$ tags $t_1, \ldots, t_n$, one per word. Tag $t_i$ names the grammatical role of word $w_i$ in this sentence, not in a dictionary. The same spelling can take different tags in different sentences, and the job is to pick the sequence that fits the whole sentence best.

The tag set

English most often uses the Penn Treebank set with 36 tags. $NN$ is a singular noun such as dog, $NNS$ a plural noun such as dogs, $NNP$ a proper noun such as London, $VB$ a base verb such as run, $VBD$ a past verb such as ran, $VBZ$ a present verb with he or she such as runs, $JJ$ an adjective such as fast, $RB$ an adverb such as quickly, $DT$ a determiner such as the, $IN$ a preposition such as in. Finer splits help the model catch small grammar habits.

How a hidden Markov tagger scores a sentence

A classic tagger uses two tables. Emission $P(w_i \mid t_i)$ is the chance of seeing this word given this tag, so $P(dog \mid NN)$ is high and $P(dog \mid VB)$ is tiny but not zero. Transition $P(t_i \mid t_{i-1})$ is the chance of this tag after the last tag, so $P(NN \mid DT)$ is high because the almost always precedes a noun or adjective, never a verb. The score of a full tag path $T = (t_1, \ldots, t_n)$ is the product of those terms, and decoding picks the path with the top product.

Checking every path is hopeless. With $|T| = 36$ tags and $n = 5$ words there are $36^5 = 60,466,176$ paths. The Viterbi algorithm keeps only the best partial path into each tag at each step, which costs $O(n \times |T|^2)$, here $5 \times 36 \times 36 = 6,480$ updates. Same best path, about nine thousand times less work.

Modern taggers swap the tables for a bidirectional encoder plus a transition layer, so each word vector sees left and right context at once. A conditional random field on top keeps the whole order valid. The search idea stays the same: score whole paths, keep the best.

Worked example

Take a toy tagger with two tags for flies. Emission $P(flies \mid VBZ) = 0.06$ and $P(flies \mid NNS) = 0.10$. Transition from a subject noun $NN$: $P(VBZ \mid NN) = 0.30$ and $P(NNS \mid NN) = 0.05$.

For "Time(NN) flies":

Score $VBZ = 0.30 \times 0.06 = 0.0180$.

Score $NNS = 0.05 \times 0.10 = 0.0050$.

$VBZ$ wins by $0.0180$ to $0.0050$, about 3.6 times larger. That's why the top panel of the diagram tags flies as $VBZ$. In "Fruit flies" the neighbour pattern flips the transition numbers toward $NNS$, the same multiplication picks $NNS$, and like must then be a verb to keep the path valid. Context decides, one multiplication at a time.

Code

This snippet repeats the toy decision and the Viterbi count, so you can check the arithmetic.

1def score(trans: float, emit: float) -> float:2    return trans * emit3
4
5vbz = score(0.30, 0.06)6nns = score(0.05, 0.10)7print(round(vbz, 4), round(nns, 4))  # -> 0.018 0.0058print("VBZ" if vbz > nns else "NNS")  # -> VBZ9n, t = 5, 3610print(n * t * t)  # -> 648011print(t**n)  # -> 60466176

Watch Out For

The Quick Version

• Part-of-speech tagging writes one grammar label on each token, so flies can be VBZ in one sentence and NNS in the next.
• The Penn set splits roles finely, with 36 tags such as NN, NNS, VBZ, JJ, RB, DT and IN.
• Classic scoring multiplies emission by transition, and Viterbi finds the best path in 6480 steps instead of 60,466,176.
• The toy check gives 0.018 for VBZ against 0.005 for NNS, so VBZ wins after a subject noun.
• News models slip on chat and specialist text, so test in domain before trusting tags.
• One wrong tag can mislead parsing, so keep probabilities where the next step is fragile.

What to Read Next

• The NLP Pipeline
• Named Entity Recognition