Named Entity Recognition

Why Does This Exist?

Most useful information in text is attached to real-world entities: who did what, when, where, and to whom? Named Entity Recognition (NER) locates and classifies named spans, turning unstructured text into structured (span, entity-type) pairs — the raw material for knowledge graph construction, question answering, and event extraction.

Without NER, a sentence like "Barack Obama visited Berlin on Tuesday" is just a string of characters. With NER, it becomes three structured facts: {person: Barack Obama}, {location: Berlin}, {date: Tuesday}. This transformation is the bridge between human-readable language and machine-actionable data.

Historically, before deep learning, extracting these entities required brittle, hand-crafted rules or extensive gazetteers (massive lists of names and places). These systems broke constantly. If a rule looked for capitalization, it failed on informal text. If it relied on a gazetteer, it failed on any new company or obscure town not in its database. Modern NER systems solve this by learning the context around an entity, allowing them to confidently identify "Snapchat" as an organization or "Zendaya" as a person even if the model has never seen those specific words before.

Think of It Like This

How It Actually Works

The BIO tagging scheme

NER is classically framed as per-token classification using BIO tags. This formulation allows a model to naturally handle multi-word entities without needing a complex bounding-box mechanism. The tags are:

• B-TYPE: Beginning of an entity span (e.g., B-PER)
• I-TYPE: Inside (continuation of) an entity span (e.g., I-PER)
• O: Outside any entity span

For the sentence "Tim Cook visited New York City":

• Tim → B-PER
• Cook → I-PER
• visited → O
• New → B-LOC
• York → I-LOC
• City → I-LOC

This scheme handles multi-token entities cleanly while remaining a standard sequence classification problem. By predicting these tags per token, the model implicitly defines both the boundary of the entity and its classification type.

Standard entity types

Instead of a markdown table, standard entity types are typically grouped into these common categories:

• PER (Person): "Marie Curie" or "Barack Obama"
• ORG (Organization): "Google DeepMind" or "United Nations"
• LOC (Location): "Mount Everest" or "Pacific Ocean"
• GPE (Geo-political entity): "France", "Paris", or "California"
• DATE (Temporal expression): "last Tuesday", "Q3 2024", or "1999"
• MONEY (Monetary value): "$4.2 billion" or "100 euros"
• PRODUCT (Product name): "iPhone 15 Pro" or "Boeing 747"

Model architecture

Modern NER typically uses a three-stage pipeline to convert text into tags:

1. Pre-trained encoder (like BERT, RoBERTa, or DeBERTa): This produces contextual token embeddings that capture meaning from both sides of each token. Unlike static embeddings, a contextual embedding knows the difference between "Apple" the fruit and "Apple" the company.
2. Linear projection: A simple feed-forward layer maps the high-dimensional embedding dimension down to the number of BIO tags, producing a set of tag scores (logits) for each position in the sentence.
3. CRF layer (Conditional Random Field): This layer enforces valid tag sequences. A neural network outputting independent tags might accidentally predict I-ORG immediately following a B-PER. A CRF learns transition probabilities and strictly enforces rules like "B-X must start a span" and "I-Y must follow B-Y or I-Y". It is trained jointly with the encoder via Viterbi decoding during inference.

State-of-the-art models built on these architectures routinely achieve an F1 score above 90 on the CoNLL-2003 benchmark for news text.

Code

This snippet demonstrates how to use the popular spacy library to perform named entity recognition. Notice how the extracted entities form exact contiguous spans of text alongside their predicted labels.

1import spacy2
3# Load the small English model4nlp = spacy.load("en_core_web_sm")5
6text = """7Satya Nadella, CEO of Microsoft, announced a $10 billion investment8in OpenAI at a conference in San Francisco on Tuesday.9"""10
11doc = nlp(text)12
13# -> Satya Nadella                       PERSON     14# -> Microsoft                           ORG        15# -> $10 billion                         MONEY      16# -> OpenAI                              ORG        17# -> San Francisco                       GPE        18# -> Tuesday                             DATE       19for ent in doc.ents:20    print(f"{ent.text:<35} {ent.label_:<10}")

For higher accuracy, transformer-based pipelines are typically used. Here is an example using Hugging Face Transformers.

1from transformers import pipeline2
3# The aggregation_strategy="simple" flag reconstructs whole words from subword tokens4ner = pipeline("ner", model="dslim/bert-base-NER", aggregation_strategy="simple")5
6results = ner("Elon Musk founded SpaceX in Hawthorne, California in 2002.")7# -> Elon Musk            PER      score=0.9988# -> SpaceX               ORG      score=0.9979# -> Hawthorne            LOC      score=0.99210# -> California           LOC      score=0.99811for entity in results:12    print(f"{entity['word']:<20} {entity['entity_group']:<8} score={entity['score']:.3f}")

Watch Out For

The Quick Version

• Named Entity Recognition locates spans of text naming real-world entities and classifies them by type (PER, ORG, LOC, DATE).
• The BIO tagging scheme frames span labeling as a token-level classification problem: B-TYPE for the beginning, I-TYPE for the continuation, and O for outside.
• Modern architectures stack a CRF layer on top of a pre-trained contextual encoder (like BERT) to enforce valid tag sequence transitions.
• Strong models achieve F1 > 90% on standard news benchmarks, but performance degrades heavily under domain shift.
• The structured output of (span, type) pairs provides the crucial raw input that feeds knowledge graphs, QA systems, and event extraction pipelines.