Skip to content
AI360XpertBeta

Named Entity Recognition

Scan raw text word by word, label each token, and group the labels into people, places, organizations, and dates that a database can store.

Seven tokens take BIO tags B-PER I-PER O B-LOC I-LOC I-LOC B-DATE, so 6 of 7 tokens score and 2 of 3 entities match exactly for F1 0.67
Seven tokens take BIO tags B-PER I-PER O B-LOC I-LOC I-LOC B-DATE, so 6 of 7 tokens score and 2 of 3 entities match exactly for F1 0.67

Why Does This Exist?

A classifier can say an article is about business, but an analyst needs which businesses appear and where. Tracking every mention of two companies across ten thousand articles needs names pulled out and typed, not whole-document labels.

Named entity recognition tags each token and groups the tags into spans such as person, organization, location, and date. Those spans feed search, knowledge graphs, and alerting. This page covers token labels, the BIO boundary scheme, and entity scoring. The word-count baseline is bag of words. Contextual embeddings that power modern taggers live in BERT. Lightweight word vectors live in fastText.

Grounding first: a token means one model input unit after splitting text, often a word. A span means consecutive tokens forming one entity, such as New York City.

Think of It Like This

Three highlighters on a contract

A reviewer reads a rental contract with a yellow marker for tenant names, a blue marker for companies, and a green marker for dates. Each word gets at most one color, multi-word names stay one continuous stroke, and unmarked words stay plain. A second reviewer checks the colors word by word, then checks whole names: getting six of seven words right can still mean missing one of three names.

Tokens are words, colors are BIO tags, and whole names are entity spans. Where it stops: highlighters never confuse Apple the fruit with Apple the company, while a tagger without context does, so the model must read surrounding words rather than matching shapes alone.

How It Actually Works

Token labels and BIO boundaries

NER assigns one label per token. The BIO scheme marks boundaries: B-PER starts a person, I-PER continues it, and O marks outside any entity. For Tim Cook visited New York City today the gold tags are B-PER I-PER O B-LOC I-LOC I-LOC B-DATE. A linear layer over each contextual embedding predicts each tag, and a sequence layer such as a conditional random field forbids illegal transitions like I-PER after O. Context decides ambiguity: visited signals people nearby, while orchard signals fruit.

Worked example: tokens high, entities lower

Say the model tags City as a new location instead of continuing New York City, splitting one place into two. Token count: 66 of 77 tags match, so token accuracy is 6/7≈0.857≈0.866 / 7 \approx 0.857 \approx 0.86. Entity count is stricter: exact span plus exact type. Predicted entities are Tim Cook, New York, and today. Gold entities are Tim Cook, New York City, and today. Two of three match exactly.

precision=2/3≈0.667≈0.67,recall=2/3≈0.67\text{precision} = 2 / 3 \approx 0.667 \approx 0.67, \quad \text{recall} = 2 / 3 \approx 0.67 F1=2×0.67×0.67/(0.67+0.67)≈0.67F1 = 2 \times 0.67 \times 0.67 / (0.67 + 0.67) \approx 0.67

Token accuracy 0.86 hides the split, entity F1 0.67 exposes it. That gap is the diagram footer: one boundary error costs little per token and one third of entities. Teams report entity F1, never token accuracy alone.

Old and new machinery

Statistical taggers used hand features plus conditional random fields for transitions. Transformer taggers pass the sentence through an encoder and classify each token embedding, keeping the transition logic as a thin top layer or dropping it when data is large. Lowercased or noisy text removes the capitalization cue and needs retraining on that style.

Code

The point is the check: rebuild the token accuracy and entity F1 from the worked split.

gold_tokens = ["B-PER", "I-PER", "O", "B-LOC", "I-LOC", "I-LOC", "B-DATE"]pred_tokens = ["B-PER", "I-PER", "O", "B-LOC", "I-LOC", "B-LOC", "B-DATE"]
token_acc: float = sum(g == p for g, p in zip(gold_tokens, pred_tokens)) / len(gold_tokens)
gold_entities = {("Tim Cook", "PER"), ("New York City", "LOC"), ("today", "DATE")}pred_entities = {("Tim Cook", "PER"), ("New York", "LOC"), ("today", "DATE")}correct = len(gold_entities & pred_entities)precision: float = correct / len(pred_entities)recall: float = correct / len(gold_entities)f1: float = 2 * precision * recall / (precision + recall)
print(f"token {token_acc:.2f} precision {precision:.2f} recall {recall:.2f} F1 {f1:.2f}")# -> token 0.86 precision 0.67 recall 0.67 F1 0.67

The printout matches the hand division: 6 of 7 is 0.86, 2 of 3 is 0.67. Change the City prediction back to I-LOC and both scores jump to 1.00 together. That joint movement is why boundary errors are the first thing to inspect.

Watch Out For

Reporting token accuracy instead of entity F1

Token accuracy counts O tags that dominate most sentences, so a model that predicts O everywhere can look strong while extracting nothing. The error that matters splits or merges a span, which costs one token but one whole entity.

Symptom: token accuracy above 0.95 with entity F1 near 0.60, especially on long locations and organizations. Fix: grade exact span plus exact type with precision, recall, and F1, and tune boundaries before touching embeddings.

Trusting capitalization on noisy text

Many taggers lean on capital letters, so all-lowercase posts or broken OCR strip their best cue. Apple and apple become identical inputs and Washington the person merges with Washington the place.

Symptom: deployed F1 falls ten or more points on social or scanned text while clean news stays flat. Fix: train and test on lowercased and noisy copies of the data, and keep context features from BERT rather than shape rules alone.

The Quick Version

  • Named entity recognition labels every token and groups B and I tags into typed spans for people, places, organizations, and dates.
  • BIO marks where each entity starts, continues, and where non-entity O resumes.
  • Our split of New York City keeps 6 of 7 tokens at 0.86 but drops entities to 2 of 3.
  • Precision 0.67 with recall 0.67 gives entity F1 0.67, the number teams report.
  • Context disambiguates Apple the company from apple the fruit, so casing alone is never the rule.
  • Statistical taggers used conditional random fields, while modern taggers classify each transformer token embedding.