NLP Concepts
Classic NLP pipeline, POS tagging, NER, sentiment analysis, and text processing
Foundations
Linguistic Analysis
- NLP Concepts
Tokenization in NLP
Tokenization cuts raw text into pieces models can index. Frequent words stay whole while rare ones split, so one word can become three tokens.
- NLP Concepts
Text Normalization
Clean and standardize text before the model sees it. Lowercasing, lemmatizing and stripping noise collapse variants of one word into one form.
- NLP Concepts
Part-of-Speech Tagging
Every word plays a grammatical role such as noun, verb or adjective. A tagger picks the most likely role for each word from its neighbours, so flies can be a verb in one sentence and a noun in the next.
- NLP Concepts
Named Entity Recognition
Named entity recognition finds name spans in text, people, places, organizations, dates, and labels each span so later steps can use it as a fact.
- NLP Concepts
Dependency Parsing
Dependency parsing draws one arrow from each word to the word it modifies, so a sentence becomes a tree headed by its main verb.
- NLP Concepts
Text Classification
Take a span of text and pick labels from a fixed list. Spam, sentiment, topic and ticket routing are one pipeline with different bin names.
- NLP Concepts
Word Embeddings
Word embeddings give each word a list of numbers that places meaning on a map. Close words sit near each other, so similarity becomes distance you can measure.
- NLP Concepts
Sentence Embeddings
Sentence embeddings turn a whole sentence into one vector, so paraphrases sit close together and strangers sit far apart, ready for fast search.
- NLP Concepts
Language Models
A language model learns which words tend to follow which, so it can score every possible next word and generate text one probable word at a time.
- NLP Concepts
Machine Translation
Machine translation turns a sentence in one language into the same meaning in another, picking each output word from the source words it attends to most.
- NLP Concepts
Text Summarization
Shrink a long document to its core claims. Either copy the best sentences word for word or write new ones that compress the same meaning.
- NLP Concepts
Question Answering
Question answering turns a plain question into one exact answer. The system either points at the span in a given passage or writes the answer from what it learned.
- NLP Concepts
Coreference Resolution
Coreference resolution groups every mention of the same entity into one chain, so Mira, She and her count as one person instead of three strangers.
- NLP Concepts
Semantic Role Labeling
Semantic role labeling finds the verb and labels who did what to whom, so the active and passive versions of one event get the same tags.
- NLP Concepts
Relation Extraction
Once you know the names in a sentence, relation extraction draws the labeled arrow between them, turning words into a triple a database can store.
- NLP Concepts
Information Extraction
Information extraction turns prose into rows, finding the names, linking the pairs, and filling a record a database can answer questions from.
- NLP Concepts
Spell Correction
Spell correction turns a typo into the word you meant, so a search for aple still finds apple by weighing nearby words and how common each one is.
- NLP Concepts
Language Detection
Language detection reads the statistical footprint of a text and names its language, so Spanish input reaches the Spanish pipeline instead of English.