NLP Concepts
Classic NLP pipeline, POS tagging, NER, sentiment analysis, and text processing
Foundations
Linguistic Analysis
- NLP Concepts
Tokenization in NLP
Tokenization breaks raw text into smaller, manageable units (tokens) — like words, subwords, or characters — so that models can process discrete elements instead of an unbroken character stream.
- NLP Concepts
Text Normalization
Text normalization cleans and standardizes text (lowercasing, removing accents, expanding contractions) so that models do not treat variations of the same word as different tokens.
- NLP Concepts
Part-of-Speech Tagging
Every word plays a grammatical role — noun, verb, adjective. A POS tagger assigns the most probable role to each word given its context, resolving ambiguities like whether "flies" is a verb or a noun.
- NLP Concepts
Named Entity Recognition
NER finds spans of text that refer to real-world entities — people, places, organizations, dates — and labels them, turning unstructured text into structured facts that downstream systems can query.
- NLP Concepts
Dependency Parsing
Dependency parsing analyzes the grammatical structure of a sentence, establishing relationships between head words and words which modify them.
- NLP Concepts
Text Classification
Text classification assigns predefined categories to free-text documents, forming the basis for spam detection, topic labeling, and intent recognition.
- NLP Concepts
Word Embeddings
Word embeddings map discrete words to continuous vectors of real numbers, allowing machine learning models to capture and mathematically manipulate semantic meaning.
- NLP Concepts
Sentence Embeddings
Sentence embeddings map entire sentences or paragraphs into dense vector spaces, capturing high-level semantic meaning for tasks like semantic search and clustering.
- NLP Concepts
Language Models
Language models learn the probability distribution of text, enabling them to predict the next word in a sequence and generate coherent human-like text.
- NLP Concepts
Machine Translation
Machine translation automatically converts text from one language to another while preserving meaning, typically using encoder-decoder architectures.
- NLP Concepts
Text Summarization
Summarization algorithms condense long documents into shorter versions while retaining the core meaning, using either extraction or abstraction.
- NLP Concepts
Question Answering
Question answering systems map a user's natural language query to a precise answer by extracting it from context or generating it from learned knowledge.
- NLP Concepts
Coreference Resolution
Coreference resolution identifies all expressions in a text that refer to the same entity, so a model knows that "he", "the CEO", and "Tim Cook" are the same person.
- NLP Concepts
Semantic Role Labeling
Semantic role labeling identifies the predicate in a sentence and its arguments, answering "who did what to whom, where, and when?".
- NLP Concepts
Relation Extraction
Relation extraction identifies semantic relationships between entities in text, such as determining that a person is employed by an organization.
- NLP Concepts
Information Extraction
Information extraction converts unstructured text into structured data formats, pulling out entities, relationships, and events.
- NLP Concepts
Spell Correction
Spell correction identifies and fixes misspelled words in text, improving the performance of downstream NLP tasks.
- NLP Concepts
Language Detection
Language detection automatically identifies the natural language a given text is written in, often serving as a routing step in multilingual pipelines.