Concise, plain-English definitions of the AI and ML terms used across the site. Filter by tag to narrow the glossary.
439 terms
A systematic evaluation process where individual components of a machine learning model are removed to measure their exact contribution to performance.
Analogy: Like removing ingredients from a recipe one by one to figure out exactly which spice is making the soup taste good.
Related: Feature Importance, Model Pruning, Association Rule Mining
A mathematical operation applied to a neural network node's output that introduces the crucial non-linearity required to learn highly complex patterns.
Analogy: Like a gatekeeper deciding how loudly a message should be shouted to the next room based on the importance of the incoming signal.
An interpretability technique that proves a network component handles a specific concept by pasting its clean activations into a corrupted forward pass and w...
Analogy: Like bypassing a broken house fuse with a battery to prove that specific fuse controls the kitchen lights.
Related: Sparse Autoencoder, Ablation Study, Agent Harness
Read the full conceptA specialized training paradigm where the model explicitly selects the most informative unlabelled data points and queries a human annotator for labels.
Analogy: Like a student who only asks the teacher for help on the hardest questions they don't understand, rather than asking about everything in the textbook.
Related: Human-in-the-Loop, Semi-Supervised Learning, CTC Loss
An incredibly popular adaptive optimization algorithm that combines momentum and scaling to dynamically adjust the learning rate of individual parameters.
Analogy: Like a driver who remembers their past speed (momentum) while adapting to the current road conditions (scaling) for each individual tire.
Related: Learning Rate Scheduling, Second-Order Optimization, Batch Normalization
A tiny set of highly trainable parameters injected into the layers of a frozen, pre-trained neural network to drastically simplify task-specific tuning.
Analogy: Like adding a custom optical filter to the front of an expensive camera lens instead of rebuilding the entire lens from scratch.
Related: PEFT, Foundation Model, Bias (parameter)
Any optimizer that gives each parameter its own effective learning rate from that parameter's own gradient history, rather than one shared rate for all of them.
Related: Second-Order Optimization, Gradient, Convex Function
Read the full conceptA square grid recording which pairs of nodes in a graph are connected, with one row and one column per node.
Related: Matrix, Sparse Matrix, Matrix Rank
Read the full conceptA specialized testing framework that securely wraps an AI agent, intercepts its tool calls, and rigorously evaluates its behavior in a sandbox environment.
Analogy: Like an aviation flight simulator that tests how a pilot reacts to fake emergencies without risking a real multi-million dollar aircraft.
Related: Agent, Tool Calling, Agent-to-Agent Protocol
A standardized communication interface that enables multiple autonomous AI agents to negotiate, share intermediate context, and collaboratively solve tasks.
Analogy: Like a structured handoff form that nurses use during shift changes so nothing gets lost in translation between team members.
Related: Agent, Byte-Pair Encoding, Causal Language Model
An autonomous AI system equipped with diverse tools and reasoning capabilities that continuously perceives its environment and takes actions to achieve a goal.
Analogy: Like a digital employee who doesn't just answer questions, but actively opens spreadsheets and browses the web to finish the job you gave them.
Related: Tool Calling, Chain-of-Thought, Citation Grounding
The inherent, unavoidable and irreducible randomness in a dataset that cannot be eliminated simply by collecting more data or building a more complex model.
Analogy: Like flipping a fair coin; no matter how closely you study the coin's physics or how many times you flip it, you still can't perfectly predict the next outcome.
Related: Expected Value, Random Variable, Conformal Prediction
A positional scheme that adds no vector to the input at all, instead subtracting a distance-proportional penalty from attention scores before softmax runs.
Related: Rotary Position Embeddings, Long-Context Extension, Cross-Attention
Read the full conceptThe complex process of ensuring that an artificial intelligence model's goals, behaviors, and outputs strictly match human values and intended instructions.
Analogy: Like training a powerful guard dog not just to stop intruders, but to also ensure it never accidentally bites the mail carrier.
Related: Proximal Policy Optimization, Constitutional AI, CUDA
Finding rare rows that do not belong to normal patterns, often a fraud or failure class, scored by precision and recall rather than accuracy alone.
Related: Outlier, Class Imbalance, Isolation Forest
Read the full conceptA highly efficient search algorithm that dramatically speeds up vector retrieval by finding close matches rather than computing the exact nearest neighbor.
Analogy: Like finding a good enough parking spot on the correct block instead of driving around for an hour to find the single absolute closest spot to the door.
Related: Embedding, Cosine Similarity, Data Poisoning
Rules describing which items co-occur more than chance predicts, scored by support, confidence, and lift, the last of which rules out spurious associations.
Related: Standardization, Class Imbalance, Anomaly Detection
Read the full conceptSpecific initial tokens in a sequence that inadvertently absorb massive amounts of attention weight simply because they serve as a dumping ground for mass.
Analogy: Like a designated junk drawer in a kitchen where you blindly toss random miscellaneous items just because they have to go somewhere.
Related: Long-Context Extension, Diffusion Transformer, Feed-Forward Network
A single scalar metric representing the complete two-dimensional area underneath the Receiver Operating Characteristic curve of a classification model.
Analogy: Like grading a metal detector not just on how many coins it finds, but how well it performs across every possible sensitivity dial setting you could use.
Related: Backtesting, Benchmark Contamination, Error Analysis
A type of sequence generation model that predicts the very next token in a series by conditioning its output on all of the previous tokens it generated.
Analogy: Like a person writing a book by only looking at the words they have written so far to decide what the very next single word should be.
Related: Next-Token Prediction, LLM, Context Compaction
The foundational algorithm used to train neural networks by calculating the gradient of the loss function with respect to every single weight in the network.
Analogy: Like tracing a flawed product on an assembly line backward through every single factory station to find exactly which worker made the mistake.
Related: Gradient Descent, Loss Function, Curriculum (data)
A rigorous simulation framework that evaluates the historical performance of a predictive model or trading strategy using strictly past observational data.
Analogy: Like reading yesterday's newspaper to see if the stock picks you would have made actually ended up making any money in reality.
Related: Data Leakage, Time Series Cross-Validation, Context Engineering
Representing a document as counts over a fixed vocabulary, discarding word order entirely.
Related: TF-IDF, Tokenization, Sparse Matrix
Read the full conceptAn ensemble learning technique that trains multiple independent models on random subsets of the training data and averages their final output predictions.
Analogy: Like asking a diverse panel of doctors for their individual diagnoses and taking a majority vote to drastically reduce the chance of a misdiagnosis.
Related: Random Forest, Bootstrap, Double Descent
A technique that standardizes the activations of a neural network layer across a training batch to drastically stabilize and accelerate the learning process.
Analogy: Like a microphone operator constantly adjusting the volume knob during a live speech so the speaker is never too quiet to hear or too loud to understand.
Related: Layer Normalization, Group Normalization, Dropout
How the number of examples averaged into one gradient estimate changes training noise, the largest stable learning rate, and final generalization.
Related: Gradient Accumulation, Learning Rate Scheduling, Gradient
Read the full conceptThe rule for updating a belief after seeing evidence, by combining how well the evidence fits with how plausible the belief was beforehand.
Related: Likelihood, Random Variable, Double Machine Learning
Read the full conceptTreating regression coefficients as probability distributions updated from a prior into a posterior as data arrives, so predictions come with honest, data-dr...
Related: Bayes' Theorem, Gaussian Process Regression, Ensemble
Read the full conceptChoosing where to evaluate an expensive function next by fitting a surrogate model to past evaluations and ranking candidates with an acquisition function.
Related: Gaussian Process Regression, Hyperparameter Tuning, Epoch
Read the full conceptA catastrophic evaluation failure where the exact test questions used to evaluate a language model were accidentally included in its massive training corpus.
Analogy: Like a student stealing the final exam answer key a week before the test; their perfect score proves they can memorize, not that they actually learned the material.
Related: Data Leakage, Emergent Ability, Fréchet Distance
A bidirectional encoder-only transformer pretrained by masked language modeling: hide some tokens and predict them from context on both sides at once.
Related: Encoder-Decoder Architectures, GPT, FlashAttention
Read the full conceptAn essential learnable parameter added to the weighted sum of inputs in a neural network node, allowing the activation function to simply shift left or right.
Analogy: Like adjusting the baseline temperature of an oven before you decide how much you want to turn the heat up or down based on the recipe.
Related: Linear Combination, Federated Learning, Flow Matching
The systematic error introduced by incorrect assumptions in a learning algorithm, causing the model to consistently miss the true underlying relationships.
Analogy: Like a poorly calibrated scale that always reads exactly five pounds too heavy, no matter who stands on it.
Related: Variance, Overfitting, Focal Loss
A highly effective bag-of-words retrieval function that ranks a set of documents based on the exact term frequency and inverse document frequency of a query.
Analogy: Like a librarian who scores a book higher if it mentions your search keyword often, but lowers the score if that keyword is something generic like 'the'.
Related: TF-IDF, Bag of Words, GAN
A powerful sequential ensemble technique that trains a series of weak models where each new model specifically focuses on correcting the errors of the last.
Analogy: Like a relay race where every new runner specifically practices the exact hurdles that the previous runner tripped over.
Related: Bagging, Full Fine-Tuning, GRPO
A fundamental resampling method that involves repeatedly drawing random samples with replacement from a single dataset to estimate a statistical population.
Analogy: Like drawing a card from a deck, writing down the suit, and putting it back before drawing again to figure out the overall composition of the deck.
Related: Bagging, Random Variable, Loss Masking
A widely used subword tokenization algorithm that iteratively merges the most frequently occurring pairs of characters or bytes into a single unique token.
Analogy: Like replacing the frequent letter pair 't' and 'h' with a single new symbol 'th' to shorten the overall length of a written message.
Related: Tokenization, LLM, Label Encoding
The crucial alignment process ensuring that a model's predicted confidence score perfectly matches its actual empirical probability of being exactly correct.
Analogy: Like an honest weather forecaster whose '80% chance of rain' prediction mathematically translates to it actually raining on 8 out of 10 identical days.
Related: Expected Value, Log-Likelihood, Grad-CAM
The number of distinct values a column contains — two for a yes/no flag, fifty thousand for a merchant ID.
Related: One-Hot Encoding, Hashing Trick, Feature
Read the full conceptA directed acyclic graph that mathematically visualizes the true cause-and-effect relationships between random variables in a complex data generating process.
Analogy: Like drawing a flowchart that strictly points out that rain causes both umbrellas and wet grass, preventing you from thinking umbrellas cause wet grass.
Related: Random Variable, QLoRA, RLVR
A neural architecture that predicts the next token in a sequence by strictly masking out all future tokens to prevent any forward-looking data leakage.
Analogy: Like trying to guess the next word in a sentence while holding a piece of paper that completely covers up the rest of the page.
Related: Autoregressive Model, GPT, GELU
A common statistical condition in survival analysis where the true time-to-event for a specific observation is completely unknown or only partially known.
Analogy: Like tracking how long lightbulbs last, but the study ends while five of them are still shining, so you only know they lasted 'at least' that long.
Related: Survival Analysis, Sequence Packing, SGD
A powerful prompting strategy that forces a large language model to explicitly generate intermediate reasoning steps before outputting its final conclusion.
Analogy: Like a math teacher forcing you to write out every step of your long division on the chalkboard instead of just shouting out the final number.
Related: Prompt Engineering, Reasoning Model, Prompt Injection
The critical preprocessing step of splitting massive documents into smaller, semantically coherent text segments for ingestion into a vector search engine.
Analogy: Like slicing a massive wedding cake into individual portions so people can actually eat it, making sure not to cut halfway through a decorative flower.
Related: Vector Database, Embedding, MinHash
A rigorous generation technique where an AI model explicitly links every factual claim it makes back to a specific sentence in a retrieved source document.
Analogy: Like a lawyer who isn't allowed to just make a claim to the jury, but must physically hold up the exact piece of evidence that proves their statement.
Related: Hallucination, Integrated Gradients, K-Fold
A classification problem where one class is far rarer than another, so accuracy stops describing anything useful.
Related: SMOTE, Stratified Sampling, Ground Truth
Read the full conceptA powerful sampling technique used in diffusion models to dramatically increase the alignment between the generated image and the specific text condition.
Analogy: Like an artist painting a portrait who constantly steps back to compare their current progress against the client's strict written description.
Related: Diffusion Model, ControlNet, ReAct
An unsupervised machine learning technique that groups completely unlabelled data points into distinct clusters based entirely on their underlying similarity.
Analogy: Like dumping a massive pile of unmarked puzzle pieces onto a table and sorting them purely by color and pattern without knowing what the final picture is.
Related: K-Means, DBSCAN, Sim-to-Real
A change in the relationship between inputs and the correct answer, so the same input now means something different.
Related: Data Drift, Covariate Shift, Ground Truth
Read the full conceptA mathematically rigorous framework that wraps any existing machine learning model to output statistically guaranteed prediction intervals rather than points.
Analogy: Like a weather forecaster saying 'It will be between 65 and 75 degrees' rather than just guessing exactly 70 degrees, guaranteeing they are rarely wrong.
Related: Calibration, Steering Vector, Supervised Learning
A safety training paradigm where an AI model is recursively fine-tuned to critique and revise its own outputs based on a strict set of written principles.
Analogy: Like an author repeatedly rewriting their own novel draft by checking each new chapter against a strict style guide provided by their demanding publisher.
Related: Alignment, RLHF, Triplet Loss
A specialized generation technique that explicitly restricts a language model's output probabilities to perfectly match a strictly defined structured schema.
Analogy: Like a strict multiple-choice test where you aren't physically allowed to write down a word, you can only fill in one of the specific letter bubbles.
Related: JSON Schema, Top-k / Top-p, Zero-Shot Learning
An optimization strategy that systematically condenses a massive input prompt into a much shorter, dense representation to save both memory and compute time.
Analogy: Like reading an entire textbook and writing down only the most important formulas on a single cheat sheet before walking into a final exam.
Related: Context Window, Unlearning, Unsupervised Learning
The deliberate architectural process of selecting, formatting, and injecting the absolute best external data into a language model's input context window.
Analogy: Like organizing the perfect dossier of strictly relevant case files for a detective before asking them to solve a complex murder mystery.
Related: Context Window, Prompt Engineering
The strictly enforced theoretical limit on the total exact number of consecutive tokens a model can simultaneously process and remember during inference.
Analogy: Like a person's short term working memory; you can only hold a certain amount of facts in your head at once before you start forgetting the first things you read.
Related: Long-Context Extension, Attention Sink, GGUF
A highly specialized scheduling algorithm that dynamically injects entirely new requests into a running inference batch exactly as older requests finish up.
Analogy: Like a busy restaurant seating a new party at a freshly cleared table immediately, rather than waiting for the entire dining room to finish eating first.
Related: vLLM, Inference, Weak Supervision
A powerful self-supervised learning paradigm where a model is trained to actively pull similar data points closer together and push dissimilar ones apart.
Analogy: Like organizing a messy photo album by simply tossing all pictures of dogs into one pile and aggressively throwing pictures of cats to the other side.
Related: Embedding, Cosine Similarity, Xavier Initialization
An architectural extension for diffusion models that dramatically improves spatial consistency by conditioning image generation on strict structural hints.
Analogy: Like giving an imaginative painter a strict architectural blueprint to trace over, ensuring the final fantasy castle actually has walls that make sense.
Related: Diffusion Model, Classifier-Free Guidance
A function shaped like a bowl, where any straight line between two points on it stays above the surface — so its only stationary point is the global minimum.
Analogy: A single valley with no side pockets: walk downhill and you cannot end up anywhere but the bottom.
Related: Hessian Matrix, Stationary Point, Hierarchical Forecasting
Read the full conceptCovariance rescaled by both standard deviations, giving a unit-free strength of linear relationship between minus one and one.
Related: Covariance, Standard Deviation, Integer Programming
Read the full conceptThe cosine of the angle between two vectors, measuring agreement in direction while ignoring length entirely.
Related: Unit Vector, Euclidean Distance, Orthogonal Vectors
Read the full conceptA measure of whether two variables tend to be above or below their means at the same time.
Related: Variance, Correlation Coefficient, Expected Value
Read the full conceptThe mix of inputs changing while the input-to-answer relationship stays the same.
Related: Data Drift, Concept Drift, Train-Test Split
Read the full conceptA highly specialized attention mechanism where the queries come directly from the decoder, while the keys and values are explicitly pulled from the encoder.
Analogy: Like a chef (decoder) constantly looking back at a specific recipe card (encoder) to figure out exactly how much salt they need to add to the soup next.
Related: Encoder-Decoder Architectures, K-Nearest Neighbors, Latent Diffusion
A specialized loss function used heavily in speech recognition to perfectly align an input audio sequence with an output text sequence of a different length.
Analogy: Like stretching and squishing a rubber band with writing on it so the words perfectly line up with a completely different set of markings on the table.
Related: Encoder-Decoder ASR, Online/Offline Skew, Point-in-Time Correctness
A parallel computing platform and programming interface created by NVIDIA that allows software developers to directly access the immense power of modern GPUs.
Analogy: Like finally getting the master key to a factory full of thousands of workers, allowing you to give them all instructions simultaneously instead of one by one.
Related: Mixed Precision Training
A training strategy that artificially orders the presentation of data, starting with the simplest examples and gradually increasing the difficulty over time.
Analogy: Like teaching a child basic addition and subtraction before suddenly dropping a calculus textbook on their desk.
Related: Few-Shot Learning, Uplift Modeling, Vector Quantization
The way data becomes uninformative as columns are added, because volume grows exponentially while your row count does not.
Related: Euclidean Distance, Cosine Similarity, Principal Component Analysis
Read the full conceptA technique used to artificially expand the size of a training dataset by applying random transformations like rotations or flips to existing data examples.
Analogy: Like a basketball player practicing trick shots from different angles to prepare for any unexpected situation during a real game.
Related: Synthetic Data, Overfitting, Yeo-Johnson Transform
A structured document that provides detailed metadata about a dataset, covering its origins, intended use cases, known limitations, and ethical constraints.
Analogy: Like a nutrition label on food packaging that tells you exactly what ingredients were used and potential allergens you should watch out for.
Related: Data Leakage, Off-Policy Evaluation, Out-of-Distribution Detection
A version-controlled statement of the shape a data producer promises to deliver, checked automatically on every batch.
Related: Data Drift, Ground Truth, Feature Matrix
Read the full conceptA change in the inputs a model receives after it was trained, measured by comparing production data against the training window.
Related: Concept Drift, Covariate Shift, Data Contract
Read the full conceptThe model having access at training time to information it will not have when it makes a real prediction.
Related: Train-Test Split, K-Fold Cross-Validation, Ground Truth
Read the full conceptA malicious attack where adversaries intentionally inject misleading or corrupted samples into a training dataset to degrade the resulting model's performance.
Analogy: Like sneaking fake facts into a student's textbook before an exam so they end up learning the wrong material and failing the test.
Related: Data Augmentation, Z-Score Normalization
A density-based clustering algorithm that groups together closely packed points while marking points that lie alone in low-density regions as noise outliers.
Analogy: Like finding cities on a map by looking for areas with many buildings close together, while ignoring isolated cabins in the middle of nowhere.
Related: Clustering, Outlier, Kernel
An efficient sampling algorithm for diffusion models that accelerates the image generation process by skipping steps in the reverse diffusion trajectory path.
Analogy: Like taking an express train that skips several local stops to reach the final destination much faster without significantly changing the route.
Related: DDPM, Diffusion Model, Lagrange Multiplier
A foundational class of generative models that learns to create data by slowly reversing a process that gradually adds Gaussian noise to the original inputs.
Analogy: Like slowly unscrambling a television static screen pixel by pixel until a perfectly clear image eventually emerges from the random noise.
Related: DDIM, Diffusion Model, Model Context Protocol
An ordered protocol for isolating why a training run is failing, moving from the simplest possible check to the more expensive ones rather than guessing.
Related: Early Stopping, Gradient Clipping, Gradient
Read the full conceptThe process of identifying and removing exact or near-exact duplicate examples from a dataset to prevent a model from memorizing repeated training sequences.
Analogy: Like going through your photo gallery and deleting all the identical burst-mode shots so you only keep one copy of each unique moment.
Related: Data Leakage, Memorization
The rate at which a function's output changes per unit change in its input, measured at one point.
Analogy: A speedometer, not an odometer — it reads the rate right now, not the total so far.
Related: Partial Derivative, Gradient, Stationary Point
Read the full conceptA guarantee that a result looks statistically almost identical whether or not any single individual's data was included.
Related: K-Anonymity, Synthetic Data, Normal Distribution
Read the full conceptA type of generative artificial intelligence that creates novel outputs by learning to iteratively remove random noise from a completely unstructured signal.
Analogy: Like a sculptor starting with a random block of marble and slowly chiseling away the rough edges until a perfectly detailed statue is revealed.
Related: DDPM, Generative Model, Model Merging
An advanced generative architecture that replaces the traditional convolutional U-Net backbone in a diffusion model with a highly scalable transformer network.
Analogy: Like upgrading a factory assembly line from specialized custom robotic arms to a unified, programmable grid of robots that can scale infinitely.
Related: Diffusion Model, Vision Transformers, Multi-Head Latent Attention
A data transformation technique that reduces the number of input variables while preserving the most important structural information and underlying patterns.
Analogy: Like summarizing a dense 500-page novel into a 10-page plot outline that captures all the main characters and key events without the extra fluff.
Related: Principal Component Analysis, Feature Selection, Naive Bayes
Converting a continuous column into a small set of ordered bands — turning an age of 34 into the bucket 30 to 39.
Related: Ordinal Encoding, One-Hot Encoding, Feature
Read the full conceptA machine learning algorithm that focuses on finding the boundary that separates different classes of data rather than understanding how the data was created.
Analogy: Like a bouncer at a club who only checks if you are on the VIP list, rather than caring about where you were born or what your life story is.
Related: Generative Model, Logistic Regression, Posterior
The automated process of extracting structured text, metadata, and visual elements from unstructured files like PDFs to prepare them for machine processing.
Analogy: Like a librarian reading through a messy pile of handwritten letters and neatly typing out the author, date, and main points into a spreadsheet.
Closing the gap when a model trained on one data distribution gets deployed on a shifted one, without collecting new labels for the new distribution.
Related: Multi-Task Learning, Meta-Learning
Read the full conceptA phenomenon where a model's test error initially decreases, then spikes as it interpolates the data, and finally decreases again as model capacity expands.
Analogy: Like learning to ride a bike: you improve, then suddenly crash a lot as you try advanced tricks, before finally mastering it perfectly.
Related: Overfitting, Generalization, Partial Dependence
A causal inference framework that uses machine learning to control for confounding variables while estimating the exact causal effect of a specific treatment.
Analogy: Like using two different referees to double-check that a player's skill, rather than the wind or the shoes, was the true cause of the winning goal.
Related: Causal Graph, Covariate Shift, Perplexity
An alignment technique that directly optimizes a language model using human preference data without needing a separate reinforcement learning reward network.
Analogy: Like teaching a dog a trick by immediately giving a treat for the right move, skipping the need to build a complex scoring rubric first.
Related: RLHF, Reward Model, Alignment
A regularization technique where randomly selected neurons are temporarily disabled during training to prevent the network from relying on specific connections.
Analogy: Like a sports team practicing with different starting players every day so they don't become overly dependent on just one superstar to win the game.
Related: Regularization, Overfitting, Multi-Label Classification
Halting training once validation performance stops improving for a set number of checks, then restoring the weights from whichever checkpoint scored best.
Related: K-Fold Cross-Validation, Train-Test Split, Debugging Training Runs
Read the full conceptRunning a compressed machine learning model directly on a user's phone, laptop, or IoT sensor instead of in a cloud data center.
Analogy: Like training a waiter to memorize the most common menu questions so they don't have to walk back to the chef for every single request.
Related: Quantization-Aware Training, Multi-Tenancy, Named Entity Recognition
Read the full conceptA specialized neural network designed to convert text, images, or audio into dense numerical vectors that capture the underlying semantic meaning of the data.
Analogy: Like translating a library of books into a universal mathematical language where books with similar themes are placed right next to each other.
Related: Embedding, Cosine Similarity, Nested Embedding
A short list of numbers standing in for something complicated, arranged so that similar things end up close together.
Related: Cosine Similarity, Latent Space, Vector
Read the full conceptA complex capability that spontaneously appears in a machine learning model only after it reaches a critical threshold of scale in parameters or training data.
Analogy: Like water turning into ice: the temperature drops gradually with no visible change until a specific point is reached and a new state suddenly forms.
Related: Scaling Laws, Foundation Model, Reward Hacking
The three ways to arrange transformer blocks — encoder-only for unrestricted understanding, decoder-only with causal masking for generation, and encoder-deco...
Read the full conceptAn offline speech recognition model that reads the entire audio clip to grasp the full context before translating it to text word by word using cross-attention.
Analogy: Like a translator listening to a full recorded speech before writing down the translation, allowing them to use future context to decipher mumbled words.
Related: RNN Transducer, ONNX, Perceptron
Read the full conceptA machine learning technique that combines the predictions from multiple distinct models to produce a single, more accurate, and highly robust final output.
Analogy: Like a jury where multiple people with different perspectives discuss a case to reach a verdict that is fairer than any single person's decision.
Related: Bagging, Boosting, Random Forest
The data processing task of identifying and merging records from different sources that refer to the exact same real-world object, person, or organization.
Analogy: Like realizing that 'Bob Smith', 'Robert S.', and 'Bobby Smith' in your contact list are actually all the exact same friend and merging their cards.
Related: Deduplication, Knowledge Graph
A mathematical measure of randomness or unpredictability in a probability distribution, frequently used in machine learning to quantify information content.
Analogy: Like the difference between a loaded die that always rolls a six and a fair die where you have absolutely no idea what number will come up next.
Related: Information Gain, Power (statistical), Propensity Score
A type of uncertainty caused by a lack of knowledge or training data, which can be systematically reduced by simply gathering more observations for the model.
Analogy: Like guessing the layout of a dark room; you are uncertain because you cannot see, but turning on a flashlight immediately resolves your confusion.
Related: Aleatoric Uncertainty, Active Learning, Rejection Sampling
One complete pass through the entire training dataset during optimization of a machine learning algorithm, allowing the model to update its internal weights.
Analogy: Like reading through an entire textbook from cover to cover once before an exam; you might need to read it multiple times to fully understand it.
Related: Gradient Descent
The systematic process of reviewing mistakes made by a machine learning model to understand its weaknesses and guide future improvements in the architecture.
Analogy: Like a chess player reviewing their lost games to figure out exactly which moves led to their defeat so they can avoid making them in the future.
Related: Debugging Training Runs, ROC, Semantic Entropy
Straight-line distance between two points, the L2 norm of the vector between them.
Analogy: What a bird flies, not what a taxi drives.
Related: L2 Norm, Cosine Similarity, Vector
Read the full conceptThe probability-weighted average of a random variable — the long-run mean, and the centre of mass of its distribution.
Related: Random Variable, Variance, Standard Deviation
Read the full conceptA scaling technique for mixture of experts models where neural network experts are distributed across separate GPUs to increase computational efficiency.
Analogy: Like managing a massive hospital by sending heart patients to the cardiology floor and brain patients to neurology so no single doctor gets overwhelmed.
Related: Mixture of Experts, FSDP, Pipeline Parallelism
The degree to which the internal mechanics, logic, and decision-making processes of a machine learning model can be understood and interpreted by human experts.
Analogy: Like showing your work on a complex math test rather than just writing down the final answer, so the teacher knows exactly how you solved the problem.
Related: Explainable AI, Mechanistic Interpretability, Prefix Caching
A field of artificial intelligence focused on developing tools and techniques that allow humans to understand exactly how and why a model makes its predictions.
Analogy: Like having an interpreter translate a highly technical medical diagnosis into simple language so the patient can fully understand the reasoning behind it.
Related: Explainability, SHAP, QK-Normalisation
Gradients growing exponentially large as they propagate backward through many layers, producing huge parameter updates that destabilize or crash training.
Related: Vanishing Gradients, Gradient Clipping, Gradient
Read the full conceptAn evaluation metric for classification models that calculates the harmonic mean of precision and recall, providing a balanced measure for imbalanced datasets.
Analogy: Like judging a fisherman not just by how many fish they caught, but also by making sure they didn't catch too much garbage in the process of fishing.
An open-source software library developed by Meta that enables extremely fast and highly efficient similarity search and clustering for dense numerical vectors.
Analogy: Like a highly organized library index that lets you instantly find a specific book out of millions without having to read every single title on the shelf.
Related: Approximate Nearest Neighbor, Embedding Model, Query Expansion
A ranking of how much each feature drives predictions, computed free from split counts or, more honestly, from the drop in held-out score after shuffling it.
Related: Stacking, Cardinality, Slice-Based Evaluation
Read the full conceptThe grid of numbers a model actually trains on: one row per example, one column per feature.
Related: Feature, Matrix, Sparse Matrix
Read the full conceptThe process of identifying and retaining the most relevant and informative input variables in a dataset while discarding redundant or noisy data attributes.
Analogy: Like packing for a trip and only bringing the clothes you will actually wear, leaving behind the heavy items that will just weigh your suitcase down.
Related: Dimensionality Reduction, Feature Importance
A centralized data management system designed to compute, store, and serve machine learning features consistently across training and production environments.
Analogy: Like a pre-prepped ingredient station in a restaurant kitchen where chefs can grab exactly what they need without chopping vegetables from scratch every time.
Related: MLOps, Query (attention), Receptive Field
One input column a model reads — a single measured or constructed number describing each row.
Related: Feature Matrix, Ground Truth, Interaction Feature
Read the full conceptA decentralized training approach where a shared global model is collaboratively learned across many distributed devices without exchanging raw, sensitive data.
Analogy: Like a group of chefs perfecting a recipe by practicing in their own kitchens and only sharing their notes, rather than bringing all their ingredients together.
Related: Differential Privacy, Edge Deployment, Rectified Flow
A fundamental artificial neural network architecture where information travels strictly in one direction from the input layer to the final output layer nodes.
Analogy: Like water flowing down a series of stepped waterfalls; the water always moves forward and never flows backward up the stream to a previous level.
Related: Neural Network, Backpropagation, Self-Attention
Classifying a new category correctly after seeing only a handful of labeled examples of it, instead of the thousands an ordinary classifier needs per class.
Related: Multi-Task Learning, Proximal Policy Optimization
Read the full conceptThe process of taking a pre-trained model and further training it on a smaller, task-specific dataset to adapt its broad knowledge to a specialized domain.
Analogy: Like taking a doctor who has already completed general medical school and giving them specialized training to become an expert cardiovascular surgeon.
Related: Transfer Learning, LoRA
An optimized algorithm that dramatically speeds up transformer models and reduces memory usage by reordering the standard attention computation on the GPU.
Analogy: Like organizing a messy workshop so you can build a project without constantly walking back and forth to the tool shed to grab different supplies.
Related: Semantic Search, Sentence Embedding, Seq2Seq
A generative modeling framework that trains continuous normalizing flows by regressing vector fields, offering an efficient alternative to standard diffusion.
Analogy: Like teaching a boat how to navigate a complex river by mapping out the exact speed and direction of the water currents at every single point along the way.
Related: Diffusion Model, Normalizing Flow, Score Matching
A dynamically scaled loss function designed for classification tasks that automatically down-weights the importance of easy examples to focus on harder cases.
Analogy: Like a teacher ignoring the simple math problems a student always gets right to spend all their tutoring time helping them master the most difficult concepts.
Related: Class Imbalance, Singular Value Decomposition, VC Dimension
A massive artificial intelligence model trained on broad, generalized data that can be adapted and fine-tuned for a wide variety of specific downstream tasks.
Analogy: Like a multi-tool pocket knife that comes with general-purpose blades out of the box but can be modified with specific attachments for specialized jobs.
Related: LLM, Pre-training, Sliding Window Attention
A mathematical measure used to calculate the similarity between two curves, frequently applied to evaluate the quality of images produced by generative models.
Analogy: Like measuring the shortest leash needed for a person walking on one path to walk a dog on a separate path without either of them moving backward.
Related: Validation Set, Watermarking
A memory optimization technique that splits model parameters, gradients, and optimizer states across multiple GPUs to enable the training of massive networks.
Analogy: Like a group of friends splitting the cost of a massive dinner bill so no single person has to carry the entire financial burden on their own credit card.
Related: ZeRO, Sparse Retrieval, Speculative Decoding
A rigorous training method where every single parameter in a pre-trained neural network is updated, requiring significant computational power and system memory.
Analogy: Like completely rebuilding a house from the foundation up to change its layout, rather than just remodeling a single room or painting the exterior walls.
Related: Fine-Tuning, LoRA, PEFT
A generative architecture where a generator and discriminator network compete against each other to produce highly realistic synthetic data samples for tasks.
Analogy: Like an art forger constantly trying to paint better fake masterpieces while an art detective continuously gets better at spotting the microscopic flaws.
Related: Generative Model, Diffusion Model, Tabular Foundation Model
A model that places a prior directly over the space of possible functions, defined by a kernel, and predicts by conditioning that prior on observed data.
Related: Bayesian Linear Regression, Polynomial Regression
Read the full conceptA novel rendering technique that represents a 3D scene as a collection of Gaussian ellipsoids, allowing for incredibly fast and photorealistic view synthesis.
Analogy: Like painting a hyper-realistic landscape by rapidly throwing thousands of tiny, soft-edged paint balloons onto a canvas instead of drawing hard lines.
Related: Neural Radiance Field, T5 & Text-to-Text Transformers, Time Series Foundation Model
A widely used activation function in modern transformers that weights neural inputs by their value, smoothing decision boundaries and improving model training.
Analogy: Like a bouncer who strictly denies people with terrible attitudes, lets great people in immediately, and takes a nuanced approach to everyone in between.
Related: ReLU, SwiGLU, Activation Function
The fundamental ability of a machine learning model to accurately interpret and perform well on new, previously unseen data that was not part of its training.
Analogy: Like a student who learns the core concepts of algebra and can solve brand new equations on a test, rather than just memorizing the textbook examples.
Related: Overfitting, Underfitting
An artificial intelligence system designed to understand the underlying distribution of a dataset so it can create completely original and realistic outputs.
Analogy: Like a talented composer who listens to hundreds of classical symphonies and then writes a brand new piece of music that matches the exact same style.
Related: Discriminative Model, Diffusion Model, Tool Retrieval
A specialized file format designed for storing and running large language models efficiently, optimized for fast loading and low memory usage on standard CPUs.
Analogy: Like compressing a massive movie file into a highly optimized format so it can play smoothly on an older laptop without stuttering or crashing the system.
Related: vLLM, Two-Tower Model, U-Net
A statistical metric used in decision trees to evaluate the probability that a randomly chosen element would be incorrectly classified if labeled automatically.
Analogy: Like looking into a bowl of mixed fruit and calculating the odds of blindly grabbing a grape when you were actually hoping to pull out a strawberry.
Related: Entropy
A decoder-only, causally-masked transformer trained on one objective: predict the next token, everywhere, in any text.
Related: Encoder-Decoder Architectures, BERT, Multi-Token Prediction
Read the full conceptAn explainability technique that uses gradients flowing into the final convolutional layer to produce a heat map highlighting important regions of an image.
Analogy: Like eye-tracking software that records exactly where a person is looking on a screen to understand why they clicked on a specific advertisement.
Related: Explainable AI, Vision-Language-Action Model, Voice Agent
Summing the gradients from several small mini-batches before running one optimizer step, simulating a larger batch size on limited GPU memory.
Related: Batch Size Effects, Mixed Precision Training, Gradient
Read the full conceptCapping the size of a gradient before the optimizer step, so one unusually large batch can't throw the parameters somewhere training can't recover from.
Related: Exploding Gradients, Gradient, L2 Norm
Read the full conceptA foundational optimization algorithm used in machine learning to minimize loss functions by iteratively adjusting model parameters in the steepest direction.
Analogy: Like a blindfolded hiker trying to find the bottom of a deep valley by repeatedly taking small steps in whichever direction feels like the steepest downhill.
Related: Backpropagation
The vector of every partial derivative of a function, pointing in the direction of steepest increase.
Related: Partial Derivative, Derivative, Jacobian Matrix
Read the full conceptA network that builds each node's representation by repeatedly combining it with messages gathered from its graph neighbors, learning from relationships, not...
Related: Adjacency Matrix, Vector-Quantized Models, World Model
Read the full conceptAn advanced retrieval technique that enhances large language models by structuring knowledge into interconnected entities and relations instead of flat texts.
Analogy: Like studying for a history test by drawing a complex mind map of how historical figures influenced each other, rather than just reading paragraphs in order.
Related: Knowledge Graph, Document Parsing
A simple and fast text generation strategy where a language model always selects the highest probability token at each step without exploring any alternates.
Analogy: Like driving across the country and always taking the immediate road with the least traffic, even if it eventually leads you into a massive dead end.
Related: Temperature, XGBoost
The label you treat as correct when training and scoring a model — your best available answer, not necessarily the real one.
Related: Feature, Inter-Annotator Agreement, Data Leakage
Read the full conceptNormalizing a fixed number of channel groups per example, independent of batch size, which is why it replaces batch normalization on small-batch vision models.
Related: Layer Normalization, RMSNorm, Standardization
Read the full conceptAn optimized attention mechanism that speeds up inference by grouping query heads to share a single key and value head, reducing memory bandwidth usage.
Analogy: Like having a team of researchers all share a single master filing cabinet of reference documents rather than printing individual copies for every person.
A reinforcement learning technique for alignment that optimizes policy updates using grouped responses, improving sample efficiency without memory overhead.
Analogy: Like a teacher grading a group project by comparing students against their peers in the same group rather than against the entire school district standard.
A phenomenon where an AI model generates seemingly plausible but factually incorrect, nonsensical, or unverified information as if it were absolute truth.
Analogy: Like a confident student who didn't study but convincingly invents an answer to pass an oral exam.
Related: Ground Truth
A training example that is visually or semantically very similar to the anchor but belongs to a different class, forcing the model to learn fine details.
Analogy: Like identical twins where you have to learn subtle differences to tell who is who.
Related: Contrastive Learning
Mapping categories or tokens into a fixed number of buckets with a hash function, so the feature width never grows.
Related: One-Hot Encoding, Cardinality, Sparse Matrix
Read the full conceptThe matrix of all second partial derivatives, describing how a surface curves rather than how it slopes.
Related: Jacobian Matrix, Convex Function, Gradient
Read the full conceptAn internal vector representation in a neural network that captures and retains information from previous inputs in a sequential data stream over time.
Analogy: Like a runner's momentum in a marathon, carrying the cumulative effect of all the steps taken so far.
Related: LSTM, RNN Transducer
A predictive modeling approach for time series data organized in a structure, ensuring predictions at lower levels sum up to match higher-level totals.
Analogy: Like ensuring the predicted sales of every individual store perfectly add up to the predicted national sales.
Related: Time Series Cross-Validation
Restricting self-attention to local windows and merging patches across stages, rebuilding a CNN-like multi-resolution pyramid at linear rather than quadratic...
Related: Vision Transformers, Patch Embeddings
Read the full conceptHierarchical Navigable Small World is an efficient graph-based algorithm for approximate nearest neighbor search in high-dimensional vector spaces.
Analogy: Like an express elevator system where you take the fast elevator to the right zone before switching to local floors.
Related: Cosine Similarity, Embedding
A system design where human feedback or intervention is integrated into the model training, evaluation, or decision-making process to ensure safety.
Analogy: Like a self-driving car that handles highway cruising but alerts a human driver to take the wheel in chaotic city traffic.
Related: Active Learning, Alignment
A search strategy combining traditional keyword-based exact matching with modern dense vector semantic search to retrieve the most relevant documents.
Analogy: Like searching a library using both the exact title of a book and a general description of what the book is about.
Related: Embedding Model, BM25
Hypothetical Document Embeddings is a retrieval technique where an LLM generates a fake ideal document to answer a query, which is then vectorized.
Analogy: Like sketching a picture of a suspect based on a description, and then using that sketch to search a photo database.
Related: Embedding Model, Hallucination
Searching for the combination of a model's untrained settings, like depth or learning rate, that scores best on validation, rather than its learned parameters.
Related: K-Fold Cross-Validation, Bayesian Optimization
Read the full conceptA statistical method used to determine if there is enough evidence in a sample of data to infer that a certain condition holds true for a population.
Analogy: Like a courtroom trial where you assume innocence until the evidence strongly proves guilt beyond a reasonable doubt.
Related: Expected Value, Variance
A square matrix with ones down the diagonal and zeros elsewhere, which leaves any matrix it multiplies unchanged.
Analogy: Multiplying by one, for matrices.
Related: Matrix, Transpose, Matrix Rank
Read the full conceptUser behavior data collected indirectly, such as clicks, watch time, or purchase history, used to infer preferences without asking for direct ratings.
Analogy: Like guessing a friend's favorite food by noticing what they always order, rather than asking them directly.
Related: Feature, Generative Model
Replacing a missing value with an estimate, so a row with a gap can still be used.
Related: Standard Deviation, Variance, Feature
Read the full conceptThe ability of a language model to learn a new task simply from instructions and examples provided within its prompt, without updating its weights.
Analogy: Like showing a chef three examples of how you want your sandwich cut, and they immediately understand the pattern.
Related: Few-Shot Learning, Chain-of-Thought
An unmixing method recovering independent source signals from linear mixtures, a stronger condition than PCA's decorrelation, able to separate blended signals.
Related: Covariance, Standardization, Matrix Factorization
Read the full conceptA security vulnerability where malicious instructions are hidden within external data that a language model ingests, hijacking its intended behavior.
Analogy: Like slipping a secret note into a textbook that tricks the reader into performing a prank when they read that specific page.
The phase in machine learning where a trained model is deployed to make predictions or generate outputs on new, previously unseen data in production.
Analogy: Like a student finally taking the final exam after weeks of rigorous studying and practice tests.
Related: Continuous Batching, KV Cache
A metric used in decision trees to measure the reduction in entropy or uncertainty about a target variable after splitting data on a specific feature.
Analogy: Like playing Twenty Questions and asking a brilliant question that instantly eliminates half of the possible answers.
Related: Entropy, Gini Impurity
A fine-tuning process that trains a language model to follow natural language commands and instructions, bridging the gap between text completion and dialogue.
Analogy: Like sending a brilliant but rambling philosopher to a boot camp to learn how to answer direct questions clearly.
Related: Fine-Tuning, DPO
A variable used in causal inference to estimate the causal effect of an input when there is hidden confounding between the input and the target outcome.
Analogy: Like using weather patterns to study the effect of fish supply on prices, since weather affects supply but not demand directly.
Related: Causal Graph, Expected Value
A mathematical optimization technique where some or all of the decision variables are strictly constrained to take on integer values.
Analogy: Like trying to buy exactly the right combination of whole apples and oranges with a specific budget, since you can't buy half an apple.
Related: Linear Combination
An explainable AI technique that attributes a model's prediction to its input features by integrating gradients along a path from a baseline to the input.
Analogy: Like tracing the exact flow of water from a mountain spring down into a valley to understand which streams contributed the most.
Related: Explainability, Gradient
How often two or more people independently give the same label to the same item — and therefore how well the task is defined.
Related: Ground Truth, Class Imbalance
Read the full conceptA column built by combining two others — typically their product or ratio — so a model can use an effect that depends on both.
Related: Feature, Lag Feature, Feature Matrix
Read the full conceptIntersection over Union is an evaluation metric used in object detection to measure the overlap between a predicted bounding box and the ground truth.
Analogy: Like checking how perfectly a freshly placed sticker aligns with the faded outline of an old sticker on a laptop.
Related: Ground Truth, F1-Score
A tree-based anomaly detector scoring points by how few random splits isolate them, with short paths meaning anomalous and long paths meaning typical.
Related: Outlier, Curse of Dimensionality, Anomaly Detection
Read the full conceptThe matrix of all first partial derivatives of a function with several inputs and several outputs.
Related: Gradient, Hessian Matrix, Partial Derivative
Read the full conceptA declarative language that defines the expected structure, constraints, and data types for JSON documents to ensure consistency and validation.
Analogy: Like a strict bouncer at a club checking that every ID has the right format, birthdate, and state seal before letting anyone in.
Related: Constrained Decoding, Tool Calling
A systematic flaw where an LLM acting as an evaluator disproportionately favors certain outputs based on style, length, or position rather than quality.
Analogy: Like a tired teacher giving higher grades to essays that are longer or simply placed at the top of the grading stack.
Related: LLM-as-a-Judge, Bias (statistical)
A dataset property where every combination of identifying-ish columns is shared by at least k people, so no row stands alone.
Related: Differential Privacy, Cardinality
Read the full conceptSplitting data into K parts and training K times, each run validating on a different part, so every row is validated exactly once.
Related: Train-Test Split, Stratified Sampling, Data Leakage
Read the full conceptA cross-validation technique where the dataset is divided into 'k' equal-sized subsets, and the model is trained and evaluated 'k' times iteratively.
Analogy: Like a sports team playing every other team in the league once to ensure their overall ranking isn't just a fluke based on one lucky game.
Related: K-Fold Cross-Validation, Variance
An unsupervised clustering algorithm that partitions data into 'k' distinct groups by minimizing the distance between points and their cluster centers.
Analogy: Like organizing a chaotic closet by throwing all clothes into three distinct piles based on which pile is physically closest.
Related: Clustering, Euclidean Distance
A simple, non-parametric machine learning algorithm that classifies a new data point based on the majority class of its 'k' closest neighbors in space.
Analogy: Like figuring out a neighborhood's political leaning by just looking at the yard signs of the five houses closest to you.
A mathematical function used in support vector machines and other algorithms to map data into a higher-dimensional space where it becomes linearly separable.
Analogy: Like realizing you can't draw a straight line to separate red and blue marbles on a table, so you throw them in the air to separate them in 3D.
Related: One-Class SVM
Karush-Kuhn-Tucker conditions are first-order necessary mathematical conditions for a solution in nonlinear programming to be optimal under constraints.
Analogy: Like a set of strict physical laws that guarantee a ball rolling down a bumpy hill has genuinely reached the absolute lowest valley.
Related: Gradient, Stationary Point
A measure of how much information is lost when one distribution is used to approximate another.
Related: Log-Likelihood, Softmax, Entropy
Read the full conceptA model compression technique where a small 'student' model is trained to mimic the outputs and probability distributions of a large 'teacher' model.
Analogy: Like a master chef passing down their refined recipes to an apprentice so the apprentice can cook just as well but much faster.
Related: Fine-Tuning, Model Pruning
A structured semantic network that models entities, events, and abstract concepts, explicitly defining the complex relationships connecting them.
Analogy: Like a detective's corkboard covered in photos and string, showing exactly how all the suspects and locations are connected.
Related: Graph RAG, Entity Resolution
A memory optimization in transformer models that stores previously computed Key and Value vectors during generation to prevent redundant calculations.
Analogy: Like a mathematician keeping intermediate calculations on a scratchpad so they don't have to restart the whole equation for every new step.
Related: Continuous Batching, PagedAttention
The sum of absolute values of a vector's components, also called Manhattan or taxicab length.
Analogy: The distance a taxi drives on a grid, not the distance a bird flies.
Read the full conceptThe square root of the sum of squares — ordinary straight-line length, and the default meaning of a vector's magnitude.
Related: L1 Norm, Euclidean Distance, Unit Vector
Read the full conceptA data preprocessing technique that converts categorical text variables into integer values so they can be processed by machine learning algorithms.
Analogy: Like assigning a unique jersey number to every player on a sports team so the referee can easily log penalties.
Related: One-Hot Encoding, Ordinal Encoding
A column holding an earlier value of the same series, so a model can see the recent past from the current row.
Related: Interaction Feature, Stationarity, Feature
Read the full conceptA mathematical strategy used to find the local maxima and minima of a function subject to equality constraints by introducing a new scalar variable.
Analogy: Like trying to find the highest point on a mountain, but you are strictly forbidden from leaving the hiking trail.
Related: KKT Conditions, Gradient
A neural search architecture where queries and documents are encoded independently, and interaction between them happens only at the very final scoring stage.
Analogy: Like a blind taste test where the chef and the critic prepare in total isolation and only meet at the moment of tasting.
Related: Embedding Model, Reranking
A highly efficient generative model architecture that performs the diffusion process (adding/removing noise) in a compressed lower-dimensional latent space.
Analogy: Like an artist sketching a complex scene as a rough, tiny thumbnail before meticulously painting the full high-resolution canvas.
Related: Diffusion Model, Latent Space
The compressed coordinate system a model learns, in which each example is a short vector rather than raw data.
Related: Embedding, Manifold Hypothesis, Principal Component Analysis
Read the full conceptStandardizing every feature of a single example to mean zero and variance one, computed independently per example rather than across a batch of examples.
Related: RMSNorm, Group Normalization, Standardization
Read the full conceptChanging the learning rate over the course of training on a fixed schedule, usually shrinking it, instead of holding one value for every step.
Related: Learning Rate Warmup, Adaptive Optimizers, Gradient
Read the full conceptIncreasing the learning rate from near zero up to its target value over the first steps of training, before any decay schedule takes over.
Related: Learning Rate Scheduling, Adaptive Optimizers, Weight Initialization
Read the full conceptA class of supervised machine learning techniques designed to optimize the ordering of a list of items rather than predicting an exact score or class.
Analogy: Like a DJ arranging a playlist so the best songs play first, rather than just rating each song individually.
Related: Recommender System, NDCG
A predictive metric estimating the total net profit a business can expect to generate from a single customer throughout their entire relationship.
Analogy: Like estimating the total apples an orchard tree will produce over its lifespan, rather than just counting this season's harvest.
Related: Propensity Model, Survival Analysis
The probability of the observed data under a particular choice of parameters, read as a function of those parameters.
Related: Log-Likelihood, Bayes' Theorem, Random Variable
Read the full conceptLocal Interpretable Model-agnostic Explanations is a technique that explains individual predictions by fitting a simple proxy model locally around the input.
Analogy: Like understanding a complex curved road by looking at a tiny, perfectly straight one-meter segment right in front of you.
Related: Explainability, SHAP
A weighted sum of vectors: scale each one by a number and add the results.
Related: Vector, Matrix Rank, Scalar
Read the full conceptA supervised projection maximizing between-class scatter over within-class scatter, separating known classes rather than maximizing overall data variance.
Related: Covariance, Orthogonal Vectors, Mean Shift
Read the full conceptAn automated evaluation paradigm where a powerful language model is prompted to score, critique, or rank the outputs of other AI models.
Analogy: Like having a master chef taste and grade the dishes prepared by culinary students during a cooking exam.
Related: Judge Bias, Alignment
A Large Language Model is a massive neural network trained on vast text corpora, capable of understanding, generating, and reasoning with human language.
Analogy: Like a highly literate librarian who has read millions of books and can predict exactly what word should naturally come next in a sentence.
Related: Foundation Model
The logarithm of the likelihood, which turns an unusable product of probabilities into a well-behaved sum.
Related: Likelihood, KL Divergence
Read the full conceptA rearrangement that computes the log of a sum of exponentials without ever overflowing, by factoring out the largest term first.
Related: Numerical Overflow, Softmax, Machine Epsilon
The inverse of exponentiation: the power a base must be raised to in order to produce a given number.
Related: Log-Likelihood, Log-Sum-Exp, Entropy
Read the full conceptA fundamental statistical algorithm that models the probability of a binary outcome by passing a linear combination of inputs through a sigmoid function.
Analogy: Like predicting if it will rain by assigning weights to clouds and humidity, then squishing the final score between 0% and 100%.
Related: Sigmoid
An interpretability technique that decodes the intermediate hidden states of a transformer model directly into vocabulary words to observe its 'thinking'.
Analogy: Like putting a stethoscope on a person's brain to hear their half-formed thoughts before they actually speak a full sentence.
Related: Mechanistic Interpretability, Hidden State
Techniques that let a model trained on a shorter sequence generalize to much longer ones, by rescaling its positional signal rather than retraining it.
Related: Rotary Position Embeddings, ALiBi Attention Bias
Read the full conceptLow-Rank Adaptation is a parameter-efficient fine-tuning technique that injects small, trainable rank-decomposition matrices into frozen model weights.
Analogy: Like learning a new language by adding a tiny cheat sheet to a massive dictionary, rather than rewriting the entire dictionary.
Related: PEFT, Fine-Tuning
A mathematical formula that calculates the difference between a model's predicted output and the actual ground truth, guiding the optimization process.
Analogy: Like a coach's harsh grading system that tells an athlete exactly how far they missed the target so they can correct their aim.
Related: Gradient Descent
A technique used during LLM fine-tuning where the loss is ignored for the instruction prompt and only calculated for the model's generated response.
Analogy: Like a teacher grading a student's test answer but completely ignoring any spelling mistakes in the printed question itself.
Related: Instruction Tuning
Long Short-Term Memory is a specialized recurrent neural network architecture designed to mitigate the vanishing gradient problem in long sequences.
Analogy: Like a diary with a smart index that automatically decides which old entries to remember and which obsolete ones to forget.
Related: Hidden State, Vanishing Gradients
The smallest gap a float type can represent near one — the resolution limit of its arithmetic.
Related: Numerical Overflow, Derivative
Read the full conceptA state-space model architecture that achieves transformer-like performance on long sequences while maintaining linear, rather than quadratic, scaling.
Analogy: Like a super-reader who can process an entire book linearly without ever needing to look back at previous pages to understand the plot.
Related: State Space Model, Attention Sink
The claim that real data occupies a tiny, smooth, low-dimensional subset of the space used to store it.
Related: Latent Space, Curse of Dimensionality, Embedding
Read the full conceptMaximum A Posteriori estimation is a Bayesian method for finding the most likely model parameters by combining observed data with a prior distribution.
Analogy: Like a detective who uses new crime scene evidence to update their strong initial hunch about who committed the crime.
Related: Bayes' Theorem, MLE
A pre-training objective where random words in a sequence are hidden, and the model must predict them using context from both the left and the right.
Analogy: Like a fill-in-the-blank test where you use the rest of the sentence to guess the missing word in the middle.
Related: BERT, Self-Supervised Learning
Approximating a large, mostly unobserved matrix as the product of two smaller matrices, fit only on observed entries so missing ones are filled in for free.
Related: Standardization, Covariance, Independent Component Analysis
Read the full conceptThe number of genuinely independent directions in a matrix — how much distinct information it actually carries.
Related: Matrix, Linear Combination
Read the full conceptA rectangular grid of numbers addressed by two indices — rank 2, shape (m, n).
Related: Vector, Tensor, Transpose
Read the full conceptA clustering method where each point climbs toward the nearest density peak, cluster count and shape both falling out of the data instead of being chosen.
Related: Euclidean Distance, Curse of Dimensionality, Linear Discriminant Analysis
Read the full conceptA research field dedicated to reverse-engineering neural networks down to individual neurons and circuits to understand exactly how they compute answers.
Analogy: Like taking apart a complex digital watch piece by piece to understand precisely how the gears turn to produce the correct time.
Related: Superposition, Logit Lens
A picture of a sound: time along one axis, perceptually-spaced frequency bands up the other, brightness for energy.
Read the full conceptA privacy attack where an adversary analyzes a model's output to determine if a specific data record was used in its original training dataset.
Analogy: Like tasting a famous chef's secret soup and confidently knowing if they used your specific brand of tomatoes to make it.
Related: Differential Privacy, Memorization
A phenomenon where a model learns to perfectly recall exact examples from its training data rather than generalizing abstract patterns to new data.
Analogy: Like a student who memorizes the exact answers to a practice test but fails the real exam because the questions changed slightly.
Related: Overfitting, Membership Inference
Training a shared starting point across many related tasks, rather than a solution to one, so adapting to any new task takes only a handful of gradient steps.
Related: Proximal Policy Optimization, Domain Adaptation
Read the full conceptRescaling a column into a fixed range, usually 0 to 1, by subtracting the minimum and dividing by the span.
Related: Standardization, Outlier
Read the full conceptA probabilistic algorithm used to quickly estimate the Jaccard similarity between two large sets, heavily used for massive dataset deduplication.
Analogy: Like comparing two massive books by just checking if the first random words on ten specific pages are identical.
Related: Hashing Trick, Deduplication
Running the forward and backward pass in a lower-precision format for speed and memory savings, while keeping a full-precision master copy of the weights.
Related: Numerical Overflow, Gradient Accumulation, Gradient Clipping
Read the full conceptA neural network architecture where a gating network dynamically routes tokens to a sparse selection of specialized sub-networks during a forward pass.
Analogy: Like a hospital triage system that routes a patient with a broken bone directly to orthopedics rather than a general doctor.
Related: Expert Parallelism, Model Routing
The loop a project actually runs — problem, data, model, evaluation, shipping, monitoring — where production findings feed back into reframing the problem, n...
Related: Multicollinearity, No Free Lunch
Read the full conceptMaximum Likelihood Estimation is a statistical method for estimating the parameters of a probability distribution that makes the observed data most probable.
Analogy: Like adjusting a telescope's focus knob until the blurry star in your lens looks as sharp and bright as possible.
Related: MAP Estimation, Log-Likelihood
Machine Learning Operations encompasses the practices, tools, and culture required to reliably build, deploy, and monitor ML models in production.
Analogy: Like the pit crew and logistics team that keeps a high-performance Formula 1 race car running smoothly on the track.
Related: Pipeline, Data Drift
A standardized, transparent documentation framework that details a machine learning model's intended use cases, performance metrics, and inherent biases.
Analogy: Like a detailed nutrition label on a food package, telling you exactly what is inside, how it was made, and allergy warnings.
Related: Data Card, Explainability
An open standard protocol that securely connects AI agents and models to external data sources, APIs, and tools, standardizing how context is integrated.
Analogy: Like a universal power adapter that lets your appliances plug into and draw electricity from any wall socket in the world.
Related: Agent, Tool Calling
A technique to combine the weights of multiple fine-tuned models into a single, unified model without requiring expensive retraining or gradient updates.
Analogy: Like melting down two specialized swords to forge a single, master weapon that inherits the strengths of both.
Related: Fine-Tuning, LoRA
Removing weights or whole structures from a trained network, usually the smallest-magnitude ones first, to shrink it with minimal loss in accuracy.
Related: Quantization-Aware Training, L1 Norm, Sparse Matrix
Read the full conceptAn architecture pattern in LLM applications where a dynamic gateway analyzes incoming queries and routes them to the most appropriate, cost-effective model.
Analogy: Like a telephone operator who connects a simple question to a junior clerk, but routes a complex legal issue to a senior partner.
Related: Mixture of Experts, Pipeline
An optimization technique that accelerates gradient descent by adding a fraction of the previous update vector to the current one, dampening oscillations.
Analogy: Like a heavy bowling ball rolling down a hill, easily smashing through small bumps that would stop a lightweight marble.
Related: Gradient Descent, Adaptive Optimizers
A core mechanism in transformers that runs multiple attention calculations in parallel, allowing the model to simultaneously focus on different features.
Analogy: Like reading a complex legal document while one lawyer checks for loopholes, another for grammar, and a third for context.
An advanced attention mechanism that drastically compresses the KV cache by projecting keys and values into a shared, lower-dimensional latent space.
Analogy: Like saving a massive library of high-res photos as compressed thumbnails in RAM, only expanding them precisely when viewed.
Related: KV Cache, Multi-Query Attention
A predictive modeling task where a single data instance can be simultaneously assigned to multiple overlapping categories or classes.
Analogy: Like tagging a movie on Netflix as both 'Action', 'Sci-Fi', and 'Thriller' rather than forcing it into one single genre.
Related: Sigmoid
An optimized transformer mechanism where all attention heads share a single set of Key and Value projections while maintaining multiple distinct Queries.
Analogy: Like a dozen different customers asking unique questions to a single librarian who shares the exact same reference book with all of them.
Related: KV Cache, Grouped-Query Attention
Training one shared network on several related tasks at once, so what one task teaches the shared layers can help the others, for one inference cost instead ...
Related: Few-Shot Learning, Domain Adaptation
Read the full conceptA software architecture where a single instance of a model or database serves multiple distinct customers while securely isolating their data and usage.
Analogy: Like an apartment building where everyone shares the same plumbing and foundation, but each family has a secure, private room.
Related: Edge Deployment, Data Leakage
A training technique that adds extra heads predicting several future tokens per position, not just the next one, then discards those heads at inference.
Related: GPT
Read the full conceptHigh correlation between input features, which lets a regression's individual coefficients swing wildly across resamples while its actual predictions stay st...
Related: Correlation Coefficient, Matrix Rank
Read the full conceptAn AI model designed to simultaneously process, understand, and generate data across multiple distinct modalities, such as text, images, and audio.
Analogy: Like a skilled human who can watch a video, listen to a speaker, and read a chart all at the same time to understand a presentation.
Related: Vision Transformers, Embedding Model
A simple yet highly effective probabilistic classifier based on applying Bayes' theorem with strict independence assumptions between all input features.
Analogy: Like a detective assuming that finding a suspect's fingerprint and shoe print are completely unrelated coincidences.
Related: Bayes' Theorem, Logistic Regression
A natural language processing task that automatically identifies and classifies key informational elements in text into predefined structured categories.
Analogy: Like highlighting all the people in blue, locations in green, and organizations in yellow while reading a newspaper.
Related: Tokenization
Normalized Discounted Cumulative Gain is an evaluation metric that measures the quality of a ranking system by penalizing highly relevant items placed low.
Analogy: Like scoring a librarian higher if they hand you the exact book you want first, rather than handing it to you tenth.
Related: Learning to Rank, Recommender System
Two nested loops: an inner one tunes a hyperparameter using only outer training rows, an outer one scores the chosen setting on rows the tuning never saw.
Related: K-Fold Cross-Validation, Hyperparameter Tuning, Time Series Cross-Validation
Read the full conceptA representation technique where lower-dimensional embeddings are strictly contained as a prefix within the dimensions of a larger, full-size embedding.
Analogy: Like a Russian nesting doll, where the core miniature doll perfectly fits inside and forms the center of every larger doll.
Related: Embedding Model, Dimensionality Reduction
Automatically searching a defined space of possible network architectures, using a search strategy plus a cost-aware score, instead of hand-designing one.
Related: Semi-Supervised Learning, Curse of Dimensionality
Read the full conceptA foundational machine learning architecture inspired by biological brains, composed of interconnected layers of artificial neurons that process complex data.
Analogy: Like a massive committee of advisors where each person listens to colleagues, applies a rule, and passes their opinion up the chain.
Related: Activation Function, Backpropagation
A deep learning technique that trains a neural network to represent a continuous 3D scene from a sparse set of 2D images by predicting light and density.
Analogy: Like a sculptor learning the exact shape of a statue just by looking at five photographs taken from different angles.
Related: Gaussian Splatting, Generative Model
The fundamental pre-training objective of autoregressive language models, where the network learns to guess the subsequent word in a given text sequence.
Analogy: Like a game of autocomplete where you have to correctly guess the next word someone is going to say based on their sentence so far.
A theorem stating that, averaged over every possible problem, no learning algorithm outperforms another — an edge on real problems comes from matching their ...
Related: Statistical Learning Theory, The ML Workflow
Read the full conceptThe symmetric bell-shaped continuous distribution described entirely by its mean and standard deviation.
Related: Random Variable, Standard Deviation, Expected Value
Read the full conceptA class of generative models that transforms a simple base probability distribution into a highly complex one using a sequence of invertible functions.
Analogy: Like molding a perfectly round ball of clay into a complex statue in a way that you can precisely reverse back into a ball.
Related: Generative Model, VAE
A decoding strategy for text generation that truncates the vocabulary to the smallest set of tokens whose cumulative probability exceeds a threshold 'p'.
Analogy: Like letting a chef choose from the top 90% most likely ingredients for a dish, completely ignoring the bizarre 10% outliers.
Related: Temperature, Greedy Decoding
When a computed value exceeds the largest number its float type can hold and becomes infinity.
Related: Log-Sum-Exp, Machine Epsilon, Softmax
Read the full conceptA computer vision task that goes beyond identifying what is in an image by precisely locating multiple distinct objects using explicit bounding boxes.
Analogy: Like drawing a precise red rectangle around every single car, pedestrian, and traffic light you spot in a photograph.
A method in reinforcement learning used to estimate the performance of a new target policy using historical data collected by a different, older policy.
Analogy: Like trying to predict how well a new basketball strategy will work by only watching tapes of last year's games played differently.
Related: Reinforcement Learning, Expected Value
A boundary learned from only normal training examples, flagging anything outside it as novel, without ever seeing an example of what abnormal looks like.
Related: Outlier, Class Imbalance, Isolation Forest
Read the full conceptTurning a categorical column into one binary column per category, with exactly one set to 1 in every row.
Related: Ordinal Encoding, Cardinality, Sparse Matrix
Read the full conceptA critical failure mode where a model's performance degrades in production because the real-time inference data differs from the batch training data.
Analogy: Like practicing for a spelling bee using a written dictionary, but failing on stage because you can't understand the announcer's accent.
Related: Data Drift, MLOps
Open Neural Network Exchange is an open standard format designed to represent machine learning models, enabling interoperability across different frameworks.
Analogy: Like a universal translator that lets a book written in PyTorch be seamlessly read and executed by a TensorFlow or C++ engine.
Mapping ordered categories to integers that respect their order — small, medium, large becoming 0, 1, 2.
Related: One-Hot Encoding, Discretization, Cardinality
Read the full conceptTwo vectors at right angles, whose dot product is exactly zero.
Analogy: Pushing a trolley directly sideways: maximum effort, zero forward progress.
Related: Unit Vector, Cosine Similarity, Vector
Read the full conceptThe capability of a machine learning system to reliably identify when an input significantly deviates from the distribution of its original training data.
Analogy: Like a dog that knows exactly how to herd sheep, but immediately stops and barks for help when asked to herd a flock of ostriches.
Related: Epistemic Uncertainty, Anomaly Detection
A value far from the rest of the data — which may be a broken measurement or the rare event you are trying to predict.
Related: Standard Deviation, Min-Max Scaling, Imputation
Read the full conceptA modeling error that occurs when a neural network memorizes the random noise in its training data, resulting in poor generalization to new, unseen data.
Analogy: Like memorizing the exact answers to a specific practice test, but failing the final exam because the numbers in the questions changed.
Related: Regularization, Variance
Probably Approximately Correct learning is a mathematical framework that defines the conditions under which a machine learning algorithm will reliably succeed.
Analogy: Like mathematically proving that if you ask enough random people for directions, you will almost certainly end up very close to your destination.
Related: Statistical Learning Theory, Generalization
A memory management technique inspired by operating systems that partitions the KV cache of large language models into non-contiguous blocks of memory.
Analogy: Like storing a massive file by splitting it into small chunks scattered across a hard drive, rather than requiring one huge block of free space.
Related: KV Cache, Continuous Batching
An explainability technique that shows the marginal effect one or two specific features have on the predicted outcome of a machine learning model.
Analogy: Like checking how the price of a house changes if you only increase the square footage, while keeping the location and age exactly the same.
Related: Explainability, Feature Importance
The derivative with respect to one input while every other input is held still.
Related: Derivative, Gradient
Read the full conceptFlattening a fixed-size image patch into a raw pixel vector, then projecting it with one shared learned matrix into a transformer's token embedding dimension.
Related: Vision Transformers, Hierarchical Vision Transformers
Read the full conceptPrincipal Component Analysis is a dimensionality reduction algorithm that projects high-dimensional data onto lower-dimensional orthogonal axes of variance.
Analogy: Like shining a flashlight on a complex 3D object to find the specific 2D shadow that captures the most recognizable shape.
Related: Dimensionality Reduction, Orthogonal Vectors
Parameter-Efficient Fine-Tuning encompasses methods that adapt massive pre-trained language models to new tasks by updating only a tiny fraction of weights.
Analogy: Like customizing a massive, factory-built car by just swapping out the steering wheel and seats, rather than rebuilding the entire engine.
Related: LoRA, Fine-Tuning
The simplest form of an artificial neural network, consisting of a single layer that outputs a binary decision based on a linear combination of inputs.
Analogy: Like a strict bouncer who multiplies your age, height, and attire by specific weights, and only lets you in if the total score beats a threshold.
Related: Neural Network, Activation Function
A crucial evaluation metric for language models that quantifies how surprised the model is by a sequence of text; lower perplexity indicates better understan...
Analogy: Like measuring a psychic's skill by how rarely they are completely shocked by an unfolding sequence of completely predictable events.
Related: Entropy
A distributed training technique that splits a massive neural network's layers sequentially across multiple GPUs to fit models larger than a single memory bank.
Analogy: Like a bucket brigade where one person handles the first half of the water transfer and immediately passes the bucket to the next person.
Related: Tensor
An automated, sequential chain of processing steps where the output of one machine learning component serves directly as the input to the next.
Analogy: Like a factory assembly line where one robot cleans the part, the next paints it, and the final one inspects it for quality.
Related: MLOps, Data Leakage
A critical data engineering principle ensuring that historical training data only contains information that was actually knowable at the exact time of predic...
Analogy: Like making sure you don't use tomorrow's newspaper to help you place a bet on today's horse race.
Related: Data Leakage, Feature Store
Fitting ordinary linear regression on top of powers of the input, letting a linear method draw curved shapes instead of only straight lines.
Related: Quantile Regression, Bayesian Linear Regression
Read the full conceptA mechanism used in transformer networks to inject information about the absolute or relative order of tokens into their dense vector representations.
Analogy: Like numbering the pages of a scattered manuscript so the reader inherently understands the chronological flow of the story.
Related: RoPE
In Bayesian statistics, the updated probability distribution of a parameter or hypothesis after observing new evidence or training data.
Analogy: Like your revised opinion of a restaurant's quality after you actually eat a meal there, combining your prior expectations with the new experience.
Related: Bayes' Theorem, MAP Estimation
The probability that a statistical test will correctly reject a false null hypothesis, effectively measuring the test's ability to detect a true effect.
Analogy: Like a metal detector's sensitivity; higher power means you are much less likely to accidentally walk right over a buried gold coin.
Related: Hypothesis Testing, Variance
Proximal Policy Optimization is a highly stable reinforcement learning algorithm that restricts how much a model's policy can change in a single update step.
Analogy: Like a golf instructor forcing you to fix your swing in tiny, safe increments rather than trying a wildly different technique on every single shot.
Related: Reinforcement Learning, RLHF
The initial, computationally massive phase where a model learns fundamental patterns, features, and knowledge from a vast dataset without task-specific labels.
Analogy: Like a child spending years learning how to walk, talk, and understand gravity before they eventually specialize in playing a sport.
Related: Fine-Tuning, Foundation Model
A classification metric that measures the proportion of true positive predictions out of all the positive predictions a machine learning model actually made.
Analogy: Like a sniper's accuracy metric: out of all the shots they actually took, what percentage successfully hit the intended target?
A performance optimization technique that stores the computed KV cache of common system prompts to instantly reuse them across multiple inference requests.
Analogy: Like a barista pre-measuring exactly 100 cups of the house blend every morning so they don't have to grind beans for every single customer.
Related: KV Cache, Continuous Batching
Finding the directions along which data varies most, so it can be described with fewer numbers.
Related: Covariance, Latent Space, Curse of Dimensionality
Read the full conceptAn advanced reward model used in RLHF that evaluates and scores each individual step of a model's reasoning chain, rather than just the final answer.
Analogy: Like a math teacher giving a student partial credit for correctly executing the first three steps of a formula, even if they messed up the final calculation.
Related: Reward Model, Chain-of-Thought
The systematic practice of designing, refining, and optimizing the natural language inputs given to an AI model to steer it toward a desired behavior.
Analogy: Like writing an incredibly specific, unambiguous recipe for a talented but literal-minded chef who will do exactly what you say, nothing more.
Related: In-Context Learning, Chain-of-Thought
A critical security vulnerability where a malicious user deliberately inputs text designed to override or hijack a language model's original instructions.
Analogy: Like a stage hypnotist telling someone to ignore all previous commands and immediately act like a chicken whenever they hear a specific bell ring.
Related: Indirect Prompt Injection, Alignment
A predictive machine learning model designed to calculate the statistical probability that a specific user or entity will perform a predefined action.
Analogy: Like a marketing manager looking at your past shopping habits to calculate the exact percentage chance you'll buy a new jacket this winter.
Related: Logistic Regression, Lifetime Value
In causal inference, the conditional probability that an individual receives a specific treatment based on their observed background characteristics.
Analogy: Like calculating the likelihood that a patient would have chosen to take a new experimental drug based purely on their age, weight, and prior medical history.
Related: Instrumental Variable
A policy-gradient method that clips its objective so a single update can't push the policy too far from where it started, making it safe to reuse one batch f...
Related: Meta-Learning, Few-Shot Learning
Read the full conceptA stabilization technique in transformers where Layer Normalization is applied directly to the Query and Key vectors before calculating attention scores.
Analogy: Like putting sunglasses on two incredibly bright spotlights so you can actually measure their overlap without completely blinding the camera.
Related: Layer Normalization
Quantized Low-Rank Adaptation is a highly memory-efficient fine-tuning method that trains low-rank adapters over a frozen, 4-bit quantized base model.
Analogy: Like modifying a massive physical encyclopedia using sticky notes, while leaving the actual heavy pages compressed in a tight, unreadable format.
Related: LoRA, Quantization-Aware Training
Fitting a line that a chosen fraction of outcomes falls below, using an asymmetric pinball loss instead of squared error, to predict a spread rather than onl...
Related: Polynomial Regression, Gaussian Process Regression
Read the full conceptSimulating low-precision rounding during the forward pass while training, so a network's weights adapt to quantization error before the model ever ships.
Related: Semi-Supervised Learning, Neural Architecture Search, Numerical Overflow
Read the full conceptA retrieval optimization technique that automatically rewrites or augments a user's original search query with synonyms or related contextual terms.
Analogy: Like an experienced librarian hearing you ask for 'car fixing books' and automatically checking the catalog for 'auto repair manuals' as well.
Related: Hybrid Search, HyDE
In the attention mechanism, a vector representing the current token's specific informational needs, used to search against the Keys of all other tokens.
Analogy: Like a shopper handing a detailed grocery list to a clerk, hoping the clerk can match the requested items to the store's inventory.
Related: Multi-Head Attention, KV Cache
A powerful ensemble learning algorithm that builds a multitude of independent decision trees during training and merges their predictions to output a result.
Analogy: Like asking fifty different experts for their opinion on a complex problem and going with the most popular consensus answer to ensure accuracy.
Related: Bagging
A quantity whose value is decided by a random process, described by the distribution over the values it can take.
Related: Expected Value, Normal Distribution, Variance
Read the full conceptA mathematical property defining the maximum number of linearly independent column or row vectors within a matrix, representing its true dimensionality.
Analogy: Like measuring the true size of a team by counting only the people who bring unique skills, ignoring those who just perfectly copy someone else.
Reasoning and Acting is a prompt engineering paradigm that forces an LLM to generate explicit internal thoughts before taking an action with an external tool.
Analogy: Like a detective thinking out loud, 'The safe is locked, so I need a key,' before actually reaching out to grab the key from the desk.
Related: Agent, Chain-of-Thought
A specialized language model designed and trained to prioritize deep, multi-step logical deduction and planning over immediate, reactive text generation.
Analogy: Like consulting a master chess player who sits quietly to calculate ten moves ahead, rather than a commentator who just quickly reacts to the board.
Related: Chain-of-Thought, Process Reward Model
A classification metric that measures the proportion of actual positive cases that a machine learning model successfully identified out of all true positives.
Analogy: Like a radar system's effectiveness: out of all the enemy planes currently in the sky, what percentage did the system successfully detect?
The specific region of the input space that a particular neuron or layer in a neural network is exposed to and can draw contextual information from.
Analogy: Like looking through a cardboard tube; you can only see a small circle of the world, and anything outside that circle cannot affect your judgment.
An automated information filtering system designed to predict a user's preferences and actively suggest highly relevant content, products, or services.
Analogy: Like a wildly observant bookstore clerk who remembers every book you've ever bought and instantly hands you the perfect new novel when you walk in.
Related: Learning to Rank, Implicit Feedback
An advanced generative modeling technique that connects data and noise distributions using completely straight, linear paths rather than curved trajectories.
Analogy: Like building a perfectly straight superhighway between two cities instead of relying on the winding, complex dirt roads that naturally formed there.
Related: Diffusion Model, Generative Model
A set of mathematical techniques used to artificially penalize a model's complexity during training, forcing it to generalize rather than memorize the data.
Analogy: Like a strict referee giving a sprinter a heavy backpack during practice so they are forced to build genuine muscle rather than relying on a tailwind.
Related: Overfitting, Weight Decay
A paradigm of machine learning where an agent learns to make sequential decisions in an environment by performing actions and receiving numerical rewards.
Analogy: Like training a dog by giving it a treat when it successfully fetches a ball, and ignoring it when it completely misses the mark.
Related: PPO, Reward Model
A Monte Carlo algorithm used to generate data samples from a complex target probability distribution by sampling from an easier, bounding distribution.
Analogy: Like a casting director auditioning everyone who walks through the door, but immediately rejecting anyone who doesn't perfectly fit the character description.
Related: Generative Model, MLE
Rectified Linear Unit is the most widely used activation function in deep learning, which outputs the input directly if positive, and exactly zero otherwise.
Analogy: Like a strict bouncer at a club who lets everyone with money in normally, but instantly kicks out anyone carrying a negative balance.
Related: Activation Function, Vanishing Gradients
A two-stage search optimization process where a fast retriever fetches a broad list of candidates, and a heavier model meticulously re-orders them for accuracy.
Analogy: Like a casting assistant quickly grabbing 100 headshots that loosely fit the description, and the director carefully sorting them to find the perfect star.
Related: Late Interaction, Learning to Rank
An architectural shortcut in deep neural networks that entirely bypasses one or more layers by directly adding the original input to the layer's output.
Analogy: Like a highway bypass that lets cars skip the congested city center and merge directly back onto the main road on the other side.
Related: Vanishing Gradients
A critical alignment failure where an AI agent finds a clever, unintended loophole to maximize its given reward signal without actually completing the task.
Analogy: Like a factory worker paid per assembled widget who discovers they can get rich by breaking apart finished widgets and reassembling them.
Related: Reinforcement Learning, Alignment
A secondary machine learning model trained explicitly to evaluate and score the outputs of a primary AI agent based on complex human preferences.
Analogy: Like a strict food critic who tastes every dish a chef makes and assigns a numerical score based on presentation, taste, and creativity.
Related: RLHF, Process Reward Model
Ridge regression is a variant of linear regression that introduces an L2 penalty term to the loss function to heavily shrink excessively large coefficients.
Analogy: Like a strict manager forcing every employee to share the workload evenly, rather than letting one superstar employee do absolutely everything alone.
Related: Regularization, Logistic Regression
Reinforcement Learning from Human Feedback is a crucial alignment technique that uses human preference data to optimize a language model's behavior and tone.
Analogy: Like an etiquette coach watching a brilliant but wildly inappropriate scholar interact with people, heavily rewarding them when they speak politely and helpfully.
Related: PPO, Reward Model
Reinforcement Learning with Verifiable Rewards is a training methodology that optimizes models by automatically checking logical outputs against absolute tru...
Analogy: Like teaching a student math by having a computer instantly verify if their final answer matches the known solution, rather than asking a human grader.
Related: Reinforcement Learning, Process Reward Model
A normalization layer that rescales a vector by its own root-mean-square magnitude alone, skipping the mean-centering step layer normalization performs.
Related: Layer Normalization, Group Normalization
Read the full conceptA speech recognition architecture that uses an encoder for audio and a prediction network for text, combining them to stream transcriptions live without wait...
Analogy: Like a court stenographer listening to the current syllable while remembering the sentence so far to lock in the correct word instantly.
Related: Encoder-Decoder ASR
Read the full conceptReceiver Operating Characteristic is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold v...
Analogy: Like graphing exactly how many false alarms a smoke detector will trigger as you slowly increase its sensitivity to detect smaller and smaller fires.
Rotary Position Embedding is an advanced mathematical technique in transformers that encodes sequence position by physically rotating vectors in a latent space.
Analogy: Like giving two dancers specific instructions on how far to rotate so they can instantly calculate their relative angle to each other at any moment.
Related: Positional Encoding
A method that injects relative position by rotating query and key vectors by an angle proportional to position, instead of adding a separate position vector.
Related: ALiBi Attention Bias, Long-Context Extension
Read the full conceptA single number, with no axes — the smallest container of data there is.
Read the full conceptEmpirical mathematical relationships demonstrating that an AI model's performance improves predictably as compute, dataset size, and parameters increase.
Analogy: Like knowing exactly how much faster a race car will go for every single horsepower you add to its engine.
Related: Foundation Model, Test-Time Compute
The process of aligning natural language queries directly to the specific tables, columns, and relationships in a structured database.
Analogy: Like a translator converting a tourist's vague request for 'good food' into a specific address for the best local restaurant.
Related: Text-to-SQL, Entity Resolution
A statistical technique for training generative models by teaching them to predict the gradient of the log-probability density of the data.
Analogy: Like teaching someone to navigate a mountain in the dark by only telling them which specific direction points uphill at any given spot.
Related: Diffusion Model, Generative Model
Optimization that uses the Hessian, or an approximation of it, to account for curvature and take a more direct step than the gradient alone allows.
Related: Gradient, Jacobian Matrix, Hessian Matrix
Read the full conceptA mechanism in transformers that allows every token in a sequence to dynamically weigh the importance of every other token to understand full context.
Analogy: Like reading a mystery novel where your brain automatically connects a clue on page 10 directly to the villain revealed on page 300.
Related: Multi-Head Attention
A machine learning paradigm where a model automatically generates its own training labels from the raw input data by hiding and predicting parts of it.
Analogy: Like a student studying for an exam by covering up random words in their textbook and guessing what they are.
Related: Masked Language Modeling, Contrastive Learning
An advanced metric used to measure a language model's uncertainty by evaluating if its varied generated responses actually mean the same thing.
Analogy: Like asking a witness the same question five times; if their answers use different words but tell the exact same story, they are certain.
Related: Epistemic Uncertainty, Hallucination
An information retrieval technique that finds documents based on their underlying meaning and context, rather than relying on exact keyword matches.
Analogy: Like asking a librarian for books about 'felines' and receiving books about 'cats' even though the word 'feline' isn't in the title.
Related: Embedding Model, Vector Database
Training on a small labeled set plus a much larger unlabeled pool, typically by having the model label the unlabeled data itself and keeping only confident g...
Related: Knowledge Distillation, Model Pruning, Ground Truth
Read the full conceptA dense mathematical vector representation of an entire sentence or paragraph, capturing its full semantic meaning in a fixed-dimensional space.
Analogy: Like boiling down a 500-page novel into a single, highly dense summary paragraph that captures the entire core plot.
Related: Embedding Model, Cosine Similarity
A neural network framework designed to transform an input sequence of varying length into a completely new output sequence of varying length.
Analogy: Like a professional translator listening to a long, winding speech in English and delivering a concise, structured summary in Spanish.
Related: Encoder-Decoder Architectures
A crucial data optimization technique during LLM training where multiple short documents are concatenated into a single max-length context window.
Analogy: Like tightly packing multiple small packages into a single large shipping container so the cargo ship doesn't waste any empty space.
Related: Attention Sink
Stochastic Gradient Descent is a foundational optimization algorithm that updates model weights iteratively using the gradient of a single random data batch.
Analogy: Like a blindfolded hiker taking a quick step downhill based only on the slope right under their foot, rather than surveying the whole mountain.
Related: Gradient Descent, Momentum
SHapley Additive exPlanations is a game-theoretic approach that assigns a specific importance value to every feature for a particular model prediction.
Analogy: Like fairly distributing a team bonus by mathematically proving exactly how much each player contributed to the final winning score.
Related: Explainability, LIME
A mathematical activation function that smoothly maps any real-valued number into a strict range between 0 and 1, forming an S-shaped curve.
Analogy: Like a dimmer switch that takes an infinitely powerful electrical current and smoothly caps it to a maximum brightness of 100%.
Related: Logistic Regression, Activation Function
The process of training an AI model entirely in a fast, safe virtual simulation and successfully transferring that learned behavior to the physical world.
Analogy: Like learning to fly a jet perfectly in a highly realistic video game simulator before ever stepping foot into a real cockpit.
Related: Reinforcement Learning, Domain Adaptation
A fundamental linear algebra technique that factorizes any complex matrix into three simpler matrices, isolating its most important structural components.
Analogy: Like breaking down a complex musical chord into the three pure, individual tuning fork tones that perfectly combine to create it.
Related: PCA, Rank (matrix)
An architectural feature that routes the output of one neural network layer past several subsequent layers to be directly added to a deeper layer.
Analogy: Like an express train track that allows commuters to skip the congested local stops and merge directly into the downtown terminal.
Related: Residual Connection, Vanishing Gradients
A rigorous testing methodology where model performance is measured across specific critical data subsets, rather than just relying on a global average.
Analogy: Like a teacher checking if a student passed both the math and history sections individually, rather than just looking at their overall average GPA.
Related: Model Card, Bias (statistical)
A localized attention mechanism where tokens only interact with a fixed number of neighboring tokens, drastically reducing compute for long sequences.
Analogy: Like reading a scroll where you only keep the last 50 words in your active memory to understand the current sentence, ignoring the beginning.
Related: Attention Sink, Sparse Model
Synthetic Minority Over-sampling Technique: inventing new minority-class rows by interpolating between a real row and one of its neighbours.
Related: Class Imbalance, Euclidean Distance, Synthetic Data
Read the full conceptA function that turns a vector of arbitrary scores into a probability distribution that sums to one.
Related: Log-Sum-Exp, KL Divergence, Numerical Overflow
An interpretability tool that untangles the dense, multi-meaning activations of an LLM into a massive dictionary where each feature corresponds to one clean,...
Analogy: Like a sound engineer taking a chaotic stereo recording of a crowded room and isolating 50 individual, crystal-clear vocal tracks.
Related: Activation Patching
Read the full conceptA matrix stored as only its non-zero entries, because almost all of its cells are zero.
Related: Matrix, Adjacency Matrix, Curse of Dimensionality
Read the full conceptA neural network where a vast majority of the weights are zero or dynamically inactive during any given forward pass, drastically reducing active compute.
Analogy: Like a massive university with thousands of specialized professors, but a student only ever talks to three of them for a specific major.
Related: Mixture of Experts, Model Pruning
A traditional search methodology that relies on exact keyword matching, representing documents as high-dimensional but mostly empty vectors.
Analogy: Like looking up a specific term in a textbook's index to find exactly which pages contain that exact word.
An inference acceleration technique where a small, fast draft model predicts several tokens ahead, and a larger target model verifies them in parallel.
Analogy: Like an assistant quickly drafting a whole paragraph, and the boss reading it instantly to approve or rewrite it all at once.
Training a meta-model on the out-of-fold predictions of several diverse base models, so the combination learns which model to trust rather than a fixed average.
Related: Feature Importance, Data Leakage
Read the full conceptThe square root of the variance, putting spread back into the same units as the data.
Related: Variance, Expected Value, Normal Distribution
Read the full conceptRescaling a column by subtracting its mean and dividing by its standard deviation, so it ends up centred at 0 with unit spread.
Related: Min-Max Scaling, Standard Deviation, Variance
Read the full conceptA mathematical framework for modeling sequence data that defines how a hidden internal state evolves continuously over time based on incoming inputs.
Analogy: Like a complex weather simulation predicting tomorrow's storm by mathematically updating today's exact pressure and temperature readings.
A property of a series whose statistical behaviour does not drift over time — its mean and variance stay put.
Related: Lag Feature, Variance, Data Drift
Read the full conceptA point where the gradient is zero, so the surface is momentarily flat — a minimum, a maximum, or a saddle.
Related: Derivative, Gradient, Convex Function
Read the full conceptThe field proving generalization bounds on how far true error can exceed training error — too loose to use directly, but explaining why data and capacity bot...
Related: No Free Lunch, Multicollinearity
Read the full conceptA test comparing two models scored on the same rows by using only the disagreements, since agreement carries no information about which model is better.
Related: K-Fold Cross-Validation, Hyperparameter Tuning, Validation Curves
Read the full conceptA specialized vector added to the internal hidden states of an LLM during inference to actively push its behavior toward a specific concept or tone.
Analogy: Like adding a tiny weight to a bowling ball to ensure it always naturally curves toward the center of the lane no matter how it is thrown.
Splitting or sampling so each group appears in the same proportion as it does in the full dataset.
Related: Train-Test Split, Class Imbalance, K-Fold Cross-Validation
Read the full conceptA phenomenon where a neural network represents far more abstract concepts than it has physical dimensions by storing them in nearly orthogonal directions.
Analogy: Like packing 100 different outfits into a tiny suitcase by carefully folding them so their shapes perfectly interlock without ruining each other.
A fundamental machine learning paradigm where an algorithm is trained on a dataset containing strict pairs of inputs and their perfectly correct outputs.
Analogy: Like a student doing math homework with the teacher's answer key sitting right next to them, constantly checking if they got the problem right.
Related: Fine-Tuning, Ground Truth
A branch of statistics used to analyze and predict the expected duration of time until a specific event of interest occurs, handling censored data.
Analogy: Like predicting exactly how many years a newly built bridge will last before requiring major repairs, based on weather and traffic data.
Related: Censoring, Lifetime Value
A gated feed-forward activation: one linear projection of the input is scaled elementwise by a Swish-activated projection before a final output projection.
Related: Encoder-Decoder Architectures
Read the full conceptData produced by a model, a simulator, or a set of rules rather than measured from the world.
Related: SMOTE, Differential Privacy, Variance
Read the full conceptT-distributed Stochastic Neighbor Embedding is a non-linear dimensionality reduction algorithm designed explicitly to visualize high-dimensional data in 2D.
Analogy: Like squishing a complex 3D galaxy of stars onto a flat map so that stars that were close together remain tightly grouped on the paper.
Related: UMAP, Dimensionality Reduction
A large, pre-trained neural network explicitly designed to process, understand, and generate data across diverse, structured relational database tables.
Analogy: Like an expert accountant who has analyzed thousands of different company spreadsheets and can instantly spot trends in a brand-new financial layout.
Related: Foundation Model, Random Forest
A hyperparameter in generative models that scales the output logits before the softmax function, controlling the randomness and creativity of the output.
Analogy: Like adjusting a dial between a strict, by-the-book bureaucrat (low) and a wild, free-associating poet (high).
Related: Nucleus Sampling, Softmax
A container of numbers with any number of axes; scalars, vectors and matrices are its rank 0, 1 and 2 cases.
Related: Matrix, Vector, Scalar
Read the full conceptThe allocation of additional processing power and time during inference to allow a model to search, plan, or reason before outputting a final answer.
Analogy: Like giving a student two full hours to carefully outline and draft an essay, rather than forcing them to answer in thirty seconds.
Related: Reasoning Model, Chain-of-Thought
An AI task where a natural language query from a user is automatically translated into a syntactically correct SQL query to interact with a database.
Analogy: Like having a bilingual assistant who listens to you ask 'How many cars did we sell?' and types out the complex database code to fetch the answer.
Related: Schema Linking, Agent
Framing every NLP task, including classification, as generating a text string from a text string, on top of an ordinary encoder-decoder transformer.
Related: Encoder-Decoder Architectures
Read the full conceptWeighting each word count by how rare the word is across documents, so common words stop dominating.
Related: Bag of Words, Logarithm, Cosine Similarity
Read the full conceptA cross-validation split where training always sits strictly before the validation block it's scored on, since a shuffled fold would let a model see the future.
Related: K-Fold Cross-Validation, Stationarity, Nested Cross-Validation
Read the full conceptA massive, pre-trained neural network capable of performing zero-shot forecasting or anomaly detection on novel sequential data without task-specific tuning.
Analogy: Like an experienced meteorologist who can predict weather patterns in a brand-new city just by looking at a few days of their thermometer data.
Related: Foundation Model
Cutting text into the discrete units a model actually reads, which are usually word fragments rather than words.
Related: Bag of Words, Embedding, Mel-Spectrogram
Read the full conceptThe capability of an AI model to intentionally halt text generation, format an API request to execute an external function, and ingest the result.
Analogy: Like a chef realizing they don't know the exact temperature of the oven, so they pause cooking to check a thermometer before continuing.
Related: Agent, JSON Schema
A dynamic system where an agent uses semantic search to fetch the most relevant tools from a massive library instead of loading all tools simultaneously.
Analogy: Like a handyman walking into a giant hardware store and only picking up the exact three tools needed for the current repair job.
Related: Tool Calling, Agent
Two primary decoding strategies used during text generation to truncate the vocabulary distribution, balancing model creativity and factual coherence.
Analogy: Like a manager asking a team for ideas but instantly dismissing the bottom 90% worst suggestions before picking one randomly.
Related: Nucleus Sampling, Temperature
Dividing data so a model is fitted on one part and measured on another it has never seen.
Related: K-Fold Cross-Validation, Stratified Sampling, Data Leakage
Read the full conceptA foundational ML concept where a model developed for one specific task is reused as the starting point for a model on a second related task.
Analogy: Like a professional tennis player using their deeply ingrained hand-eye coordination and footwork to quickly master playing badminton.
Related: Fine-Tuning, Pre-training
Swapping a matrix's rows and columns, turning an m by n matrix into an n by m one.
Related: Matrix, Identity Matrix
Read the full conceptA loss function used to train embedding models by mathematically pulling a baseline example closer to a matching example while pushing away a mismatch.
Analogy: Like a teacher moving two best friends to sit closer together in class, while intentionally moving a disruptive student far away from them.
Related: Contrastive Learning, Hard Negative
A neural network architecture with two independent encoder pathways that process different data types into a shared vector space for fast comparison.
Analogy: Like an interpreter who translates English into a universal sign language and French into the exact same sign language so both sides can communicate.
Related: Late Interaction, Recommender System
A convolutional neural network architecture uniquely shaped like a 'U', designed for extremely precise, pixel-level image segmentation tasks.
Analogy: Like a factory line that shrinks a car down to inspect its engine, then perfectly reassembles the body panels back to full size.
Related: Diffusion Model, Skip Connection
Uniform Manifold Approximation and Projection is a fast, highly scalable algorithm for dimensionality reduction that preserves global data structure well.
Analogy: Like taking a 3D scan of a sprawling city and creating a highly accurate 2D paper map that preserves the relative distances of all the neighborhoods.
Related: t-SNE, Dimensionality Reduction
A modeling failure where an algorithm is far too simple to capture the underlying complex patterns in the data, resulting in poor accuracy everywhere.
Analogy: Like trying to explain the entire plot of a massive fantasy book series using only three very generic bullet points.
Related: Overfitting, Bias (statistical)
A vector of length exactly one, carrying direction with no magnitude.
Related: Vector, L2 Norm, Cosine Similarity
Read the full conceptThe theorem that a sufficiently wide single-hidden-layer network can approximate any continuous function on a bounded domain to within arbitrary accuracy.
Related: Weight Initialization
Read the full conceptThe complex process of intentionally removing specific knowledge, facts, or behaviors from a trained model's weights without degrading general performance.
Analogy: Like trying to perfectly remove just the salt from a fully baked cake without ruining the texture, flavor, or structural integrity of the dessert.
Related: Memorization, Alignment
A broad machine learning paradigm where models attempt to find hidden structures, patterns, or clusters in data entirely without human-provided labels.
Analogy: Like dropping a child into a room full of random mixed Lego blocks and watching them naturally group them by color and size.
Related: Clustering, K-Means
A predictive modeling technique that calculates the incremental, purely causal impact of a specific intervention or marketing action on an individual.
Analogy: Like figuring out if handing someone a flyer actually caused them to buy a product, or if they were already going to buy it anyway.
Related: Causal Graph, Propensity Model
A Variational Autoencoder is a generative model that learns to compress data into a structured probabilistic latent space from which new data is sampled.
Analogy: Like a compression program that doesn't just store files, but learns the 'idea' of the files so it can randomly generate brand-new, similar ones.
Related: Generative Model, Latent Space
A plot of training and validation error against one hyperparameter with data fixed, whose validation curve traces a U marking over- and underfitting.
Related: K-Fold Cross-Validation, Hyperparameter Tuning, Statistical Model Comparison
Read the full conceptA specific subset of data held back during model training, used to iteratively evaluate performance, tune hyperparameters, and detect overfitting.
Analogy: Like a student taking a series of practice tests to gauge their readiness, entirely separate from their study notes and the final exam.
Related: Train-Test Split, Early Stopping
Gradients shrinking toward zero as they propagate backward through many layers, leaving early layers with almost no signal to learn from during training.
Related: Exploding Gradients, Weight Initialization, Gradient
Read the full conceptThe average squared distance of a random variable from its own mean — how widely values scatter.
Related: Expected Value, Standard Deviation, Covariance
Read the full conceptVapnik-Chervonenkis dimension is a theoretical mathematical measure of the capacity, complexity, and expressive power of a machine learning algorithm.
Analogy: Like rating a highly flexible yoga instructor by the absolute maximum number of complex poses they can physically contort themselves into.
Related: PAC Learning, Statistical Learning Theory
A specialized data system purpose-built to efficiently store, index, and query massive collections of high-dimensional embeddings for semantic search.
Analogy: Like an impossibly massive library organized not by alphabet, but by grouping books whose plots, themes, and feelings are geometrically close together.
A data compression technique that replaces massive, continuous floating-point vectors with integer pointers mapped to a discrete, limited codebook.
Analogy: Like mapping every exact shade of paint in the world to the closest available option in a standard 64-color crayon box.
Related: Quantization-Aware Training, FAISS
A model that replaces an encoder's continuous output with the nearest entry in a fixed, learned codebook, turning a latent space into a finite set of discret...
Related: Graph Neural Networks, Latent Space
Read the full conceptAn ordered list of numbers, addressed by a single index — rank 1, shape (n,).
Analogy: A row of numbered lockers: one number gets you to any of them.
Related: Scalar, Matrix, Unit Vector
Read the full conceptA multimodal neural network that processes visual and text inputs directly into executable robotic control commands and physical world actions.
Analogy: Like a human brain interpreting what the eyes see and what the ears hear, instantly translating it into precise instructions for the hands to move.
Related: Multimodal, Agent
Splitting an image into fixed-size patches, embedding each as a token, and running ordinary self-attention over the resulting sequence instead of convolution.
Related: Patch Embeddings, Hierarchical Vision Transformers
Read the full conceptAn open-source, ultra-high-throughput inference engine specifically designed for massive language models, utilizing novel memory management techniques.
Analogy: Like swapping a slow, single-lane toll booth for an automated, multi-lane express toll system to process thousands of cars instantly.
Related: PagedAttention, Continuous Batching
An AI system that interacts with users entirely through spoken natural language, processing audio input and generating synthesized speech output.
Analogy: Like a highly intelligent telephone operator who can hear your question, understand the nuance in your tone, and speak back the correct answer.
Related: Multimodal, Pipeline
A cryptographic technique that subtly alters the probability distribution of an LLM's output to invisibly embed a detectable signature in generated text.
Analogy: Like a printer secretly adding microscopic yellow dots to every page it prints so the manufacturer can trace the origin of the document later.
Related: Alignment
A machine learning approach that rapidly generates vast amounts of noisy, lower-quality training labels using heuristics rather than human annotators.
Analogy: Like a teacher quickly grading 1,000 math tests by just glancing to see if the final number is even, rather than checking the complex work.
Related: Data Augmentation, Semi-Supervised Learning
A crucial regularization technique built directly into optimization algorithms that penalizes large parameter values by shrinking them slightly every step.
Analogy: Like a gardener continuously pruning the wildest, fastest-growing branches of a bush to ensure it maintains a balanced and healthy shape.
Related: Regularization, Adam / AdamW
The scheme for picking a network's starting weights, chosen so activations and gradients keep a stable scale across layers instead of collapsing or exploding.
Related: Vanishing Gradients, Exploding Gradients, Universal Approximation
Read the full conceptA foundational NLP technique that maps individual words into a dense mathematical vector space where geometric distance represents semantic similarity.
Analogy: Like placing cities on a map; New York and Boston are placed geographically close together, while Tokyo is mapped far away.
Related: Word2Vec, Embedding Model
A pioneering neural network model developed by Google in 2013 that revolutionized NLP by efficiently learning high-quality word embeddings from text.
Analogy: Like a detective figuring out the true identity of a masked person strictly by observing exactly who they choose to hang out with.
Related: Word Embedding, Self-Supervised Learning
An AI architecture that builds a complex internal representation of the physical laws, dynamics, and causality of its environment to predict the future.
Analogy: Like a basketball player instantly calculating exactly where the ball will bounce based on their deep intuitive understanding of gravity and physics.
Related: Reinforcement Learning, Multimodal
A mathematical formula for setting the initial random weights of a neural network to ensure signal variance remains perfectly constant across all layers.
Analogy: Like tuning every instrument in an orchestra to the exact same base volume before the concert starts so no section drowns out the others.
Related: Weight Initialization, Vanishing Gradients
Extreme Gradient Boosting is an incredibly powerful, highly optimized ensemble learning algorithm that builds a strong model from a series of weak trees.
Analogy: Like a team of students taking a test where each new student focuses exclusively on fixing the exact mistakes made by the previous student.
Related: Random Forest
A statistical data preprocessing technique that normalizes skewed features into a more Gaussian (bell-shaped) distribution, securely handling zero and negati...
Analogy: Like reshaping a lopsided, squished piece of clay into a perfectly symmetric sphere so it rolls smoothly down a hill.
Related: Standardization, Feature
You Only Look Once is a groundbreaking family of computer vision models that performs real-time object detection in a single forward pass of the network.
Analogy: Like a security guard scanning a crowd and instantly pointing out every suspicious person simultaneously, rather than staring at individuals one by one.
Related: Object Detection
A standard data scaling technique that centers a feature's distribution around a mean of exactly zero with a standard deviation of exactly one.
Analogy: Like converting everyone's test scores from raw points into a percentage above or below the class average so you can compare them fairly.
Related: Standardization, Standard Deviation
The capability of a machine learning model to correctly perform a complex task without ever being explicitly trained on labeled examples of that task.
Analogy: Like a chef who has never seen or cooked a specific exotic dish before, but perfectly creates it just by reading a quick description.
Related: Few-Shot Learning, Foundation Model
Zero Redundancy Optimizer is an advanced memory optimization technique that massively partitions model states across multiple GPUs to enable training giants.
Analogy: Like tearing a massive textbook into chapters and giving one chapter to each student to hold, rather than forcing every student to carry the whole book.