Paper breakdown

Word2Vec

Introduced Word2Vec, a highly efficient method for learning dense word embeddings that captured semantic meaning and analogical relationships.

Paper: Efficient Estimation of Word Representations in Vector Space

Authors: Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean · 2013

Read the paper

Word2Vec learns embeddings by either predicting a target word from its context (CBOW) or predicting the context from a target word (Skip-gram).

The Problem

Before 2013, words were typically represented as 'one-hot encodings' (a massive vector of zeros with a single '1' for the specific word). This representation was sparse, extremely high-dimensional, and most importantly, it captured absolutely zero semantic relationship between words (e.g., the vectors for 'cat' and 'dog' were mathematically as far apart as 'cat' and 'refrigerator').

The Idea

The authors applied the 'distributional hypothesis': words that appear in the same contexts share semantic meaning. They designed a simple, shallow neural network tasked with predicting a word based on its neighbors (or vice versa). The goal wasn't the prediction itself; the goal was to extract the internal weights the network learned along the way. These weights became the dense 'word embeddings'.

How It Works

The paper proposed two architectures:

Continuous Bag-of-Words (CBOW): The model looks at a window of surrounding context words (e.g., 'The cat sat on the _') and tries to predict the missing center word ('mat').
Skip-gram: The model takes a single center word ('sat') and tries to predict the surrounding context words ('The', 'cat', 'on', 'the', 'mat').

By training this simple network on massive amounts of Google News text, it learned 300-dimensional continuous vectors for every word. Because the network had no hidden non-linear layers, it was incredibly computationally efficient.

Why It Mattered

Word2Vec revolutionized Natural Language Processing. The resulting embeddings possessed mind-blowing mathematical properties. They captured analogies via simple vector arithmetic: vector('King') - vector('Man') + vector('Woman') resulted in a vector remarkably close to vector('Queen').

Suddenly, NLP models could understand synonyms, analogies, and semantic groupings right out of the box, drastically improving performance on every downstream NLP task.

What Came After

Word2Vec killed one-hot encodings. It was quickly followed by GloVe (Global Vectors) and fastText (which handled subwords). While modern LLMs (like BERT and GPT) now learn dynamic, context-dependent embeddings instead of static Word2Vec dictionaries, the fundamental concept of mapping discrete tokens to dense continuous vectors remains the foundation of all modern AI language models.