Skip to content
AI360Xpert
Core ML

FastText

An extension of Word2Vec that represents words as bags of character sub-words, allowing it to understand morphology and generate vectors for words it has never seen before.

FastText breaks words down into overlapping character n-grams, combining their vectors to form the final word embedding.
FastText breaks words down into overlapping character n-grams, combining their vectors to form the final word embedding.

Why Does This Exist?

Word2Vec and GloVe have two critical flaws.

First, they treat every word as an atomic unit. To Word2Vec, the words "jump", "jumps", and "jumping" are three completely independent vectors. It does not understand that they share the root "jump". This forces the model to waste parameters learning the same semantic concept three different times.

Second, they crash on Out-Of-Vocabulary (OOV) words. If a Word2Vec model encounters a typo like "awesomme" or a novel compound word in production, it throws an error because it has no vector for it.

FastText (created by Facebook AI in 2016) solves both problems by looking inside the word.

Think of It Like This

Think of It Like This

Imagine you are learning a foreign language, like German. You encounter the word Krankenhaus (hospital) for the very first time.

If you are a Word2Vec model, you throw your hands up. You have never seen this exact string of letters before, so it means nothing to you.

But if you are a human, you look at the sub-components. You know that Kranken means "sick" and Haus means "house". By combining the meaning of the sub-words, you can guess that a "sick house" is a hospital. FastText uses this exact logic, breaking words into overlapping chunks of characters to infer the meaning of novel words.

How It Actually Works

FastText is built directly on top of the Word2Vec Skip-gram architecture, but instead of learning a vector for the whole word, it learns vectors for character n-grams (sub-words).

1. Breaking words into n-grams

Given the word "apple", FastText adds boundary brackets < and > so the model knows where the word starts and ends. It then extracts all character sequences of length nn (usually n=3n=3 to n=6n=6).

For n=3n=3, <apple> becomes: <ap, app, ppl, ple, le>

It also includes the whole word <apple> as a special token.

2. The Vector Arithmetic

During training, FastText assigns a vector to every one of those 3-letter chunks. To get the final vector for the word "apple", it simply adds the vectors of its constituent chunks together.

apple⃗=<ap⃗+app⃗+ppl⃗+ple⃗+le>⃗+<apple>⃗\vec{\text{apple}} = \vec{\text{<ap}} + \vec{\text{app}} + \vec{\text{ppl}} + \vec{\text{ple}} + \vec{\text{le>}} + \vec{\text{<apple>}}

3. Solving Out-Of-Vocabulary

If FastText encounters the typo "appple" in production, it doesn't crash. It breaks the novel word into <ap, app, ppp, ppl, ple, le>. Because the model has vectors for almost all of those individual sub-words from training, it adds them together and generates a highly accurate vector that sits very close to "apple" in the vector space.

Show Me the Code

In Python, the gensim library provides a FastText implementation with the exact same API as Word2Vec.

from gensim.models import FastText
# Toy corpussentences = [    ["the", "cat", "is", "jumping"],    ["the", "dog", "is", "running"]]
# Train FastText (min_n and max_n define the character sub-word lengths)model = FastText(sentences, vector_size=10, min_n=3, max_n=6, min_count=1)
# Get the vector for a word in the vocabularyprint("Vector for 'jumping':\n", model.wv['jumping'])
# Get a vector for a word the model has NEVER seen# (Word2Vec would throw a KeyError here)print("\nVector for 'jumped' (OOV):")print(model.wv['jumped'])
# Check semantic similarityprint("\nSimilarity between jumping and jumped:")print(model.wv.similarity('jumping', 'jumped'))# -> 0.82 (They are correctly identified as highly similar!)

Watch Out For

Watch Out For

Massive memory consumption. Because FastText learns a vector for every possible 3, 4, 5, and 6-letter sequence across the entire corpus, its vocabulary size (and therefore RAM usage) is significantly larger than Word2Vec.

Watch Out For

Not context-aware. Like GloVe and Word2Vec, FastText is still a static embedding model. It generates the same vector for the word "bank" regardless of the surrounding sentence. Modern NLP relies on transformers for context-dependent embeddings.

The Quick Version

  • Word2Vec and GloVe treat words as atomic units, failing on typos, unseen words, and morphological variations.
  • FastText represents every word as a sum of its character n-grams (sub-words).
  • If it encounters an Out-Of-Vocabulary word in production, it can dynamically generate a vector for it by summing the vectors of its sub-words.
  • It is highly effective for morphologically rich languages (like German or Turkish) where compound words and suffixes are common.

Related concepts