FastText
FastText spells each word from overlapping letter chunks, so jump and jumping share most of their parts and a typo still looks familiar.
Why Does This Exist?
Word2Vec treats each word as one atom. Jump, jumps and jumping get three independent vectors, and the shared root is learned three times. Worse, a typo like appple or a new compound has no vector at all, so production code hits a missing key.
GloVe has the same blind spot: it learns from co-occurrence counts of whole words, never looking inside the spelling. FastText looks inside. It learns vectors for letter chunks and builds each word from its chunks, so morphological cousins share most of their parts and unseen words still get a sensible sum.
This page covers the chunking rule, the sum, and the typo case. Whole word training lives in Word2Vec. Count based vectors live in GloVe. Phrase and sentence vectors live in sentence embeddings.
Think of It Like This
Lego bricks that spell words
Imagine each word arrives as a Lego model built from overlapping three stud bricks. Apple uses five bricks, and jumping shares long stretches of bricks with jump and jumped. A typo swaps one brick for a wrong one, but four bricks still match, so you recognize the model at a glance.
That maps directly: bricks are character trigrams, the finished model is the word vector, and shared bricks are the shared morphology that lets cousins sit near each other.
Where it stops: Lego bricks snap in fixed shapes with no meaning of their own. Letter chunks gain meaning only from training across many words, and order beyond three letters still needs the full word token to disambiguate.
How It Actually Works
Call a word, meaning one vocabulary entry like apple. Pad it to , meaning the same letters with start and end marks so edge chunks differ from middle ones. For , slide a three letter window across the padded form. Each window is one character trigram, and the model learns one vector per trigram plus one for the whole word.
For , which is seven characters long, the five trigrams are , , , and . The word vector is the plain sum of its part vectors:
An unseen typo like appple pads to eight characters and yields six trigrams, four of which overlap the apple set. Summing the known part vectors gives a vector near apple with no retraining and no missing key.
Worked example: toy two dimensional part vectors
Give each apple trigram a small 2D vector. Use for the first axis and for the second:
Add the parts:
Add the parts:
So . The typo appple keeps , , , and in large part, so its sum lands within a small distance of instead of missing. That shared sum is the diagram: five chunk boxes feeding one word box labeled 0.7, 0.7.
Why the whole word token stays
Pure chunks confuse anagrams and short words, so FastText also learns the full token and adds it to the sum. Frequent words thus keep an identity beyond their parts, while rare words lean on the shared chunks.
Code
The point is the check: split apple into trigrams by hand positions and confirm the vector sum matches 0.7, 0.7.
word: str = "<apple>"tri_x: float = 0.2 + 0.3 - 0.1 + 0.1 + 0.0 + 0.2tri_y: float = 0.1 - 0.1 + 0.4 + 0.2 + 0.1 + 0.0
print(word[0:3], word[1:4], word[2:5], word[3:6], word[4:7]) # -> <ap app ppl ple le>print(f"{tri_x:.1f} {tri_y:.1f}") # -> 0.7 0.7The first line prints the five slices at positions 0 through 4, exactly the trigram set from the worked example. The second line prints the summed toy vector, matching the hand addition. Change apple to appple and most slices still match, which is why the typo gets a vector.
Watch Out For
Holding every n-gram table in memory
FastText learns a vector for each distinct 3 to 6 letter chunk, so the embedding table dwarfs a whole word table on large corpora.
Symptom: training RAM climbs several times past a Word2Vec run on the same data, mostly from rare chunks seen once. Fix: hash the chunk space to a fixed bucket count, raise minimum chunk frequency, and keep the full table only when morphology like German compounds or Turkish suffixes pays for it.
Expecting one vector to know context
FastText assigns apple the same vector in apple pie and Apple stock, because the sum never sees neighboring words.
Symptom: the word bank sits midway between river and money senses and nearest neighbors mix both meanings. Fix: use FastText for morphology and typos, then move to sentence embeddings or a contextual model when the surrounding sentence changes the meaning.
The Quick Version
- FastText pads apple to seven characters and splits it into five trigrams:
<ap,app,ppl,pleandle>. - The word vector is the sum of its trigram vectors plus the whole word token.
- Our toy trigram vectors sum to 0.7 on x and 0.7 on y, so apple is [0.7, 0.7].
- A typo like appple shares most trigrams, so its sum lands near apple instead of missing.
- Chunk tables cost far more memory than whole word tables, and vectors stay blind to surrounding context.