Language Detection

Language detection reads the statistical footprint of a text and names its language, so Spanish input reaches the Spanish pipeline instead of English.

Trigram footprints score distance 6 to Spanish and 30 to English, so the input routes to the Spanish pipeline.

Why Does This Exist?

Support tickets arrive in Spanish, comments in Hindi, reviews in Japanese, and none of them carry a reliable language tag. Feed Spanish into an English sentiment model and you get confident noise. Every multilingual product hits this wall on day one: before any real analysis, it must ask what language it is looking at.

Language detection answers that question and routes. It sends each text to the right tokenizer, the right tagger, the right translation engine. Without this triage step, multilingual systems misfire silently, because the downstream models assume the language they were built for.

This page covers the identification step itself: footprints, profiles, and the distance that picks a winner. The method is classification, so the parent is text classification: one text in, one label out. The full chain this step opens belongs to the NLP pipeline. The one prerequisite is small: a character n-gram is a sliding window of $n$ characters, so "canta" gives "can", "ant", "nta" at $n = 3$ . Everything else builds on counting those windows.

Think of It Like This

A mail sorting facility

Millions of unaddressed letters flood a sorting hall. Nobody reads them fully. A scanner glances at letter shapes: many words ending in "cion" go left to the Spanish team, dense clusters of "the" and "ing" go right to the English team. Meaning never enters the decision. Shape routes the mail.

Detection works the same way. It never parses the sentence. It counts character windows, compares the footprint to stored language shapes, and drops the text in the right queue. Fast, shallow, and almost always right on long enough input.

Where it stops: a postcard saying only "no" could be English, Spanish, or Italian, and no scanner can tell. Shape needs enough ink. Below a dozen characters the footprints overlap and the router must admit it does not know.

How It Actually Works

Detection is classification over footprints. Each language owns a stored profile, the input grows its own, and the closest profile wins.

Character trigram profiles

Training text in each language is sliced into overlapping character trigrams, and the most frequent ones form that language's profile with their ranks. Spanish profiles rank " la", "cio", "ion" family windows near the top; English profiles rank " th", "he ", "the" first and bury the Spanish windows deep. Whole words are never compared, which keeps the method independent of tokenizers that differ per language.

The distance rule

Let $g$ be a trigram, $r(g)$ its rank in the input profile, and $q(g)$ its rank in a language profile. The out-of-place distance is:

D = \sum_g |r(g) - q(g)|

A trigram missing from a language profile takes a penalty rank instead of zero, so absent evidence counts against the language rather than silently passing. The language with the smallest $D$ wins, usually with a confidence gap the pipeline can threshold.

Worked example with real numbers

Input: "la niña canta una canción". Its top four trigrams by first appearance rank as " la" at $0$ , "a c" at $1$ , "ció" at $2$ , "ión" at $3$ .

Spanish profile ranks: " la" at $1$ , "a c" at $6$ , "ció" at $2$ , "ión" at $3$ . Distance:

D_{\text{es}} = |0-1| + |1-6| + |2-2| + |3-3| = 1 + 5 + 0 + 0 = 6

English profile ranks: " la" at $7$ , "a c" at $8$ , "ció" at $10$ , "ión" at $11$ . Distance:

D_{\text{en}} = |0-7| + |1-8| + |2-10| + |3-11| = 7 + 7 + 8 + 8 = 30

Six against thirty is no contest. The text routes to Spanish, which is exactly the verdict the diagram shows.

What neural models change

Shallow nets like fastText keep the same framing and swap counting for learning: trigram embeddings average into one document vector, then a hierarchical softmax predicts the label in milliseconds across hundreds of languages. Profiles become weights, but the decision is still closest footprint wins.

Code

This computes the out-of-place distance from the worked example with rank tables, using only the standard library.

1def out_of_place(input_ranks: dict[str, int], lang_ranks: dict[str, int], missing: int = 12) -> int:2    return sum(abs(r - lang_ranks.get(g, missing)) for g, r in input_ranks.items())3inp = {" la": 0, "a c": 1, "ció": 2, "ión": 3}4es = {" la": 1, "a c": 6, "ció": 2, "ión": 3}5en = {" la": 7, "a c": 8, "ció": 10, "ión": 11}6print(out_of_place(inp, es))  # -> 67print(out_of_place(inp, en))  # -> 308print("route to Spanish")  # -> route to Spanish

Watch Out For

Overconfidence on very short inputs

A single word like "no" lives in English, Spanish, and Italian, yet the classifier still returns one label with a straight face. The symptom is confident wrong routes on titles, names, and acronyms. Fix it with a minimum length around 15 to 20 characters plus a confidence floor: below either, label the text unknown and ask or default, never guess.

Forcing one label onto mixed text

Code-switched sentences ("voy al store to buy leche") get exactly one tag from a document classifier, usually the dominant language, and the other half misroutes. The symptom is half-right downstream output on bilingual traffic. Fix it with segment-level detection that tags spans, not documents, when your traffic actually mixes languages mid-sentence.

The Quick Version

Language detection names the language of a text so each input reaches the pipeline built for it.
The classic method slices text into character trigrams, builds a ranked profile, and picks the closest stored language by out-of-place distance.
In the worked example, the input scores distance 6 to Spanish against 30 to English and routes to Spanish.
Neural models keep the same decision and learn the profiles as embeddings for speed across hundreds of languages.
Short strings and mixed-language sentences need thresholds and segment tagging, not a bolder single guess.