Deduplication
The process of identifying and removing exact or near-exact duplicate examples from a dataset to prevent a model from memorizing repeated training sequences.
Think of It Like This
Like going through your photo gallery and deleting all the identical burst-mode shots so you only keep one copy of each unique moment.
In the context of large language models, deduplication is a critical data preprocessing step that significantly improves generalization. When a network sees the same text hundreds of times, it is prone to regurgitating that exact string rather than learning underlying linguistic patterns. Removing duplicates reduces training time and boosts quality.