Stemming vs Lemmatization in NLP: Difference With Examples
Stemming and lemmatization are two text normalization techniques that reduce related words to a shared base form; stemming cuts off suffixes with fixed rules and may output a non-word, while lemmatization uses a dictionary and grammar context to return a real word called a lemma. Both shrink the vocabulary a model has to handle, but they trade speed against linguistic accuracy. They usually run after tokenization in a natural language processing (NLP) pipeline.
What stemming does
Stemming applies a fixed list of rules to remove or shorten common prefixes and suffixes without checking whether the result is a real word. The output is called a stem, and several words can map to the same stem even if that stem does not appear in any dictionary.
A widely used example is the Porter stemmer, first published by Martin Porter in 1980. It runs an ordered cascade of rewrite rules, so the same process is applied to every word. Because the logic is rule-based and does not look up a dictionary, it is fast and predictable on large text collections.
The trade-off is visible on irregular words. A stemmer may leave better unchanged because no simple suffix rule connects it to good, and it can reduce universe to univers, which is not a valid English word.
What lemmatization does
Lemmatization aims to return the lemma, the canonical dictionary form of a word. To do that it considers the part of speech and the surrounding context, then checks a lexical resource such as WordNet. This lets it handle irregular forms that rules miss:
am,are, andisall reduce tobe.betterreduces togood.micereduces tomouse.
The output is a meaningful word, which helps tasks that depend on correct grammar and meaning, such as sentiment analysis or question answering. The cost is that lemmatization needs dictionary resources and more computation, so it is generally slower than stemming.
Side-by-side comparison
| Factor | Stemming | Lemmatization |
|---|---|---|
| Method | Rule-based affix removal | Dictionary and morphology lookup |
| Output | May not be a valid word | A real dictionary word (lemma) |
| Context awareness | Low; works on a single token | Uses part of speech and sentence context |
| Speed | Usually faster | Usually slower |
| Irregular forms | Often misses them | Handles common irregular forms |
| Typical use | Search indexing, large-scale preprocessing | Meaning-sensitive NLP tasks |
| Resources needed | A rule set | A lexical database such as WordNet |
Examples on the same words
| Original | Stemmed form | Lemmatized form |
|---|---|---|
| studies | studi | study |
| studying | studi | study |
| played | play | play |
| caring | care | care |
| mice | mice | mouse |
| better | better | good |
On regular words such as playing or played, the two methods often agree. The difference becomes clear on irregular words and on words whose base form depends on grammar, where the dictionary lookup gives a result that suffix rules cannot reach.
Where each fits in a pipeline
Neither technique replaces the other in every project; the choice depends on the task and the data volume.
- Search and information retrieval often favors stemming because speed matters across many documents, and grouping words by a shared stem can improve recall even when the stem itself is not a word.
- Chatbots, translation, and sentiment work often favor lemmatization because an incorrect base form can change meaning, and a readable lemma keeps later steps accurate.
- Large corpora with limited time may use stemming first, then add lemmatization only for the parts of the system where precision matters.
Languages with rich morphology present a harder case: a rule set designed for English may not transfer, and a dictionary for the target language may be limited. In such situations teams often test both methods on their own data rather than assume one is universally preferable.
How it relates to other NLP concepts
These normalization steps sit between tokenization, which splits text into units, and the word embeddings that represent those units numerically. Older word-level methods such as Word2Vec benefit when inflected forms are collapsed before training. The broader set of tasks and terminology is covered in the NLP overview and the machine learning glossary.
FAQ
Can a stem be a word that does not exist?
Yes. Stemming follows rules rather than a dictionary, so outputs such as studi or univers are common even though they are not dictionary words. Lemmatization is the method that returns a valid word.
Which is faster, stemming or lemmatization?
Stemming is usually faster because it applies a fixed rule set without dictionary lookups or part-of-speech analysis. Lemmatization does more linguistic work and takes more time, especially on large datasets.
Do large language models still use these techniques?
Modern language models mostly rely on subword tokenization and learned embeddings rather than classic stemmers, but stemming and lemmatization remain common in search systems, traditional NLP toolkits, and projects where a readable, explainable base form is useful.
Does lemmatization give more accurate results?
It is more accurate at returning a dictionary base form and handling irregular words, but it depends on having a lexical resource for the language. On a language without good dictionary coverage, a rule-based stemmer may be the more practical option.
What is over-stemming and under-stemming?
Over-stemming happens when two different words are reduced too aggressively to the same stem. Under-stemming happens when related words are not merged enough. Lemmatization reduces these errors by checking meaning and grammar, though it cannot remove them entirely.
Key takeaways
Stemming is a fast, rule-based way to merge words by chopping affixes, while lemmatization returns a dictionary lemma using grammar and context. Choose stemming for speed and search recall on large data, and lemmatization when meaning and irregular forms matter. Both reduce vocabulary size after tokenization and help downstream models treat related words as the same concept.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free