AI Notes
Master stemming and lemmatization. Porter stemmer, WordNet. NLP 2024.
Why Normalize Word Forms?
In natural language, a single concept appears in many surface forms: "running," "runs," "ran," "runner" all relate to the concept of "run." Without normalization, a search for "running" would miss documents containing "runs," and a classifier would treat each form as a completely different feature. Text normalization reduces this vocabulary explosion by mapping related word forms to a common base.
Two primary approaches exist: stemming (crude affix stripping) and lemmatization (linguistically informed base-form reduction). Understanding when to use each—and their trade-offs—is essential for NLP preprocessing pipelines.
Stemming: Rule-Based Affix Removal
What is Stemming?
Stemming applies heuristic rules to strip suffixes (and sometimes prefixes) from words, producing a "stem" that may not be a real word but groups related forms together.
| "running" | "run" |
| "computation" | "comput" (not a real word, but consistent) |
| "computational" | "comput" (same stem — grouped!) |
| "computing" | "comput" |
| "studies" | "studi" |
| "studying" | "studi" |
Porter Stemmer (Most Famous)
The Porter stemmer applies 5 phases of suffix-stripping rules:
| Phase 1 | Plurals and past participles |
| SSES | SS: "caresses" → "caress" |
| IES | I: "ponies" → "poni" |
| S | ∅: "cats" → "cat" |
| (SS | SS: "caress" → "caress" — don't strip) |
| Phase 2 | Derivational suffixes |
| ATIONAL | ATE: "relational" → "relate" |
| TION | T: "adoption" → "adopt" (wait — actually ATION→ATE) |
| IVENESS | IVE: "effectiveness" → "effective" |
| IZER | IZE: "digitizer" → "digitize" |
| Phase 3 | More derivational |
| ICATE | IC: "triplicate" → "triplic" |
| ATIVE | ∅: "formative" → "form" |
| ALIZE | AL: "formalize" → "formal" |
| Phase 4 | Remove further suffixes |
| AL | ∅: "revival" → "reviv" |
| ENCE | ∅: "inference" → "infer" |
| MENT | ∅: "adjustment" → "adjust" |
| Phase 5 | Clean up |
| E | ∅ (if stem > 1 syllable): "probate" → "probat" |
| LL | L: "controlling" → "controll" → "control" |
Worked Example: Porter Stemmer Trace
| Word | "generalization" |
| Phase 1: No plural/past participle rule applies | "generalization" |
| Phase 2: IZATION | IZE: "generalize" → wait, IZATION → IZE |
| Actually: IZATION | IZE: "generalization" → "generalize" |
| Phase 3: Check further... ALIZE | AL: "generalize" → "general" |
| Phase 4: AL | ∅ (if long enough): "general" → "gener" |
| Result | "gener" |
| Word | "running" |
| Phase 1: ING | ∅ (if stem has vowel): "runn" has vowel → "runn" |
| Then: double consonant rule: "runn" | "run" |
| Result | "run" |
Other Stemmers
| Snowball Stemmer | Improved Porter, supports multiple languages |
| Lancaster Stemmer | More aggressive stripping |
| "presumably" | "presum" (Porter) vs "pres" (Lancaster) |
| Lovins Stemmer | Single-pass, longest-match suffix removal |
Lemmatization: Linguistically Informed Normalization
What is Lemmatization?
Lemmatization reduces words to their dictionary form (lemma) using vocabulary lookup and morphological analysis. Unlike stemming, it always produces valid words and considers part-of-speech.
| "running" | "run" (verb) |
| "better" | "good" (adjective) — irregular form! |
| "mice" | "mouse" (noun) — irregular plural! |
| "was" | "be" (verb) — suppletive form! |
| "studies" | "study" (noun or verb) |
WordNet Lemmatizer (NLTK)
| lemmatize("better", pos='a') | "good" (adjective) |
| lemmatize("better", pos='v') | "better" (verb: "to better oneself") |
| lemmatize("saw", pos='v') | "see" |
| lemmatize("saw", pos='n') | "saw" (a cutting tool) |
| lemmatize("running", pos='v') | "run" |
| lemmatize("running", pos='n') | "running" (the activity) |
SpaCy Lemmatizer
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The children were running quickly to their better hiding spots")
for token in doc:
print(f"{token.text:12} → {token.lemma_:12} (POS: {token.pos_})")
# Output:
# The → the (DET)
# children → child (NOUN) — irregular plural
# were → be (AUX) — irregular verb
# running → run (VERB)
# quickly → quickly (ADV)
# to → to (ADP)
# their → their (PRON)
# better → good (ADJ) — comparative
# hiding → hide (VERB)
# spots → spot (NOUN)Stemming vs. Lemmatization Comparison
| Aspect | Stemming | Lemmatization |
|---|---|---|
| Output | May not be a real word | Always a dictionary word |
| Speed | Very fast (rules only) | Slower (dictionary lookup) |
| Accuracy | Lower (over/under-stems) | Higher (linguistically correct) |
| POS needed | No | Yes (for ambiguous words) |
| "better" | "better" or "bett" | "good" (adj) or "better" (verb) |
| "mice" | "mice" (can't handle) | "mouse" |
| "was/were/is" | Different stems! | All → "be" |
| Use case | Search/IR, fast prototyping | NLP pipelines, analysis |
Common Errors
Over-Stemming (False Positives)
| "universe" | "univers" |
| "university" | "univers" |
| "wander" | "wand" |
| "wand" | "wand" |
Under-Stemming (False Negatives)
| "absorb" | "absorb" |
| "absorption" | "absorpt" |
| "alumnus" | "alumnus" |
| "alumni" | "alumni" |
When to Use Which
Use Stemming When:
Use Lemmatization When:
Impact on NLP Tasks
Vocabulary reduction
Raw vocabulary: 50,000 unique words
After stemming: ~30,000 stems (40% reduction)
After lemmatization: ~35,000 lemmas (30% reduction)
Search engine example
Query: "running shoes"
Without normalization: matches only "running shoes"
With stemming: matches "run shoes", "runner shoe", "runs shoe"
With lemmatization: matches "run shoe" forms
Sentiment analysis
"better" → "good" (lemma) connects to positive lexicon
"worse" → "bad" (lemma) connects to negative lexicon
Stemming misses these: "better"→"better", "worse"→"wors"
Modern Approaches: Subword Tokenization
| "unhappiness" | ["un", "##happiness"] or ["un", "happi", "ness"] |
| BPE (Byte-Pair Encoding) | learns frequent character sequences |
| WordPiece (BERT) | similar to BPE, likelihood-based |
| SentencePiece | language-agnostic, works on raw text |
Interview Questions
Q: Why might stemming hurt a machine learning model? A: Over-stemming merges unrelated words ("wander"/"wand" → same stem), introducing noise. The model receives false similarity signals. For modern embeddings (Word2Vec, BERT), stemming is usually unnecessary—the model learns morphological relationships automatically.
Q: How does lemmatization handle ambiguous words? A: Through POS tag context. "Leaves" as a noun lemmatizes to "leaf" (tree leaves). "Leaves" as a verb lemmatizes to "leave" (she leaves). Without POS, lemmatizers typically default to the most common form or return the word unchanged.
Q: Do modern NLP systems still need stemming/lemmatization? A: For transformer-based models (BERT, GPT), typically no—subword tokenization handles morphology implicitly. But for classical ML pipelines (bag-of-words, TF-IDF), stemming/lemmatization remains important for dimensionality reduction and improving feature matching.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Stemming & Lemmatization - Normalization.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.
Search Terms
artificial-intelligence, artificial intelligence, artificial, intelligence, natural, language, processing, stemming
Related Artificial Intelligence Topics