AI Notes

Text Preprocessing - Cleaning & Preparation

Complete preprocessing guide. Normalization, stop words, stemming. NLP 2024.

Why Preprocess Text?

Raw text is messy: it contains HTML tags, special characters, inconsistent casing, abbreviations, misspellings, and irrelevant tokens. Before any NLP model can effectively learn patterns, this noise must be reduced. Text preprocessing transforms raw, unstructured text into a clean, standardized format suitable for analysis.

Preprocessing is not one-size-fits-all. The steps you apply depend on your task, domain, and model. For a modern transformer, minimal preprocessing is best (the model handles complexity). For classical bag-of-words models, aggressive preprocessing is essential. Understanding each step's purpose helps you make informed decisions about your pipeline.

The Preprocessing Pipeline

Example

Raw Text ↓ HTML/markup removal ↓ Unicode normalization ↓ Lowercasing ↓ Tokenization ↓ Stop word removal ↓ Punctuation removal ↓ Number handling ↓ Stemming/Lemmatization ↓ Noise removal (URLs, emails, @mentions) Clean Tokens

Step-by-Step Preprocessing

1. Text Cleaning

Example

# Remove HTML tags import re text = "<p>Check out <a href='url'>this link</a>!</p>" clean = re.sub(r'<[^>]+>', '', text) # Result: "Check out this link!" # Remove URLs text = "Visit https://example.com for more info" clean = re.sub(r'https?://\S+', '', text) # Result: "Visit for more info" # Remove special characters (keep alphanumeric + spaces) clean = re.sub(r'[^a-zA-Z0-9\s]', '', text) # Unicode normalization (curly quotes → straight, accents) import unicodedata text = unicodedata.normalize('NFKD', text)

2. Tokenization

Breaking text into individual tokens (words, subwords, or characters):

Whitespace tokenization

"I can't believe it!" → ["I", "can't", "believe", "it!"]

Problem: "can't" and "it!" aren't clean tokens

Word tokenization (NLTK)

"I can't believe it!" → ["I", "ca", "n't", "believe", "it", "!"]

Handles contractions and punctuation

Sentence tokenization

"Dr. Smith went home. He was tired." →

["Dr. Smith went home.", "He was tired."]

Must handle abbreviations (Dr., Mr., etc.)

Subword tokenization (BPE/WordPiece)

"unhappiness" → ["un", "##happi", "##ness"]

No out-of-vocabulary problem

3. Lowercasing

Example

"Apple" → "apple" "URGENT: Meeting NOW" → "urgent: meeting now" When to lowercase: ✓ Sentiment analysis (case rarely matters) ✓ Topic classification ✓ Search/information retrieval When NOT to lowercase: ✗ Named Entity Recognition ("Apple" company vs "apple" fruit) ✗ Machine Translation (German nouns capitalized) ✗ Sentiment emphasis ("LOVE" vs "love")

4. Stop Word Removal

Stop words	extremely common words with little semantic content
English	"the", "is", "at", "which", "on", "a", "an", "and"
Before	["the", "cat", "sat", "on", "the", "mat"]
After	["cat", "sat", "mat"]
Common stop word lists	NLTK (179 words), spaCy, sklearn
✗ "To be or not to be"	"be" loses all meaning with removal
✗ Phrase search	"not good" vs "good" (negation lost!)
✗ Language models	need all words for grammar
✗ Transformers	pre-trained with stop words

5. Handling Numbers

Strategies

- Remove all: "born in 1879" → "born in"

- Replace with token: "born in 1879" → "born in <NUM>"

- Keep as-is: important for dates, quantities, financial data

- Normalize: "1,000,000" → "1000000"

Domain-dependent

Medical: "take 2 tablets" → numbers critical!

Movie reviews: numbers rarely matter

Financial: all numbers are essential

6. Handling Contractions

"can't"	"cannot"
"won't"	"will not"
"I'm"	"I am"
"they're"	"they are"
"it's"	"it is" (or "it has" — ambiguous!)

Task-Specific Preprocessing

For Sentiment Analysis

Keep	emoticons (:-) = positive), exclamation marks (!!!)
Keep	capitalization emphasis (AMAZING vs amazing)
Remove	URLs, @mentions, #hashtags (or extract as features)
Handle: negation scope ("not good"	"not_good")

For Topic Modeling (LDA)

Remove	stop words, numbers, short words (< 3 chars)
Apply	lemmatization (group word forms)
Remove	very rare words (< 5 occurrences)
Remove	very common words (> 80% of documents)

For Transformers (BERT, GPT)

Example

Minimal preprocessing: - Remove HTML/markup - Fix encoding issues - Truncate to max length (512 tokens for BERT) - DO NOT remove stop words - DO NOT stem/lemmatize - Let the tokenizer handle subwords The model was pre-trained on real text — match that format!

Practical Implementation

Example

import re from nltk.tokenize import word_tokenize from nltk.corpus import stopwords from nltk.stem import WordNetLemmatizer def preprocess_for_classification(text): # 1. Lowercase text = text.lower() # 2. Remove URLs text = re.sub(r'https?://\S+', '', text) # 3. Remove HTML text = re.sub(r'<[^>]+>', '', text) # 4. Remove special characters (keep letters and spaces) text = re.sub(r'[^a-z\s]', '', text) # 5. Tokenize tokens = word_tokenize(text) # 6. Remove stop words stop = set(stopwords.words('english')) tokens = [t for t in tokens if t not in stop and len(t) > 2] # 7. Lemmatize lemmatizer = WordNetLemmatizer() tokens = [lemmatizer.lemmatize(t) for t in tokens] return tokens # Example raw = "<p>I can't believe how AMAZING this product is! https://buy.it</p>" clean = preprocess_for_classification(raw) # Result: ['believe', 'amazing', 'product']

Common Mistakes

Example

1. Over-preprocessing for deep learning models BERT doesn't need stop word removal or stemming! 2. Removing negation words as stop words "not" is a stop word but critical for sentiment! 3. Case-folding before NER "Apple Inc." → "apple inc." loses entity signal 4. Applying English preprocessing to other languages Chinese has no spaces! Arabic reads right-to-left! 5. Preprocessing training and test data differently Must apply identical pipeline to both

Interview Questions

Q: What preprocessing would you apply for a BERT-based classifier? A: Minimal: remove HTML, fix encoding, maybe lowercase (depends on task). Let the BERT tokenizer handle subwords. Do NOT remove stop words or punctuation—BERT was trained with them and uses them for understanding. Truncate to 512 tokens if needed.

Q: Why might removing stop words hurt a model? A: Stop words carry grammatical and semantic information. "Not good" without "not" becomes "good"—opposite meaning. "The dog bit the man" vs "the man bit the dog"—word order (stop words help) changes meaning entirely. Modern models leverage these words for understanding.

Q: How do you preprocess text in a language you don't speak? A: Use language-specific tokenizers (spaCy supports 60+ languages, Stanza). Don't apply English rules (no English stop word list). Use Unicode normalization. For Chinese/Japanese, use segment-based tokenizers (jieba, MeCab). Multilingual models (mBERT) handle most languages with their built-in tokenizers.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Text Preprocessing - Cleaning & Preparation.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.

Search Terms

artificial-intelligence, artificial intelligence, artificial, intelligence, natural, language, processing, text

Continue learning this concept

Natural Language ProcessingConversational AI - ChatbotsMaster conversational AI. Dialogue systems, intent recognition. NLP 2024.Natural Language ProcessingMachine Translation - Neural MTComplete translation guide. Seq2seq, attention, transformers. NLP advanced 2024.Natural Language ProcessingNER - Information ExtractionMaster NER. Entity tagging, BIO, CRF models. NLP 2024.Natural Language ProcessingNLP Fundamentals - Complete GuideMaster NLP basics. Text processing, tokenization, embeddings. BTech guide 2024.Natural Language ProcessingSentiment Analysis - Opinion MiningComplete sentiment guide. Polarity classification, neural approaches. NLP 2024.Natural Language ProcessingStemming & Lemmatization - NormalizationMaster stemming and lemmatization. Porter stemmer, WordNet. NLP 2024.

PreviousStemming & Lemmatization - Normalization NextTokenization - Text Processing

AI Notes

Text Preprocessing - Cleaning & Preparation

Complete preprocessing guide. Normalization, stop words, stemming. NLP 2024.

Why Preprocess Text?

The Preprocessing Pipeline

Example

Step-by-Step Preprocessing

1. Text Cleaning

Example

2. Tokenization

Breaking text into individual tokens (words, subwords, or characters):

Whitespace tokenization

"I can't believe it!" → ["I", "can't", "believe", "it!"]

Problem: "can't" and "it!" aren't clean tokens

Word tokenization (NLTK)

"I can't believe it!" → ["I", "ca", "n't", "believe", "it", "!"]

Handles contractions and punctuation

Sentence tokenization

"Dr. Smith went home. He was tired." →

["Dr. Smith went home.", "He was tired."]

Must handle abbreviations (Dr., Mr., etc.)

Subword tokenization (BPE/WordPiece)

"unhappiness" → ["un", "##happi", "##ness"]

No out-of-vocabulary problem

3. Lowercasing

Example

4. Stop Word Removal

Stop words	extremely common words with little semantic content
English	"the", "is", "at", "which", "on", "a", "an", "and"
Before	["the", "cat", "sat", "on", "the", "mat"]
After	["cat", "sat", "mat"]
Common stop word lists	NLTK (179 words), spaCy, sklearn
✗ "To be or not to be"	"be" loses all meaning with removal
✗ Phrase search	"not good" vs "good" (negation lost!)
✗ Language models	need all words for grammar
✗ Transformers	pre-trained with stop words

5. Handling Numbers

Strategies

- Remove all: "born in 1879" → "born in"

- Replace with token: "born in 1879" → "born in <NUM>"

- Keep as-is: important for dates, quantities, financial data

- Normalize: "1,000,000" → "1000000"

Domain-dependent

Medical: "take 2 tablets" → numbers critical!

Movie reviews: numbers rarely matter

Financial: all numbers are essential

6. Handling Contractions

"can't"	"cannot"
"won't"	"will not"
"I'm"	"I am"
"they're"	"they are"
"it's"	"it is" (or "it has" — ambiguous!)

Task-Specific Preprocessing

For Sentiment Analysis

Keep	emoticons (:-) = positive), exclamation marks (!!!)
Keep	capitalization emphasis (AMAZING vs amazing)
Remove	URLs, @mentions, #hashtags (or extract as features)
Handle: negation scope ("not good"	"not_good")

For Topic Modeling (LDA)

Remove	stop words, numbers, short words (< 3 chars)
Apply	lemmatization (group word forms)
Remove	very rare words (< 5 occurrences)
Remove	very common words (> 80% of documents)

For Transformers (BERT, GPT)

Example

Practical Implementation

Example

Common Mistakes

Example

Interview Questions

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Text Preprocessing - Cleaning & Preparation.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.

Search Terms

artificial-intelligence, artificial intelligence, artificial, intelligence, natural, language, processing, text

Continue learning this concept

PreviousStemming & Lemmatization - Normalization NextTokenization - Text Processing