ML Notes
HubAI vs ML vs Deep Learning — Machine LearningApplications of Machine LearningHistory of Machine LearningMachine Learning RoadmapWhat is Machine Learning?Calculus for Machine LearningData Preprocessing BasicsLinear Algebra for Machine LearningProbability Basics for Machine LearningPython for Machine LearningStatistics Basics for Machine LearningBias vs Variance TradeoffModel Evaluation OverviewOverfitting and UnderfittingReinforcement Learning Fundamentals — Machine LearningSupervised LearningTraining vs Testing DataTypes of Machine LearningUnsupervised LearningData Cleaning — Machine LearningData Collection for Machine LearningData Visualization for MLEncoding Categorical DataFeature EngineeringFeature ScalingHandling Missing ValuesTrain-Test SplitElastic Net RegressionLasso Regression (L1 Regularization)Multiple Linear RegressionPolynomial RegressionIntroduction to RegressionRegression ProjectsRidge Regression (L2 Regularization)Simple Linear RegressionIntroduction to ClassificationClassification ProjectsDecision TreesK-Nearest Neighbors (KNN)Logistic RegressionNaive Bayes ClassifierRandom Forest ClassificationSupport Vector Machine (SVM)XGBoost for ClassificationClustering in Machine Learning: Complete Guide to Unsupervised LearningClustering Projects: Real-World Applications and Implementation GuidesDBSCAN ClusteringGaussian Mixture Models (GMM): Probabilistic Clustering for Advanced Data AnalysisHierarchical ClusteringK-Means ClusteringDimensionality Reduction: Comprehensive Guide to Feature Reduction TechniquesFeature Selection Techniques: Choosing Informative Variables for ML ModelsLinear Discriminant Analysis (LDA): Supervised Dimensionality ReductionPrincipal Component Analysis (PCA)t-SNE: Non-Linear Dimensionality Reduction for Data VisualizationUMAP: Uniform Manifold Approximation and Projection for Modern Data VisualizationAdaBoost: Adaptive Boosting Algorithm Explained with ExamplesBagging: Bootstrap Aggregating for Reducing Model VarianceBoosting: Sequential Ensemble Learning for Bias and Variance ReductionEnsemble Learning: Complete Guide to Combining ML Models for Better PredictionsGradient Boosting: Building Powerful ML Models Through Sequential OptimizationRandom Forest: Complete Guide to Tree Ensembles and Feature ImportanceStacking: Advanced Ensemble Method Using Meta-Learners for Superior ML PredictionsActivation Functions in Neural NetworksBackpropagation: Complete Algorithm for Training Neural NetworksIntroduction to Deep Learning: Neural Networks, Architectures, and ApplicationsDeep Learning Projects: Hands-On Applications and Case StudiesNeural Networks Basics: Perceptron, Architecture, and TrainingPerceptron: Foundational Single-Neuron Classifier for Binary ClassificationPyTorch Basics: Building and Training Neural NetworksTensorFlow Basics: Building and Training Neural NetworksIntroduction to Natural Language Processing: Fundamentals and ApplicationsNLP Projects: Practical Applications from Sentiment Analysis to Machine TranslationStemming and Lemmatization: Reducing Words to Root Forms in NLPText Preprocessing: Essential Steps for NLP ApplicationsTF-IDF: Term Frequency-Inverse Document Frequency for Text Feature ExtractionTokenization in NLP: Breaking Text into Meaningful UnitsTransformers: The Architecture Behind Modern NLP ModelsWord Embeddings: Representing Words as Vectors for NLPIntroduction to Computer VisionComputer Vision ProjectsImage Classification: Building and Deploying CNN ModelsImage Processing BasicsImage Segmentation — Machine LearningObject DetectionOpenCV BasicsAgents and Environments in RLDeep Q-Networks (DQN)Policy Gradient Methods — Machine LearningQ-Learning AlgorithmReinforcement Learning: IntroductionReinforcement Learning ProjectsAccuracy, Precision, Recall: Classification Metrics Explained for ML InterviewConfusion Matrix: Complete Guide to Classification EvaluationCross-Validation: Comprehensive Guide to Robust Model EvaluationF1 Score: Balanced Classification Metric for Imbalanced DataGrid Search: Complete Guide to Hyperparameter Tuning in Machine LearningHyperparameter Tuning: Complete Guide to Optimizing Machine Learning ModelsModel Selection: Choosing the Best Algorithm for Your ML ProblemROC and AUC: Complete Guide to Classification Threshold EvaluationBuild an AI Assistant with LLMsBuild a Conversational ChatbotCustomer Churn PredictionFace Recognition System — Machine LearningHouse Price PredictionRecommendation System ProjectSentiment Analysis Project — Machine LearningSpam Email ClassifierML Case Study Questions for InterviewsML Coding Questions for Technical InterviewsMachine Learning Interview CheatsheetEssential Machine Learning Interview QuestionsStatistics for Machine Learning InterviewsDatasets for ML PracticeImportant ML FormulasMachine Learning GlossaryRecommended ML Books and CoursesCI/CD for Machine LearningDocker for Machine LearningFastAPI for Machine Learning APIsFlask API for Machine LearningMLOps: IntroductionModel Deployment StrategiesML Model Monitoring & Observability
Complete guide to encoding categorical variables for ML including one-hot encoding, label encoding, target encoding, and handling high-cardinality features.
Machine learning algorithms work with numbers, not text. Categorical features like "color", "country", or "product_type" must be converted to numerical representations. The encoding method you choose can significantly impact model performance.
Types of Categorical Variables
| │ ├── Color | red, blue, green |
| │ ├── Country | USA, UK, India |
| │ └── Gender | male, female, other |
| ├── Education | high school < bachelor < master < PhD |
| ├── Size | S < M < L < XL |
| └── Rating | poor < average < good < excellent |
Label Encoding
Assigns integers to categories. Use ONLY for ordinal data or tree-based models.
Example
from sklearn.preprocessing import LabelEncoder, OrdinalEncoder
import pandas as pd
import numpy as np
df = pd.DataFrame({
'size': ['S', 'M', 'L', 'XL', 'M', 'S', 'L'],
'color': ['red', 'blue', 'green', 'red', 'blue', 'green', 'red'],
'price': [10, 20, 30, 40, 25, 15, 35]
})
# Ordinal encoding (preserves order)
size_order = [['S', 'M', 'L', 'XL']]
ordinal_enc = OrdinalEncoder(categories=size_order)
df['size_encoded'] = ordinal_enc.fit_transform(df[['size']])
print("Ordinal Encoding (size has natural order):")
print(df[['size', 'size_encoded']])
# Label encoding (arbitrary assignment)
le = LabelEncoder()
df['color_label'] = le.fit_transform(df['color'])
print("\nLabel Encoding (⚠️ implies false ordering!):")
print(df[['color', 'color_label']])
print("Problem: model might think green(1) > blue(0), which is meaningless!")
One-Hot Encoding
Creates binary columns for each category. The standard for nominal data.
Example
import pandas as pd
from sklearn.preprocessing import OneHotEncoder
df = pd.DataFrame({
'color': ['red', 'blue', 'green', 'red', 'blue'],
'size': ['S', 'M', 'L', 'XL', 'M'],
'price': [10, 20, 30, 40, 25]
})
# Method 1: pandas get_dummies (quick and easy)
df_dummies = pd.get_dummies(df, columns=['color'], drop_first=True)
print("pd.get_dummies (drop_first=True to avoid multicollinearity):")
print(df_dummies)
# Method 2: sklearn OneHotEncoder (for pipelines)
ohe = OneHotEncoder(drop='first', sparse_output=False)
encoded = ohe.fit_transform(df[['color']])
feature_names = ohe.get_feature_names_out(['color'])
print(f"\nsklearn OneHotEncoder features: {feature_names}")
print(encoded)
Target Encoding (Mean Encoding)
Replaces categories with the mean of the target variable. Powerful but risky (can overfit).
Example
import pandas as pd
import numpy as np
df = pd.DataFrame({
'city': ['NYC', 'LA', 'NYC', 'Chicago', 'LA', 'NYC', 'Chicago', 'LA'],
'income': [80000, 65000, 90000, 55000, 70000, 85000, 60000, 68000]
})
# Target encoding: replace city with mean income per city
city_means = df.groupby('city')['income'].mean()
df['city_target_encoded'] = df['city'].map(city_means)
print("Target Encoding:")
print(df)
print(f"\nCity means:\n{city_means}")
print("\n⚠️ Must use cross-validation or smoothing to prevent overfitting!")
Handling High Cardinality
When a feature has many unique values (e.g., zip codes, product IDs):
Example
import pandas as pd
import numpy as np
# High cardinality example: 1000 unique cities
np.random.seed(42)
n = 5000
cities = [f"city_{i}" for i in range(1000)]
df = pd.DataFrame({
'city': np.random.choice(cities, n),
'target': np.random.normal(50, 10, n)
})
print(f"Unique cities: {df['city'].nunique()}")
print("One-hot encoding would create 1000 columns! Too many.\n")
# Strategy 1: Keep top-k categories, group rest as "Other"
top_k = 20
top_cities = df['city'].value_counts().head(top_k).index
df['city_grouped'] = df['city'].where(df['city'].isin(top_cities), 'Other')
print(f"After grouping: {df['city_grouped'].nunique()} categories")
# Strategy 2: Frequency encoding
freq_map = df['city'].value_counts(normalize=True)
df['city_freq'] = df['city'].map(freq_map)
print(f"Frequency encoding - unique values: {df['city_freq'].nunique()}")
Encoding Comparison
| Method | Best For | Pros | Cons |
|---|---|---|---|
| One-Hot | Nominal, low cardinality | No false ordering | Many columns, sparse |
| Label/Ordinal | Ordinal features, trees | Compact, single column | False ordering for nominal |
| Target | High cardinality | Compact, powerful | Overfitting risk |
| Frequency | High cardinality | Simple, no overfitting | Loses category meaning |
| Binary | Moderate cardinality | Compact | Less interpretable |
Interview Questions
- Why can't you use label encoding for nominal categorical features with linear models?
Label encoding assigns arbitrary integers (red=0, blue=1, green=2), which implies a mathematical relationship (blue > red, green > blue). Linear models would learn false patterns from this ordering.
- What is the dummy variable trap?
If you one-hot encode without dropping one category, you create perfect multicollinearity (columns sum to 1). Linear models can't handle this. Solution: use drop_first=True.
- How do you handle a categorical feature with 10,000 unique values?
Options: target encoding with smoothing, frequency encoding, embedding layers (for neural nets), hashing trick, or grouping rare categories into "Other".
- When would you use target encoding over one-hot encoding?
When cardinality is high (>20 categories) and you need a compact representation. Always use with cross-validation to prevent overfitting.
- How do you handle unseen categories in production?
Set handle_unknown='ignore' in OneHotEncoder (encodes as all zeros), use a fallback value for target encoding, or map to "Other" category.
Continue the lesson path
Study next in this module
01Data PreprocessingData Visualization for MLEssential data visualization techniques for machine learning including EDA plots, feature analysis, distribution analysis, and correlation visualization with Python.02Data PreprocessingFeature EngineeringMaster feature engineering for ML including creating new features, polynomial features, interaction terms, domain-specific features, and automated feature generation.03Data PreprocessingData Collection for Machine LearningLearn effective data collection strategies for ML projects including web scraping, APIs, databases, surveys, and data quality assessment techniques.04Data PreprocessingFeature ScalingComplete guide to feature scaling techniques including standardization, normalization, and when to use each method with practical sklearn implementations.