Machine Learning Glossary

Quick reference for common machine learning terms and concepts.

A

Activation Function: Non-linear function applied to neuron outputs in neural networks (ReLU, Sigmoid, Tanh).

AdaBoost: Adaptive Boosting - ensemble method that weights misclassified samples more heavily.

Accuracy: (TP+TN)/(TP+FP+TN+FN) - proportion of correct predictions.

Anomaly Detection: Identifying outliers or unusual patterns in data.

Attention Mechanism: Neural network component that weights importance of different inputs (used in Transformers).

Augmentation: Creating new training samples by applying transformations (rotation, flip, crop) to existing data.

Autoencoder: Neural network that learns compressed representations of data by reconstructing inputs.

B

Backpropagation: Algorithm for computing gradients through neural networks using chain rule.

Batch: Subset of training data processed together during gradient descent.

Batch Normalization: Normalizing batch inputs to improve training stability.

Bias: Model's tendency to systematically make errors (underfitting). Also, bias term in linear models.

Bias-Variance Tradeoff: Balance between model simplicity (high bias) and fitting noise (high variance).

Boosting: Ensemble method combining weak learners sequentially (AdaBoost, Gradient Boosting).

C

Classification: Supervised learning task of predicting discrete classes.

Clustering: Unsupervised learning task of grouping similar data points.

Confusion Matrix: Table showing TP, FP, TN, FN for classification model.

Convolutional Neural Network (CNN): Neural network with convolutional layers for image processing.

Cross-Entropy Loss: Loss function for classification (negative log-likelihood).

Cross-Validation: Dividing data into k folds for robust model evaluation.

D

Data Augmentation: Creating synthetic training samples through transformations.

Data Drift: Change in input data distribution over time in production.

Dimensionality Reduction: Reducing number of features while preserving information (PCA, t-SNE).

Dropout: Regularization technique randomly deactivating neurons during training.

E

Embedding: Mapping categorical variables to dense numerical vectors.

Ensemble: Combining predictions from multiple models for better performance.

Epoch: One complete pass through entire training dataset.

Explainability: Ability to interpret and understand model decisions (LIME, SHAP).

F

Feature: Input variable for ML model.

Feature Engineering: Creating new features from raw data to improve model.

Feature Importance: Ranking which features most influence model predictions.

Feature Scaling: Normalizing features to same scale (StandardScaler, MinMaxScaler).

Fine-tuning: Adapting pre-trained model to new task with small dataset.

F1-Score: 2*(Precision*Recall)/(Precision+Recall) - harmonic mean.

G

Generalization: Model's ability to perform well on unseen data.

Gradient: Partial derivative of loss with respect to parameters.

Gradient Descent: Optimization algorithm updating parameters in direction of negative gradient.

Gradient Boosting: Sequential ensemble method fitting new models to residuals (XGBoost, LightGBM).

H

Hyperparameter: Configuration parameter set before training (learning rate, tree depth).

Hyperparameter Tuning: Finding optimal hyperparameter values (Grid Search, Random Search).

I

Imbalanced Dataset: Classification data with unequal class distributions.

Inference: Making predictions on new data with trained model.

Information Gain: Reduction in entropy from splitting on feature (Decision Trees).

J

Jaccard Index: |A∩B|/|A∪B| - similarity between sets.

K

K-Fold Cross-Validation: Dividing data into k equal folds for evaluation.

K-Means: Clustering algorithm partitioning data into k clusters.

K-Nearest Neighbors (KNN): Classification/regression using k nearest training samples.

Kernel: Function computing similarity/distance in high-dimensional space (SVM).

L

Learning Curve: Plot of training/validation error vs training size.

Learning Rate: Step size for parameter updates in gradient descent (critical hyperparameter).

LIME: Local Interpretable Model-agnostic Explanations - technique for explaining predictions.

Linear Regression: Regression using linear relationship between features and target.

Logistic Regression: Classification using logistic/sigmoid function.

Loss Function: Measures difference between predictions and actual values (MSE, cross-entropy).

M

Machine Learning: Learning patterns from data without explicit programming.

Mean Absolute Error (MAE): Average absolute difference between predictions and actual.

Mean Squared Error (MSE): Average squared difference between predictions and actual.

Metric: Quantitative measure of model performance (accuracy, precision, recall).

Mini-batch: Small subset of data processed together.

Model Drift: Degradation of model performance over time due to changing data/concept.

Momentum: Optimizer accumulating gradient history for smoother updates.

N

Neural Network: Model inspired by biological neurons with layers of interconnected nodes.

Normalization: Scaling features to [0,1] range.

Null Hypothesis: Assumption of no effect in statistical testing.

O

Optimization: Process of improving model parameters.

Optimizer: Algorithm for updating weights (SGD, Adam, RMSprop).

Overfitting: Model memorizing training data, poor generalization.

Oversampling: Creating synthetic samples of minority class (SMOTE).

P

P-value: Probability of observing result if null hypothesis true (<0.05 = significant).

Perceptron: Simplest neural network with single layer.

Precision: TP/(TP+FP) - proportion of positive predictions that are correct.

Preprocessing: Data cleaning and transformation before modeling.

Probability: Likelihood of event, [0, 1].

Q

Quantization: Converting model weights to lower precision (int8) for faster inference.

R

Recall (Sensitivity): TP/(TP+FN) - proportion of actual positives correctly identified.

Recurrent Neural Network (RNN): Neural network with feedback connections for sequences.

Regularization: Technique preventing overfitting (L1, L2, dropout).

Regression: Supervised learning task predicting continuous values.

Reinforcement Learning: Learning through interaction and rewards/penalties.

Residual: Difference between actual and predicted values.

Root Mean Squared Error (RMSE): Square root of MSE, same units as target.

S

Sampling: Selecting subset of data for training/evaluation.

Sigmoid: Activation function σ(x)=1/(1+e^(-x)) mapping to [0,1].

Silhouette Score: -1 to 1 metric for cluster quality.

Softmax: Activation converting logits to probability distribution.

Specificity: TN/(TN+FP) - proportion of negatives correctly identified.

Standardization: Scaling to mean=0, std=1.

Stratified Split: Maintaining class distribution when splitting data.

Supervised Learning: Learning from labeled data (classification, regression).

Support Vector Machine (SVM): Classification finding optimal separating hyperplane.

T

Target: Dependent variable we're predicting.

T-test: Statistical test comparing means of two groups.

Temperature (Softmax): Parameter controlling randomness of softmax distribution.

Tensor: Multi-dimensional array (matrix generalization).

Test Set: Data used only for final model evaluation.

Time Series: Sequence of observations over time.

Tokenization: Breaking text into words/tokens.

Training Set: Data used to train model.

Transfer Learning: Using pre-trained model as starting point for new task.

Transformer: Neural architecture using attention (BERT, GPT).

U

Underfitting: Model too simple, poor performance on training and test data.

Undersampling: Removing samples from majority class (balancing).

Unsupervised Learning: Learning from unlabeled data (clustering, dimensionality reduction).

V

Validation Set: Data used to tune hyperparameters and select models.

Vanishing Gradient: Gradients becoming very small in deep networks.

Variance: Model's sensitivity to small fluctuations in training data (overfitting).

Vectorization: Converting operations to matrix form for efficiency.

W

Weight: Parameter multiplying feature in model.

Word Embedding: Representation of words as dense vectors (Word2Vec, GloVe).

X

XGBoost: Extreme Gradient Boosting - fast, scalable gradient boosting.

Y

Yield: Efficiency metric (True Positives / total positives detected).

Z

Zero-shot Learning: Making predictions on classes never seen during training.

Quick Lookup by Category

Supervised Learning

Classification, Regression, Logistic Regression, Linear Regression, SVM, Decision Trees, Ensemble, Boosting, Bagging

Unsupervised Learning

Clustering, K-Means, Dimensionality Reduction, PCA, Autoencoder

Evaluation Metrics

Accuracy, Precision, Recall, F1-Score, Confusion Matrix, ROC-AUC, MSE, RMSE, MAE

Neural Networks

Activation Function, Backpropagation, Dropout, Batch Normalization, Optimizer, Convolutional Neural Network, Recurrent Neural Network, Transformer

Model Improvement

Regularization, Cross-Validation, Hyperparameter Tuning, Feature Engineering, Data Augmentation

Challenges

Overfitting, Underfitting, Bias-Variance Tradeoff, Imbalanced Dataset, Vanishing Gradient