ML Notes

ML Model Monitoring & Observability

Monitoring ML models in production: performance metrics, data drift detection, alerting, and logging.

Monitoring ensures models perform as expected in production and alerts to issues before they impact users.

What to Monitor

1. Model Performance Metrics

Predictions:

Latency: time to generate prediction (target: <100ms)
Throughput: predictions per second (scale requirements)
Error rate: crashes, exceptions (target: <0.1%)

Quality:

Accuracy on recent data (baseline)
Precision, recall for classification
RMSE, MAE for regression

2. Data Quality Metrics

Input validation:

Missing values percentage
Out-of-range values
Null/NaN counts

Data distribution:

Feature mean, std deviation
Min, max, percentiles
Expected range violations

3. Data Drift Detection

Input distribution changed from training:

Training data

Age: mean=45, std=15, range=[20,70]

Income: mean=$50k, range=[$20k, $200k]

Production (Week 1)

Age: mean=47, std=16 ← small change, OK

Income: mean=$48k

Production (Week 8)

Age: mean=52, std=20 ← significant drift!

Income: mean=$35k ← significant drift!

→ Model may perform worse

Detection methods:

Statistical tests: Kolmogorov-Smirnov test (p < 0.05 = drift)
Distribution metrics: Compare CDFs
Statistical distances: Wasserstein distance, Maximum Mean Discrepancy

4. Model Drift Detection

Model performance degrading over time:

Week 1	Accuracy 95%
Week 2	Accuracy 93% (OK, natural variation)
Week 3	Accuracy 91%
Week 4	Accuracy 88% (significant drop!)
Week 5	Accuracy 82% (critical, retrain needed)

5. Operational Metrics

System health:

CPU, memory usage
Request queue length
Model loading time
Database latency

Monitoring Implementation

Logging Predictions

Example

import json import logging from datetime import datetime logger = logging.getLogger(__name__) def log_prediction(user_id, features, prediction, model_version, actual=None): """Log prediction with metadata""" log_entry = { 'timestamp': datetime.utcnow().isoformat(), 'user_id': user_id, 'model_version': model_version, 'features': {k: float(v) for k, v in zip(feature_names, features)}, 'prediction': float(prediction), 'actual': float(actual) if actual else None } logger.info(json.dumps(log_entry)) @app.post("/predict") def predict(request): features = request.features prediction = model.predict(features) log_prediction( user_id=request.user_id, features=features, prediction=prediction, model_version=os.environ['MODEL_VERSION'] ) return {"prediction": prediction}

Performance Metrics with Prometheus

Example

from prometheus_client import Counter, Histogram, Gauge import time # Define metrics predictions_total = Counter( 'ml_predictions_total', 'Total predictions made', ['model_version', 'status'] ) prediction_latency_seconds = Histogram( 'ml_prediction_latency_seconds', 'Prediction latency in seconds', buckets=[0.01, 0.05, 0.1, 0.5, 1.0], labelnames=['model_version'] ) model_accuracy = Gauge( 'ml_model_accuracy', 'Model accuracy on recent data', ['model_version'] ) @app.post("/predict") def predict(request): start_time = time.time() try: prediction = model.predict(request.features) status = "success" except Exception as e: predictions_total.labels( model_version=VERSION, status="error" ).inc() raise # Record metrics latency = time.time() - start_time predictions_total.labels( model_version=VERSION, status=status ).inc() prediction_latency_seconds.labels( model_version=VERSION ).observe(latency) return {"prediction": prediction}

Data Drift Detection

Example

from scipy.stats import ks_2samp import numpy as np def detect_data_drift(X_new, X_reference, threshold=0.05): """ Detect if input distribution has changed Args: X_new: New data from production X_reference: Reference data (training set) threshold: p-value threshold for drift detection Returns: dict: {column: p_value} for drifted columns """ drifts = {} for column in X_new.columns: # Kolmogorov-Smirnov test statistic, p_value = ks_2samp(X_reference[column], X_new[column]) if p_value < threshold: drifts[column] = p_value return drifts # Monitor daily def monitor_data_drift(): X_today = load_today_data() X_baseline = load_baseline_data() drifts = detect_data_drift(X_today, X_baseline) if drifts: message = "Data drift detected:\n" for col, p_val in drifts.items(): message += f" {col}: p-value={p_val:.4f}\n" alert(message, severity="warning") # Trigger retraining if significant if len(drifts) > 3: trigger_retraining()

Performance Degradation Detection

python example

class PerformanceMonitor:
    def __init__(self, baseline_accuracy, alert_threshold=0.05):
        self.baseline = baseline_accuracy
        self.alert_threshold = alert_threshold
    
    def check_performance(self, y_true, y_pred):
        """Check if performance has degraded"""
        from sklearn.metrics import accuracy_score
        
        current_acc = accuracy_score(y_true, y_pred)
        degradation = (self.baseline - current_acc) / self.baseline
        
        # Alert if >5% degradation
        if degradation > self.alert_threshold:
            self.send_alert(
                f"Performance degradation: {degradation:.1%}",
                f"Accuracy dropped from {self.baseline:.3f} to {current_acc:.3f}",
                severity="critical"
            )
            
            # Trigger retraining
            trigger_retraining()
        
        return {
            'current_accuracy': current_acc,
            'baseline_accuracy': self.baseline,
            'degradation': degradation,
            'alert_triggered': degradation > self.alert_threshold
        }
    
    def send_alert(self, title, message, severity="warning"):
        """Send alert (email, Slack, PagerDuty, etc.)"""
        pass

# Use it
monitor = PerformanceMonitor(baseline_accuracy=0.95, alert_threshold=0.05)

# Check after getting predictions
results = monitor.check_performance(y_test, y_pred)

Monitoring Dashboard

python example

# Example: Grafana dashboard queries

# Query 1: Prediction latency over time
SELECT
    timestamp,
    avg(ml_prediction_latency_seconds) as latency
FROM metrics
WHERE model_version = 'v1.0'
GROUP BY time(1m)

# Query 2: Prediction volume
SELECT
    timestamp,
    sum(ml_predictions_total) as total_predictions
FROM metrics
WHERE model_version = 'v1.0'
GROUP BY time(1h)

# Query 3: Model accuracy trend
SELECT
    timestamp,
    ml_model_accuracy
FROM metrics
WHERE model_version = 'v1.0'
ORDER BY timestamp

Alerting Strategies

Alert Rules

Example

IF accuracy_drop > 5% THEN Alert to data_science_team with severity=HIGH IF latency_p99 > 500ms THEN Alert to backend_team with severity=MEDIUM IF data_drift_detected THEN Trigger automatic retraining Alert to data_science_team IF error_rate > 1% THEN Alert to ops_team with severity=CRITICAL Trigger automatic rollback

Alert Fatigue Prevention

Set reasonable thresholds (not too sensitive)
Group related alerts
Require consecutive failures (not single anomaly)
Time-based alert deduplication (max 1 alert per hour)

Monitoring for Specific Use Cases

Classification Model Monitoring

Metrics

- Class distribution (any class disappearing?)

- Precision/recall per class

- Confusion matrix changes

- Calibration (predicted proba vs actual)

Alerts

- Minority class disappears

- Precision drops >X%

- Model overconfident (high proba, wrong predictions)

Regression Model Monitoring

Metrics

- Residual distribution

- Mean absolute error

- Root mean squared error

- Prediction range vs actual range

Alerts

- RMSE increased >X%

- Predictions biased (mean error != 0)

- Outlier predictions (outside expected range)

Ranking/Recommendation Monitoring

Metrics

- NDCG@10 (ranking quality)

- Click-through rate

- Conversion rate

- Recommendation diversity

Alerts

- CTR dropped >X%

- Diversity decreased (recommending same items)

- User satisfaction score decreased

Summary

Monitoring ensures production reliability:

Track performance metrics
Monitor data quality
Detect drift (data and model)
Alert on issues
Trigger retraining when needed

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for ML Model Monitoring & Observability.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.

Search Terms

machine-learning, machine learning, machine, learning, mlops, model, monitoring, ml model monitoring & observability

Continue learning this concept

MlopsModel Deployment StrategiesDifferent approaches to deploying ML models: batch, real-time, shadow, canary, blue-green, and A/B testing.MlopsCI/CD for Machine LearningImplementing continuous integration and continuous deployment pipelines for ML models including testing, validation, and automated deployment.MlopsDocker for Machine LearningContainerizing ML models and workflows using Docker for reproducibility, scalability, and deployment.MlopsFastAPI for Machine Learning APIsBuilding high-performance ML APIs with FastAPI, async support, and automatic documentation.MlopsFlask API for Machine LearningBuilding ML model serving APIs with Flask, request handling, and deployment.MlopsMLOps: IntroductionIntroduction to MLOps: productionizing ML systems, monitoring, deployment, and lifecycle management.

PreviousModel Deployment Strategies

ML Notes

ML Model Monitoring & Observability

Monitoring ML models in production: performance metrics, data drift detection, alerting, and logging.

Monitoring ensures models perform as expected in production and alerts to issues before they impact users.

What to Monitor

1. Model Performance Metrics

Predictions:

Latency: time to generate prediction (target: <100ms)
Throughput: predictions per second (scale requirements)
Error rate: crashes, exceptions (target: <0.1%)

Quality:

Accuracy on recent data (baseline)
Precision, recall for classification
RMSE, MAE for regression

2. Data Quality Metrics

Input validation:

Missing values percentage
Out-of-range values
Null/NaN counts

Data distribution:

Feature mean, std deviation
Min, max, percentiles
Expected range violations

3. Data Drift Detection

Input distribution changed from training:

Training data

Age: mean=45, std=15, range=[20,70]

Income: mean=$50k, range=[$20k, $200k]

Production (Week 1)

Age: mean=47, std=16 ← small change, OK

Income: mean=$48k

Production (Week 8)

Age: mean=52, std=20 ← significant drift!

Income: mean=$35k ← significant drift!

→ Model may perform worse

Detection methods:

Statistical tests: Kolmogorov-Smirnov test (p < 0.05 = drift)
Distribution metrics: Compare CDFs
Statistical distances: Wasserstein distance, Maximum Mean Discrepancy

4. Model Drift Detection

Model performance degrading over time:

Week 1	Accuracy 95%
Week 2	Accuracy 93% (OK, natural variation)
Week 3	Accuracy 91%
Week 4	Accuracy 88% (significant drop!)
Week 5	Accuracy 82% (critical, retrain needed)

5. Operational Metrics

System health:

CPU, memory usage
Request queue length
Model loading time
Database latency

Monitoring Implementation

Logging Predictions

Example

Performance Metrics with Prometheus

Example

Data Drift Detection

Example

Performance Degradation Detection

python example

class PerformanceMonitor:
    def __init__(self, baseline_accuracy, alert_threshold=0.05):
        self.baseline = baseline_accuracy
        self.alert_threshold = alert_threshold
    
    def check_performance(self, y_true, y_pred):
        """Check if performance has degraded"""
        from sklearn.metrics import accuracy_score
        
        current_acc = accuracy_score(y_true, y_pred)
        degradation = (self.baseline - current_acc) / self.baseline
        
        # Alert if >5% degradation
        if degradation > self.alert_threshold:
            self.send_alert(
                f"Performance degradation: {degradation:.1%}",
                f"Accuracy dropped from {self.baseline:.3f} to {current_acc:.3f}",
                severity="critical"
            )
            
            # Trigger retraining
            trigger_retraining()
        
        return {
            'current_accuracy': current_acc,
            'baseline_accuracy': self.baseline,
            'degradation': degradation,
            'alert_triggered': degradation > self.alert_threshold
        }
    
    def send_alert(self, title, message, severity="warning"):
        """Send alert (email, Slack, PagerDuty, etc.)"""
        pass

# Use it
monitor = PerformanceMonitor(baseline_accuracy=0.95, alert_threshold=0.05)

# Check after getting predictions
results = monitor.check_performance(y_test, y_pred)

Monitoring Dashboard

python example

# Example: Grafana dashboard queries

# Query 1: Prediction latency over time
SELECT
    timestamp,
    avg(ml_prediction_latency_seconds) as latency
FROM metrics
WHERE model_version = 'v1.0'
GROUP BY time(1m)

# Query 2: Prediction volume
SELECT
    timestamp,
    sum(ml_predictions_total) as total_predictions
FROM metrics
WHERE model_version = 'v1.0'
GROUP BY time(1h)

# Query 3: Model accuracy trend
SELECT
    timestamp,
    ml_model_accuracy
FROM metrics
WHERE model_version = 'v1.0'
ORDER BY timestamp

Alerting Strategies

Alert Rules

Example

Alert Fatigue Prevention

Set reasonable thresholds (not too sensitive)
Group related alerts
Require consecutive failures (not single anomaly)
Time-based alert deduplication (max 1 alert per hour)

Monitoring for Specific Use Cases

Classification Model Monitoring

Metrics

- Class distribution (any class disappearing?)

- Precision/recall per class

- Confusion matrix changes

- Calibration (predicted proba vs actual)

Alerts

- Minority class disappears

- Precision drops >X%

- Model overconfident (high proba, wrong predictions)

Regression Model Monitoring

Metrics

- Residual distribution

- Mean absolute error

- Root mean squared error

- Prediction range vs actual range

Alerts

- RMSE increased >X%

- Predictions biased (mean error != 0)

- Outlier predictions (outside expected range)

Ranking/Recommendation Monitoring

Metrics

- NDCG@10 (ranking quality)

- Click-through rate

- Conversion rate

- Recommendation diversity

Alerts

- CTR dropped >X%

- Diversity decreased (recommending same items)

- User satisfaction score decreased

Summary

Monitoring ensures production reliability:

Track performance metrics
Monitor data quality
Detect drift (data and model)
Alert on issues
Trigger retraining when needed

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for ML Model Monitoring & Observability.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Machine Learning topic.

Search Terms

machine-learning, machine learning, machine, learning, mlops, model, monitoring, ml model monitoring & observability

Continue learning this concept

PreviousModel Deployment Strategies