Monitoring and Logging

Monitoring and logging are essential practices for maintaining reliable software systems in production. Monitoring provides real-time visibility into system health through metrics and alerts, while logging captures detailed event records for debugging and analysis. Together with distributed tracing, they form the three pillars of observability — the ability to understand what is happening inside a system by examining its external outputs.

Why Monitoring and Logging Matter

Software in production is a living system. Servers experience hardware failures, networks partition, databases run out of connections, memory leaks accumulate, and traffic spikes overwhelm capacity. Without monitoring, teams discover problems only when users complain — or worse, when revenue drops. With effective monitoring, teams detect and resolve issues before users are affected.

Consider an e-commerce platform during a holiday sale. Traffic increases tenfold. Without monitoring, the team might not realize that response times have degraded from 200 milliseconds to 8 seconds until customers abandon their carts and complaints flood customer support. With monitoring, an alert fires when response time exceeds 500 milliseconds, the on-call engineer investigates, identifies a database connection pool exhaustion, scales the connection pool, and resolves the issue in minutes — all before most customers notice.

The Three Pillars of Observability

Metrics

Metrics are numerical measurements collected at regular intervals. They answer quantitative questions: How many requests per second? What is the 95th percentile latency? How much CPU is being consumed? Metrics are compact, cheap to store, and excellent for dashboards and alerting.

Common categories of metrics include:

• Infrastructure metrics: CPU usage, memory utilization, disk I/O, network throughput
• Application metrics: Request rate, error rate, response latency, queue depth
• Business metrics: Orders per minute, user signups, revenue per hour

The RED method provides a useful framework for service monitoring: Rate (requests per second), Errors (failed requests per second), and Duration (response time distribution).

Logs

Logs are timestamped records of discrete events. They provide context and detail that metrics cannot capture. When a metric tells you that error rates spiked at 2:47 PM, logs tell you exactly which requests failed, what error messages were generated, and what parameters were involved.

Effective log entries include structured data:

Structured logging (JSON format) enables powerful searching, filtering, and aggregation compared to unstructured plain-text logs.

Distributed Tracing

In microservice architectures, a single user request might traverse dozens of services. When that request is slow, which service is the bottleneck? Distributed tracing answers this by assigning a unique trace ID to each request and propagating it through every service. Each service records a "span" with timing information, creating a complete picture of the request's journey.

Monitoring Architecture

A typical monitoring stack includes:

Collection: Agents on each server collect metrics and forward them to a central system. Prometheus uses a pull model (scraping endpoints), while StatsD and Telegraf use a push model.

Storage: Time-series databases (InfluxDB, Prometheus TSDB, TimescaleDB) store metrics efficiently. Log aggregation systems (Elasticsearch, Loki) store and index log data.

Visualization: Grafana dashboards display metrics in real-time graphs, heat maps, and gauges. Kibana provides log exploration and visualization.

Alerting: Rules define conditions that trigger notifications. PagerDuty, OpsGenie, and VictorOps manage on-call rotations and escalation policies.

Alerting Best Practices

Poor alerting creates "alert fatigue" — when teams receive so many false or unimportant alerts that they start ignoring them, including genuine emergencies. Effective alerting follows these principles:

Alert on symptoms, not causes. Alert when users experience degraded service (high latency, errors), not when a single server has high CPU. High CPU might be perfectly normal during peak hours.

Ensure alerts are actionable. Every alert should have a clear response action. If the on-call engineer cannot do anything about an alert at 3 AM, it should not page them.

Set appropriate thresholds. Use historical data to understand normal variation. Set thresholds that trigger for genuine anomalies, not routine fluctuations. Consider using anomaly detection rather than static thresholds.

Include context in notifications. Alert messages should include what is happening, which service is affected, relevant dashboard links, and suggested investigation steps.

Log Management at Scale

Production systems generate enormous volumes of logs — large applications produce gigabytes per hour. Managing this requires strategy:

Log levels control verbosity: DEBUG (development only), INFO (normal operations), WARN (potential issues), ERROR (failures requiring attention), FATAL (system cannot continue). Production typically runs at INFO level, with the ability to dynamically enable DEBUG for specific components during investigation.

Log rotation prevents disk exhaustion by archiving and deleting old log files. Most systems retain recent logs locally and ship historical logs to centralized storage.

Centralized aggregation collects logs from all servers into a searchable system. The ELK Stack (Elasticsearch, Logstash, Kibana) and Grafana Loki are popular choices.

Retention policies balance storage costs with debugging needs. Hot storage (fast search) for recent days, warm storage for weeks, cold archive for compliance requirements.

Real-World Monitoring Stack Example

A medium-sized SaaS company might implement:

• Prometheus for metrics collection and alerting rules
• Grafana for dashboards displaying system health
• Elasticsearch + Kibana for log aggregation and searching
• Jaeger for distributed tracing across microservices
• PagerDuty for on-call management and escalation
• Synthetic monitoring (Pingdom or Datadog) for external uptime checks

Service Level Objectives (SLOs)

Modern monitoring practices center around SLOs — quantitative reliability targets. An SLO might state: "99.9% of API requests will complete successfully within 500 milliseconds, measured over a rolling 30-day window." This translates to an error budget: the team is allowed 43.2 minutes of downtime per month. Monitoring tracks error budget consumption and alerts when the burn rate exceeds sustainable levels.

Interview Q&A

Q: What is the difference between monitoring and observability? A: Monitoring checks predefined conditions and alerts when thresholds are breached — it answers known questions. Observability enables investigation of novel, unforeseen issues by providing rich telemetry (metrics, logs, traces) that allows engineers to ask arbitrary questions about system behavior without deploying new instrumentation.

Q: How do you handle alert fatigue? A: Reduce noise by alerting on user-facing symptoms rather than internal causes, set thresholds based on statistical analysis rather than guesses, regularly review and tune alerts, suppress duplicate alerts during known incidents, and ensure every alert has clear ownership and documented response procedures.

Q: What is structured logging and why use it? A: Structured logging outputs log entries in a machine-parseable format (typically JSON) with consistent fields rather than free-form text. This enables powerful filtering, aggregation, and correlation across services — essential for debugging in distributed systems where you need to trace a request across dozens of log streams.