Observability (Logging, Monitoring, Alerting)

Mental Model

Connecting isolated components into a resilient, scalable, and observable distributed web.

In a distributed system, when something goes wrong, it's often impossible to tell why without a robust Observability stack. Observability is the measure of how well you can understand the internal state of your system based on its external outputs.

1. The Three Pillars

graph LR
    Producer[Producer Service] -->|Publish Event| Kafka[Kafka / Event Bus]
    Kafka -->|Consume| Consumer1[Consumer Group A]
    Kafka -->|Consume| Consumer2[Consumer Group B]
    Consumer1 --> DB1[(Primary DB)]
    Consumer2 --> Cache[(Redis)]

• Logs: Immutable, timestamped records of discrete events. (Loki, ELK).
• Metrics: Aggregated numerical data (counters, gauges, histograms) over time. (Prometheus).
• Traces: The journey of a single request through the entire system. (OpenTelemetry, Jaeger).

2. Real-world Analogy: The Flight Recorder

• Monitoring is the alarm that beeps when the plane is too low.
• Observability is the Black Box (Flight Recorder) that tells you the engine temperature, oil pressure, and pilot actions that led to the alarm.

3. High-Level Architecture

1. Instrumentation: Services emit telemetry.
2. Collector: Aggregates and filters data (e.g., OpenTelemetry Collector).
3. Storage: Time-series DB for metrics, Document DB for logs, Trace DB.
4. Visualization: Dashboards (Grafana) and Alerting (AlertManager).

Final Takeaway

You can't fix what you can't see. Invest in observability early to reduce MTTR (Mean Time to Resolution).

Technical Trade-offs: Messaging Systems

Key Takeaways

• Logs: Immutable, timestamped records of discrete events. (Loki, ELK).
• Metrics: Aggregated numerical data (counters, gauges, histograms) over time. (Prometheus).
• Traces: The journey of a single request through the entire system. (OpenTelemetry, Jaeger).

Production Readiness Checklist

Before deploying this architecture to a production environment, ensure the following Staff-level criteria are met:

• High Availability: Have we eliminated single points of failure across all layers?
• Observability: Are we exporting structured JSON logs, custom Prometheus metrics, and OpenTelemetry traces?
• Circuit Breaking: Do all synchronous service-to-service calls have timeouts and fallbacks (e.g., via Resilience4j)?
• Idempotency: Can our APIs handle retries safely without causing duplicate side effects?
• Backpressure: Does the system gracefully degrade or return HTTP 429 when resources are saturated?

Read Next

• Module 14: Security Basics (Auth, Encryption)
• Case Study: Design a Logging & Monitoring System

Verbal Interview Script

Interviewer: "How would you ensure high availability and fault tolerance for this specific architecture?"

Candidate: "To achieve 'Five Nines' (99.999%) availability, we must eliminate all Single Points of Failure (SPOF). I would deploy the API Gateway and stateless microservices across multiple Availability Zones (AZs) behind an active-active load balancer. For the data layer, I would use asynchronous replication to a read-replica in a different region for disaster recovery. Furthermore, it's not enough to just deploy redundantly; we must protect the system from cascading failures. I would implement strict timeouts, retry mechanisms with exponential backoff and jitter, and Circuit Breakers (using a library like Resilience4j) on all synchronous network calls between microservices."