Observability vs monitoring — a practical SRE guide

There is a particular kind of dread that every on-call engineer knows intimately. It is 2 AM. Your phone buzzes. The dashboard is a wall of green — CPU nominal, memory fine, disk healthy, all health checks passing. Yet the error tracker is filling up with user complaints: checkout broken, search returning empty results, payments timing out after 30 seconds. Everything looks healthy. Nothing works.

This gap — between what your dashboards tell you and what your users experience — is exactly the gap that observability exists to close. If monitoring is the smoke detector, observability is the ability to trace the smell of smoke back to the exact wire that is overheating inside the wall. This guide walks through the fundamental differences, the tooling landscape in 2026, and a concrete implementation roadmap for engineering teams ready to make the shift.

Monitoring — necessary but not sufficient

Monitoring has served the industry well for decades. Nagios, Zabbix, PRTG, and their descendants built the foundation: measure known quantities, set thresholds, fire alerts when things cross the line. The mental model is straightforward — define what “healthy” looks like, and get notified when reality deviates.

This works under one critical assumption: you know in advance what can go wrong.

In a monolithic application running on three servers, that assumption holds. The failure modes are well-understood: the database runs out of connections, the disk fills up, a process leaks memory. You can enumerate these scenarios, build dashboards for each, and sleep relatively well at night.

But modern architectures have shattered that assumption. A single user request in a microservices environment might traverse 15 services, 4 message queues, 2 caches, 3 databases, and an external API. The number of possible failure modes is combinatorial. You cannot pre-build a dashboard for every way this system can break — because most of those failure modes have never happened before and exist only as emergent properties of component interactions.

The known-unknowns problem

Monitoring is excellent at known-knowns (things you expect and measure) and decent at known-unknowns (things you know could go wrong but haven’t seen yet). It fails entirely at unknown-unknowns — failure modes you have never imagined.

In distributed systems, unknown-unknowns dominate. A subtle clock skew between two services causes an ordering guarantee to silently break. A configuration change in service A causes service B to retry aggressively, which cascades into service C running out of database connections. No single component is “down.” Every individual health check passes. But the system as a whole is degraded in a way no one predicted.

What observability actually means

The term comes from control theory, introduced by Rudolf Kalman in 1960. A system is observable if you can determine its internal state from its external outputs. Translated to software: a system is observable if you can understand what is happening inside it — including scenarios you never anticipated — by examining the telemetry data it produces.

The key distinction:

Aspect	Monitoring	Observability
Question type	Pre-defined (“Is X healthy?”)	Ad-hoc (“Why did user 4821 see a timeout?”)
Failure mode	Known, enumerated	Unknown, emergent
Debugging	Look at dashboard, check known metrics	Explore data, form hypotheses, drill down
Scales with complexity	Poorly — more services = more dashboards	Well — same tools, deeper exploration
Answers	THAT something broke	WHY it broke

Monitoring is a subset of observability. You need monitoring. But monitoring alone does not give you observability.

The three pillars of observability

Observability rests on three complementary types of telemetry data. Each pillar answers different questions, and the real power emerges when you correlate across all three.

Logs — the narrative record

Logs are timestamped records of discrete events. A request arrived. A database query took 450ms. A user authenticated. A payment was declined. Logs tell the story of what happened, in sequence, with context.

The shift from unstructured to structured logging has been transformative. Instead of parsing free-text strings with regex, structured logs (JSON format) enable filtering, aggregation, and machine processing at scale.

A well-structured log entry includes:

{
  "timestamp": "2026-04-15T10:23:45.123Z",
  "level": "ERROR",
  "service": "checkout-service",
  "version": "2.4.1",
  "trace_id": "7f3a8b2c1d4e5f6a",
  "span_id": "a1b2c3d4e5f6",
  "user_id": "usr_48291",
  "event": "payment_declined",
  "gateway": "stripe",
  "decline_code": "insufficient_funds",
  "amount_cents": 15900,
  "currency": "EUR",
  "latency_ms": 1243
}

The trace_id field is what connects this log entry to a distributed trace. The user_id lets you find every event for a specific user session. These correlation identifiers are what transform logs from isolated text files into queryable, navigable data.

The volume challenge: a medium-scale system easily generates hundreds of gigabytes of logs per day. Retention policies, tiered storage (hot/warm/cold), and selective sampling are essential to keep costs manageable.

Metrics — the quantitative pulse

Metrics are numerical measurements collected over time. CPU utilization, request latency, error count, queue depth, active connections, memory usage. Unlike logs, metrics are inherently aggregatable and storage-efficient — a single metric time series with 15-second resolution costs a fraction of equivalent log data.

Four fundamental metric types:

• Counter — monotonically increasing value (total requests, total errors)
• Gauge — value that goes up and down (CPU usage, active connections, temperature)
• Histogram — distribution of values across buckets (latency percentiles: p50, p90, p99)
• Summary — client-side computed quantiles, similar to histogram

Two frameworks that bring discipline to metric selection:

The RED method (for request-driven services):

• Rate — requests per second
• Errors — percentage of failed requests
• Duration — distribution of request latencies

The USE method (for infrastructure resources):

• Utilization — percentage of resource capacity in use
• Saturation — amount of work queued, waiting
• Errors — count of error events

Together, RED for services and USE for infrastructure cover the essential health signals of most systems.

Traces — the distributed detective

Distributed tracing follows a single request as it propagates through multiple services. One trace represents the entire journey — from the moment a user clicks “Buy Now” through the API gateway, order service, payment processor, inventory check, notification service, and back to the response.

A trace consists of spans. Each span represents one operation: an HTTP call, a database query, a cache lookup, a message publish. Spans have parent-child relationships, forming a tree (or directed acyclic graph) that visualizes the request’s path through the system.

[API Gateway] ─── 15ms
  └── [Order Service] ─── 52ms
        ├── [User Service] ─── 6ms (cache hit)
        ├── [Payment Service] ─── 4200ms  ← bottleneck
        │     ├── [Fraud Check] ─── 89ms
        │     └── [Stripe API] ─── 4050ms  ← root cause
        ├── [Inventory Service] ─── 12ms
        └── [Notification Service] ─── 3ms (async)

Without tracing, you see: “the checkout request took 4.3 seconds.” With tracing, you see precisely that the Stripe API call took 4 seconds, inside the payment service, after the fraud check. You know where to look, who to call, and what to fix.

Correlation — where the magic happens

Each pillar alone has blind spots. Metrics show you a latency spike at 10:23 but not which requests were affected. Logs show individual events but not the causal chain across services. Traces show request flow but not system-wide resource pressure.

Correlation connects them: click a latency spike in a metric chart, see the associated error logs, open the trace_id from a log entry, and view the full request waterfall. This requires shared context — trace_id and span_id propagated through all three signal types. This is exactly what OpenTelemetry standardizes.

OpenTelemetry — instrument once, export everywhere

OpenTelemetry (OTel) is a CNCF project that provides a unified standard for generating, collecting, and exporting telemetry data. Born from the merger of OpenTracing and OpenCensus in 2019, it has become the dominant instrumentation standard across the industry.

Gartner forecasts that 70% of organizations will adopt OpenTelemetry by 2027 as their primary instrumentation framework. The momentum is undeniable — every major observability vendor supports OTel natively.

How OpenTelemetry works

1. SDKs and auto-instrumentation

OTel provides SDKs for every major language: Java, Python, Go, Node.js, .NET, Ruby, PHP, Rust. Auto-instrumentation libraries automatically capture HTTP calls, database queries, gRPC calls, and message queue operations — often with zero code changes.

// Go — auto-instrumentation for net/http
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

handler := otelhttp.NewHandler(mux, "server")
http.ListenAndServe(":8080", handler)
// Traces and metrics captured automatically

2. The OTel Collector

The Collector is a vendor-agnostic proxy that sits between your applications and your observability backend. It receives telemetry data, processes it (filtering, sampling, enriching, batching), and exports it to one or more destinations.

This architecture is the key to avoiding vendor lock-in. Migrating from Jaeger to Grafana Tempo? Change the Collector’s exporter configuration. Your application code stays untouched. Sending data to both Datadog and an internal Prometheus? Add a second exporter. The Collector handles fan-out.

3. OTLP — the universal protocol

OpenTelemetry Protocol (OTLP) is the native wire format for all three signal types. It supports gRPC and HTTP transport, is efficient on the wire, and is implemented by virtually every observability tool on the market.

Why this matters economically

Before OTel, every vendor had proprietary SDKs, proprietary agents, and proprietary formats. Instrumenting for Datadog meant importing Datadog libraries into every service. Switching to New Relic meant re-instrumenting everything. The switching cost was so high that most organizations stayed locked in.

OTel decouples instrumentation from backend. You invest once in instrumenting your services — and that investment is portable across any backend, forever. This is a structural shift in the economics of observability.

Tools landscape — choosing your stack

The observability tooling market is mature and competitive. The right choice depends on team size, budget, operational capability, and data retention requirements.

Tool	Type	Metrics	Logs	Traces	Cost model	Best for
Prometheus + Grafana	Open-source	Yes	Loki	Tempo	Free + infra	Teams with ops expertise
Datadog	SaaS	Yes	Yes	Yes	$15-50/host/mo	All-in-one, fast setup
New Relic	SaaS	Yes	Yes	Yes	Free tier, then $/GB	Startups, mid-size
Elastic/ELK	Hybrid	Yes	Yes	Yes (APM)	Free + infra / SaaS	Heavy log users
Jaeger	Open-source	No	No	Yes	Free + infra	Tracing only
Grafana LGTM	Open-source	Mimir	Loki	Tempo	Free + infra / Cloud	Full open-source stack
Splunk	SaaS/On-prem	Yes	Yes	Yes	Premium	Enterprise, compliance

The Grafana LGTM stack

The open-source combination of Loki (logs), Grafana (visualization), Tempo (traces), and Mimir (metrics) has emerged as the leading open-source alternative to SaaS platforms. Each component is designed for horizontal scalability, and Grafana provides seamless correlation across all three data types.

Strengths: no per-host licensing, full data ownership, active community, OTel-native. Weaknesses: requires operational investment to run at scale. Scaling Mimir/Loki/Tempo on Kubernetes is a non-trivial operational task.

Datadog — the convenience premium

Datadog covers metrics, logs, traces, profiling, synthetics, Real User Monitoring, and security in a single platform. The correlation between signal types works out of the box. Hundreds of built-in integrations. Excellent UX.

The trade-off is cost. At scale, companies routinely discover that Datadog consumes 25-35% of their total cloud budget. Per-host pricing, per-GB log ingestion fees, and per-million-span charges compound quickly. And despite OTel support for ingestion, exporting your data out of Datadog for analysis elsewhere remains limited.

Choosing wisely

A practical heuristic: if your team has fewer than 5 engineers and limited ops experience, start with SaaS (New Relic’s free tier is generous). If you have a platform team comfortable with Kubernetes, the Grafana LGTM stack gives you full control and dramatically lower costs at scale.

SRE practices — making observability actionable

Data without a decision framework is just noise. Site Reliability Engineering, as codified by Google, provides the operational framework that turns observability data into informed decisions.

SLIs — measuring what matters

Service Level Indicators are the metrics that genuinely reflect user experience. Not server CPU — users do not care about server CPU. Instead:

• Availability — proportion of successful requests (HTTP 2xx/3xx vs total)
• Latency — response time distribution (p50 for typical, p99 for tail)
• Correctness — proportion of responses that return the right data
• Freshness — how stale the data is (critical for data pipelines and search indexes)

The art is choosing SLIs that align with what users actually experience. An e-commerce site’s primary SLI might be “percentage of checkout flows completed successfully within 3 seconds.” A search engine’s might be “percentage of queries returning relevant results within 200ms.”

SLOs — setting internal targets

Service Level Objectives are targets set against SLIs. Example: “99.9% of checkout requests complete successfully within 3 seconds, measured over a rolling 30-day window.”

SLOs are an internal agreement — the engineering team’s commitment to a quality bar. They should be ambitious enough to keep users happy, but not so aggressive that they paralyze development.

SLAs — external commitments

Service Level Agreements are contractual obligations to customers, with financial penalties for breach. SLAs should always be less aggressive than SLOs. If your SLO is 99.95%, your SLA might be 99.9% — giving you a safety buffer before contractual penalties kick in.

Error budgets — resolving the velocity-stability tension

The error budget is the inverse of the SLO. If your SLO is 99.9% availability, your error budget is 0.1%. Over a 30-day month (43,200 minutes), that is 43.2 minutes of allowed downtime.

Error budgets resolve the eternal conflict:

• Development wants to ship fast (new features carry risk)
• Operations wants stability (no changes = no risk)

The rule: while error budget remains, ship freely. When the budget is exhausted, freeze deployments and invest in reliability. This replaces subjective arguments with an objective, data-driven decision criterion.

Alert fatigue — the silent reliability killer

The average SRE team receives 200-500 alerts per week. Most are false positives, non-actionable, or duplicates. Engineers learn to ignore alerts. And the one critical alert that matters gets lost in the noise. This is alert fatigue, and it is one of the most common causes of extended outages.

Principles for effective alerting

1. Alert on symptoms, not causes

Instead of “CPU > 80%,” alert on “error rate > 1% for 5 minutes.” Users do not experience CPU usage — they experience errors and slowness. Symptom-based alerts have higher signal-to-noise ratios because they directly reflect user impact.

2. Use SLO burn rate alerts

Instead of static thresholds, monitor the rate at which you are consuming your error budget. An alert that says “at the current error rate, you will exhaust your monthly error budget in 4 hours” provides both urgency and context. Google’s SRE workbook recommends multi-window, multi-burn-rate alerting as the gold standard.

3. Tiered severity with clear escalation

• P1 (page immediately): burn rate > 14x (budget exhausted in < 1 hour)
• P2 (respond within 30 min): burn rate > 6x (budget exhausted in < 6 hours)
• P3 (next business day): burn rate > 1x (slow but steady budget drain)
• Info (no notification): anomalies, minor deviations, logged for review

4. Every alert needs a runbook

An alert without a runbook is an alert that will be ignored. The runbook should answer: What is broken? What should I check first? What commands should I run? Who should I escalate to? What is the customer impact?

5. Monthly alert hygiene reviews

Review every alert that fired in the past month. Which were actionable? Which were noise? Delete the noise. It is better to have 15 alerts that always require action than 300 that are usually ignored.

The cost of observability — managing the data explosion

Observability is not free. Companies regularly discover they are spending 25-35% of their cloud budget on observability tooling and data storage. As microservices proliferate and traffic grows, telemetry volume grows superlinearly — more services means more inter-service communication, which means more traces, more logs, more metrics.

Where the costs hide

• Data volume: a single service instance can generate thousands of metric time series, megabytes of logs, and hundreds of trace spans per minute. Multiply by 100 services and 500 instances.
• Retention: storing 7 days vs 90 days vs 1 year — the cost differences are orders of magnitude.
• Cardinality explosion: metrics with high-cardinality labels (user_id, session_id, request_id) multiply storage requirements exponentially.
• SaaS pricing models: per-host, per-GB-ingested, per-million-spans — costs compound across multiple dimensions.

Cost control strategies

Sampling: you do not need 100% of traces. Head-based sampling (random 10%) drastically reduces volume for healthy traffic. Tail-based sampling (keep 100% of error traces, 1% of successful) gives you the best of both worlds — full visibility into problems, minimal storage for normal operations.

Tiered retention: full-resolution data for 7 days (hot), downsampled aggregates for 30 days (warm), metrics-only for 1 year (cold). Most debugging happens within the first 72 hours.

Cardinality discipline: never use high-cardinality values as metric labels. user_id as a Prometheus label with 10 million users creates 10 million time series — a storage and query disaster. Instead, put user_id in logs and trace attributes, where high cardinality is expected and handled.

Team-level budgets: allocate observability cost budgets per team. The team generating 70% of log volume should know about it and have incentives to optimize. Make observability cost visible in the same way cloud compute cost is visible.

Implementation roadmap — from zero to observable

Observability is not a weekend project. It is a journey that matures alongside your organization’s operational capability. The following phased approach has proven effective across many teams.

Phase 1: Foundation (weeks 1-4)

1. Choose your stack — strategic decision: open-source (Grafana LGTM) or SaaS (Datadog, New Relic). Consider team skills, budget, and scale trajectory.
2. Deploy metrics collection — Prometheus with node_exporter and kube-state-metrics. Build initial Grafana dashboards: USE method per node, RED method per service.
3. Standardize logging — mandate JSON format with required fields: timestamp, level, service, version, trace_id. Deploy centralized log aggregation (Loki or Elasticsearch).
4. Define initial SLIs and SLOs — start with your 2-3 most critical user-facing services. Involve product owners: what does “the service is working” mean from a user perspective?

Phase 2: Distributed tracing (weeks 5-8)

5. Deploy OpenTelemetry SDKs — start with auto-instrumentation on the critical request paths that generate the most user complaints. Validate that trace context propagates across service boundaries.
6. Configure the OTel Collector — centralized receiver, processor, and exporter. Enable tail-based sampling at 10-20% for healthy traffic, 100% for errors.
7. Connect the pillars — ensure trace_id appears in structured logs. Configure Grafana data links so you can click from a metric spike to correlated logs to the full trace waterfall.

Phase 3: Maturation (weeks 9-16)

8. SLO-based alerting — replace static threshold alerts with burn rate alerts. Build error budget dashboards visible to both engineering and product teams.
9. Runbooks — create runbooks for every P1 and P2 alert. Store them alongside alert definitions. Review and update after every incident.
10. Cost optimization — implement tail-based sampling. Move older data to cheaper storage tiers. Identify and eliminate high-cardinality metric labels.

Phase 4: Culture (ongoing)

11. Blameless postmortems — every significant incident ends with a written postmortem. The question is “what in the system allowed this to happen?” not “who caused this?”
12. Toil tracking and reduction — identify repetitive manual operational work and automate it. SRE target: no more than 50% of time spent on toil.
13. Chaos engineering — deliberately inject failures (Chaos Monkey, Litmus, Gremlin) to validate that your observability stack actually enables fast diagnosis. If a chaos experiment reveals a blind spot, fix the instrumentation before the gap causes a real incident.

The MTTR argument — justifying the investment

The strongest business case for observability is Mean Time to Resolution (MTTR) — the average time from incident detection to resolution.

Organizations with mature observability practices consistently achieve 40-60% MTTR reduction compared to traditional monitoring approaches. When downtime costs thousands of dollars per minute — in e-commerce, fintech, SaaS, adtech — the return on investment is compelling.

A simplified calculation:

• 10 incidents/month, average MTTR 60 minutes = 600 minutes of degradation
• 50% MTTR reduction = 300 minutes saved
• Downtime cost $500/minute = $150,000/month saved
• Grafana LGTM stack infrastructure cost for 200 services: ~$15,000/month
• First-month ROI: 10x

Even at smaller scale, reducing debugging time from hours to minutes materially improves engineering quality of life, reduces burnout, and frees time for feature development instead of firefighting.

What is changing in 2026

The observability landscape continues to evolve rapidly. Several trends are reshaping the field:

eBPF-powered observability — eBPF allows kernel-level telemetry collection without any application instrumentation. Tools like Cilium, Pixie (now part of New Relic), and Grafana Beyla generate HTTP, DNS, and TCP traces from kernel events alone. This is transformative for legacy systems, third-party software, and polyglot environments where adding SDK instrumentation to every service is impractical.

AI-assisted root cause analysis — LLMs and ML models are being applied to correlate anomalies across signals, suggest probable root causes, and auto-generate runbook steps. Datadog’s Watchdog, New Relic’s AI monitoring, and Dynatrace’s Davis AI are leading examples. The technology is still maturing, but the trajectory is clear: AI will not replace SREs, but SREs who use AI will work faster than those who do not.

Continuous profiling as the fourth pillar — OpenTelemetry is adding continuous profiling (CPU, memory allocation, lock contention, I/O wait) to its specification. Tools like Pyroscope and Grafana Alloy already support profiling alongside metrics, logs, and traces. Profiling answers the “what is consuming resources” question that the other three pillars cannot answer efficiently.

FinOps for observability — as observability costs have grown to rival compute costs, a dedicated discipline is emerging around managing observability spend. Tools that attribute telemetry costs to specific teams, services, and environments are becoming essential at scale.

Conclusion

Monitoring tells you THAT something is broken. Observability tells you WHY. In the era of microservices, distributed systems, and increasing architectural complexity, monitoring alone is insufficient.

Key takeaways:

• Three pillars — logs, metrics, traces — together paint the full picture. In isolation, each leaves critical blind spots.
• OpenTelemetry — instrument once, export anywhere. The standard that eliminates vendor lock-in and makes your instrumentation investment portable.
• SLO-based alerting — alert on what impacts users, not on what is easy to measure. Burn rate alerts provide both urgency and context.
• Start small — instrument your 2-3 most critical services first, prove value, then expand. Do not attempt a big-bang rollout.
• Control costs relentlessly — sampling, tiered retention, cardinality discipline. Observability without cost governance is budget combustion.
• Culture matters — blameless postmortems, toil reduction, chaos engineering. Tools are necessary but not sufficient. The practices around the tools determine whether observability actually improves reliability.

Observability is not a product you buy or a project you complete. It is an ongoing practice — a way of building and operating systems that prioritizes understanding over guessing. The teams that invest in it sleep better. Or at least, when they get paged at 2 AM, they resolve the issue in minutes instead of hours.

---

Ready to build SRE and observability capabilities in your team? EITT offers hands-on training in Site Reliability Engineering covering practical implementation of monitoring, observability, SLIs/SLOs, and incident management. Explore our current training catalog at eitt.academy/en/trainings/.