Observabilité et Surveillance : Guide Complet pour les Équipes d'Ingénierie

Every engineering team monitors production. Few can answer the question "why is the system behaving this way?" without a lengthy debugging session. The difference between monitoring and observability is not a tooling decision — it is a design philosophy that determines how quickly your team can understand and respond to unknown failure modes.

Traditional monitoring assumes you know what to watch. You set CPU thresholds, check endpoint status codes, and page someone when disk usage exceeds 90%. This works for known failure modes. But modern distributed systems fail in ways that no dashboard anticipated: a subtle latency regression in one service cascades into a queue backlog, which manifests as a database connection pool exhaustion that only surfaces under peak traffic. Without the ability to ask arbitrary questions about your system's internal state, you are flying blind.

Why Observability Matters Beyond Basic Monitoring

Observability is the property of a system that allows you to infer its internal state from its external outputs. Monitoring tells you something is wrong. Observability lets you debug what went wrong, even when you never predicted that particular failure mode. The distinction is crucial: monitoring is alert-driven and dashboard-bound; observability is inquiry-driven and data-rich.

When your system emits high-cardinality structured data across logs, metrics, and traces, you can correlate events, drill into specific users or requests, and discover the root cause of incidents that no threshold-based alert would ever catch. Teams that invest in observability reduce mean time to resolution by an order of magnitude because they spend less time guessing and more time acting on evidence.

"Monitoring tells you if a system is working. Observability lets you ask why it is not. The latter is a prerequisite for operating systems you do not fully understand — which is every production system."

The Three Pillars: Logs, Metrics, and Traces

The observability ecosystem is built on three complementary data types, each serving a distinct purpose. Treating them as a unified signal rather than separate silos is the key to effective debugging.

Logs: Immutable Records of Events

Logs are timestamped records of discrete events. They are the most granular observability signal, capturing exactly what happened at a specific point in time. A well-structured log line contains enough context — request ID, service name, duration, error details — to reconstruct the execution path without needing to correlate across multiple sources.

The most common mistake teams make with logs is treating them as unstructured text. Grepping through flat files worked when you had three servers. At scale, unstructured logs are noise. Every log line must be parseable, include structured metadata, and follow a consistent schema across every service in your architecture.

Metrics: Aggregated Measurements Over Time

Metrics are numeric representations of your system's state measured at intervals. They are designed for efficient storage and fast aggregation. Metrics answer questions like "how many requests per second?" and "what is the p99 latency?" Unlike logs, metrics discard individual request data — they trade granularity for compression and speed.

The standard metric types — counters, gauges, histograms, and summaries — each serve different use cases. Counters track cumulative values like total requests. Gauges record point-in-time values like memory usage. Histograms sample observations into configurable buckets for latency distributions. Choosing the right metric type prevents misleading aggregations and wasted cardinality.

Traces: End-to-End Request Lifecycles

Distributed traces follow a single request as it propagates across service boundaries. Each trace is composed of spans — named operations with start and end timestamps that capture the work done by a specific service or function. Traces are the only signal that can reconstruct the full lifecycle of a request in a microservice architecture.

Without traces, a slow page load might be blamed on any of the dozens of services involved. With traces, you can identify that the bottleneck is the user-service database query that takes 800 milliseconds, while every other span completes in under 50 milliseconds. This precision is impossible with logs or metrics alone.

"Logs tell you what happened. Metrics tell you how many times it happened. Traces tell you how it all fits together. You need all three to navigate a production incident without making assumptions."

OpenTelemetry Setup and Instrumentation

OpenTelemetry is the industry standard for generating, collecting, and exporting telemetry data. It provides a single set of APIs and SDKs across languages that emit logs, metrics, and traces in a vendor-neutral format. Adopting OpenTelemetry eliminates vendor lock-in and ensures your instrumentation works whether you use Datadog, Grafana, Honeycomb, or an in-house pipeline.

Automatic vs. Manual Instrumentation

Most OpenTelemetry SDKs support automatic instrumentation — zero-code hooks into popular frameworks and libraries. A single initializer call can instrument HTTP servers, database clients, message queues, and gRPC calls. Automatic instrumentation covers the common paths, but manual instrumentation is required for business-critical code paths, custom middleware, and domain-specific operations.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-grpc";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { ExpressInstrumentation } from "@opentelemetry/instrumentation-express";
import { PgInstrumentation } from "@opentelemetry/instrumentation-pg";
import { Resource } from "@opentelemetry/resources";
import { SEMRESATTRS_SERVICE_NAME } from "@opentelemetry/semantic-conventions";

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: "payment-service",
  }),
  traceExporter: new OTLPTraceExporter({
    url: "http://otel-collector:4317",
  }),
  metricExporter: new OTLPMetricExporter({
    url: "http://otel-collector:4317",
  }),
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
    new PgInstrumentation(),
  ],
});

sdk.start();
process.on("SIGTERM", () => sdk.shutdown());

The OpenTelemetry Collector Architecture

The OpenTelemetry Collector is a vendor-agnostic proxy that receives, processes, and exports telemetry data. It is the backbone of any production observability pipeline. Deploying a collector per host or as a cluster in your Kubernetes infrastructure decouples instrumentation from backend choice, allowing you to batch, filter, sample, and transform data before it reaches your observability platform.

Collectors can be configured with processors that handle tail-based sampling — retaining complete traces for high-latency or error requests while discarding the majority of healthy traffic. This dramatically reduces storage costs without sacrificing debugging capability for the requests that matter most.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  filter:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.target"] == "/healthz"'
        - 'attributes["http.target"] == "/metrics"'
  attributes:
    actions:
      - key: environment
        value: production
        action: upsert

exporters:
  otlp:
    endpoint: "api.honeycomb.io:443"
    headers:
      "x-honeycomb-team": "${HONEYCOMB_API_KEY}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter, batch, attributes]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch, attributes]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, attributes]
      exporters: [otlp]

Structured Logging Patterns

Structured logging means emitting log entries in a machine-parseable format — typically JSON — with consistent field names across services. It transforms logs from a search problem into a query problem. When every team uses the same field conventions for request IDs, service names, error codes, and durations, you can run ad-hoc queries across your entire infrastructure without writing custom parsers.

Schema Design for Logs

Every structured log event should include a minimum set of attributes: timestamp, severity level, service name, trace ID, span ID, and message. Beyond these, include domain-specific fields but enforce naming conventions. For example, always use camelCase, prefix user-related fields with user, and store error details in a nested error object rather than concatenating them into the message string.

• Always include trace_id and span_id to correlate logs with traces
• Use a log level (debug, info, warn, error) that maps to actionable signals, not developer convenience
• Never log sensitive data — PII, secrets, or tokens — even in development environments
• Keep log messages static and put variable data in structured fields to enable aggregation
• Standardize timestamp format across all services to RFC 3339 or Unix milliseconds

Common Structured Logging Pitfalls

The most expensive logging mistake is logging too much. High-cardinality fields like user IDs or IP addresses can balloon log volume by orders of magnitude. Use sampling strategies for high-volume debug logs and reserve verbose logging for specific trace IDs or error conditions. The second mistake is inconsistent field naming — one service using user_id, another using customerId, and a third using user.id makes cross-service queries impossible without transformation layers.

"A log line that cannot be queried might as well not exist. Structured logging is not about readability — it is about making every log entry a first-class citizen in your observability platform."

Distributed Tracing in Microservices

Distributed tracing is the single most effective tool for debugging microservice architectures. A trace that spans twenty services tells you exactly where time is spent, which requests fail, and how errors propagate. Without traces, debugging a slow checkout flow means checking logs across a dozen services and guessing at causality.

Trace Context Propagation

Trace context must be propagated across every service boundary — HTTP headers, message queue metadata, gRPC metadata, and even across async boundaries like scheduled jobs or background workers. OpenTelemetry handles this automatically when you use its instrumentation libraries for HTTP, gRPC, and messaging frameworks. The trace ID flows from the ingress gateway through every downstream service, collecting spans along the way.

• Instrument all ingress points: API gateways, load balancers, ingress controllers
• Propagate trace context through message queues using headers or message metadata
• Include trace IDs in log output to enable log-to-trace correlation
• Use sampling strategies that preserve traces with errors or high latency
• Add custom attributes to spans for business context — user tier, product SKU, region

Reading and Interpreting Traces

A well-annotated trace reveals the critical path of a request. Focus on the span with the longest duration — that is your bottleneck. Look for spans with error events or high attribute cardinality. Compare traces across successful and failed requests to identify patterns. If traces consistently show a specific downstream call timing out, you have a dependency problem, not an application bug.

Alerting: What to Alert On and How to Avoid Alert Fatigue

Alert fatigue is the single biggest threat to incident response. When a team receives fifty alerts per shift, every alert is ignored. The goal of a healthy alerting strategy is not to detect every anomaly — it is to produce a small, high-signal set of notifications that require human judgment. Everything else should be a dashboard or a log query.

The Alert Tier System

Organize alerts into three tiers. Tier 1 alerts page an on-call engineer immediately because they indicate a user-facing issue — high error rate, complete service unavailability, or SLO burn rate exceeding threshold. Tier 2 alerts create a ticket for next-business-day triage — elevated latency, degraded but not failing components. Tier 3 alerts are informational — certificate nearing expiry, approaching storage limits.

• Only page on symptoms, not causes. The user sees a 5xx error — page on that, not on CPU usage
• Use multi-condition alerts that require sustained deviation before firing (e.g., 5 minutes above threshold)
• Set a maximum of three paging alerts per engineer per shift to maintain signal quality
• Review and prune alert rules every quarter — stale alerts erode trust in the system
• Include runbook links in every alert so the responder knows the first three steps

Burn Rate Alerting

Burn rate alerting triggers when your error budget is being consumed faster than expected. Unlike static threshold alerts, burn rate alerts are directly tied to your SLOs. If your SLO is 99.9% availability over 30 days, you can afford about 43 minutes of downtime. A burn rate alert fires when the projected budget depletion would exhaust your window faster than planned, giving you time to respond before the SLO is breached.

Building Useful Dashboards

Most engineering dashboards are graveyards of unused charts. A dashboard is useful when it answers a specific question without requiring interpretation. The best dashboards are built for a single persona and a single use case: an on-call dashboard for incident triage, a weekly review dashboard for capacity planning, and a team dashboard for tracking SLO attainment.

The On-Call Dashboard

The on-call dashboard should fit on a single screen and answer four questions: is the service up? what is the error rate? what is the latency distribution? has the error budget been breached? Every chart must have a clear threshold line so the viewer can immediately tell if the current value is healthy. Avoid putting more than six charts on a single on-call dashboard — cognitive load matters during an incident.

• Start with RED metrics: Rate (requests per second), Errors (failed requests), Duration (latency percentiles)
• Add USE metrics for infrastructure: Utilization, Saturation, Errors for each resource
• Include SLO attainment and burn rate as prominent status indicators
• Link every chart to its underlying logs or trace query for one-click drilldown
• Use logarithmic scales for latency charts — linear scales hide important variation at the tails

Common Dashboard Anti-Patterns

The most common anti-pattern is the dashboard that shows every metric your infrastructure produces. These "wall of green" dashboards create a false sense of security and make it impossible to find the signal during an incident. Other anti-patterns include using pie charts for time-series data, stacking multiple metrics on inconsistent axes, and failing to label threshold lines. If a chart requires a comment to explain, it does not belong on the dashboard.

SLIs, SLOs, and Error Budgets

Service Level Indicators, Objectives, and Error Budgets form the contract between your engineering team and your users. SLIs are the raw measurements — request latency, error rate, throughput. SLOs are the targets you commit to — 99.9% of requests complete in under 200 milliseconds. Error budgets are the allowable margin of failure — the 0.1% headroom that gives teams permission to deploy, experiment, and iterate without fear of violating commitments.

Choosing Meaningful SLIs

Good SLIs are user-facing, measurable, and actionable. Availability (the fraction of successful requests), latency (the fraction of requests under a threshold), and freshness (the age of served data) are the most common SLIs for web services. The key is to measure from the user's perspective — a request that succeeds in 50 milliseconds from your server but takes two seconds on a slow mobile network is a failure from the user's point of view.

Setting SLO Targets Realistically

A 99.999% SLO sounds impressive but comes with enormous engineering cost. Each additional nine requires about ten times the investment in redundancy, testing, and operational tooling. Start with 99.9% for most services and reserve higher SLOs for customer-facing critical paths. Be honest about what you can achieve — an SLO that is consistently violated is worse than no SLO at all, because it normalizes failure and erodes engineering trust in the metrics.

How Error Budgets Enable Velocity

Error budgets transform reliability from a constraint into a measurable risk. When the budget is full, teams can deploy freely, knowing they have headroom. When the budget is depleted, the team halts non-critical releases and focuses exclusively on reliability work. This creates a clear, data-driven decision process that replaces subjective debates about whether a release is "safe enough."

Observability for Serverless and Edge

Serverless and edge computing introduce unique observability challenges. Functions are ephemeral, infrastructure is abstracted, and the execution environment is distributed across global points of presence. Traditional agent-based monitoring fails because there is no host to run an agent on. Observability in these environments requires a different approach: push-based telemetry, fine-grained distributed tracing for cold starts, and careful management of cardinality across global deployments.

Serverless-Specific Patterns

Cold starts are the dominant latency factor in serverless functions. Instrument function initialization separately from request handling so you can distinguish cold-start latency from business logic latency. Use structured logging to capture invocation context — request ID, region, function version, execution environment — and emit custom metrics for invocation count, duration, and error rate asynchronously to avoid blocking the response path.

• Instrument cold start duration as a custom metric — this is invisible in standard cloud provider metrics
• Use OpenTelemetry propagators compatible with Lambda extensions or Cloudflare Workers
• Batch telemetry export to avoid adding latency to function execution time
• Set up log-based metrics for environments where custom metric APIs are unavailable
• Tag all telemetry with deployment environment, region, and function version for effective filtering

Edge Computing Considerations

Edge platforms like Cloudflare Workers or Vercel Edge Functions execute at dozens of global locations. This geo-distribution makes traditional centralized telemetry collection impractical. Use edge-native observability tools that collect telemetry at each PoP and aggregate centrally with minimal overhead. Monitor region-level error rates — a regional ISP issue could degrade your service in one geography while the global average looks healthy.

Building an Observability Culture

Tools and pipelines are necessary but insufficient. The hardest part of observability is cultural. Your team must value instrumented code as much as tested code. Every new feature should include observability instrumentation in its definition of done, just as it includes unit tests and code review. This requires infrastructure — shared libraries, documented conventions, and a observability champion in each team.

Making Observability a First-Class Concern

Start by creating a telemetry design document that defines the standard fields, metric naming conventions, and tracing requirements for every service. Make this document part of the onboarding process for new services. Set up automated checks in CI that enforce minimum instrumentation — for example, reject pull requests that add new HTTP handlers without corresponding trace instrumentation or structured log entries.

• Include observability instrumentation in the definition of done for every feature
• Run regular game days where the team practices debugging using only dashboards and trace queries
• Celebrate observability wins — share incident postmortems that highlight how good telemetry led to fast resolution
• Assign rotating observability champions who review telemetry quality across teams each sprint
• Invest in shared instrumentation libraries that make it easier to do the right thing than the wrong thing

The Feedback Loop

Observability is not a one-time implementation. It is a continuous feedback loop. Instrument, ship, observe, learn, and improve. When an incident exposes a blind spot — a metric that was not tracked, a log field that was missing, a trace that was dropped — treat that as a bug in your observability platform and fix it. Over time, your telemetry becomes more complete, your debugging becomes faster, and your team develops the confidence that comes from knowing they can understand any failure mode, even the ones they have never seen before.

"The goal of observability is not to prevent incidents. It is to ensure that when incidents happen, your team has the data, tools, and confidence to resolve them before your users notice."

Adopting observability is a journey, not a purchase. Start small — instrument one critical service end-to-end, build a single useful dashboard, and write one meaningful SLO. Let the value speak for itself. Once your team experiences the difference between guessing and knowing during an incident, they will never want to go back.