ADR-014 Accepted 2026-05-28

OpenTelemetry as the Unified Observability Standard

All logs, metrics, and traces emitted by madcatalog services (SAM Lambdas, MCP Go server) use the OpenTelemetry SDK and export via OTLP to a local OTel Collector. The collector fans out to Loki (logs), Tempo (traces), and Prometheus (metrics). Grafana serves as the single pane of glass across all three signals. No vendor-specific agents, no direct Prometheus push, no CloudWatch SDK calls.

Context

At the start of this work, madcatalog had zero observability: 10 of 11 SAM Lambda functions emitted no telemetry at all; the 11th (stripe-webhook) had three bare console.log statements; the Go MCP server used log/slog with two lines. There was no metrics collection, no distributed tracing, and no log aggregation.

As the system grows to handle real Stripe payments via the MPP flow, the absence of observability creates blind spots across three dimensions:

Business health — no visibility into payment success/failure rates, revenue, or webhook processing latency.
Operational health — Lambda cold starts, DynamoDB latency spikes, and error rates are invisible until a user reports a problem.
Debugging — multi-hop request paths (browser → SAM API → DynamoDB → Stripe) cannot be traced end-to-end.

The decision required choosing both an instrumentation standard (how services emit telemetry) and a backend stack (where telemetry is stored and queried).

Decision

Instrumentation standard: OpenTelemetry

All madcatalog services must instrument using the OpenTelemetry SDK and export via OTLP (OpenTelemetry Protocol) to a local OTel Collector. The collector is the single egress point for all telemetry signals.

SAM Lambdas (Node.js) — a shared Lambda layer bootstraps @opentelemetry/sdk-node with auto-instrumentation for AWS SDK, HTTP, and DynamoDB clients. stripe-webhook additionally emits custom OTel metrics for business KPIs (payment counters, amount histograms, webhook duration).
MCP Go server — go.opentelemetry.io/otel with OTLP gRPC exporter; HTTP server wrapped with otelhttp middleware.

Backend stack: LGTM (Loki + Grafana + Tempo + Prometheus)

A self-contained observability/docker-compose.yml runs the full LGTM stack locally. Grafana is the single pane of glass; all three datasources are provisioned as code.

Service	Role	Port	Receives from
OTel Collector	OTLP gateway / fan-out	:4317 gRPC / :4318 HTTP	Lambdas, MCP server
Prometheus	Metrics storage	:9090	Collector :8889 scrape
Loki	Log aggregation	:3100	Collector loki exporter
Tempo	Distributed traces	:3200	Collector OTLP forward
Grafana	Single pane of glass	:3000	Prometheus, Loki, Tempo

Alternatives Considered

Direct Prometheus push from each service
Rejected. Ties every service directly to Prometheus; changing the metrics backend requires touching all services. OTel Collector decouples instrumentation from backend choice.

AWS CloudWatch + X-Ray
Rejected. Vendor lock-in, cost at scale, and requires AWS credentials in all local dev paths. OTel is cloud-agnostic.

Grafana Cloud (remote backends, local collector only)
Rejected for now. Free tier limits are reasonable but adds a network dependency to local dev. The self-hosted LGTM stack can be replaced with Grafana Cloud by changing collector exporter endpoints — the instrumentation layer is unchanged.

Grafana Mimir instead of Prometheus
Rejected for local dev. Mimir is Prometheus’ horizontally-scalable successor and requires object storage configuration. Prometheus is right-sized for a single-developer local stack. Mimir remains the correct choice if this stack is deployed to a shared environment.

Consequences

Positive

Single instrumentation API across Node.js and Go runtimes
Backend-agnostic: swapping Prometheus for Mimir or Grafana Cloud requires only collector config changes, not service changes
Auto-instrumentation handles DynamoDB, HTTP, and AWS SDK spans with no per-Lambda code changes beyond layer attachment
Business KPIs (Stripe revenue, payment failures) visible in Grafana alongside operational metrics — correlated by time
Trace-to-log and trace-to-metric correlation in Grafana out of the box via provisioned datasource links

Negative / Watch

Five new Docker containers add ~400 MB RAM to local dev baseline (task obs:down reclaims if needed)
Lambda layer adds ~30 MB to each function’s deployment package; cold-start overhead of OTel SDK initialisation (~50–100 ms) must be monitored
All new Lambdas must initialise the OTel SDK — new contributors must follow the layer pattern
OTel Collector is a required dev dependency; task obs:up must precede any local testing that exercises telemetry assertions

Implementation Phases

P1 — Done: observability/docker-compose.yml, OTel Collector config, Prometheus/Loki/Tempo configs, Grafana provisioning, Taskfile obs:* tasks, tmuxinator obs pane.
P2: Shared Lambda layer (sam/layers/otel-bootstrap/), SAM template env vars, stripe-webhook custom metrics.
P3: MCP Go server OTel SDK, otelhttp middleware, slog bridge.
P4: Provisioned Grafana dashboards as JSON, dashboard schema validation in pre-commit, PromQL lint.