Beyond Logs: Why Metrics, Traces, and Cardinality Matter Most
A practical breakdown of metrics vs traces vs logs in observability, cardinality pitfalls, and why logs-to-metrics pipelines are an anti-pattern.
When debugging a production incident, most engineers reach for logs first — yet logs are the least structured and most expensive-at-scale of the four observability signals (metrics, traces, logs, profiling). The source content targets a common anti-pattern: deriving metrics from log lines (logs_to_metrics pipelines), which is costly, laggy, and fragile compared to instrumenting metrics directly at the source.
Metrics work because of bounded cardinality — labels like HTTP method, route, or status code stay cheap and fast to query. But adding unbounded labels like user_id or request_id can explode a metrics backend's time-series count and take it down. This is exactly where traces should take over: spans aren't pre-aggregated, so they can carry arbitrary high-cardinality, per-request context without a cardinality tax.
The practical takeaway for engineers: a per-request log line with identical shape emitted thousands of times a minute is really a metric in disguise and should become a counter or histogram instead. Logs should be reserved for exceptional, narrative-worthy events. Matching each signal to its actual job — rather than defaulting to logs for everything — directly cuts observability cost and complexity.