From StatsD to OpenTelemetry: Scaling Metrics with vmagent

A team migrated a large-scale metrics platform, mostly instrumented with StatsD, to OpenTelemetry (OTLP) backed by a Prometheus-based storage system. They standardized on OTLP for internal services while keeping StatsD as a fallback, using a shared metrics library to dual-emit both protocols for a low-friction rollout. The switch to OTLP cut metrics-related CPU usage from 10% to under 1%, improved reliability over UDP-based StatsD, and unlocked Prometheus-native features like exponential histograms.

However, their highest-volume, most critical services saw memory pressure and increased garbage collection under OTLP due to extremely high-cardinality metrics. Switching those services to delta temporality reduced memory overhead, trading occasional data gaps for improved stability.

For aggregation, the team evaluated and rejected Veneur, recording rules, m3aggregator, the OTel Collector, and Vector before settling on VictoriaMetrics' vmagent. They built a two-layer router/aggregator architecture for consistent sharding and stateful aggregation. The result: a single production cluster scaling to hundreds of aggregators, ingesting over 100 million samples per second, while cutting costs by an order of magnitude.