Streaming CloudWatch Metrics to VPC OpenTelemetry Collectors via Lambda
AWS explains how a Lambda transform bridges CloudWatch Metric Streams and Firehose to deliver metrics into private VPC-based OpenTelemetry collectors.
AWS describes a solution built for a customer struggling with API throttling and data loss caused by a Prometheus-style pull-based monitoring setup. The fix streams CloudWatch metrics through Amazon Data Firehose to an OpenTelemetry collector running inside a private VPC, but Firehose can only deliver to public HTTP endpoints, not private ones.
To bridge that gap, the team uses Firehose's data transformation feature to synchronously invoke a Lambda function, which securely forwards metrics through a Network Load Balancer into the VPC-hosted collector. The architecture has four parts: CloudWatch Metric Streams, Data Firehose, the Lambda transform function, and an OpenTelemetry collector running as a container on EC2.
This push-based design eliminates third-party licensing costs, achieves sub-minute latency for alerting, and lets the customer consolidate metrics from AWS, other accounts, and on-prem systems into a single dashboard. For engineers, it offers a concrete reference architecture for building scalable, vendor-neutral observability pipelines using open standards without sacrificing strict VPC-based data privacy requirements.