CloudWatch to OTel: Tearing Down the Observability Bridge Pattern

Every organization that grows beyond two or three engineering teams faces the same tension: AWS services emit metrics natively to CloudWatch — Lambda, RDS, EKS, API Gateway, MSK — but the corporate observability backend speaks OTLP. The result is a split world: SREs need to open two consoles to correlate an incident, alerts live in different namespaces, and business dashboards become impossible to build without manual ETL.

The bridge pattern exists to solve exactly this. The core idea is simple: a Lambda function (or an OTel collector running on ECS/EKS) subscribes to CloudWatch metric streams via CloudWatch Metric Streams or polls the GetMetricData API, transforms the payload to OTLP format, and forwards it to a collector endpoint. In theory, you get a single observability control plane. In practice, the complexity hides in the details.

What makes this problem especially treacherous in financial environments is the combination of three factors: (1) metric volume — a mid-size AWS account with EKS, Multi-AZ RDS, and Lambda can easily generate 50,000+ metric series per minute; (2) business latency — anomaly detection SLOs require freshness of 60 seconds or less; (3) API cost — each GetMetricData call has a direct cost and a quota of 50 metrics per request, meaning naive polling at scale breaks both the budget and AWS rate limits.

The pattern has two ingestion flavors, and the choice between them defines everything that follows.

CloudWatch Metric Streams + Kinesis Firehose is the low-latency path. Streams deliver data in OpenTelemetry 0.7 (protobuf) or JSON format with 2–3 second latency. The cost is predictable: $0.003 per 1,000 metric updates. For an account with 100k active series, that's roughly $300/month before Firehose costs. The critical architectural advantage is that you're not polling — data flows, and the Lambda at the Firehose destination receives batches, not individual calls.

Polling via GetMetricData is the fine-grained control path. You choose exactly which metrics, at what resolution, and what lookback period. But the API has a quota of 50 metrics per request and 500 requests per second per account (soft limit). In a large account, hitting that limit is a matter of minutes if the polling code doesn't implement exponential backoff with jitter and doesn't batch correctly. I've seen financial production systems break critical alerts because the poller entered throttling at 09:00 on a Monday morning — exactly when the market opens and transaction volume explodes.

The transformation Lambda needs three non-negotiable capabilities: (1) idempotency — Firehose can re-deliver batches on failure; the Lambda must detect duplicates via payload hash; (2) circuit breaker for the external OTel endpoint — if the collector is down, the Lambda cannot loop consuming concurrency; (3) DLQ with alarm — batches that fail after 3 attempts go to SQS DLQ and a CloudWatch alarm must fire within 5 minutes.

In financial environments, the observability bridge is an underestimated data exfiltration vector. Business metrics — transaction volume, payment latency, authentication error rates — are sensitive information. A misconfigured OTel endpoint can leak this data out of the AWS account without any alarm.

The three control layers I implement in every deployment of this pattern:

IAM with resource conditions: The Lambda role must have cloudwatch:GetMetricData and cloudwatch:ListMetrics permission with condition aws:ResourceTag/Environment: production — never a wildcard. For Metric Streams, the Firehose role needs cloudwatch:PutMetricStream and firehose:PutRecord, but the destination Lambda role must be separate and have only s3:GetObject on the buffer bucket (if using S3 as fallback) and invocation permission.

KMS CMK for payload in transit: Firehose must be configured with ServerSideEncryption using a CMK managed by the security team. The transformation Lambda must decrypt with the same key. This ensures that even unauthorized access to the Firehose stream doesn't expose readable data.

VPC Endpoint for the OTel collector: If the collector runs on ECS inside the VPC, the Lambda must be in the same VPC and use private DNS. If the collector is external (SaaS), traffic must exit through a NAT Gateway with a fixed IP whitelisted in the vendor's firewall — never through an Internet Gateway without egress control. Add a WAF rule on API Gateway if the external collector exposes an HTTP endpoint.

A detail that burned one of my clients: the bridge Lambda running with a 15-minute timeout (maximum) and no reserved concurrency can consume the entire account concurrency pool during an ingestion spike, bringing down critical business functions. Reserve explicit concurrency — typically 10–20 instances are sufficient for a Firehose with a 5MB batch.

After implementing and debugging this pattern in three different financial environments, the design that works has these concrete characteristics:

Ingestion via Metric Streams with namespace filter: Configure the stream to include only relevant namespaces — AWS/Lambda, AWS/EKS, AWS/RDS, AWS/ApiGateway, AWS/MSK — and explicitly exclude high-cost, low-value namespaces like AWS/Billing and AWS/CloudFront (unless you monitor CDN). The format should be opentelemetry0.7 (protobuf), not JSON, to reduce payload size by ~40%.

Firehose with 60s/5MB buffer and S3 as fallback: Configure BufferingHints with IntervalInSeconds: 60 and SizeInMBs: 5. The fallback S3 bucket should have a lifecycle policy to expire objects after 7 days — it exists only for manual replay in case of external collector failure, not as permanent storage.

Transformation Lambda in Python/Rust with OTel SDK: Use opentelemetry-sdk to build the OTLP payload. Add fixed resource attributes at transformation time: cloud.account.id, cloud.region, deployment.environment. These attributes enable cross-account correlation in the backend. Timeout should be 3 minutes (not 15) — if transforming a batch takes more than 3 minutes, there's a volume problem that needs to be solved in the stream filter, not in the timeout.

Observability of the bridge itself: Instrument the Lambda with custom metrics: bridge.metrics.transformed.count, bridge.export.latency.p99, bridge.export.errors.count. Create a CloudWatch dashboard for the bridge itself — it's ironic but necessary: you need to observe the observability system. Define an SLO of bridge.export.errors.count < 0.1% with a 5-minute burn rate alarm.