ML Observability on EKS: Logs, Metrics and Tracing Head-to-Head

A stateless inference pod generates predictable telemetry: a few hundred log lines per minute, CPU/memory metrics, request latency. A distributed training job with PyTorch DDP or Ray Train across 32 GPU nodes is an entirely different category.

First, log volume is non-linear with worker count. Each rank emits epoch progress, checkpointing, gradient norm and NCCL collective logs. With 32 workers and 10-minute epochs, it is common to see 50–200 MB/min of unstructured stdout arriving at the collection DaemonSet — before any business-application log.

Second, GPU metrics have high cardinality. DCGM Exporter exposes ~40 metrics per GPU (SM utilization, memory bandwidth, NVLink throughput, tensor core activity, ECC errors). On a p4d.24xlarge node with 8 A100s that is 320 time series per node, scraped every 15 s. Across 20 nodes you have 6,400 active series — within CloudWatch's limit (10 custom metrics per namespace free up to 10 k) but dangerously close to blowing the custom-metrics budget if you do not filter at the source.

Third, distributed tracing in training differs from microservice tracing. There is no request trace; there is an execution graph of CUDA operations, collective communication and data I/O. OpenTelemetry still lacks native semantics for this — you must manually instrument PyTorch hooks or use MLflow Tracing, which is a separate layer.

These three characteristics — explosive log volume, high-cardinality GPU metrics and absent native tracing — define the evaluation criteria for this bake-off.

Fluent Bit (native EKS DaemonSet) is the default log collector in the aws-for-fluent-bit add-on. It tails /var/log/containers/*.log, parses JSON or regex, and routes to CloudWatch Logs, S3, Kinesis Data Streams or Firehose. Backpressure configuration via mem_buf_limit and storage.type filesystem is critical: without it, a 200 MB/min burst of training logs will OOM the DaemonSet on memory-constrained nodes. Its strength is lightness — ~50 MB memory in normal operation — and native IAM integration via IRSA. Its weakness is that it collects no metrics or traces; it is a pure log collector.

OpenTelemetry Collector (OTel) is the most versatile contender. Deployed as a DaemonSet or Deployment via opentelemetry-operator, it unifies logs (via filelog receiver), metrics (via prometheusreceiver scraping DCGM) and traces (via otlp receiver) in a single pipeline. Operational cost is higher: the pipeline needs tuning of batch processor (send_batch_size, timeout), memory_limiter and parallel exporters. But the ability to route to multiple backends simultaneously — CloudWatch, S3 via OTLP/Parquet, Jaeger — without duplicating agents is the real architectural differentiator.

CloudWatch Container Insights with Enhanced Observability for EKS is the AWS managed option. Enabled via the amazon-cloudwatch-observability add-on, it automatically installs a pre-configured OTel Collector, DCGM Exporter and Fluent Bit. Operational cost is zero, but financial cost is high: Enhanced Observability charges $0.009 per vCPU-hour and $0.009 per GB memory-hour per monitored node — on a 20-node p4d.24xlarge cluster (96 vCPUs each), that is ~$17,280/month in observability alone.

Datadog Agent with DogStatsD is the most complete enterprise option. The datadog/datadog Helm chart installs the Agent as a DaemonSet with log collection, metrics (including native DCGM integration), APM and NPM. The differentiator is ML Observability (formerly Weights & Biases integration) and automatic correlation between logs, metrics and traces. Cost is $15–23/host/month depending on tier, plus log ingestion at $0.10/GB after the free tier.

The amazon-cloudwatch-observability add-on with Enhanced Observability is the simplest option to enable — one eksctl enable addon and you have DCGM, Fluent Bit and OTel Collector pre-configured. But the pricing model is a trap for ML clusters.

Enhanced Observability charges per vCPU-hour and GB-memory-hour per monitored node, regardless of how many metrics you actually use. A p4d.24xlarge has 96 vCPUs and 1,152 GB RAM. At $0.009/vCPU-hour, the vCPU dimension alone costs $0.864/hour per node. With 20 nodes running 24/7 that is $12,441/month — and the memory dimension adds another ~$4,976/month. Total: ~$17,417/month.

For comparison, the GPU cost of those 20 nodes is ~$460,800/month (at $32.77/hour per node). So observability represents ~3.8% of compute cost — which might seem reasonable until you realise that a standalone OTel Collector with EMF exporter to CloudWatch Metrics covers 90% of the same use cases for ~$800/month.

Practical recommendation: use Enhanced Observability only on dev/staging clusters with smaller nodes (m5, c5) where the per-vCPU cost is negligible. On production clusters with large GPU instances, build the OTel pipeline manually with memory_limiter set to 80% of the container limit, batch processor with send_batch_size: 1000 and timeout: 10s, and prometheusreceiver scraping DCGM every 30 s (not 15 s — you halve metric volume without meaningful diagnostic loss).

The OpenTelemetry Collector is the only one of the four contenders that solves the problem of multiple backends without agent duplication. In regulated financial environments this is often mandatory: you need to send logs to CloudWatch (7-year regulatory retention via S3 Glacier), metrics to an internal analytics backend (Prometheus + Thanos or Amazon Managed Prometheus) and traces to Jaeger or Tempo — all simultaneously, with different delivery guarantees.

The pipeline configuration I use in production for ML workloads has three critical stages:

1. Parallel receivers with memory isolation: filelog receiver with start_at: beginning disabled (prevents re-reading historical logs on pod restart), prometheusreceiver with scrape_interval: 30s and target_allocator enabled to distribute scraping across multiple Collectors when the cluster has >50 nodes.

2. Chained processors with circuit breaker: memory_limiter as the first processor (limit at 80% of the container's resources.limits.memory), followed by resource processor to add k8s.cluster.name, ml.job.id and gpu.node.type attributes — essential for cross-signal correlation. The batch processor comes last, not first: placing batch before memory_limiter is the most common mistake I see in architecture reviews.

3. Exporters with retry and persistent queue: awscloudwatchlogs exporter with log_stream_name derived from k8s.pod.name (avoids per-stream throttling), awsemf exporter with explicit metric_declarations to avoid sending all 320 DCGM series to CloudWatch (select the 8–12 that actually matter: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_MEM_COPY_UTIL, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL). retry_on_failure with max_elapsed_time: 300s and sending_queue with storage: file_storage ensures a temporary CloudWatch throttle does not lose training data.

In financial environments, training logs frequently contain information that should not leave the security perimeter: customer IDs in validation datasets, feature values derived from transactions, or simply the model name and version (which is sensitive competitive information). This adds a layer of requirements that most observability comparisons ignore.

CloudWatch Logs with KMS: all four collectors support sending to CloudWatch Logs, but only if you configure kms_key_id on the log group. Fluent Bit does this via auto_create_group Off and pre-creating the group with aws logs create-log-group --kms-key-id arn:aws:kms:.... OTel Collector requires the log group to exist before the first PutLogEvents — a common race condition in clusters that scale rapidly.

IAM with context conditions: the collector's IRSA policy must have Condition: StringEquals: aws:RequestedRegion: [region] to prevent cross-region exfiltration, and aws:SourceVpc if you use VPC Endpoints for CloudWatch. The Datadog Agent stores the API key in a Kubernetes Secret — use External Secrets Operator with AWS Secrets Manager and 90-day automatic rotation, not a manual kubectl create secret.

Data masking in the pipeline: OTel Collector has the transform processor with OTTL (OpenTelemetry Transformation Language) that allows masking fields before sending: replace_pattern(body, "customer_id=\\d+", "customer_id=REDACTED"). Fluent Bit has the lua filter for the same purpose. Datadog has Sensitive Data Scanner on the backend side, but that means the data has already left AWS — for regulated environments, masking must happen in the collector, not the backend.

Retention and immutability: for compliance with financial regulations (BACEN, CVM, SOX), configure S3 Object Lock in COMPLIANCE mode on log archive buckets with a 7-year retention period. Fluent Bit with S3 output supports s3_key_format /%Y/%m/%d/%H/ for temporal partitioning that facilitates audit queries with Athena.