Migrating to Stateful Cloud Native Platforms on AWS EKS

The original platform ran on r6i.4xlarge EC2 instances with directly mounted gp2 EBS disks, a per-tenant PostgreSQL database on RDS Single-AZ, and a manually managed Redis cache layer bootstrapped via shell scripts. The model worked — until it didn't. Each tenant had its own instance set, which provided strong isolation but made operational cost prohibitive: 40 tenants meant 40 infrastructure stacks, 40 patch pipelines, 40 failover runbooks. The operations team spent more time managing infrastructure than delivering value.

State was implicit everywhere: user sessions in local memory, processing queues in unpartitioned PostgreSQL tables, and tenant configurations scattered across manually versioned .env files. There was no reliable inventory of which instance served which tenant at any given moment. When a node failed, recovery involved SSH, manual log inspection, and EBS snapshot restoration — an average MTTR of 47 minutes for data incidents.

Pressure to migrate came from two directions: cost (the EC2 bill grew linearly with tenant count) and compliance (a SOC 2 Type II audit identified the lack of auditable logical separation between tenants as a high-risk finding). The decision to move to EKS was not made out of trend-chasing — it was made because the Kubernetes operator model offered the lifecycle automation the team needed to scale without proportionally growing the operations headcount.

Namespace per tenant is necessary, but far from sufficient in a financial environment. The namespace is a naming and RBAC boundary — it does not prevent a misconfigured pod from accessing another tenant's network, nor does it prevent an encryption key from being inadvertently shared.

The isolation model we implemented has four layers:

1. Network isolation: A default NetworkPolicy denies all ingress and egress traffic within the cluster, except explicitly permitted routes. Each tenant can only communicate with its own namespace and the platform-services namespace (containing the egress proxy and authentication service). We used Calico as the CNI for GlobalNetworkPolicy support, which applies rules before namespace-level policies.

2. Identity isolation: Each tenant ServiceAccount has a dedicated IAM Role via IRSA (eks.amazonaws.com/role-arn annotation). The IAM Role has a StringEquals condition on sts:AssumeRoleWithWebIdentity that verifies the OIDC token sub — ensuring only pods from the correct namespace can assume that role. This prevents privilege escalation even if a pod manages to create a ServiceAccount in another namespace.

3. Encryption isolation: Each tenant has its own KMS Customer Managed Key (CMK). EBS volumes, Secrets Manager secrets, and S3 data for each tenant are encrypted with the tenant's CMK. The CMK key policy uses kms:ViaService and aws:SourceAccount conditions to ensure only calls originating from the correct services can use the key.

4. Resource isolation: Per-namespace ResourceQuota and LimitRange prevent a tenant from monopolizing CPU or memory on the shared node group. For Gold tenants, taints on dedicated nodes ensure no pod from another tenant is scheduled there, even under scheduling pressure.

The decision to invest in a custom operator rather than Helm + kubectl scripts was the most internally debated. The argument against was legitimate: operators have high development cost, introduce a critical component into the control plane, and are difficult to debug when the reconciliation loop enters an inconsistent state.

The argument in favor, which prevailed, was based on three properties scripts do not have:

Continuous reconciliation: A script runs once. An operator continuously observes desired state versus actual state. When an engineer accidentally deleted a tenant's NetworkPolicy in production (it happened), the operator recreated it within 30 seconds — before any unauthorized traffic could occur. With scripts, that would have been a security incident.

Full lifecycle management: The operator handles creation, update, and deletion (with finalizers to ensure data is not deleted before backup). A provisioning script rarely includes safe deprovisioning logic — and when it does, it is tested with far less rigor.

Native status and observability: The TenantWorkspace CRD has a .status.conditions field with conditions like StorageProvisioned, NetworkPolicyApplied, SecretsSync. Any tool consuming the Kubernetes API can observe each tenant's state in real time. This was critical for the support team: instead of SSH-ing into nodes, they run kubectl get tenantworkspace -n platform-ops and see every tenant's state in seconds.

The real cost of the operator was approximately 3 development sprints and 1 sprint of integration test hardening. Operational break-even was reached in month four, when the tenant count exceeded 60 and management complexity would have required hiring an additional operations engineer.

PersistentVolumeClaims backed by EBS have a fundamental limitation that frequently surprises migrating teams: an EBS volume can only be mounted on a single node at a time (ReadWriteOnce). This has direct implications for deployment strategies.

With a Deployment (not StatefulSet), a rolling update may attempt to create the new pod on a different node before the old pod releases the volume — resulting in the new pod stuck in ContainerCreating indefinitely, waiting for the volume to detach. In financial environments with short maintenance windows, this can be catastrophic.

The solution we adopted was threefold: (1) use StatefulSet for all EBS-backed PVC workloads, which guarantees the old pod is terminated before the new one is created; (2) configure podManagementPolicy: Parallel only for StatefulSets where startup order does not matter (Redis cache), keeping OrderedReady for those with initialization dependencies; (3) implement PodDisruptionBudget with minAvailable: 1 to ensure node drains (during EKS upgrades) do not remove the sole pod of a critical tenant without a ready replacement.

For the use case of artifacts shared across multiple pods of the same tenant (asynchronously generated reports, for example), EFS with ReadWriteMany was the right choice — but with an important caveat: EFS has significantly higher write latency than EBS for synchronous operations (typically 1-5ms vs. 0.1-0.5ms for gp3). Any code path that wrote to EFS synchronously was refactored to asynchronous writes via SQS, with EFS as the final destination after processing.

The most common mistake I see in multitenant platforms is treating observability as an infrastructure concern — CPU, memory, and latency metrics aggregated at the cluster level. This is useless when a customer calls complaining of slowness: you cannot isolate the problem without tenant dimensions.

The most impactful architectural decision in the observability phase was defining tenant_id as a mandatory dimension on all platform metrics, traces, and logs. This was implemented in three layers:

Application: The OTel SDK was configured with a Resource that includes tenant.id and tenant.tier as attributes. All spans inherit these attributes automatically. On the metrics side, we used exemplars to link high-cardinality metrics to specific traces — essential for investigating P99 for a specific tenant.

Infrastructure: CloudWatch Container Insights was configured with enhanced observability on EKS, collecting per-pod metrics with Kubernetes labels. A CloudWatch Logs Insights metric filter extracts tenant_id from structured logs and creates custom metrics with that dimension.

Per-tenant SLOs: In Datadog, each tenant has an independent SLO monitor. The burn rate alert uses a 1-hour window (fast alert) and 6-hour window (trend alert). When the 1-hour burn rate exceeds 14.4x (consuming 2% of the budget in 1 hour), a PagerDuty alert is created with tenant_id in the title — the on-call engineer immediately knows which tenant is impacted without investigating dashboards.

The ingestion cost of high-cardinality metrics was a real concern. The solution was using histogram_quantile in Prometheus (running as a Managed Service via AMP) for latency metrics, rather than per-percentile gauges — reducing cardinality by 80% with equivalent precision for SLOs.