Training and inference on one GPU fleet: four ways to share 10,000 accelerators

The common mistake is framing the problem as "training versus inference". It is not a fight between workloads — it is a fight between two demand regimes that never line up in time. Training is batch: it arrives in bursts, wants 64 cards for days, tolerates waiting in a queue and tolerates being interrupted if there is a checkpoint. Inference is online: it arrives as a traffic curve, wants a predictable p99, tolerates neither waiting nor preemption. When each regime gets its own pool, each pool is sized for its own peak, and the sum of both peaks is the bill. CMB's 35% before the change is exactly that sum: half the fleet idle, waiting for the neighbor's peak.

The architecture decision, then, is who arbitrates the boundary between the two regimes — and at what granularity. There are four honest answers:

1. Nobody arbitrates: separate pools, boundary fixed at provisioning time.
2. An open scheduler arbitrates inside the cluster: Kueue for quota-aware admission, KEDA to scale inference on metrics, HAMi to slice the card, Fluid so the data lands before the GPU goes idle. This is CMB's design.
3. AWS arbitrates with the same scheduler, packaged: SageMaker HyperPod with EKS orchestration and task governance, which installs Kueue and exposes quota, fair-share and borrowing as console policy.
4. A managed service arbitrates outside the cluster: inference on Bedrock or SageMaker endpoints, training as SageMaker jobs — no fleet of your own to share.

The rest of this article compares the four. The answer is not the same for a bank with 10,000 cards and a fintech with 40.

The announcement names five projects and gives each a role. It is worth translating each role into the parameter it controls, because that is where utilization is won or lost.

Kueue — quota-aware admission: each team submits to a LocalQueue in its own namespace; the LocalQueue points to a ClusterQueue with a nominalQuota per ResourceFlavor (one flavor per card type — that is how 10,000 heterogeneous accelerators become one queue). ClusterQueues in the same cohort lend idle quota to each other up to borrowingLimit, and spec.preemption.reclaimWithinCohort: Any takes the loan back when the owner needs it. The main gain is what the announcement calls "not reserving capacity prematurely": the job only creates pods once its whole quota is guaranteed, so 64 cards do not sit half-allocated waiting for the other 32.

KEDA — inference scales on a demand signal: a ScaledObject with the prometheus scaler (serverAddress, query, threshold, activationThreshold) scales model-server replicas by queue depth or tokens per second, not CPU. When traffic drops, replicas disappear and quota flows back to the cohort for training.

HAMi — the card becomes a fractional resource: a CNCF Incubating project that exposes nvidia.com/gpumem (in MB) and nvidia.com/gpucores alongside nvidia.com/gpu, supporting NVIDIA, Ascend, Cambricon and others. It is what lets five LoRA tenants share one base-model instance — Twinkle, the bank's in-house framework, does this by default and reports 80% fewer accelerators for the same work.

Fluid — data before GPU: a CNCF Incubating project with Dataset + cache runtime (Alluxio, JuiceFS) and warm-up. Checkpoint and model are already on the node when the pod starts. A GPU waiting on object storage is zero utilization that shows up as 100% "allocated".

Prometheus closes the loop: it is the source of truth both for KEDA and for the dashboard that proved the 60%.

Every percentage point of utilization above 50% is bought with less isolation. At a bank that is not a detail — it is the first thing the auditor asks.

There are three mechanisms to put two pods on the same card, and they are not equivalent:

Time-slicing: the device plugin or DRA driver advertises N slots per GPU and the CUDA scheduler takes turns. It works on every NVIDIA instance on AWS, needs no special hardware — and the EKS documentation is explicit: it offers no memory or compute isolation between pods. A tenant with a memory leak takes the neighbor down with CUDA out of memory. Fine for dev and sparse inference; not fine for two distinct business tenants.

MIG: hardware partitioning, up to 7 instances per A100/H100/H200/B200 (fewer on Blackwell g7/g7e), each with dedicated memory, compute and bandwidth. It is the only one of the three with hardware fault isolation — the kind PCI-DSS or a BACEN 4.893 audit trail accepts without debate. The price: the partition is static per node (nvidia.com/mig.config.state must be success) and only exists on the P family and Blackwell G instances.

HAMi: software virtualization with per-pod memory and core limits, enforced where the device backend allows. Finer than MIG (any fraction, not just 7 profiles), safer than time-slicing (the limit exists), and the only one of the three that speaks to non-NVIDIA accelerators — relevant for CMB with a heterogeneous fleet, less so if you are 100% on EC2. Fault isolation remains software-level.

The rule I use: MIG for multi-tenant inference with an SLO, HAMi or time-slicing for LoRA fine-tuning and experimentation, whole card for pre-training. Option B lets you choose per ResourceFlavor; option C only offers MIG or whole card. If your case is "five LoRA tenants on one base", that decides it.

Unified-platform diagrams show the happy path. The two incidents I have watched happen sit outside the drawing.

Preemption without checkpoints. When inference scales up at 9 AM and the cohort reclaims borrowed quota, Kueue evicts the training job that was using someone else's GPUs. If the job does not checkpoint every N steps, it loses everything since the start — and the next day someone will ask to "turn off that preemption", which sends the platform back to 35%. The configuration that prevents this is not in Kueue; it is in the job: checkpoint at an interval short enough that average rework stays under 15 minutes, written to the Fluid cache with write-back to S3. In Kueue, withinClusterQueue: LowerPriority protects training from being interrupted by another training job from the same team; reclaimWithinCohort: Any says borrowed quota can be taken back at any time. Those two lines together are the contract.

Oscillating scale. KEDA with a tight threshold and a short cooldownPeriod produces flapping: replicas go up, training gets preempted, traffic drops, replicas go down, training restarts from checkpoint, traffic goes up. Each cycle costs a model cold start (tens of seconds for 30 GB of weights without cache) and training rework. Use an activationThreshold above zero so noise does not wake a replica, and a cooldownPeriod measured in minutes, not seconds.

Cold data. Without Fluid or an equivalent, the inference pod starts, asks S3 for 30 GB of weights and sits allocated for two minutes at 0% use. On the dashboard that counts as "GPU in use". That is the difference between allocated and real utilization — measure DCGM_FI_DEV_GPU_UTIL, not the pod's requests.