Kubernetes and GPU clusters: handing GPUs to jobs

The cluster has 24 GPUs and the dashboard shows 16 of them idle, yet a training job that needs 16 GPUs stays Pending; worse, half of its Pods start, the other half do not, and the half that started holds its GPUs while waiting. By default Kubernetes finds a node for one Pod at a time, while distributed training needs "either the whole group of Pods is placed, or none of them holds anything"; and GPUs can only be handed out whole, ideally with topology in mind. The total number of idle GPUs tells you little about whether a large job can run.

Draft, pending review. The site owner has not reviewed this note yet; facts, emphasis and wording may change. Check the primary sources linked in the text before relying on it.

For how the program inside a Pod hands work to the GPU, see The framework layer; for why every rank of a training job must be present, see Multi-GPU training. This note follows the official kubernetes.io and NVIDIA documentation, checked on 2026-10-15; the latest Kubernetes release at that time was 1.37.1 (released 2026-09-15, see the releases page).

The few Kubernetes concepts you need

The limits for CPU and memory can add up to more than a node's capacity (overcommit): a container over its CPU limit is throttled, one over its memory limit is OOM-killed. GPUs are different. They are extended resources, which the docs say must be integers, and "Extended resources cannot be overcommitted, so request and limit must be equal …" when both are given (resource management page). One device also cannot be shared between containers. Per the GPU scheduling page, GPUs are specified only in limits, and requests default to the limit.

How GPUs become schedulable resources

Through a device plugin. The vendor's plugin registers with the kubelet over gRPC and reports the devices it manages through ListAndWatch; the kubelet writes the count into the node status, such as "this machine has 8 nvidia.com/gpu". The scheduler uses this count to decide whether a Pod fits; when a container is created, the kubelet calls the plugin's Allocate so it can do device-specific setup. An 8-GPU training Pod therefore looks like this:

apiVersion: v1
kind: Pod
metadata:
  name: trainer-0
spec:
  containers:
  - name: trainer
    image: registry.example.com/trainer:v1
    resources:
      limits:
        nvidia.com/gpu: 8

A plugin alone is not enough: the node also needs a driver, a way for containers to use the GPU, labels with the GPU model, and exported metrics. The NVIDIA GPU Operator uses the operator pattern to manage these components together: the GPU driver (run as a container by default, or the node's preinstalled one), the Container Toolkit, the device plugin, GPU Feature Discovery to label nodes, the MIG Manager to configure MIG, and DCGM Exporter to collect metrics for Prometheus. As of 2026-10-15, its latest release on GitHub is v26.7.2 (released 2026-10-14).

Dynamic Resource Allocation (DRA) is another route. It requests devices the way a PersistentVolumeClaim requests storage: drivers publish ResourceSlices describing the available devices, administrators define DeviceClasses, and workloads write ResourceClaims, which can filter devices by attributes with CEL expressions, let several Pods share a device by referencing the same claim, and carry per-claim device configuration. Compared with a device plugin, it is no longer just one integer of "how many GPUs".

Official sources disagree on which version made DRA stable (checked 2026-10-15). The v1.34 release announcement (2025-08-27) says the core of DRA went GA in v1.34, with the resource.k8s.io/v1 APIs graduating to stable and available by default; the DRA concept page is labelled "Stable since Kubernetes v1.35", says it first appeared in v1.30, and says its feature gate is locked and can no longer be turned off. The feature gate reference lists DynamicResourceAllocation as alpha in 1.30–1.31, beta in 1.32–1.33, and with one stable row from 1.34 and another from 1.35; the concept page's v1.35 may refer to the release that locked the gate, but the page does not say so. What is certain is that it is a stable feature in the current 1.37. The concept page also lists a limitation: the scheduler does not support preemption for DRA resources, so a higher-priority Pod has to wait until the Pod holding the device finishes. On NVIDIA's side, the GPU Operator documentation describes installing the DRA Driver for NVIDIA GPUs v0.5.1 as a newer alternative to the device plugin; features such as dynamic MIG, share-based sharing and MPS are still alpha there, and a cluster can have either the GPUCluster resource used for DRA or the ClusterPolicy resource used for the device plugin, not both.

Several jobs on one GPU: MIG, time-slicing, MPS

Small-model inference, notebooks and evaluation scripts often do not fill a whole GPU. There are three ways to share one, and the key difference is what gets isolated:

NVIDIA's device plugin README (checked 2026-10-15) adds: time-slicing and MPS are mutually exclusive; MPS sharing has been marked experimental since v0.15.0 and is not supported on devices with MIG enabled; all GPUs on a node use the same sharing method. The GPU Operator's MIG page describes two strategies: single means MIG is enabled on all GPUs of a node, mixed means it is not enabled on all of them; in the docs example, after an H100 80GB is split into 7 instances of 1g.10gb, the node label shows a GPU count of 7.

Use the MIG profile table for some arithmetic: on the H100 80GB (PCIe and SXM5), 1g.10gb takes 1/7 of the SMs and 1/8 of the memory, up to 7 instances; 1g.20gb also takes 1/7 of the SMs but 1/4 of the memory, up to 4 instances. A 7B-parameter model with BF16 weights needs $7\times10^9\times2=14$ GB, which does not fit a 10 GB instance; in a 20 GB instance, only about 6 GB is left for the KV cache and runtime. This is a teaching calculation counting weights only; for the full memory budget see KV cache inference budget.

Why distributed training needs gang scheduling

Take an example you can work by hand. Three nodes A, B and C, 8 GPUs each, 24 in total. Four jobs arrive in order: two single-Pod 4-GPU jobs S4a and S4b, a 16-GPU training job T16 (2 Pods of 8 GPUs each; training starts only when all ranks are present), and a single-Pod 8-GPU job T8. Total demand is 32 GPUs, more than 24, so someone must queue; the question is who queues and whether the queued job holds GPUs.

With the v1.37 defaults, the scheduler places one Pod at a time; the NodeResourcesFit plugin scores with LeastAllocated by default, weighting only CPU and memory (see the scheduler configuration reference), so GPU usage does not enter this score. After S4a lands on A, S4b is more likely to go to B or C, whose CPU and memory are less used; assume B. The table compares the final state under three approaches:

The first row is the worst: T16's first Pod takes C, its second Pod finds no node with 8 free GPUs, and the processes that did start sit waiting for the other half of the ranks; A and B have 4 free GPUs each, 8 in total, but cannot form one 8-GPU Pod, so T8 waits too. Only 8 of the 24 GPUs are doing work. If a second identical T16 arrives, the two jobs may each take one node and each wait for its other half, and neither can ever start: a deadlock.

Second row: gang scheduling sees that T16's two Pods cannot both be placed, so it binds neither, C goes to T8, and 16 GPUs are doing work. The cost is that T16 waits for two whole nodes to become free at the same time, which without priorities or reservations may take indefinitely long. The third row switches the scheduler to the MostAllocated bin-packing strategy with a weight on GPUs: the two small jobs squeeze onto one node, leaving two whole nodes for T16, and all 24 GPUs are doing work; now T8 waits for A to free up. Which order is better depends on priorities and quotas; bin-packing is not better in every case.

Fitting is not enough; it must also be close

T16's 16 ranks do a gradient all-reduce every step. In an ideal ring each rank sends $2\frac{P-1}{P}S$ bytes (derived in Multi-GPU training), and every edge of the ring carries that much data. Order the 16 ranks into a ring by node, and the number of cross-node edges equals the number of nodes used: whole nodes on 2 machines means 2 edges over the network and the rest over the intra-node interconnect; if the job may be split into 4-GPU Pods spread over 3 machines, 3 edges go over the network and each machine competes for its network card with more neighbours. One level up, if the two nodes sit in different racks, the path between racks is usually longer. This is an idealized teaching ring; which algorithm NCCL actually chooses, and how much intra-node and inter-node bandwidth differ, depend on the hardware and on NCCL configuration.

This is what topology-aware scheduling addresses: it asks not only "how many GPUs are left" but also "how close those GPUs are". The projects below all deal with gangs and topology in different ways; they are listed neutrally from their official documentation:

The Kubernetes v1.37 blog post (2026-09-08) also introduces CompositePodGroup, at alpha, for expressing multi-level gang and topology constraints, and mentions that it targets workloads managed by higher-level APIs such as JobSet and LeaderWorkerSet.

Slurm or Kubernetes

Another way to run a training cluster is to schedule jobs with Slurm. The two start from different places; the main differences:

The last row is the easiest to misread: the same term means "take turns" in Slurm and "start together" in the Kubernetes ecosystem. The table compares mechanisms, not which is better; the choice depends on the team's existing tools, how much online serving it runs, and its multi-tenancy needs.

Inference serving: scaling is more than adding replicas

An inference service is usually a Deployment, scaled by a HorizontalPodAutoscaler (HPA). The core HPA formula is:

$$ \text{desiredReplicas}=\left\lceil \text{currentReplicas}\times\frac{\text{currentMetricValue}}{\text{desiredMetricValue}}\right\rceil . $$

For example, with 4 replicas, an average queue of 30 requests per replica and a target of 10, you need $\lceil4\times30/10\rceil=12$ replicas. If the ratio is within the tolerance (0.1 by default), nothing happens; scale-down can be given a stabilization window, 300 seconds in the docs example. Where do the metrics come from? Per the docs, metrics other than CPU and memory are served through the custom or external metrics APIs, which usually requires an adapter; GPU metrics can reach Prometheus through DCGM Exporter, for example memory usage as DCGM_FI_DEV_FB_USED. Which metric to choose matters more. Google Cloud's GKE best practices for autoscaling inference (updated 2026-10-13) recommends the server's queue size and batch size, and points out that GPU utilization only measures how much of the time the GPU is active, not how much work it does while active.

Now the cold start. A 70B-parameter model stored in BF16 has $70\times10^9\times2=140$ GB of weights. A new replica must at least read these 140 GB from storage, which takes about $140\text{ GB}/B$, where $B$ is the effective read bandwidth (every row below is a teaching assumption, not the spec of any storage system):

Change one condition: 8 replicas start at once and share storage with 10 GB/s of total bandwidth; together they read $8\times140=1120$ GB, about 112 s, 8 times slower than a single replica. Besides reading weights there is adding nodes, pulling images, initializing CUDA and warming up the engine (for example capturing CUDA Graphs or compiling), none of which is counted here. Back to the scaling example above: assume a new replica needs 2 minutes in total before it can take requests (a teaching assumption); once the decision to go from 4 to 12 replicas is made, the queue keeps growing for those 2 minutes. Common responses are to keep a minimum of warm replicas, trigger scale-up on earlier signals, and scale down slowly; how well they work has to be verified by replaying real requests.

What to ask first when debugging

The scheduler only decides who gets which GPUs; once they are assigned, how a request queues and is batched inside the engine is covered in One LLM request, and how the code in a Pod hands work to the GPU is back in The framework layer.