Writing

Kubernetes GPU scheduling: plugins, DRA, gangs

Published 26 September 2026

Kubernetes GPU scheduling has three separate jobs. A device plugin or Dynamic Resource Allocation (DRA) exposes GPUs to the scheduler. Device selection places a Pod on a node with the right accelerator and nearby hardware. Gang scheduling waits until a distributed job can place enough Pods to run. Configuring one layer does not solve the other two.

Choose the scheduling layer that matches the problem

A Pod can have a valid GPU and still wait on peers, cross a slow network boundary, or move data through a remote NUMA node. The scheduler needs explicit information for each of those constraints.

The stable baseline: device plugins and extended resources

Kubernetes has stable GPU support through device plugins. The cluster operator installs the vendor's host driver and device plugin on GPU nodes. The plugin advertises an extended resource such as nvidia.com/gpu or amd.com/gpu. A Pod requests an integer quantity in its resource limits:

apiVersion: v1
kind: Pod
metadata:
  name: inference-worker
spec:
  restartPolicy: OnFailure
  containers:
    - name: worker
      image: registry.example/inference-worker:v1
      resources:
        limits:
          gpu-vendor.example/gpu: 1

GPU resources are specified in limits. Kubernetes uses the same value as the request; a separate GPU request without a limit is not allowed. The scheduler then places the Pod only where that extended resource is allocatable.

That answers quantity, not quality. If nodes have different GPU models, label them through a controlled process or Node Feature Discovery and add node affinity. Keep the label tied to observed hardware, not a hand-edited spreadsheet. A stale label can schedule a workload successfully onto the wrong accelerator.

What Kubernetes Dynamic Resource Allocation adds

Kubernetes Dynamic Resource Allocation treats devices more like storage claims. A DRA driver publishes devices in ResourceSlices. A DeviceClass defines a category the cluster offers, and a ResourceClaim or ResourceClaimTemplate describes what the workload needs. The scheduler allocates a matching device and places the Pod on a node that can access it.

Core DRA has been stable since Kubernetes 1.35. It can filter devices with CEL expressions, pass per-workload device configuration, share a device where the driver supports sharing, and centralize categories such as cost-optimized or high-memory accelerators. A DRA-compatible driver is still required; turning on the API does not make an existing device plugin publish ResourceSlices.

Kubernetes 1.37 made migration less abrupt. DRA Extended Resource support reached general availability, so a DRA driver can satisfy a traditional extended-resource request such as example.com/gpu without a second device plugin. Existing Pod manifests can stay simple while the device inventory moves to DRA.

GPU and NIC locality now has a common name

A distributed training worker moves data between GPU memory and the network. If the accelerator and high-speed NIC sit behind different CPU sockets, traffic may cross the inter-socket link before it reaches the NIC. The Pod still runs, but locality can become part of the job's tail latency.

Kubernetes 1.37 defines the stable resource.kubernetes.io/numaNode device attribute so different DRA drivers can publish the same locality concept. The release also adds alpha derived attributes: CEL expressions can map vendor-specific GPU and NIC attributes to one virtual value, then require a match. Derived attributes remain alpha, so they need the relevant feature gate and a test cluster before production use.

Locality inside the server is only half the path. Once traffic leaves the NIC, RoCE congestion, packet loss and path selection decide whether remote accelerators can exchange data predictably.

Why distributed jobs need gang scheduling

The default scheduler places Pods one at a time. Suppose a training job asks for eight workers and the cluster has capacity for four. Those four can hold four GPUs while waiting for peers that cannot start. Another job may hold the remaining capacity while waiting for the same thing. Every individual placement is valid and no job can make progress.

Gang scheduling treats the workers as a group and binds them only when at least the configured count can run. Native gang scheduling graduated to beta in Kubernetes 1.37 through the Workload and PodGroup APIs. It is still disabled by default behind the GenericWorkload feature gate. Workload-aware preemption is part of the same beta gate, so preemption can consider whether removing lower-priority work creates enough capacity for the group rather than helping one stranded Pod.

Topology is a capacity constraint

Eight free GPUs do not form one useful pool if four are in another zone or behind an oversubscribed network tier. Topology-aware scheduling lets a workload request co-location in a domain such as a zone or rack. Kubernetes 1.37 adds multi-level placement for workloads that need the whole job in one zone and each worker group within a rack, but that hierarchy is alpha.

Choose topology keys from the physical design and maintain them as infrastructure data. A label named rack is useful only if it matches where the data center network actually changes failure domain or oversubscription. Excessively tight affinity can leave expensive GPUs idle; no affinity can make a job pay for avoidable network distance.

Diagnose a Pending GPU Pod in this order

1. Read the scheduler event. kubectl describe pod should say which resource or predicate failed.
2. Compare node capacity with allocatable values. A detected GPU is not necessarily advertised to Kubernetes.
3. Check the vendor driver and device-plugin or DRA-driver DaemonSet on the affected node.
4. Confirm the resource name and integer count in the Pod match what the node publishes.
5. Inspect node affinity, taints and tolerations before blaming GPU capacity.
6. For DRA, inspect the DeviceClass, ResourceClaim and ResourceSlices, including CEL evaluation errors.
7. For a distributed job, check PodGroup admission and topology constraints. One free GPU does not satisfy an eight-worker gang.

kubectl describe pod inference-worker
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable
kubectl get resourceclaims,resourceslices,deviceclasses
kubectl get pods -A -o wide

Use the commands as a path to the authoritative object, not as a complete dashboard. Vendor resource names and DRA objects vary by driver, and the empty custom column is itself evidence that the requested field was not published.

A practical adoption order

Keep the stable device-plugin path when workloads need whole GPUs and node labels express the required model. Add DRA when device attributes, per-workload configuration, sharing or GPU-to-NIC locality justify the driver change. Add gang scheduling when partial placement consumes devices without useful work. Test alpha topology features only after the cluster's labels reflect real racks and zones.

Device placement does not size container RAM. A worker can land on the right accelerator and still restart under its cgroup limit; the Kubernetes OOMKilled guide separates that failure from node pressure and explains the MemoryQoS controls in 1.37.

A Pending Pod with a clear event is cheap to diagnose. A running job that occupies accelerators while it waits on peers or crosses the wrong hardware path costs far more.

Sources

• Schedule GPUs: official device-plugin resource rules and node-selection guidance
• Dynamic Resource Allocation: stable DRA model, benefits and limitations
• Kubernetes v1.37: DRA updates: published 3 September 2026, including extended resources, NUMA attributes and derived attributes
• Kubernetes v1.37: advancing workload-aware scheduling: published 8 September 2026, feature states and topology behavior
• Kubernetes 1.37 release announcement: gang scheduling beta and release-wide feature status

Kubernetes feature states were checked on 26 September 2026.