GPUs are the most expensive, most contended, and worst-utilized resource on an AI platform — idle 60–80% in most un-managed clusters while teams complain they can't get one.
Tier: Advanced · Workflow area: Platform-for-AI: observability & cost
This playbook applies scheduler discipline: Kueue for queueing and preemption between teams, MIG and time-slicing for fractional GPUs, Karpenter for spot-backed batch capacity, and DCGM metrics to run utilization as an SLO rather than a rumor.
nvidia-smi shows single-digit utilization.Skip the full machinery for a single team with one workload class — right-size the node group and revisit at the second tenant.
nvidia.com/gpu requests.Measure before you manage. From DCGM exporter, chart per-GPU utilization (DCGM_FI_DEV_GPU_UTIL) and memory use, grouped by namespace/team. A week of this data tells you whether your problem is fragmentation (many small underusing pods), contention (queue of pending pods), or waste (allocated-but-idle) — each has a different fix below.
Split inference from batch at the node level. Label and taint GPU node groups: workload=inference (on-demand, stable) vs workload=batch (spot-eligible). Inference Deployments tolerate only their pool. This single move ends the worst incidents — a training job evicting the chatbot.
Put batch behind Kueue. Kueue makes GPU demand visible and preemption governed instead of accidental:
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata: {name: gpu-batch}
spec:
namespaceSelector: {}
resourceGroups:
- coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
flavors:
- name: spot-l4
resources:
- {name: "nvidia.com/gpu", nominalQuota: 8}
preemption:
withinClusterQueue: LowerPriority
Give each team a LocalQueue pointing at the ClusterQueue; jobs carry kueue.x-k8s.io/queue-name and get admitted (or wait, visibly) instead of pending mysteriously. Use WorkloadPriorityClasses so eval jobs yield to time-critical retrains. Verify current API fields with kubectl explain — Kueue's API is still v1beta1 and moves.
Slice big cards for small workloads. If utilization data shows many pods using a fraction of an A100/H100-class card: enable MIG via the GPU Operator (mig.strategy=single or mixed) and request slices as resources, e.g. nvidia.com/mig-1g.10gb: 1 — hardware-isolated, right-sized shares for small models, notebooks, and embedding services. For dev/notebook workloads where isolation matters less, time-slicing is the cruder, driver-level alternative. Don't slice inference nodes running models that need the whole card.
Buy batch capacity on spot. Karpenter NodePool for the batch pool: spot capacity type, several GPU instance families for diversity, sensible disruption budgets. Spot + Kueue is the pairing that works — interruptions become requeues, not failures — provided jobs checkpoint. Keep inference on on-demand/reserved; spot inference is how you turn a capacity event into a customer incident.
Publish utilization SLOs and review monthly. Two numbers per pool: allocation rate (requested ÷ capacity) and true utilization (DCGM busy ÷ allocated). Targets like >70% true utilization on batch pools change behavior when they're on a dashboard next to the team names. Feed the $ view from pb-46's FinOps dashboards; use an AI assistant to draft the monthly narrative from the metrics export — and verify its numbers against the dashboard before it goes to leadership.
Queueing, slicing, and spot are the same playbook you already know from CPU capacity management — the only new part is that the unit costs 30x more, which makes the discipline mandatory instead of nice.
kubectl get workloads) rather than pending silently; it runs when quota frees.nominalQuota from measured usage; let Kueue borrowing absorb bursts.DCGM_FI_DEV_GPU_UTIL at 90% can still mean inefficient kernels. It's a capacity signal, not a performance guarantee — pair with throughput metrics from pb-46.GPU quotas are org politics rendered as YAML — get the priority ladder (whose jobs preempt whose) agreed in writing before you encode it in a PriorityClass.