How to Set Up Kubernetes AI Workload Management in 2026

AI jobs expose every undocumented difference between Kubernetes nodes. A web pod may run fine while a training job stays pending because a GPU label, taint, driver, or webhook doesn’t match.

This setup helps you find those gaps safely. You’ll run a repeatable ML-style job, trace failures, detect drift, and record enough state to reproduce a run. Test it in a non-production cluster first. Several steps change cluster-wide controls.

What Is Kubernetes AI Workload Management?

Kubernetes AI workload management combines resource requests, GPU placement, admission controls, monitoring, and GitOps. There isn’t one package to install. You need several practices and components for training jobs, pipeline tasks, distributed workers, and model-serving pods.

Normal applications often run on interchangeable CPU nodes. AI jobs may require a specific GPU, driver, kernel, device plugin, storage path, scheduler, operator, or webhook. One mismatch can leave an ML job pending while the rest of the cluster looks healthy.

This baseline uses native Kubernetes objects. It also covers where queue managers such as Kueue and workload operators such as KubeRay fit. Check their official compatibility docs before installation. No Ray Operator and Kueue version matrix works safely across every cluster.

These commands target an existing Kubernetes cluster. You can run kubectl from Windows, macOS, or Linux.

Prerequisites

Make sure you have:

• An existing Kubernetes cluster running a supported Kubernetes release
• A non-production namespace where you can create Jobs, ConfigMaps, and service accounts
• Permission to read nodes, events, pods, Jobs, quotas, admission webhooks, and controller logs
• kubectl within one supported minor version of the API server
• Git 2.x for storing desired configuration
• Helm 3.x if you plan to add Kueue, Argo CD, or another packaged controller
• At least 2 CPU cores and 4 GiB of free cluster memory for the CPU-only test
• GPU worker nodes, a vendor-supported driver, and a device plugin for the optional GPU test
• Access to monitoring data, ideally Prometheus and Grafana
• Access to the approved GitOps repository
• A reviewed backup or recovery procedure before changing cluster-wide components

Requirement	Details
Client platforms	Windows 11 or Windows Server 2022+, macOS 14+, or a web administration console
Cluster	Kubernetes version supported by your provider and installed add-ons
CLI	kubectl; Git; optional Helm 3.x
Permissions	Namespace write access plus read access to nodes and relevant controllers
Hardware	CPU nodes for the base test; compatible GPU nodes for accelerator workloads
Network	HTTPS access to the Kubernetes API and approved image registries
Recommended infrastructure	Dedicated GPU nodes, reliable NAS storage, a 2.5GbE switch or faster fabric, Cat6 Ethernet cable, and a UPS

Before choosing a version, check the Kubernetes releases page and patch-release schedule. Managed services often trail upstream Kubernetes. Add-ons may support fewer releases. You need the overlap between those support windows.

Test Environment

The examples use this logical environment:

Component	Test baseline
Kubernetes	A currently supported release verified at deployment time
Client	A kubectl release allowed by the Kubernetes version-skew policy
Namespace	ai-workloads
Queue manager	Optional; native Jobs work without Kueue
Workload	CPU-only Python Job, followed by an optional GPU Job
Desired-state storage	Git repository
Monitoring	Kubernetes metrics plus optional Prometheus, Grafana, and GPU exporter

Don’t copy a version number from an old guide into production. Pin the exact Kubernetes patch, container digest, Helm chart, operator, CRD, and webhook versions your team tested. “Close enough” often ends with an afternoon spent reading controller logs.

Step-by-Step Guide

Step 1: Install the Kubernetes Client

Install only the client on your workstation. Keep the Kubernetes API server and GPU nodes on Linux servers or a managed service. You only need a local kubectl binary for these steps.

Windows

Open Windows Terminal as your normal user. Install kubectl with Windows Package Manager:

winget install --exact --id Kubernetes.kubectl

Close and reopen the terminal so it loads the updated PATH. Then check the client:

kubectl version --client

Expected output resembles:

Client Version: v1.xx.x
Kustomize Version: v5.x.x

If winget isn’t available, follow the official Install and Set Up kubectl on Windows guide. Avoid random download mirrors. This binary handles cluster credentials.

macOS

Install kubectl with Homebrew:

brew install kubectl

Check it:

kubectl version --client

Expected output resembles:

Client Version: v1.xx.x
Kustomize Version: v5.x.x

Use the official macOS kubectl installation guide if your organization doesn’t allow Homebrew.

Linux administration host

On Ubuntu or Debian, follow the current Kubernetes package repository guide. Old repository names tend to fail at the least useful time. The official Linux installation guide has the current signing key and repository steps.

After installation:

kubectl version --client

Step 2: Obtain Cluster Access and Select the Correct Context

Ask the cluster administrator for an approved way to receive the kubeconfig. Don’t paste kubeconfig data into tickets, chat, screenshots, or Git. It can contain credentials or references that are just as useful to an attacker.

List the available contexts:

kubectl config get-contexts

Expected output resembles:

CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* ai-staging ai-staging platform-user
ai-prod ai-prod platform-user

Select the non-production context:

kubectl config use-context ai-staging

Expected output:

Switched to context “ai-staging”.

Confirm the active context and your access:

kubectl config current-context
kubectl auth can-i get pods --all-namespaces
kubectl auth can-i get nodes

Expected output:

ai-staging
yes
yes

If either check returns no, request the narrow Role-Based Access Control (RBAC) permission you need. Shared administrator credentials fix one problem and create a worse one.

Step 3: Record the Cluster Baseline Before Changing Anything

Record versions, API resources, nodes, add-ons, admission components, and resource policies before installation. You’ll need that baseline if scheduling behavior changes later.

Run these read-only commands:

kubectl version
kubectl get nodes -o wide
kubectl get namespaces
kubectl get resourcequota --all-namespaces
kubectl get limitrange --all-namespaces
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations
kubectl get customresourcedefinitions

Expected node output resembles:

NAME STATUS ROLES AGE VERSION
cpu-node-01 Ready worker 21d v1.xx.x
gpu-node-01 Ready worker 21d v1.xx.x

Record at least these versions before troubleshooting or upgrading:

• Kubernetes API server and node patch versions
• Container runtime and operating-system image
• Linux kernel on each node pool
• GPU model, firmware, driver, runtime, and device-plugin image
• Queue manager and scheduler
• Workload operator, such as KubeRay
• Installed CRD versions
• Mutating and validating admission webhooks
• Cluster Autoscaler or provider autoscaler
• Container images and immutable digests
• Helm chart versions and values
• Kubeflow Pipelines version, if installed

This list looks excessive until two supposedly identical GPU nodes act differently. Then the kernel patch and device-plugin digest become useful clues.

List common controller images without exposing Secrets:

kubectl get deployments --all-namespaces \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image'

Expected output resembles:

NAMESPACE NAME IMAGES
kube-system metrics-server registry.example/metrics-server:vX.Y.Z
kueue-system kueue-controller registry.example/kueue:vX.Y.Z
ray-system kuberay-operator registry.example/kuberay-operator:vX.Y.Z

Redact private registry hostnames before sharing the output. They usually aren’t secrets, but they expose internal names and supply-chain details for no useful gain.

Step 4: Create an Isolated AI Workload Namespace

Create a manifest directory on your workstation. Keeping these files together makes later Git and drift checks predictable.

Windows PowerShell:

New-Item -ItemType Directory -Force -Path "$PWD\ai-platform"
Set-Location "$PWD\ai-platform"

macOS or Linux:

mkdir -p ./ai-platform
cd ./ai-platform

Create namespace.yaml:

# ./namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ai-workloads
  labels:
    app.kubernetes.io/part-of: ai-platform
    environment: staging

Apply it:

kubectl apply -f ./namespace.yaml

Expected output:

namespace/ai-workloads created

Set the namespace on the current context:

kubectl config set-context --current --namespace=ai-workloads

Expected output resembles:

Context “ai-staging” modified.

Check both context and namespace before each diagnostic session:

kubectl config view --minify \
  -o jsonpath='{.current-context}{" namespace="}{..namespace}{"\n"}'

Expected output:

ai-staging namespace=ai-workloads

That check takes less than a second. Recovering a Job deleted from the wrong namespace takes longer.

Step 5: Add Resource Guardrails

A ResourceQuota stops one test from consuming the whole namespace. A LimitRange adds cautious defaults when a container omits CPU or memory values. Defaults are a safety net. Measured requests are still better than guesses.

Create guardrails.yaml:

# ./guardrails.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: ai-workload-quota
  namespace: ai-workloads
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 32Gi
    limits.cpu: "16"
    limits.memory: 64Gi
    pods: "20"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: ai-container-defaults
  namespace: ai-workloads
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 250m
        memory: 256Mi
      default:
        cpu: "1"
        memory: 1Gi

Apply and inspect the policies:

kubectl apply -f ./guardrails.yaml
kubectl describe resourcequota ai-workload-quota
kubectl describe limitrange ai-container-defaults

Expected output begins with:

resourcequota/ai-workload-quota created
limitrange/ai-container-defaults created

GPU resource names vary by vendor. Don’t add a GPU quota until you’ve checked the extended resource on your nodes. A quota with the wrong resource name provides paperwork, not protection.

Step 6: Inventory Node Capacity and Detect GPU Drift

Start with a compact inventory:

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,STATUS:.status.conditions[-1].type,KERNEL:.status.nodeInfo.kernelVersion,OS:.status.nodeInfo.osImage,RUNTIME:.status.nodeInfo.containerRuntimeVersion'

Then inspect labels and allocatable resources:

kubectl get nodes --show-labels
kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory,GPUS:.status.allocatable.nvidia\.com/gpu'

Expected GPU inventory resembles:

NAME CPU MEMORY GPUS
cpu-node-01 7500m 30218448Ki <none>
gpu-node-01 15500m 64300120Ki 2

For AMD or another vendor, replace nvidia.com/gpu with the resource name from its device plugin. Kubernetes schedules the exact advertised name. It won’t infer the vendor.

Check taints because GPU pools often block general workloads:

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'

Inspect a specific GPU node:

kubectl describe node gpu-node-01

Compare nodes in the same pool for:

• Kernel version
• OS image and container runtime
• GPU allocatable count
• Accelerator labels
• Topology labels
• Taints
• Capacity versus allocatable resources
• Node conditions
• Device-plugin availability
• GPU driver and exporter versions

A node with a different kernel, an extra manual label, or a missing GPU resource is a snowflake. Rebuild it through the approved node-pool process. Repeated hand repairs make the next failure harder to explain and reproduce.

Step 7: Run a Deterministic CPU Baseline Job

Test native scheduling before adding GPUs, queues, or an operator. This test separates basic cluster faults from accelerator problems.

Create cpu-smoke-test.yaml:

# ./cpu-smoke-test.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ai-cpu-smoke-test
  namespace: ai-workloads
  labels:
    app.kubernetes.io/name: ai-cpu-smoke-test
    ml.example/code-revision: "demo-2026-09-03"
    ml.example/dataset-version: "synthetic-v1"
spec:
  backoffLimit: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ai-cpu-smoke-test
    spec:
      restartPolicy: Never
      containers:
        - name: python
          # Pin an immutable digest in production.
          image: python:3.13-slim
          command:
            - python
            - -c
            - |
              import hashlib
              values = list(range(100000))
              result = sum(x * x for x in values)
              print(f"result={result}")
              print("dataset_sha256=" + hashlib.sha256(b"synthetic-v1").hexdigest())
          resources:
            requests:
              cpu: 250m
              memory: 128Mi
            limits:
              cpu: "1"
              memory: 512Mi

Apply it and wait for it to finish:

kubectl apply -f ./cpu-smoke-test.yaml
kubectl wait \
  --for=condition=complete \
  job/ai-cpu-smoke-test \
  --timeout=120s

--for=condition=complete waits for the Job condition. --timeout=120s stops the client from waiting forever, which has never fixed a cluster.

Expected output:

job.batch/ai-cpu-smoke-test created
job.batch/ai-cpu-smoke-test condition met

Read the log:

kubectl logs job/ai-cpu-smoke-test

Expected output resembles:

result=333328333350000
dataset_sha256=2bd62416…

If this works while a GPU job stays pending, several parts are probably sound. The API, namespace, basic scheduler, image pull, and CPU capacity all worked. Check GPU capacity, placement, taints, admission, queues, and AI-specific controllers next.

Step 8: Add GPU-Aware Placement

Continue only if a node advertises the accelerator resource. Never fake a GPU label or allocatable resource. The scheduler may accept the fiction, but the container runtime won’t.

Ask the administrator which stable labels identify the GPU pool. Production labels should come from node provisioning or a maintained feature-discovery controller. Manual labels vanish during node replacement and quietly create drift.

Create gpu-smoke-test.yaml:

# ./gpu-smoke-test.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ai-gpu-smoke-test
  namespace: ai-workloads
  labels:
    app.kubernetes.io/name: ai-gpu-smoke-test
    ml.example/code-revision: "demo-2026-09-03"
    ml.example/dataset-version: "synthetic-v1"
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ai-gpu-smoke-test
    spec:
      restartPolicy: Never
      nodeSelector:
        accelerator.example.com/family: nvidia
      tolerations:
        - key: accelerator
          operator: Equal
          value: gpu
          effect: NoSchedule
      containers:
        - name: gpu-check
          # Replace with an approved CUDA image pinned by digest.
          image: nvidia/cuda:12.8.1-base-ubuntu24.04
          command:
            - nvidia-smi
          resources:
            requests:
              cpu: "1"
              memory: 2Gi
              nvidia.com/gpu: "1"
            limits:
              cpu: "2"
              memory: 4Gi
              nvidia.com/gpu: "1"

Change the image tag, node label, taint, and GPU resource name to values you checked in your environment. Then apply it:

kubectl apply -f ./gpu-smoke-test.yaml
kubectl get job ai-gpu-smoke-test
kubectl get pods -l app.kubernetes.io/name=ai-gpu-smoke-test -o wide

A working Job moves from Pending to Running, then Completed:

NAME STATUS COMPLETIONS DURATION
ai-gpu-smoke-test Complete 1/1 8s

Read the output:

kubectl logs job/ai-gpu-smoke-test

Expected output includes the driver, CUDA compatibility, and detected GPU:

NVIDIA-SMI …
Driver Version: …
CUDA Version: …
GPU Name: …

Topology matters when a job needs several GPUs, fast interconnects, or workers in one zone. A label such as gpu=true omits the model, partition mode, network fabric, and failure domain. It answers one question and leaves the expensive ones open.

Step 9: Deliberately Create and Diagnose a Pending Job

A controlled failure proves your team can separate admission, scheduling, capacity, and autoscaling problems. Do it now, while nobody is waiting for a training deadline.

Create pending-test.yaml with an impossible node label:

# ./pending-test.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ai-pending-test
  namespace: ai-workloads
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ai-pending-test
    spec:
      restartPolicy: Never
      nodeSelector:
        accelerator.example.com/family: deliberately-unavailable
      containers:
        - name: test
          image: python:3.13-slim
          command:
            - python
            - -c
            - print("This should not run")
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 1Gi

Apply it:

kubectl apply -f ./pending-test.yaml
kubectl get pods -l app.kubernetes.io/name=ai-pending-test

Expected output:

NAME READY STATUS RESTARTS AGE
ai-pending-test-xxxxx 0/1 Pending 0 10s

Capture the generated pod name:

PENDING_POD="$(kubectl get pods \
  -l app.kubernetes.io/name=ai-pending-test \
  -o jsonpath='{.items[0].metadata.name}')"

kubectl get pod "$PENDING_POD" -o wide
kubectl describe pod "$PENDING_POD"

PowerShell equivalent:

$PendingPod = kubectl get pods `
  -l app.kubernetes.io/name=ai-pending-test `
  -o jsonpath='{.items[0].metadata.name}'

kubectl get pod $PendingPod -o wide
kubectl describe pod $PendingPod

The Events section should show a similar message:

Warning FailedScheduling default-scheduler 0/3 nodes are available:
3 node(s) didn’t match Pod’s node affinity/selector.

List current namespace events in time order:

kubectl get events \
  --sort-by=.metadata.creationTimestamp

This result points to a scheduling block. Other layers leave different evidence:

Layer	Evidence
API validation	kubectl apply fails immediately with a schema error
Admission webhook	The API rejects or delays creation and names a webhook
Queue admission	A Kueue Workload exists but has an unadmitted condition
Kubernetes scheduling	A pod exists in Pending with FailedScheduling events
Capacity	Events mention insufficient CPU, memory, GPU, or pod slots
Placement	Events mention affinity, node selectors, taints, or topology
Autoscaling	Scheduler reports an unschedulable pod; autoscaler logs explain why no node was added
Runtime	The pod schedules but enters ImagePullBackOff, CreateContainerError, or CrashLoopBackOff

A Cluster Autoscaler can add a node only when an eligible node group exists and the provider can supply it. It can’t create an unavailable GPU model in the selected region or account. Autoscaling automates capacity. It doesn’t create inventory.

Step 10: Inspect Queue Admission and Distributed Capacity

If Kueue is installed, discover its API resources before assuming names or versions:

kubectl api-resources --api-group=kueue.x-k8s.io
kubectl get workloads.kueue.x-k8s.io --all-namespaces
kubectl get clusterqueues.kueue.x-k8s.io
kubectl get localqueues.kueue.x-k8s.io --all-namespaces

Inspect a queued workload:

kubectl describe workload \
  --namespace=ai-workloads \
  WORKLOAD_NAME

Replace WORKLOAD_NAME with a value from the previous command. Read status.conditions closely. An unadmitted workload hasn’t reached normal pod scheduling, so node capacity won’t explain it.

Check Kueue controller health and logs:

kubectl get deployments --namespace=kueue-system
kubectl get pods --namespace=kueue-system
kubectl logs \
  --namespace=kueue-system \
  deployment/kueue-controller-manager \
  --all-containers=true \
  --tail=200

Distributed training often needs coordinated capacity. Starting one worker while the others wait can waste a GPU and cause timeouts. Gang scheduling or workload admission holds the group until all requested capacity is available.

For example, four workers requesting one GPU each usually need four compatible GPUs. Those GPUs must also meet the queue, quota, topology, and placement rules. Four GPUs spread across unsuitable node pools may leave the workload waiting. The total count doesn’t prove the job can run.

Step 11: Check Operator, CRD, and Webhook Compatibility

KubeRay may create Ray head and worker pods. Kueue may control when an integrated Ray workload is admitted. That path crosses several versioned interfaces:

• The submitted Ray custom resource
• The KubeRay operator
• Installed Ray CRDs
• Kueue and its integration support
• Mutating or validating admission webhooks
• The Kubernetes API server

Record the deployed images:

kubectl get deployments \
  --all-namespaces \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image'

List Ray and Kueue CRDs:

kubectl get customresourcedefinitions
kubectl api-resources --api-group=ray.io
kubectl api-resources --api-group=kueue.x-k8s.io

Inspect webhook configuration:

kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations
kubectl describe validatingwebhookconfiguration WEBHOOK_NAME

Replace WEBHOOK_NAME with the relevant name from the list. Then inspect controller logs:

kubectl logs \
  --namespace=ray-system \
  deployment/kuberay-operator \
  --all-containers=true \
  --tail=200

Namespace and Deployment names vary by installation. Find them before copying these commands into production. Naming rules have no duty to match a documentation example.

Errors such as “no matches for kind,” schema failures, conversion failures, or webhook timeouts can stop a workload before worker pods exist. Don’t call two versions incompatible based on symptoms alone. Compare the exact release notes and integration docs for your Kubernetes, KubeRay, Kueue, and CRD versions.

Step 12: Put the Desired State in Git

Create a local repository for the demonstration manifests:

git init
git add namespace.yaml guardrails.yaml cpu-smoke-test.yaml gpu-smoke-test.yaml
git commit -m "Add deterministic AI workload baseline"

Expected output resembles:

[main (root-commit) abc1234] Add deterministic AI workload baseline
4 files changed, …

Don’t commit kubeconfig files, tokens, registry credentials, raw Secret values, or generated pipeline credentials. Git remembers mistakes exceptionally well.

Add a .gitignore file:

# ./.gitignore
*.kubeconfig
.env
credentials/
rendered-secrets/
*.key
*.pem

Commit it:

git add .gitignore
git commit -m "Exclude local credentials and generated secrets"

Git now records the intended namespace, policies, and workloads. A GitOps controller such as Argo CD can compare these files with live objects and report drift.

Treat manual kubectl edit, kubectl patch, and dashboard changes as emergency work. If you must change a live object, document it and open a reviewed Git change at once. Then reconcile through the normal path. Otherwise, today’s repair becomes tomorrow’s mystery.

Step 13: Detect Live-Cluster Drift

Use server-side dry-run to ask the API server how it would read a manifest:

kubectl apply \
  --server-side \
  --dry-run=server \
  -f ./guardrails.yaml

Expected output:

resourcequota/ai-workload-quota configured (server dry run)
limitrange/ai-container-defaults configured (server dry run)

The server checks current schemas and admission rules without saving the objects. This catches problems that a client-only check can miss.

Use kubectl diff to compare local desired state with the live cluster:

kubectl diff -f ./guardrails.yaml

No output means there is no meaningful difference. A non-zero exit code with a diff means the local manifest and live object differ.

GitOps runs this check all the time. In Argo CD, open Applications, select the AI platform application, and inspect App Details > Diff. Check the cluster and namespace before using Sync.

Warning: Do not click Sync, enable auto-sync, or prune resources until the diff has been reviewed. A GitOps sync can overwrite live fields or delete resources that are absent from Git.

Some differences are valid controller-owned fields. Check managed fields before arguing with the controller:

kubectl get resourcequota ai-workload-quota \
  -o jsonpath='{range .metadata.managedFields[*]}{.manager}{"\t"}{.operation}{"\n"}{end}'

Expected output resembles:

kubectl-client-side-apply Update
kube-controller-manager Update

Move intentional changes into Git. Add comparison rules for generated or controller-owned fields only after identifying their owner and purpose. Ignoring the whole resource makes the dashboard quiet by hiding useful drift.

Step 14: Monitor Node and GPU Behavior

Start with built-in metrics if Metrics Server is installed:

kubectl top nodes
kubectl top pods --namespace=ai-workloads --containers

Expected output resembles:

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
cpu-node-01 720m 9% 6200Mi 20%
gpu-node-01 2100m 13% 18400Mi 29%

CPU and memory metrics don’t prove that GPUs are healthy or busy. For NVIDIA nodes, a common stack uses the vendor-supported GPU metrics exporter, Prometheus, and Grafana. Track:

• GPU utilization and memory use
• GPU temperature, power, and hardware errors
• Pending pod count and scheduling latency
• Node Ready state and pressure conditions
• Allocatable versus requested accelerators
• Device-plugin availability
• Worker startup skew
• Storage throughput and latency
• Network throughput between distributed workers
• Queue admission wait time

In Grafana, open the approved node or GPU dashboard. Select the staging cluster and a useful range, such as Last 6 hours. Compare nodes in the same pool. One pool-wide average can hide a slow or unhealthy worker.

An allocated GPU may sit idle because another worker didn’t start, input data is slow, or one node performs poorly. Check capacity and use together. Allocation shows where the GPU went. Utilization shows whether the workload is feeding it.

Step 15: Record Everything Needed to Reproduce a Pipeline Run

A repeatable ML run needs more than a container tag. Record this metadata with every run:

# ./run-record.yaml
run:
  id: training-2026-09-03-001
  created_at_utc: "2026-09-03T12:00:00Z"
  code_git_commit: "REPLACE_WITH_FULL_COMMIT_SHA"
  pipeline_definition_commit: "REPLACE_WITH_FULL_COMMIT_SHA"
  dataset:
    name: training-dataset
    version: "2026-09-01"
    checksum: "REPLACE_WITH_DATASET_CHECKSUM"
  model:
    name: classifier
    starting_version: "v3"
  images:
    trainer: "registry.example.invalid/ml/trainer@sha256:REPLACE_WITH_DIGEST"
  dependencies:
    lockfile_checksum: "REPLACE_WITH_LOCKFILE_CHECKSUM"
  random_seed: 20260903
  cluster:
    kubernetes_version: "REPLACE_WITH_EXACT_PATCH"
    node_pool_revision: "REPLACE_WITH_IAC_REVISION"
    kernel_version: "REPLACE_WITH_KERNEL_VERSION"
    gpu_model: "REPLACE_WITH_GPU_MODEL"
    gpu_driver: "REPLACE_WITH_DRIVER_VERSION"
    device_plugin: "REPLACE_WITH_IMAGE_DIGEST"
  platform:
    scheduler: "REPLACE_WITH_NAME_AND_VERSION"
    queue_manager: "REPLACE_WITH_NAME_AND_VERSION"
    workload_operator: "REPLACE_WITH_NAME_AND_VERSION"
    pipeline_engine: "REPLACE_WITH_NAME_AND_VERSION"

Store the record in an approved metadata system or artifact store. Don’t include private dataset locations, credentials, or registry tokens. The record should identify artifacts without becoming a credential bundle.

If you use Kubeflow Pipelines, follow the current Kubeflow Pipelines documentation and its pipeline concepts. A pipeline definition describes the workflow. A run records one execution with fixed inputs and environment versions.

Check the public Kubeflow Pipelines releases page before pinning a release.

Step 16: Verify the Complete Setup

Run this final read-only check:

kubectl get namespace ai-workloads
kubectl get resourcequota,limitrange --namespace=ai-workloads
kubectl get jobs,pods --namespace=ai-workloads
kubectl get events \
  --namespace=ai-workloads \
  --sort-by=.metadata.creationTimestamp
kubectl diff -f ./namespace.yaml
kubectl diff -f ./guardrails.yaml

A working baseline has:

• An active ai-workloads namespace
• Applied quota and default limits
• A completed CPU Job
• A completed GPU Job when compatible GPU capacity exists
• An intentionally pending test whose event explains the placement failure
• No unexplained GitOps or kubectl diff differences
• Recorded component and node versions
• Monitoring visibility for node pressure and accelerator utilization

Remove only the deliberate pending test:

kubectl delete -f ./pending-test.yaml

Expected output:

job.batch “ai-pending-test” deleted

This deletion is safe for the demonstration Job. Check the context and namespace first anyway. Muscle memory has a poor sense of environment boundaries.

Configuration

Resource Requests and Limits

Requests drive scheduling. Limits control runtime use. GPU vendors often expose an integer extended resource such as nvidia.com/gpu.

Setting	Recommended pattern	Why it matters
CPU request	Set from measured demand	Prevents ambiguous placement
Memory request	Include framework and dataset overhead	Reduces eviction and out-of-memory failures
GPU request	Request the exact supported extended resource	Limits scheduling to capable nodes
Image	Pin an immutable digest	Prevents a tag from resolving to different code
Node selector	Use stable provisioned labels	Avoids accidental placement
Affinity	Express model, zone, or topology needs	Supports compatible placement
Toleration	Match an intentional GPU-node taint	Keeps general workloads off costly nodes
Priority	Use reviewed PriorityClasses	Prevents arbitrary workload preemption

Don’t set every CPU limit equal to its request without measuring the application. Tight CPU limits can throttle data loading and leave an expensive GPU waiting for batches. Kubernetes still reports the GPU as allocated. That makes the symptom look like an accelerator fault.

Topology-Aware Placement

Use requiredDuringSchedulingIgnoredDuringExecution only for hard requirements. Strict rules give the scheduler fewer options and can strand usable capacity.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: accelerator.example.com/model
              operator: In
              values:
                - approved-gpu-model
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: training-worker

Required affinity is a hard filter. Preferred anti-affinity guides placement but won’t block the job when no better option exists. Start with the weakest rule that still protects the workload’s hardware or failure-domain needs.

Version Pinning

Record and review versions as one tested stack:

Component	Pin
Kubernetes	Exact supported patch per environment
Node image	Immutable provider image or IaC revision
Kernel and GPU driver	Approved node-pool build
Device plugin	Image digest and deployment manifest revision
Kueue	Controller image, chart, CRDs, and configuration
KubeRay	Operator image, chart, and Ray CRD version
Admission webhooks	Image, certificate process, and failure policy
Kubeflow Pipelines	Version 2 release and deployment revision
Workload image	Immutable digest
Helm and Kustomize input	Chart version, values, bases, and overlays

A tag such as latest destroys useful evidence. Some registries also allow numbered tags to be replaced. Use digests for workload and controller images when you can. Digests are ugly. Debugging the wrong image is uglier.

GitOps and Infrastructure as Code

Use each tool for the layer it handles well:

• Terraform or provider-native Infrastructure as Code defines clusters, node pools, networks, identity, disks, and autoscaling boundaries.
• Helm packages controllers and applications with versioned values.
• Kustomize expresses small environment-specific overlays.
• Argo CD or Flux continuously compares Git with live Kubernetes resources.
• Policy engines enforce approved registries, resource requests, and bans on mutable tags.

Keep staging and production on the same modules and bases. Put planned differences in reviewed variables or overlays. Copying YAML between clusters and patching it later creates snowflake environments with a short setup and a long support tail.

Each tool has limits. Helm templates can become hard to trace. Terraform handles constant application reconciliation poorly. GitOps will faithfully apply a bad commit. You still need review.

Windows, macOS, and Web Workflows

Windows

Use PowerShell for kubectl, Git, and Helm. Keep kubeconfig files in the Windows user profile with inherited access restricted. Don’t commit them or expose them in terminal recordings.

PowerShell uses backticks for line continuation and $Name for variables. Translate Bash examples that use $(...), as shown in the pending-pod step. Blindly copying shell syntax causes errors that often look unrelated to Kubernetes.

macOS

Use Terminal or an approved terminal application. Homebrew is handy, but managed Macs may require signed packages or a device-management workflow. Check whether your shell is zsh or Bash before copying shell-specific profile changes.

Web

There is no universal Kubernetes AI management portal. Your organization may provide several web tools:

• Open the approved Kubernetes, GitOps, monitoring, or Kubeflow URL.
• Confirm the selected cluster and namespace.
• Open the workload details.
• Review conditions, resource requests, placement constraints, and events.
• Open the GitOps diff without synchronizing it.
• Check queue and node dashboards for the same time window.

The browser works well for comparisons and monitoring. Keep kubectl nearby because raw conditions and events often contain details that dashboards hide. Pretty status badges don’t schedule pods.

Tips and Troubleshooting

Checks to Run Before Changing or Restarting Anything

Use this order so you keep the evidence:

• Confirm the current context, cluster, and namespace
• Record the workload YAML and current status conditions
• Record recent events with timestamps
• Determine whether the workload object and pods both exist
• Record Kubernetes, node, controller, CRD, and webhook versions
• Compare requests with allocatable node resources
• Check quotas, affinity, selectors, taints, and tolerations
• Check queue admission and workload conditions
• Check scheduler, operator, webhook, and autoscaler logs
• Compare Git desired state with the live object
• Check node health, kernel, driver, device plugin, and GPU metrics
• Save evidence before deleting pods or restarting controllers

Restarting a controller may clear a brief fault while removing the best clues about its cause. Save logs and conditions first. Five minutes of restraint can prevent several hours of guessing.

The Job Is Queued but No Pods Exist

Why it happens: The workload may be waiting for queue admission. Quota, a webhook, or an unhealthy operator may also block it.

Fix:

kubectl get workloads.kueue.x-k8s.io --all-namespaces
kubectl get events \
  --namespace=ai-workloads \
  --sort-by=.metadata.creationTimestamp
kubectl get pods --namespace=kueue-system
kubectl get validatingwebhookconfigurations

If the Kueue API doesn’t exist, the first command reports that the server lacks that resource type. Check the native Job, operator CR, and admission events instead. If no pods exist, node scheduling may not have started.

Pods Exist but Remain Pending

Why it happens: Kubernetes accepted the workload, but no node meets its resource and placement needs.

Fix:

kubectl get pods --namespace=ai-workloads -o wide
kubectl describe pod POD_NAME
kubectl get nodes --show-labels
kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory,GPUS:.status.allocatable.nvidia\.com/gpu,TAINTS:.spec.taints'

Replace POD_NAME. Compare the event message with the pod requests. Don’t reduce requests until you’ve proved the workload can run safely with less. Under-requesting memory often moves the failure from scheduling to runtime.

The Autoscaler Does Not Add a GPU Node

Why it happens: The pod may lack a matching node group. Its rules may exclude that group, quota may block provisioning, or the requested GPU may be unavailable.

Fix:

• Confirm that the pod has a FailedScheduling event.
• Confirm that a configured GPU node group matches its labels and taints.
• Inspect the autoscaler status and controller logs.
• Check cloud or datacenter GPU availability through the approved provider interface.
• Review account quota and node-pool maximum size.
• Change the IaC definition through review if the node group is wrong.

Autoscaling doesn’t guarantee physical GPU capacity. It requests nodes within set limits, but the provider still needs hardware to fill that request.

Only Some GPU Nodes Work

Why it happens: Nodes may differ in kernel, driver, device-plugin state, labels, partition mode, or hardware health.

Fix:

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion,OS:.status.nodeInfo.osImage,RUNTIME:.status.nodeInfo.containerRuntimeVersion,GPUS:.status.allocatable.nvidia\.com/gpu'
kubectl get pods --all-namespaces -o wide
kubectl describe node GPU_NODE_NAME

Compare the bad node with a healthy peer in the same IaC-defined pool. Cordon and replace a node that doesn’t match. A hand repair may recover today’s job, but it leaves a machine nobody can rebuild tomorrow.

Ray Workers Do Not Start Together

Why it happens: Distributed workers need capacity at the same time. The queue, scheduler, quota, or topology rules can’t admit the full group.

Fix:

• Inspect the parent Ray resource.
• Inspect the associated Kueue Workload, if present.
• List every generated pod and its request.
• Calculate the total simultaneous CPU, memory, GPU, and pod-slot requirement.
• Check ClusterQueue and LocalQueue status.
• Check node-pool topology, taints, quotas, and autoscaler limits.
• Review KubeRay and Kueue controller logs.

Don’t assume four free GPUs are usable when they’re split across incompatible nodes, zones, partition modes, or network fabrics. Distributed jobs care where capacity sits and how much exists.

A Webhook Blocks Resource Creation

Why it happens: The webhook service may be down, its certificate may be invalid, or its schema may not match the submitted CR.

Fix:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl get services --all-namespaces
kubectl get endpoints --all-namespaces
kubectl get events --all-namespaces \
  --sort-by=.metadata.creationTimestamp

Find the named webhook in the API error. Check its Deployment, Service endpoints, certificate process, and logs. Read the release-specific operator docs before upgrading or rolling back.

Don’t disable a production webhook as a quick test. Its validation may be the last control stopping an unsafe workload. Test the fix in staging with the same versions and failure policy.

Staging Works but Production Does Not

Why it happens: The clusters may differ in node pools, policies, Helm values, controller versions, quotas, webhooks, or manual changes.

Fix:

• Compare the exact workload image digest and manifest revision.
• Compare Kubernetes and node-pool revisions.
• Compare installed CRDs and controller images.
• Compare queues, quotas, PriorityClasses, labels, and taints.
• Review the GitOps diff in both environments.
• Encode every intentional difference in reviewed IaC or an environment overlay.

Don’t copy a working live object from staging into production. Live objects contain generated and environment-specific fields, and a direct copy skips review. Compare desired state instead.

GitOps Reports OutOfSync

Why it happens: Someone may have changed the resource by hand. A webhook or controller may also own the reported field.

Fix:

kubectl diff -f ./guardrails.yaml
kubectl get resourcequota ai-workload-quota -o yaml

Review the field-level GitOps diff. If the live change was accidental, reconcile from Git after approval. If it was planned, commit it first. When a controller owns the field, add a narrow comparison rule instead of ignoring the whole resource.

GPU Capacity Exists but Utilization Is Low

Why it happens: Requested GPUs can sit idle while workers wait. Slow storage, CPU limits, or one weak node can also starve the job.

Fix:

• Compare requested GPUs with real utilization.
• Check whether all distributed workers started.
• Review CPU throttling, memory pressure, network throughput, and storage latency.
• Compare driver, kernel, and exporter versions between nodes.
• Examine the same time range in workload, node, and GPU dashboards.
• Reproduce the workload on a known-good node pool before changing application code.

A faster GPU won’t fix a slow NAS drive or network link. If use drops in a repeating pattern, check the input pipeline and worker sync before blaming the accelerator.

An Earlier Pipeline Run Cannot Be Reproduced

Why it happens: The team recorded a pipeline name but missed its inputs, immutable artifacts, or infrastructure state.

Fix: Recover and store the full Git commits, dataset checksum, model version, image digest, dependency lockfile, and random seed. Also store the Kubernetes patch, node image, kernel, driver, device plugin, operator, queue manager, and pipeline-engine versions.

Infrastructure drift, data drift, and model-quality drift can each change results. This setup checks whether the execution environment matches its intended state.

Wrapping Up

Step	Action	Applies To
1–3	Install kubectl, select a context, and capture versions	Windows, macOS, cluster
4–8	Create guardrails and verify CPU/GPU scheduling	Kubernetes
9–11	Separate admission, scheduling, capacity, and compatibility faults	Kubernetes, Kueue, KubeRay
12–14	Detect drift through GitOps, inventories, and metrics	GitOps, IaC, monitoring
15–16	Record pipeline state and verify the baseline	MLOps teams

This baseline takes about 20–30 minutes once you have cluster access and know the GPU labels. It exposes failures through conditions, events, diffs, and metrics. Queue managers and operators add more parts, so add them only when native Jobs can’t meet your scheduling needs.

Immutable images and replaceable node pools usually give the largest reliability gain. GitOps then shows what changed before one small mismatch becomes a week-long GPU debugging session.