Cloud Infrastructure

Run Kubernetes GPU jobs on Spot with an on-demand fallback

September 13, 2026

Configure DOKS Spot GPU pools, autoscaling, affinity and PDBs. Test an on-demand fallback and compare cost per completed Kubernetes GPU job.

Jump to steps ↓

Run interruptible GPU jobs with a fallback

DigitalOcean announced DOKS Spot GPU support on September 9, 2026. This guide uses that capacity for restartable Kubernetes jobs, with an on-demand GPU pool available when Spot cannot serve them. The goal is a lower cost per completed job without making Spot availability a requirement for progress.

Spot GPU pools are in public preview. Reclamation affects a whole pool. DigitalOcean targets two hours of email notice on a best-effort basis, but emergencies can mean less or none. DOKS cordons and drains affected nodes and exposes Kubernetes events. Treat advance notice as time to reduce lost work, never as your only recovery mechanism.

Disclosure: This guide contains DigitalOcean affiliate links. I may earn a commission if you purchase through them, at no extra cost to you. Sources checked September 13, 2026. These are documentation-based examples; I have not provisioned paid GPU pools or observed a live reclaim event for this article.

Check workload fit and account capacity

Use this pattern for batch inference, independent experiments or training that already saves resumable checkpoints. A job that loses hours of progress on restart needs application changes before cheaper nodes will help. Keep a latency-sensitive service’s minimum serving capacity on on-demand nodes.

You need an existing DOKS test cluster, a CPU node pool, kubectl access and authenticated doctl. Keep system services on CPU capacity; DigitalOcean recommends at least two CPU nodes for their availability. In the Control Panel, confirm your team, GPU quota, supported region and the prices for both pools. A signup does not reserve either kind of GPU.

The Spot catalog lists MI350X, MI355X and B300 shapes. This example uses a single NVIDIA B300 Spot node. Choose a compatible on-demand NVIDIA shape in the same cluster region. H100 has less GPU memory than B300, so a model that fits B300 may fail on H100. Validate the real model, container and memory requirement on both before calling the second pool a fallback.

Confirm the kubeconfig context below matches the intended test cluster. If it does not, download that cluster’s kubeconfig through the documented DOKS connection procedure before continuing.

kubectl config current-context

Create Spot and on-demand GPU node pools

Replace YOUR_CLUSTER_ID with your test cluster ID. Confirm gpu-b300x1-288gb-spot is offered to that cluster using doctl kubernetes options sizes and the Control Panel. Replace YOUR_ON_DEMAND_NVIDIA_SIZE with an available, tested NVIDIA on-demand size. Stop if either pool cannot be provisioned.

These commands create paid resources. Each pool starts with one node and can grow to two. This is a small example ceiling, not a capacity recommendation. Size the fallback for the concurrency you must preserve if the entire Spot pool disappears.

Both pools receive our custom workload=gpu-batch label and taint. Configure these on the pool so replacement nodes inherit them. DOKS also supplies a vendor GPU taint. The Job below tolerates both restrictions. No CPU pool receives our workload label.

Choose the Spot ceiling before creation. You cannot later raise its autoscaling maximum or manually increase its fixed count. Autoscaling can regrow within the original range when capacity is available. Capacity released by scale-down may be unavailable later.

For this first test, keep one on-demand node ready. A zero-minimum fallback lowers idle cost but adds provisioning delay and capacity risk. The add-node-pools documentation warns that GPU provisioning can sometimes take hours. Do not promise immediate recovery from a cold pool.

Before continuing: Changes the target cluster. GPU resources can incur charges.

Replace every highlighted value before running this command.

doctl kubernetes cluster node-pool create YOUR_CLUSTER_ID --name gpu-spot --size gpu-b300x1-288gb-spot --count 1 --auto-scale --min-nodes 1 --max-nodes 2 --label workload=gpu-batch --taint workload=gpu-batch:NoSchedule

Before continuing: Changes the target cluster. GPU resources can incur charges.

Replace every highlighted value before running this command.

doctl kubernetes cluster node-pool create YOUR_CLUSTER_ID --name gpu-fallback --size YOUR_ON_DEMAND_NVIDIA_SIZE --count 1 --auto-scale --min-nodes 1 --max-nodes 2 --label workload=gpu-batch --taint workload=gpu-batch:NoSchedule

Tell autoscaling which pool to try first

Pod affinity scores existing nodes. The cluster autoscaler chooses which eligible pool to expand for unschedulable pods. Configure both decisions. Preferred affinity alone does not establish an autoscaler fallback order, and a busy GPU does not by itself create more pods.

Enable the priority expander below. Then run doctl kubernetes cluster node-pool list YOUR_CLUSTER_ID and record the two pool IDs. Open the existing kube-system/cluster-autoscaler-priority-expander ConfigMap with kubectl edit configmap cluster-autoscaler-priority-expander -n kube-system. Replace only the text inside data.priorities with the template shown. Substitute the real IDs and preserve the ConfigMap name and managed labels.

The larger priority tries Spot first during scale-up. Fallback remains subject to scheduling constraints and available capacity. Existing free on-demand capacity may receive a pod immediately. Neither preferred affinity nor the expander moves running pods back to Spot when it returns. Measure this behavior rather than assuming strict cheapest-first placement.

Before continuing: Changes the target cluster. GPU resources can incur charges.

Replace every highlighted value before running this command.

doctl kubernetes cluster update YOUR_CLUSTER_ID --expanders priority

Replace every highlighted value before running this command.

100:
  - "^YOUR_SPOT_POOL_ID$"
50:
  - "^YOUR_FALLBACK_POOL_ID$"
1:
  - ".*"

Run a small GPU Job that permits fallback

Save the YAML as gpu-spot-check.yaml. It runs nvidia-smi once to check that a scheduled container can see a GPU. It does not benchmark GPU compute, load a model or implement checkpoint recovery. Check the image is permitted by your registry policy; pin a verified image digest for repeated experiments.

The node selector requires our label on either GPU pool. The affinity expresses a preference for Spot, not a requirement. Do not add a required capacity-type=spot selector, because that would exclude on-demand nodes. A toleration permits scheduling through a taint; it does not select a node.

Apply the file and read the logs. Then run kubectl get pods -n default -l app=gpu-spot-check -o wide and kubectl get nodes -L doks.digitalocean.com/capacity-type,workload to identify where it ran. Expect GPU information in the logs and a Complete Job condition. Use kubectl get job gpu-spot-check -n default to check completion.

The Job allows three retries and has a 15-minute deadline. If provisioning takes longer, it can fail before a GPU becomes available. Investigate the cause before raising the deadline. For a real batch worker, use stable task IDs, write outputs idempotently and acknowledge queue work only after durable output is committed. Kubernetes may start a task more than once.

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-spot-check
  namespace: default
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 900
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels:
        app: gpu-spot-check
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 30
      nodeSelector:
        workload: gpu-batch
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: workload
          operator: Equal
          value: gpu-batch
          effect: NoSchedule
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: doks.digitalocean.com/capacity-type
                    operator: In
                    values: [spot]
      containers:
        - name: gpu-check
          image: nvidia/cuda:12.8.1-base-ubuntu24.04
          command: ["nvidia-smi"]
          resources:
            requests:
              cpu: "1"
              memory: 1Gi
            limits:
              memory: 2Gi
              nvidia.com/gpu: 1

Before continuing: Changes the target cluster. GPU resources can incur charges.

kubectl apply -f gpu-spot-check.yaml

kubectl logs -n default job/gpu-spot-check

Use PDBs for services and checkpoints for progress

A PodDisruptionBudget limits voluntary evictions. For a separate inference Deployment with at least two healthy replicas, the optional example allows one unavailable replica. Its selector must match that Deployment’s pod labels. It deliberately does not match our one-shot Job.

A PDB cannot prevent involuntary node loss or make reclaimed capacity remain available. A budget that permits no eviction can block a drain without saving the node. Give replicated services spare capacity, readiness probes and placement across nodes; distributing replicas within one Spot pool still leaves them exposed to a whole-pool reclaim.

For long jobs, save checkpoints periodically to durable storage outside the Spot nodes. Include model state, optimizer state and the input position needed to resume. Test restoring that checkpoint on the fallback GPU. Handle SIGTERM to stop accepting work and attempt a final save, but keep periodic saves because sudden loss may skip graceful shutdown. terminationGracePeriodSeconds is a pod shutdown setting, not a provider notice guarantee.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: gpu-inference
  namespace: default
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: gpu-inference

Test the fallback before trusting it

First run the diagnostic while both pools are available and record its node. For a controlled fallback eligibility test, save a second copy as gpu-fallback-check.yaml, change metadata.name to gpu-fallback-check, and add the selector doks.digitalocean.com/node-pool-id with your actual fallback pool ID under spec.template.spec.nodeSelector. Keep the existing workload selector. Apply that file and confirm its pod runs on an on-demand node.

That checks the image, tolerations and fallback capacity. It does not test Spot reclamation or the autoscaler’s capacity-error path. In a disposable cluster, separately rehearse an eviction and checkpoint resume with your actual worker. Record time to replacement, time to useful work and duplicate-output behavior. Do not force-delete a production node to validate a tutorial.

For a Pending pod, inspect kubectl describe pod POD_NAME -n default. Look for an untolerated taint, a selector excluding the fallback, insufficient nvidia.com/gpu, quota limits or unavailable provider capacity. Inspect cluster-autoscaler-status in kube-system and the events below. Capture actual reclaim event names from your cluster before creating alerts around them.

After testing, delete only the two diagnostic Jobs you created. Review both GPU pools in the Control Panel and remove tutorial-only pools once you have checked that no other workload uses them. Completed Jobs and their automatic cleanup do not stop node charges when pool minimums remain at one.

kubectl get events -A --sort-by=.metadata.creationTimestamp

Compare completed-work cost, then choose capacity

Use your account’s displayed rates instead of a headline discount. The Spot rate stays fixed for the pool’s lifetime, including autoscaled nodes. The offer at creation can change. Record node-hours for both pools, CPU nodes, storage and any network charges.

Calculate cost per successful task as total experiment cost divided by successful tasks. Count failed attempts and time spent restoring data in that total. Compare with the same input and success criteria on on-demand capacity. A cheaper node-hour can produce a more expensive completed job if restarts discard too much work.

If the recovery test passes and the measured cost is useful, use the DigitalOcean signup action below to create an account or continue with your existing team. Confirm GPU availability and the full fallback budget before provisioning. The referral link does not configure Kubernetes or guarantee promotional-credit eligibility for GPUs.

Check your result

Expected result: Diagnostic Job completes on the fallback pool.
Stop if: Inspect Pending events, selectors, taints, quota and capacity.
Next step: Test actual checkpoint recovery and measure completed-work cost.