Using NVIDIA GPU Operator

Available in VPC

You can use NVIDIA GPU Operator in Ncloud Kubernetes Service (NKS) to manage the drivers and related components of GPU nodes at the cluster level. If your node pools use different GPU models, you can apply a different driver version to each node pool.

Before you begin

Before using GPU Operator, check the following supported scope and prerequisites.

The key GPU Operator components covered in this document are as follows. Additional components may be installed depending on the chart version and settings.

GPU Operator manages the configuration of these components with a single ClusterPolicy resource.

Prepare the following in advance:

• Ncloud Kubernetes Service cluster consisting of a GPU node pool and a general node pool
• kubeconfig that can be used with kubectl
• Helm 3
• Quota request and approval for GPU server products
• Outbound path that allows nodes to pull images from nvcr.io and registry.k8s.io, or a container registry to which those images have been copied
• Path or internal mirror that allows nodes to access OS package repositories (the driver container downloads kernel header packages)
• Means of accessing the target nodes with root privileges, if you remove the pre-installed driver

The commands in this document run against the currently selected kubeconfig context. To target a different cluster, switch the context with kubectl config use-context before running them.

The values used during installation are as follows.

You cannot query the list of installable driver versions from the cluster. In GPU Operator Component Matrix, check the default driver version used by the GPU Operator version you want to install. Unless you have a specific reason to use a different version, use this value as is.

To specify a different version, check NVIDIA GPU Driver container image tags for a tag that supports your node's OS and kernel.

Driver and Container Toolkit images are provided for each node OS version.

Whether the NVIDIA driver is pre-installed depends on the GPU server image selected for the node pool. You can choose 1 of the following 2 options to manage the driver.

If you need to specify or change the driver version directly in Kubernetes, configure GPU Operator to manage the driver.

Prepare GPU node pools

Before installing GPU Operator, check the configuration and labels of the GPU node pools.

Run the following command to check whether the currently selected context is the target cluster.

kubectl config get-contexts
kubectl config current-context

Run the following command to check the node pool names and the labels of GPU nodes.

kubectl get node -L ncloud.com/nks-nodepool

NKS adds a label in the form of ncloud.com/nks-nodepool: <NODEPOOL_NAME> to each node to indicate the node pool it belongs to. This label persists even when nodes are replaced or scaled out, so you can use it to specify driver configurations per node pool. For more information about node labels, see Manage cluster node pools.

Run the following command to check the taints set on GPU nodes.

kubectl get node -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'

If you have set taints on a GPU node pool, add tolerations to the configuration so that driver pods can be scheduled on those nodes. The examples in this document use nvidia.com/gpu as the taint key. If the key you checked is different, change the tolerations field to match that key.

Install GPU Operator

Run the following command to add the NVIDIA Helm repository.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

If either of the following conditions applies, check the corresponding section before running the command. If neither applies, proceed as is.

• Using Ubuntu 20.04 (XEN) nodes: Use the command in that section instead of the command below.
• Using an internal registry: Specify the values from that section along with the command below.

Run the following command to install GPU Operator. Setting driver.nvidiaDriverCRD.enabled to true lets you manage the driver separately as a NVIDIADriver custom resource.

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --version "<OPERATOR_VERSION>" \
  --set driver.nvidiaDriverCRD.enabled=true \
  --set driver.nvidiaDriverCRD.deployDefaultCR=false \
  --wait

deployDefaultCR determines whether the chart creates the default resource for you. The default is true. This document sets it to false because you define both the default resource and the per-node-pool resources yourself. The scope of tasks varies depending on this setting. The differences are summarized in the table in Configure drivers per node pool.

Run the following command to check whether GPU Operator is running.

kubectl --namespace gpu-operator get pod -l app=gpu-operator
kubectl --namespace gpu-operator get pod -l app.kubernetes.io/name=node-feature-discovery

At this stage, it is sufficient for the gpu-operator pod and the Node Feature Discovery pods to be in the Running state.

Using an internal registry

This section describes what to do when nodes cannot access nvcr.io and registry.k8s.io. If both registries are accessible, skip this section.

Copy the images to the container registry you will use, then change the following values. Each component has its own field, so you must specify all the components you use.

The repository field is the path excluding the image name. The image name is in each component's image field. Because Node Feature Discovery is installed as a separate chart, specify the full path including the image name in node-feature-discovery.image.repository, and specify the tag with node-feature-discovery.image.tag.

Copy the following content and save it as a registry-values.yaml file. Replace <REGISTRY> with the address of the registry you will use.

operator:
  repository: <REGISTRY>/nvidia
validator:
  repository: <REGISTRY>/nvidia
driver:
  repository: <REGISTRY>/nvidia
  manager:
    repository: <REGISTRY>/nvidia/cloud-native
toolkit:
  repository: <REGISTRY>/nvidia/k8s
devicePlugin:
  repository: <REGISTRY>/nvidia
gfd:
  repository: <REGISTRY>/nvidia
dcgm:
  repository: <REGISTRY>/nvidia/cloud-native
dcgmExporter:
  repository: <REGISTRY>/nvidia/k8s
migManager:
  repository: <REGISTRY>/nvidia/cloud-native
node-feature-discovery:
  image:
    repository: <REGISTRY>/nfd/node-feature-discovery

Install by adding --values registry-values.yaml to the preceding helm install command.

If you use a registry that requires authentication, specify the secret name in each component's imagePullSecrets field. The NVIDIADriver resource uses the spec.imagePullSecrets field.

Using Ubuntu 20.04 (XEN) nodes

This section describes how to use Ubuntu 20.04 nodes. If you use a different OS version, skip this section.

For Ubuntu 20.04, some versions of the driver, Container Toolkit, and Device Plugin images are no longer provided, so you cannot use the chart defaults as is. Specify the following version combination.

Run the following command to install GPU Operator.

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --version v26.3.3 \
  --set driver.nvidiaDriverCRD.enabled=true \
  --set driver.nvidiaDriverCRD.deployDefaultCR=false \
  --set toolkit.version=v1.17.9-ubuntu20.04 \
  --set devicePlugin.version=v0.17.4 \
  --set gfd.version=v0.17.4 \
  --set dcgmExporter.version=3.2.6-3.1.9-ubuntu20.04 \
  --wait

Configure drivers per node pool

This section describes the case where GPU Operator manages the driver. If you use the driver in the server image as is, skip this section and proceed to Check driver installation method.

GPU Operator manages the driver with either the ClusterPolicy or NVIDIADriver resource. The differences between the 2 methods are as follows.

The NVIDIADriver resource defines both node selection conditions and the driver version, and GPU Operator creates a separate driver DaemonSet for each resource. In environments running node pools with different GPU models, we recommend creating a separate resource for each node pool to separate driver versions and kernel module types.

The scope of tasks depends on the values you selected during installation.

Create default configuration

The resource set to default: true handles GPU nodes that do not match the node selection conditions of the per-node-pool resources. This ensures that the driver is installed when you add a new GPU node pool, so if you plan to add node pools, define the default configuration first.

Copy the following content and save it as a nvidiadriver-default.yaml file. Replace <DEFAULT_DRIVER_VERSION> with the value you prepared.

apiVersion: nvidia.com/v1alpha1
kind: NVIDIADriver
metadata:
  name: default
spec:
  default: true
  driverType: gpu
  repository: nvcr.io/nvidia
  image: driver
  version: "<DEFAULT_DRIVER_VERSION>"
  kernelModuleType: auto
  usePrecompiled: false
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  upgradePolicy:
    autoUpgrade: true
    maxParallelUpgrades: 1
    maxUnavailable: 1

If you create dedicated resources for all GPU node pools, the default resource handles no nodes. In this case, the default resource is displayed as ready with 0 assigned nodes and does not pull the driver image. If you do not plan to add new node pools, you don't need to create the default resource.

Create per-node-pool configuration

Copy the following content and save it as a nvidiadriver-nodepool.yaml file. The first resource is a template to fill in with your values. Replace <GPU_NODEPOOL> and <DRIVER_VERSION> with the values you prepared. The second resource is an example that specifies a different driver version, kernel module type, and upgrade policy for a node pool with a different GPU model. Delete it if you have only 1 node pool. Resource names are used to specify targets in later commands, so give each resource the same name as its node pool.

apiVersion: nvidia.com/v1alpha1
kind: NVIDIADriver
metadata:
  name: "<GPU_NODEPOOL>"
spec:
  driverType: gpu
  repository: nvcr.io/nvidia
  image: driver
  version: "<DRIVER_VERSION>"
  kernelModuleType: auto
  usePrecompiled: false
  nodeSelector:
    ncloud.com/nks-nodepool: "<GPU_NODEPOOL>"
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  upgradePolicy:
    autoUpgrade: true
    maxParallelUpgrades: 1
    maxUnavailable: 1
---
apiVersion: nvidia.com/v1alpha1
kind: NVIDIADriver
metadata:
  name: gpu-l40s-01
spec:
  driverType: gpu
  repository: nvcr.io/nvidia
  image: driver
  version: "595.91.07"
  kernelModuleType: open
  usePrecompiled: false
  nodeSelector:
    ncloud.com/nks-nodepool: gpu-l40s-01
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  upgradePolicy:
    autoUpgrade: false

The key settings are as follows.

Unless you have a specific reason to use a different value, use auto for kernelModuleType.

Apply and verify configuration

Run the following command to apply the configuration. If you did not create a default configuration, skip the first command.

kubectl apply -f nvidiadriver-default.yaml
kubectl apply -f nvidiadriver-nodepool.yaml

NVIDIADriver is a cluster-scoped resource, so you do not specify a namespace. Driver pods are created in the namespace where GPU Operator is installed.

Run the following command to check the resource status.

kubectl get nvidiadriver

Immediately after the configuration is applied, the resource is displayed as notReady. After the driver container prepares and loads a module that matches the node's kernel, the remaining components restart sequentially. The run time varies depending on the number of GPUs, node specifications, and image download speed. If the status does not change, see Check driver installation method to identify the cause.

The configuration is applied when the STATUS field of all resources changes to ready. If you created a default configuration, also check that only one resource has the DEFAULT field set to true.

Run the following command to check the status of all components and ClusterPolicy.

kubectl --namespace gpu-operator get pod
kubectl get clusterpolicy cluster-policy

The configuration is complete when all pods in the gpu-operator namespace are in the Running or Completed state and the STATUS field of ClusterPolicy shows ready.

kubectl --namespace gpu-operator delete pod \
  -l 'app in (nvidia-operator-validator,nvidia-device-plugin-daemonset,gpu-feature-discovery,nvidia-dcgm-exporter)'

Run the following command to check the driver DaemonSets and their node selection conditions.

kubectl --namespace gpu-operator \
  get daemonset -l app.kubernetes.io/component=nvidia-driver \
  -o custom-columns='NAME:.metadata.name,READY:.status.numberReady,SELECTOR:.spec.template.spec.nodeSelector'

GPU Operator creates a DaemonSet for each combination of NVIDIADriver resource and node OS version. If you specify different driver versions for 2 node pools, 2 DaemonSets are created.

Run the following command to check the driver version applied to each node.

kubectl get node -l nvidia.com/gpu.present=true \
  -L ncloud.com/nks-nodepool \
  -L nvidia.com/cuda.driver-version.full \
  -L nvidia.com/gpu.product

The configuration is applied when the version specified for each node pool appears in the CUDA.DRIVER-VERSION.FULL field. If a value different from the specified version appears, see Check driver installation method to identify the cause.

Check driver installation method

Run the following command to check what manages the driver on each node.

kubectl get node -l nvidia.com/gpu.present=true \
  -L nvidia.com/gpu.deploy.driver \
  -L nvidia.com/cuda.driver-version.full

The value of the GPU.DEPLOY.DRIVER field indicates what manages the driver on that node.

On nodes displayed as pre-installed, the driver DaemonSet targets 0 nodes and no driver pods are created. Even in this state, the NVIDIADriver resource is displayed as ready, so you cannot determine from the resource status alone whether the version has been applied. Make sure to also check the CUDA.DRIVER-VERSION.FULL value. You can continue to use the driver in the server image. To switch to GPU Operator management, see Remove pre-installed driver.

kubectl get node -l nvidia.com/gpu.present=true -L nvidia.com/gpu-operator.driver.owner

If the resource is notReady or the specified version is not applied, run the following command to identify the cause.

kubectl describe nvidiadriver "<GPU_NODEPOOL>"
kubectl --namespace gpu-operator logs -l app.kubernetes.io/component=nvidia-driver --tail=100

If the output includes the ConflictingNodeSelector condition, the nodeSelector values overlap, so correct them.

If the STATUS field in the kubectl get nvidiadriver output shows disabled, the driver management method has not been switched to NVIDIADriver. In this case, creating the resource does not create a driver DaemonSet, so run the following command to check the ClusterPolicy setting.

kubectl get clusterpolicy cluster-policy -o jsonpath='{.spec.driver.useNvidiaDriverCRD}'

If the value is not true, switch the management method. If you have already completed the installation, check the installed chart version with the following command.

helm list --namespace gpu-operator

Specify the version you checked in --version to change only the settings. If you omit --version, the chart is upgraded to the latest version in the repository.

helm upgrade gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --version "<OPERATOR_VERSION>" \
  --reset-then-reuse-values \
  --set driver.nvidiaDriverCRD.enabled=true \
  --wait

If you are also upgrading the chart version, use the command in Switch driver management method.

Check GPU usage

Run the following command to check whether GPU resources are registered on the nodes.

kubectl get node -l nvidia.com/gpu.present=true \
  -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

The resources are registered when the number of GPUs appears in the GPU field. If <none> is displayed, run the following command to check the status of the Device Plugin pods.

kubectl --namespace gpu-operator get pod -l app=nvidia-device-plugin-daemonset

Copy the following content and save it as a cuda-vectoradd.yaml file. If you cannot access nvcr.io, change the image field to the address of the image copied to your registry.

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
  namespace: default
spec:
  restartPolicy: OnFailure
  nodeSelector:
    ncloud.com/nks-nodepool: "<GPU_NODEPOOL>"
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      resources:
        limits:
          nvidia.com/gpu: 1

Run the following command to create the pod and check the result.

kubectl apply -f cuda-vectoradd.yaml
kubectl --namespace default \
  wait --for=jsonpath='{.status.phase}'=Succeeded pod/cuda-vectoradd --timeout=5m
kubectl --namespace default logs pod/cuda-vectoradd

The pod performs vector addition and then terminates. If Test PASSED appears in the log, the GPU has been allocated successfully. When you're done, run the following command to delete the pod.

kubectl delete -f cuda-vectoradd.yaml

Remove pre-installed driver

Nodes whose server image includes a driver can be used as is. In this case, GPU Operator manages only the remaining components, and no additional action is required.

Remove the pre-installed driver and switch to GPU Operator management only if you want to use a different driver version for each node pool or specify the version in Kubernetes. After removal, the version specified in the NVIDIADriver resource is applied.

Proceed in the following order:

1. Run the following command to prevent new pods from being scheduled on the target node and move running pods to other nodes. Replace <NODE_NAME> with the name of the target node.

   kubectl cordon "<NODE_NAME>"
   kubectl drain "<NODE_NAME>" --ignore-daemonsets --delete-emptydir-data
   

2. Connect to the target node. This step runs directly on the node, not through kubectl, and requires root privileges. Remove the driver included in the server image with nvidia-uninstall.

   nvidia-uninstall --silent
   

3. Restart the node. Removing the driver files leaves the kernel module in memory. GPU Operator recognizes the node as having no driver only after a restart.

4. When the node is ready again, run the following command to check whether the driver management method has changed.

   kubectl get node -l nvidia.com/gpu.present=true \
     -L nvidia.com/gpu.deploy.driver \
     -L nvidia.com/cuda.driver-version.full
   

   The switch is complete when the GPU.DEPLOY.DRIVER field changes to true and the CUDA.DRIVER-VERSION.FULL field shows the version specified in the NVIDIADriver resource.

5. If the GPU.DEPLOY.DRIVER field still shows pre-installed after the restart, run the following command to update the value. Replace <NODE_NAME> with the name of the target node. This value persists only on nodes from which the driver has actually been removed. If the driver remains, it reverts to pre-installed.

   kubectl label node "<NODE_NAME>" nvidia.com/gpu.deploy.driver=true --overwrite
   

   If the value remains true, the node is included in the driver DaemonSet's targets and the driver container starts. When the installation is complete, the specified version appears in the CUDA.DRIVER-VERSION.FULL field.

6. After the driver is installed and passes Check GPU usage, run the following command to allow pods to be scheduled again.

   kubectl uncordon "<NODE_NAME>"
   

Change the driver version

To change the driver version of a node pool, modify the spec.version value of the corresponding NVIDIADriver resource. GPU Operator applies upgradePolicy only to the nodes handled by that resource and replaces the drivers sequentially. For resources with upgradePolicy.autoUpgrade set to false, modifying the version does not replace the driver, so change it to true before proceeding. When using Ubuntu 20.04 (XEN) nodes, the resource does not have the upgradePolicy field, so the driver.upgradePolicy value in ClusterPolicy is used.

If there are nodes to exclude from the replacement, run the following command to add a label before changing the version. Replace <NODE_NAME> with the name of the node to exclude. The driver on labeled nodes is not replaced until the label is removed.

kubectl label node "<NODE_NAME>" nvidia.com/gpu-driver-upgrade.skip=true --overwrite

Run the following command to change the driver version. If you keep the manifest file, also update the version value in the file. If you reapply the file without updating it, the driver reverts to the previous version.

kubectl patch nvidiadriver "<GPU_NODEPOOL>" \
  --type merge -p '{"spec":{"version":"<DRIVER_VERSION>"}}'

Run the following command to check the upgrade progress.

kubectl get node -l nvidia.com/gpu.present=true -L nvidia.com/gpu-driver-upgrade-state

The upgrade is complete when all target nodes reach the upgrade-done state.

kubectl --namespace gpu-operator logs deploy/gpu-operator | grep maxUnavailable

Switch driver management method

This procedure switches a cluster that manages the driver with ClusterPolicy to NVIDIADriver resource management. You can upgrade the chart version and switch the management method with a single command.

Run the following command to switch the management method.

helm upgrade gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --version "<OPERATOR_VERSION>" \
  --reset-then-reuse-values \
  --disable-openapi-validation \
  --set operator.upgradeCRD=true \
  --set driver.nvidiaDriverCRD.enabled=true \
  --set driver.nvidiaDriverCRD.deployDefaultCR=true \
  --set driver.upgradePolicy.autoUpgrade=true \
  --wait

When the switch is complete, apply the nvidiadriver-nodepool.yaml created in Create per-node-pool configuration to add per-node-pool resources. The default resource created by the chart continues to handle the remaining GPU nodes.

Delete GPU Operator and resources

If you no longer use GPU Operator, remove it in the following order:

Run the following command to delete the NVIDIADriver resources. If you did not create the default configuration yourself, skip the second command.

kubectl delete -f nvidiadriver-nodepool.yaml
kubectl delete -f nvidiadriver-default.yaml

If you don't have the manifest files, you can delete the resources by name with the following commands. Replace <RESOURCE_NAME> with the resource name you retrieved.

kubectl get nvidiadriver
kubectl delete nvidiadriver "<RESOURCE_NAME>"

Run the following command to remove the Helm release.

helm uninstall gpu-operator --namespace gpu-operator

Removing the Helm release does not delete the CRDs. If you no longer need the CRDs, first run the following command to check that no resources remain.

kubectl get nvidiadriver
kubectl get clusterpolicy

If the search result is empty, run the following commands to delete the CRDs. The second and third commands delete CRDs for features not covered in this document, so if you used those features, check for remaining resources before deleting them.

kubectl delete crd nvidiadrivers.nvidia.com clusterpolicies.nvidia.com
kubectl delete crd gpuclusters.nvidia.com
kubectl delete crd computedomains.resource.nvidia.com computedomaincliques.resource.nvidia.com

Some CRDs may not be installed depending on the chart version and settings. If a NotFound error appears, the CRD does not exist, so you can skip it.

To restore nodes to their original state, check the remaining labels with the following command.

kubectl get node -l nvidia.com/gpu.present=true -o name

For each node, run the following command to check the remaining label keys. Replace <NODE_NAME> with the name of the node you retrieved earlier.

kubectl get node "<NODE_NAME>" -o jsonpath='{.metadata.labels}' \
  | grep -o '"nvidia.com/[^"]*"' | tr -d '"'

Remove the keys you found with the following command. Replace <LABEL_KEY> with the label key to remove.

kubectl label node "<NODE_NAME>" <LABEL_KEY>-

If you plan to reinstall GPU Operator on the same nodes, restart the nodes after deletion. Even after the Helm release is removed, CDI configurations that reference the previously installed driver version remain in /run/cdi on the node. /run is cleared when the node restarts, so a restart alone cleans it up.

Failed to initialize NVML: Driver/library version mismatch

Limitations

Note the following limitations when using GPU Operator:

• GPU Operator is not provided as an NKS add-on, so you must manage its installation, upgrades, and removal yourself.
• Container images used by GPU Operator are subject to NVIDIA's terms of service. Review the terms before copying the images to an internal registry or redeploying them.
• The default and upgradePolicy fields of the NVIDIADriver resource are available starting with v26.7.0. In earlier versions, these 2 fields are not available, and the upgrade policy is set at the cluster level in ClusterPolicy.
• On nodes using the driver in the server image, the driver container does not run, and the version value of the NVIDIADriver resource is not applied. To use a different driver version for each node pool, you must remove the pre-installed driver, and you must repeat this task whenever you add or replace nodes.
• Support for the MIG feature varies depending on the hypervisor type of the GPU server. To check whether your product supports MIG, see General and KVM GPU.