Using CSI block storages

We are preparing a localization service for the content. We will do our best to provide the localization service as soon as possible.

Available in Classic

Before use

Every container runs with files added at build time. Since then, files newly added to the file system of a container are all deleted when the container is restarted because the process is terminated or a liveness probe of Kubernetes fails health checks.
Since then, files newly added to the file system of a container are all deleted when the container is restarted because the process is terminated or a liveness probe of Kubernetes fails health checks.

If you want to keep the files even when the container is restarted, you can use block storage, which is persistent storage that can be created to keep data via PersistentVolumeClaim (PVC) that can be created upon deployment.

NAVER Cloud Platform’s Ncloud Kubernetes Service provides a Container Storage Interface (CSI) as a volume driver. This volume driver works with Kubernetes to create, delete, attach and detach block storage volumes.

Restrictions

Supported version

NAVER Cloud Platform Kubernetes Service provides CSI as a volume driver of block storage. In order for the CSI driver and Kubernetes to work with each other, you should check if their versions are mutually compatible.

Currently, NAVER Cloud Platform supports the following versions of CSI and Kubernetes.

The Kubernetes version 1.12 supports the following storage services:

Assignable capacity

The following are the maximum and minimum size of a block storage volume that can be assigned via CSI, where the size can be assigned in increments of 10 GB.
If the volume size of less than 10 GB is requested, the minimum size, 10 GB is created. If the required volume size is not defined, 20 GB is requested by default.

• Minimum: 10 GB
• Maximum: 2000 GB

Get Started with CSI (for How to Install CSI)

This section describes how to install Container Storage Interface (CSI) in a Kubernetes cluster. If CSI is already installed, you can skip the instructions below.

Requirements

• Deploy a CSI driver compatible with the Kubernetes cluster version.
• Set the --allow-privileged flag to true in kubelet of each worker node.
• Set the feature gate flag of --feature-gates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true in kubelet of each worker node.

Deploy a CSI driver to Kubernetes clusters

Get an authentication token

Go to the following page to get an authentication token.

• Page to get a token

You need an authentication key to get a token. Go to My Page > Manage Account > Manage Auth Key in the portal to get the authentication key.

1. Click Authorize on the upper right side of the page.
2. Enter your authentication key in the Access Key and Secret Key fields. Leave the API Key field empty. Click Authorize. Check that your authentication key is added, and click Done.
3. Click /auth/token/issue. Click Try it out on the following page.
4. Add the endpoint of the cluster running in Kubernetes Service as the value of the "endpoint" key. You can get the endpoint by clicking the cluster in the Kubernetes Service page.
5. Click Execute. If the operation is successful, you can check a token in the response.
6. You can check whether the issued token is valid from /auth/token/validate.

Deploy a CSI driver

Create a secret with the issued token.
Add your token as the value of the access-token key in the following Secret example, and save it as secret.yml.

apiVersion: v1
kind: Secret
metadata:
  name: nks-access-token
  namespace: kube-system
stringData:
  access-token: "eyJhbGciO__REPLACE_ME____f9hJtQa4rb7A"

Execute kubectl with the saved file to create a secret.

$ kubectl --kubeconfig=$KUBE_CONFIG create -f ./secret.yml
secret "nks-access-token" created

You can see that a secret is created in the namespace kube-system.

$ kubectl --kubeconfig=$KUBE_CONFIG get secrets -n kube-system
NAME                  TYPE                                  DATA      AGE
nks-access-token      Opaque                                1         1m

Deploy a CSI driver and sidecars

Deploy a CSI driver and sidecars. You should deploy a CSI driver that is compatible with the running Kubernetes cluster. You can see the list of available versions in Restrictions.

Deploy in k8s 1.12 version

Kubernetes 1.12 is compatible with CSI v0.4.

$ kubectl --kubeconfig=$KUBE_CONFIG apply -f http://nks-static.ncloud.com:8443/csi/0.4/latest/csi-nks-configmap.yaml
$ kubectl --kubeconfig=$KUBE_CONFIG apply -f http://nks-static.ncloud.com:8443/csi/0.4/latest/csi-nks-driver.yaml

Deploy in k8s 1.17 version

Kubernetes 1.17 is compatible with CSI v2.0.

$ kubectl --kubeconfig=$KUBE_CONFIG apply -f http://nks-static.ncloud.com:8443/csi/0.4/latest/csi-nks-configmap.yaml
$ kubectl --kubeconfig=$KUBE_CONFIG apply -f https://nks.apigw.ntruss.com/static/v1/csi/classic/2.0/latest/csi-nks-driver.yaml

These files are the latest stable versions. If you have a problem during the installation process, execute kubectl apply -f again to apply omitted resources.

CSI Driver Object

When you need to read and write persistent data) in the Kubernetes cluster, you can use the CSI driver to mount a block storage volume onto a container. You can request the required storage size through PersistentVolumeClaim (PVC). After checking the created PVC, the CSI driver creates a block storage volume and mounts it onto the requested container.

Storage Class

In Kubernetes, StorageClass is a concept that abstracts various types of storage according to performance and purpose of use. It helps you set a policy or type for storage. NAVER Cloud Platform provides the default StorageClass that uses block storage.

You can see an object named nks-block-storage in StorageClass in the namespace kube-system.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: nks-block-storage
  namespace: kube-system
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: blk.csi.ncloud.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  type: SSD

① The default StorageClass has an annotation storageclass.kubernetes.io/is-default-class set to true. Any other value or absence of the annotation is interpreted as false.

② volumeBindingMode controls when to enable volume binding and dynamic provisioning. If not defined, it defaults to Immediate.

③ reclaimPolicy sets the reclaim policy for the PersistentVolume (PV) in use when the PVC no longer needed is deleted. If not defined, it defaults to Delete.

④ The type under parameters specifies the type of a block storage volume to create. It can be either SSD or HDD. If not defined, it defaults to SSD.

⑤ If the allowVolumeExpansion field is 'true', PVC (PersistentVolumeClaim) can be expanded.

To change the created StorageClass object, refer to the following document.

• Changing the default StorageClass

CSI Controller

CSI controller creates, deletes, assigns, and unassigns volumes, and controls and manages snapshots. You can check the CSI controller from a statefulset in the namespace kube-system.

$ kubectl --kubeconfig=$KUBE_CONFIG get statefulset -n kube-system
NAME                 DESIRED   CURRENT   AGE
csi-nks-controller   1         1         37d

CSI Node

A CSI node formats, mounts and unmounts volumes, on every Kubernetes worker node. You can check CSI nodes from DaemonSet in the namespace kube-system.

$ kubectl --kubeconfig=$KUBE_CONFIG get daemonset -n kube-system
NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
csi-nks-node   3         3         3       3            3           <none>                        44d

Assign Block Storage using CSI

PersistentVolumeClaim

You can request necessary PersistentVolume (PV) resources by using a PersistentVolumeClaim (PVC). The CSI driver checks the PVC to automatically create a necessary PersistentVolume (PV).

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pod-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nks-block-storage

AccessMode

NAVER Cloud Platform Block Storage supports the following AccessMode.

• ReadWriteOnce: the volume can be mounted as read-write by a single node.

Resources

It specifies the required storage size. If not specified, it defaults to 20 GB. The storage size can be specified in increments of 10 GB, and can be between 10 GB and 2000 GB.

Assign a new volume to a pod

Create a new block storage volume and mount it.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pod-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nks-block-storage
---
kind: Pod
apiVersion: v1
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-frontend
      image: busybox
      volumeMounts:
      - mountPath: "/data"
        name: my-volume
      command: [ "sleep", "1000000" ]
  volumes:
    - name: my-volume
      persistentVolumeClaim:
        claimName: csi-pod-pvc

Pod

You can define containers and storage volumes to bind in spec.volumes of the pod. Add them as array elements, and specify the storage name and created PVC name in claimName of persistentVolumeClaim.

Defined storage volumes can be mounted in a container. If there is a container that needs a PersistentVolume, add the defined storage name to volumeMounts under spec.container. Specify the path to mount to in mountPath.

Mount an already created block storage volume into a pod

You can use the block storage volume that has already been created to create a PersistentVolume (PV), rather than creating a PersistentVolumeClaim (PVC) to automatically create a necessary storage volume.

Create a PersistentVolume (PV) with the instance ID of an already created block storage volume.

kind: PersistentVolume
apiVersion: v1
metadata:
  name: volume-existing-01
  annotations:
    pv.kubernetes.io/provisioned-by: blk.csi.ncloud.com # Provisioner name associated with the block storage volume
spec:
  storageClassName: nks-block-storage # Storage class name of the block storage volume
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 10Gi # Block storage size
  accessModes:
    - ReadWriteOnce
  csi:
    driver: blk.csi.ncloud.com
    fsType: ext4
    volumeHandle: "952814" # Block Storage Instance ID
    volumeAttributes:
      blk.csi.ncloud.com/noformat: "true" # Do not format the block storage volume
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pod-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nks-block-storage
  volumeName: "volume-existing-01"
---
kind: Pod
apiVersion: v1
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-frontend
      image: busybox
      volumeMounts:
      - mountPath: "/data"
        name: my-volume
      command: [ "sleep", "1000000" ]
  volumes:
    - name: my-volume
      persistentVolumeClaim:
        claimName: csi-pod-pvc

PersistentVolume

To use the block storage volume that already exists in your Kubernetes cluster, specify the following information when creating a PersistentVolume.

• Enter nks-block-storage, NAVER Cloud Platform’s storage class, for storageClassName.
• Enter the size of the block storage volume that already exists for capacity.storage.
• Enter ReadWriteOnce for accessModes. The storage class supporting block storage supports only this mode.
• Enter blk.csi.ncloud.com, the storage class driver name, for driver under csi.
• Enter the instance ID of the created block storage volume for volumeHandle under csi.
• Enter blk.csi.ncloud.com/noformat: "true" for volumeAttributes under csi. It means that the block storage volume is not formatted and is mounted with the data previous stored.

PersistentVolumeClaim

Create a PersistentVolumeClaim (PVC) to bind with the created PersistentVolume (PV).

• Enter the size of the block storage volume that already exists for storage under resources.
• Enter nks-block-storage, the block storage class name, for storageClassName.
• Enter the name of the created PersistentVolume (PV) for volumeName.

Pod

When creating a pod, specify a PersistentVolumeClaim (PVC), a volume request, to mount the volume to use.

• Specify claimName to use as an element of volumes under spec.

Example: assign additional volumes

Assign multiple volumes to a pod

Create two block storage volumes and mount them into a pod. Create a PersistentVolumeClaim for each volume to create.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pod-1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nks-block-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pod-2
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nks-block-storage
---
kind: Pod
apiVersion: v1
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-csi-app
      image: busybox
      volumeMounts:
      - mountPath: "/data/pod-1/"
        name: my-volume-1
      - mountPath: "/data/pod-2/"
        name: my-volume-2
      command: [ "sleep", "1000000" ]
  volumes:
    - name: my-volume-1
      persistentVolumeClaim:
        claimName: csi-pod-1
    - name: my-volume-2
      persistentVolumeClaim:
        claimName: csi-pod-2

Use a PersistentVolume in Deployment

Create a PersistentVolumeClaim (PVC) and make a request to create a new volume. You can mount the created volume by specifying claimName under template of Deployment.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-deployment-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nks-block-storage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-csi-app
spec:
  selector:
    matchLabels:
      app: my-csi-app
  replicas: 1
  template:
    metadata:
      labels:
        app: my-csi-app
    spec:
      containers:
        - name: my-frontend
          image: busybox
          volumeMounts:
          - mountPath: "/data"
            name: my-volume
          command: [ "sleep", "1000000" ]
      volumes:
        - name: my-volume
          persistentVolumeClaim:
            claimName: csi-deployment-pvc

Use a PersistentVolume in Statefulset

In Statefulset, you can define volumeClaimTemplates to automatically create a new PersistentVolumeClaim (PVC) for each replica.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-csi-app
spec:
  selector:
    matchLabels:
      app: my-csi-app
  replicas: 1
  template:
    metadata:
      labels:
        app: my-csi-app
    spec:
      containers:
        - name: my-frontend
          image: busybox
          volumeMounts:
          - mountPath: "/data"
            name: my-volume
          command: [ "sleep", "1000000" ]
  volumeClaimTemplates:
    - metadata:
        name: my-volume
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

Create Snapshots and Restore Volumes with CSI

Create a volume snapshot

VolumeSnapshot is a custom resource definition (CRD) defined by the CSI. Refer to the following page for more information.

• Snapshot-Restore-Feature

Create a VolumeSnapshot by using a PersistentVolumeClaim (PVC) bound to a PersistentVolume (PV).

apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
  name: csi-nks-test-snapshot # New VolumeSnapShot name
spec:
  source:
    name: csi-pod-pvc # Name of the PVC bound to the PV to create a Snapshot for.
    kind: PersistentVolumeClaim

• Enter the name of a VolumeSnapShot to create for name under metadata.
• Enter the name of the PVC bound to the PV for which you want to create a snapshot, for source.name under spec.
• Enter PersistentVolumeClaim for source.kind under spec.

Check a volume snapshot

To check the created volume snapshot, execute the following kubectl command.
The resource name of the volume snapshot is VolumeSnapShot.

$ kubectl --kubeconfig=$KUBE_CONFIG get volumesnapshot
NAME                    AGE
csi-nks-test-snapshot   17h

Delete a volume snapshot

To check the created volume snapshot, execute the following kubectl command.
Delete the resource by using the name of a volume snapshot to delete.

$ kubectl --kubeconfig=$KUBE_CONFIG get volumesnapshot
NAME                    AGE
csi-nks-test-snapshot   17h

$ kubectl --kubeconfig=$KUBE_CONFIG delete volumesnapshot csi-nks-test-snapshot
volumesnapshot.snapshot.storage.k8s.io "csi-nks-test-snapshot" deleted

Restore a volume from the created volume snapshot

Create a new volume from a volume snapshot. The content of the snapshot is replicated to a new volume.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-nks-test-pvc-restore
spec:
  dataSource:
    name: csi-nks-test-snapshot # Name of the volume snapshot to restore
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nks-block-storage
---
kind: Pod
apiVersion: v1
metadata:
  name: csi-restore-app
spec:
  containers:
    - name: my-frontend
      image: busybox
      volumeMounts:
        - mountPath: "/data"
          name: my-volume
      command: [ "sleep", "1000000" ]
  volumes:
    - name: my-volume
      persistentVolumeClaim:
        claimName: csi-nks-test-pvc-restore

PersistentVolumeClaim

To restore a volume from a volume snapshot, specify the following information in spec.dataSource.

• Enter the name of a volume snapshot to restore for dataSource.name.
• Enter VolumeSnapshot, the resource name of a volume to restore, for dataSource.kind.
• Enter snapshot.storage.k8s.io, the apiGroup of the snapshot, for dataSource.apiGroup.
• Enter the value same as the size of the volume snapshot, for resources.requests.storage.

Pod

Enter the PersistentVolumeClaim (PVC) name, the requested volume information, for claimName.

Delete PersistentVolumeClaim

Kubernetes does not delete a PersistentVolumeClaim (PVC) even if the requested resources such as Deployment, StatefulSet, ReplicaSet, and Pod are deleted.

Execute the following command to delete a PersistentVolumeClaim (PVC).

# Get a PVC
$ kubectl --kubeconfig=$KUBE_CONFIG get pvc
NAME          STATUS   VOLUME                           CAPACITY   ACCESS MODES   STORAGECLASS        AGE
csi-pod-pvc   Bound    pvc-084a811bac4211e9842bf220cd   10Gi       RWO            nks-block-storage   2m8s

# Delete a PVC
$ kubectl --kubeconfig=$KUBE_CONFIG delete pvc csi-pod-pvc
persistentvolumeclaim "csi-pod-pvc" deleted