Kubernetes CSI Core Principles: From Mounting Workflow to Engineering Practice

Preface: Why Do We Need CSI?

If you have run Kubernetes in production Stateful applications(such as databases, AI training, message queues), you must have encountered this problem:The Pod has migrated—what about the data?

Kubernetes itself does not produce storage; it is only responsible for scheduling. To enable cloud vendors, open-source storage, and commercial storage to integrate seamlessly, CSI (Container Storage Interface) emerged.

Define CSI in one sentence: CSI is a unified interface standard that Kubernetes has defined for all storage vendors. With it, regardless of whether the underlying storage is AWS EBS, Ceph, or self-developed storage, K8s can operate it in the same way (create, mount, snapshot, expand).

CSI Core Components and Architecture

The implementation of CSI usually consists of two core components (running as Pods in K8s), with a clear division of labor:

1. CSI Controller (management plane)

• Deployment method: Deployment (usually 1 instance or HA multiple replicas)

• Core responsibilities: Management Storage volume lifecycle, doesn't care how data is mounted into the Pod

• Main interfaces:

• CreateVolume/DeleteVolume(Create/Delete Backend Storage)

• CreateSnapshot/DeleteSnapshot(Snapshot)

• ControllerExpandVolume(Capacity expansion)

2. CSI Node (Data Plane)

• Deployment method: DaemonSet (one Pod per Node)

• Core responsibilities: Execute Specific mounting action, mount the storage volume into the Pod's file system

• Main interfaces:

• NodePublishVolume(empty)NodeUnpublishVolume(Mount/unmount to the Pod directory)

• NodeStageVolume/NodeUnstageVolume(Format/mount to a global directory, used for multi-node shared scenarios such as Raw Block or iSCSI)

3. External auxiliary container (Sidecar)

官方提供多个 Sidecar 容器，与 CSI 驱动一起运行，简化开发：

Sidecar	Responsibilities
external-provisioner	Listen for PVC events and automatically invoke CreateVolume
external-attacher	Process VolumeAttachment, call ControllerPublishVolume
external-resizer	Listen for PVC expansion requests, call ControllerExpandVolume
external-snapshotter	Manage VolumeSnapshot CRD

A complete storage mount process

Let's trace the entire process of a PVC from creation to being used by a Pod:

How do storage vendors implement CSI plugins?

Taking the self-developed storage system as an example, when the user executes kubectl apply -f pvc.yaml When this happens, the CSI plugin internally does the following:

• Receive request: Sidecar external-provisioner Watch PVC objects and call the CSI driver's CreateVolume gRPC interface.

• Translation request: The CSI driver takes the parameters carried in the gRPC request (size=100Gi,csi.storage.k8s.io/pvc/name=mypvc) Parse it out.

• Call backend API: The driver converts these parameters into API calls to the backend storage system (e.g.CreateFileSystem(name=mypvc, capacity=100)).

• Return result: After the storage system creates a volume, it returns the volume ID and access path.

• Complete binding: The Provisioner writes the volume ID into the PV object, and the PVC is bound to the PV.

Key points: Storage vendors No need to modify a single line of Kubernetes core code. You only need to implement this set of gRPC interfaces, and any Kubernetes cluster can use this storage.

Common Engineering Problems and Deep Thinking

When using CSI in production environments, there are several issues worth exploring in depth:

1. What if a Pod is forcibly deleted before NodeUnpublish has a chance to execute?

Phenomenon: May exist on the host machine.Stale mount point, causing the mount to fail the next time the Pod is rescheduled to the same node because the directory is not empty or the device is busy.

Solution:

• Idempotency design: The CSI driver must properly handle repeated calls.NodePublishVolume scenario (check whether it is already mounted; if mounted, return success directly)

• Self-healing mechanism: Using Kubernetes' VolumeAttachment state machine, combined with external-attacher Perform unbinding and cleanup.

• Operations fallback: Provide node maintenance scripts to periodically scan for and clean up orphaned mount points.

2. How to support multi-node read-write (ReadWriteMany, RWX)?

• Challenge: Most block storage (such as AWS EBS) does not support multiple Pods mounting it for read-write at the same time.

• CSI's approach: in the CSI specification NodeStageVolume and NodePublishVolume The process allows storage vendors to implement the mount logic for NFS or parallel file systems (such as CephFS), thereby returning to Kubernetes ReadWriteMany ability

3. How to implement the simplest CSI driver from scratch?

Production-grade drivers are complex, but an educational demo can be completed in a few hours:

// Pseudocode example: a minimal CSI CreateVolume interface
func (s *identityServer) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest) (*csi.CreateVolumeResponse, error) {
    name := req.GetName()              // Volume name
    size := req.GetCapacityRange().GetRequiredBytes() // capacity

    // Fake a backend storage operation
    volumeID := "fake-vol-" + name

    // Create an empty directory on the host machine as "backend storage"
    os.MkdirAll("/tmp/csi-volumes/"+volumeID, 0755)

    return &csi.CreateVolumeResponse{
        Volume: &csi.Volume{
            VolumeId: volumeID,
            CapacityBytes: size,
        },
    }, nil
}

Production-grade drivers also need to consider: authentication, rate limiting, monitoring, graceful shutdown, error handling, and other details.

Summary

To understand CSI, the key is to master its Layered design philosophy:

• Decoupling: Kubernetes core does not care about storage implementation details.

• Standardization: A unified interface allows any storage vendor to integrate.

• Scalable: The Sidecar pattern allows functionality to be composed on demand.

Key takeaways from this article:

• The two core components of CSI: Controller (management plane) and Node (data plane)

• Complete mounting process: PVC → Provisioner → Controller → backend storage → PV → Pod

• Storage vendors only need to implement the CSI interface, without modifying K8s code.

• In engineering, attention needs to be paid to practical issues such as idempotency, residual cleanup, and RWX support.

After reading this, you will not only know how to deploy and use CSI drivers, but also understand the design principles and engineering practices behind them.

Further reading:

• Kubernetes CSI official documentation

• CSI Spec on GitHub

• Open-source CSI driver source code reference:csi-driver-nfs,csi-driver-smb