Why can’t Kubernetes CSI storage plugins do without host kernel modules?

• Preface
• 1. The “Sense of Powerlessness” of CSI: Why Can’t It Handle Storage Mounts?
• II. FinalFile Architecture: User-Space CSI + Kernel-Space Driver
• III. Why can't the driver be placed in the container?
• IV. Three Major “Pitfalls” in Production Environments and Their Solutions
  • Pitfall 1: Kernel version mismatch (most common, most fatal)
  • Pitfall 2: Missing module dependencies (modprobe vs insmod）
  • Pitfall 3: Insufficient permissions for the CSI container
• V. Summary of Best Practices
• Conclusion

Preface

In the practice of Kubernetes containerized storage, our team developed an in-house solution named FinalFile distributed parallel file system. In the process of integrating it into the K8s cluster through CSI (Container Storage Interface), we encountered a difficult problem that almost all non-NFS storage products face:Why must the CSI plugin load a .ko kernel module on the host machine?

This article will combine FinalFile's implementation practices to deeply analyze the dependencies between CSI and kernel modules, common pitfalls, and best practices in production environments.

1. The “Sense of Powerlessness” of CSI: Why Can’t It Handle Storage Mounts?

The Kubernetes CSI specification defines a set of standard gRPC interfaces that allow storage vendors to write plugins to manage the lifecycle of volumes (create, delete, mount, unmount). But many people easily overlook one fact:

The CSI plugin itself runs in user space (inside a container), and it cannot directly make the host kernel "recognize" a new filesystem type.

When you specify in StorageClass fstype: finalfile At this point, the host kernel must be able to parse this string and find the corresponding filesystem driver. Otherwise,mount -t finalfile The command will throw an error directly:

mount: unknown filesystem type 'finalfile'

This is the fundamental reason why FinalFile (and all other self-developed storage) must provide kernel modules.

II. FinalFile Architecture: User-Space CSI + Kernel-Space Driver

Let's look at the overall architecture of FinalFile to understand where the .ko fits in:

Key points:

• Userspace CSI plugin Only responsible for "initiating mount requests" and "passing parameters" (such as storage backend IP and authentication information).

• Kernel mode finalfile.ko Responsible for "doing the actual work": communicating with storage backends through network protocols, and registering at the VFS (virtual file system) layer finalfile type, and mount the remote storage to a local path on the host machine.

III. Why can't the driver be placed in the container?

Many beginners ask: "Since containers are isolated, can't I just package the .ko into the CSI container image and then load it?"

The answer is:Absolutely not, and it can't be done.

The reason is simple:

• container What is shared is the host machine's kernel., not having an independent kernel of its own.

• Load kernel module (insmod/modprobe) is privileged operation, requires direct operation /lib/modules/$(uname -r)/ and kernel symbol table.

• Even if the container uses privileged: true When running, it can only "invoke" modules already loaded on the host machine, and cannot "load" a .ko file from inside a container into the host kernel.

So the correct approach is:Load finalfile.ko in advance on the host machine; the CSI plugin is only responsible for triggering the mount operation.

IV. Three Major “Pitfalls” in Production Environments and Their Solutions

Pitfall 1: Kernel version mismatch (most common, most fatal)

The FinalFile product delivery package will contain multiple precompiled .ko files, stored in separate directories by kernel version:

finalfile-package/
├── 3.10.0-1160.el7.x86_64/
│   └── finalfile.ko
├── 4.18.0-348.el8.x86_64/
│   └── finalfile.ko
├── 5.4.0-150-generic/
│   └── finalfile.ko
└── install.sh

The installation script will execute uname -r Get the current kernel version, then load the corresponding .ko.

phenomenon: If a certain K8s node executed yum update kernel and restart. Under the new kernel, there is no corresponding finalfile.ko, so all Pods using FinalFile storage will fail to mount.

Solution:

• Lock node kernel version In production environments, disable automatic kernel updates on K8s nodes.

• Upgrade product package: Before each kernel upgrade, confirm that the product package already contains the .ko files corresponding to the new kernel.

• Troubleshooting commands:

# Check the vermagic string required by the .ko
modinfo /path/to/finalfile.ko | grep vermagic

# Compare current kernel version
uname -r

# If loading fails, check the kernel log.
dmesg | tail -20

Pitfall 2: Missing module dependencies (modprobe vs insmod）

If finalfile.ko depends on other kernel symbols (such as encryption modules, network submodules), directly use insmod It will report an error.

Solution:

• Copy .ko to /lib/modules/$(uname -r)/extra/ Under the directory.

• Execute depmod -a Update module dependencies.

• use modprobe finalfile Load it, and it will automatically handle dependencies.

Pitfall 3: Insufficient permissions for the CSI container

When the CSI Node Plugin mounts, if it does not have sufficient permissions, it will report permission denied.

Solution: In the Pod Spec of a DaemonSet, the following must be configured:

securityContext:
  privileged: true
volumeMounts:
  - name: host-dev
    mountPath: /dev
  - name: host-sys
    mountPath: /sys
  - name: host-proc
    mountPath: /proc

Also mount the host machine's /dev,/proc,/sys, so that the container's inside mount The command can correctly interact with the host kernel.

V. Summary of Best Practices

Based on the implementation experience of FinalFile, we have summarized the following golden rules:

• Precompilation vs. dynamic compilation: If the product package supports dynamic compilation (i.e., compiling .ko at the customer site based on the kernel source code), it can greatly reduce the risk of version mismatch, but requires the customer to install kernel-devel Bag.

• Health check: Add pre-start check logic in the CSI Node Plugin to confirm lsmod | grep finalfile Exists; otherwise, actively try.modprobe.

• Documentation Clearly state in the deployment documentation the list of supported Linux distributions and kernel versions to prevent customers from deploying on unsupported systems.

• Log retention: Record in the installation script uname -a,modinfo Output,dmesg logs, to facilitate later troubleshooting.

Conclusion

Kubernetes CSI greatly lowers the barrier to connecting storage to containers, but it cannot solve all problems. When a storage system requires high performance, proprietary protocols, or kernel-level caching,Host kernel modules (.ko) remain an irreplaceable foundational cornerstone..

Understanding the collaborative relationship between CSI and kernel modules not only makes your storage plugin more stable and reliable, but also helps you quickly identify the root cause when mount failures occur. I hope the practical summary in this article can be helpful to you if you are working in the field of container storage.