RDMA Network Beginner's Detailed Guide: Zero Copy, Kernel Bypass, and the Three Mainstream Solutions RoCE/iWARP

To better understand RDMA, let's start with a scenario from everyday life:

At midnight on Double 11, you rush to buy, and the instant you click Pay, the money is deducted from your account, the merchant's inventory decreases by one, and the shipping label is automatically printed. All of this is almost Instant Completed.

But if the underlying network is poor, this instant will turn into a spinning wheel.

What we're going to talk about today RDMA, that is the super technology that can make data in data centers flow like "teleportation." Don't worry, we won't pile on jargon; I'll use a Express parcel sorting center the story, helping you see through RDMA's underpants.

Traditional network: just like "unload the goods first, then load the truck"

In traditional networks without RDMA (TCP/IP), the CPU is like a warehouse manager who micromanages everything.

When the network card (delivery truck) pulls in a truckload of data packages:

• Unloading (CPU copy): The CPU must itself move the package out of the NIC's small cache and put it into the system main memory (the warehouse).

• Unpacking check (kernel processing): The CPU needs to unpack the packet to see which application it is actually for (protocol stack unpacking).

• Load again (CPU copy): After verification, the CPU again has to move these packages from the system warehouse to the application's private warehouse (user-space memory).

pain point: The same parcel was moved back and forth by the CPU twice. This is not only slow (high latency), but also keeps the CPU extremely busy (high utilization), leaving it no time to handle the real business logic (such as calculating coupons).

RDMA: like a "direct conveyor belt"

RDMA (Remote Direct Memory Access, remote direct memory access) has arrived. It has done something ruthless:Bypass the CPU and let the NIC directly read and write application memory..

Still that courier sorting center:

• Preauthorization (memory registration): The application tells the network card in advance: "My warehouse address is 0x8888; when you get the package, just put it in directly, no need to ask me."

• Direct transfer (DMA transfer): The delivery truck (network card) has arrived; it goes directly through the conveyor belt and precisely delivers the package to shelf 0x8888 in the application.

• Zero disturbance: Throughout the entire process, the CPU supervisor just drank tea and read reports, without having to lift a finger at all.

Result: Data has been realized Zero-copy and kernel bypass. Latency drops directly from the millisecond level to Microsecond-level CPU load dropped significantly.

Three "business forms" of RDMA

Concepts alone aren't enough. In reality, RDMA has three deployment approaches; you just need to remember their characteristics:

1. InfiniBand (IB): premium dedicated line

• Features: "Perfect" standalone hardware designed specifically for RDMA—from NICs to switches, all homegrown.

• Evaluation: The performance ceiling, but the price is jaw-dropping. Generally only top-tier supercomputing centers (such as TaihuLight) can afford it.

2. RoCE (RDMA over Converged Ethernet): The People's Hero (v2 Version)

• Features: Let RDMA run on ordinary Ethernet switches without switching to expensive dedicated InfiniBand links.

• Evaluation:Currently the most mainstream. All you need is a RoCE-capable network card and to enable PFC flow control, and you can enjoy the thrill of RDMA on your existing network. The king of cost-effectiveness.

3. iWARP (Internet Wide Area RDMA): WAN coolie

• Characteristics: Based on the TCP protocol, it can run RDMA over ordinary IP networks and even the Internet.

• Evaluation: Although deployment is simple, it is limited by TCP congestion control and performs worse than RoCE v2. It is suitable for cross-datacenter scenarios where performance requirements are not so extreme.

Why should you pay attention to RDMA now?

Don't assume RDMA is hidden away. In fact, it has already permeated the services you use every day:

• AI large model training Models like ChatGPT require thousands of GPUs to compute in parallel, and the GPUs exchange gradient data frantically. Without RDMA? The GPUs can only sit idle and wait, with disastrously low utilization. With RDMA, the interconnection efficiency across thousands of GPUs is maxed out.

• Distributed storage: The cloud drive photos stored on your phone are backed by distributed storage systems. With RDMA, read speeds are as fast as accessing a local hard drive (NVMe over Fabrics).

• Database separation: Databases such as Oracle and SQL Server use RDMA to achieve memory pooling, and transaction processing speed soars.

Your first "Hello RDMA"

If you are a developer, how do you verify whether RDMA is working? It’s very simple, install it under Linux perftest toolkit, run two commands:

Server side (receiver):

ib_read_bw -a -d mlx5_0

Client (sender):

ib_read_bw -a -d mlx5_0 192.168.1.100

If you see bandwidth as high as tens to hundreds of Gbps, with extremely low CPU utilization, congratulations—you have successfully mastered RDMA.

Final words

To understand RDMA, you actually only need to remember three keywords:Zero-copy,Kernel bypass,CPU load reduction.

It is no longer a black technology that only top-tier giants can afford. With the maturity of RoCE technology, RDMA is becoming standard in next-generation data centers. Next time you marvel at AI’s instant responses and cloud drives’ instant transfers, don’t forget the RDMA heroes quietly contributing behind the scenes.

Next step recommendation: If you already understand the concept, you can go look it up next.PFC (Priority Flow Control) and ECN (Explicit Congestion Notification), this is the golden partner for ensuring RoCE does not drop packets in production environments.