RDMA (remote direct memory access) lets one host read or write memory on another host without going through either kernel or the remote CPU. The adapter does the copy directly from and to registered application buffers, so latency drops to a few microseconds and CPU use stays low at 100 to 400 Gb/s. It carries NVMe over Fabrics, iSER, NFS over RDMA, SMB Direct, vSAN, MPI and GPU-to-GPU traffic for AI training.

Verbs

The programming interface (libibverbs). Applications create objects on an RDMA device and post work requests to them.

Queue pair (QP)

A send queue and a receive queue that form one end of a connection. RC (reliable connected) is the usual type; UD is datagram.

Completion queue (CQ)

Where the adapter reports that a work request has finished.

Memory region (MR)

A buffer registered and pinned with the adapter. The remote side needs its rkey to read or write it.

GID and GID index

The RDMA address of a port. On RoCE each IP address creates a v1 and a v2 entry in the GID table, so tools must use the index of the v2 entry.

RDMA CM

Connection manager that sets up QPs from IP addresses, the way sockets do. NVMe, iSER, NFS and rping use it.

A lossless RoCEv2 recipe. RoCE retransmits by going back to the lost packet, so even small loss cuts throughput sharply. The usual design is:

1. One priority for RoCE, most often 3, marked with DSCP 26 (ToS 104, or 106 with the ECN-capable bits). Hosts and switches trust DSCP.
2. PFC (802.1Qbb) enabled on that priority only, on every host and switch port in the path, with the PFC watchdog on the switches. Global 802.3x pause off.
3. ECN marking on the switch queue for that priority, and DCQCN congestion control on the adapters. The receiver answers ECN marks with CNP packets, usually DSCP 48, which the switches forward in a strict-priority queue.
4. The same jumbo MTU on every hop (9000 on hosts, 9216 on switches), which gives a RoCE MTU of 4096.
5. Either static settings everywhere or DCBX learning from the switch, not a mix.

Newer adapters with selective retransmission tolerate some loss better, and a few vendors support RoCE over lossy networks, but check the vendor design guide before dropping PFC.