Originally published on the HostingPodium Enterprise Engineering Blog.

Free SRE & Developer Diagnostics: Test your stack live with our DMARC Record Checker, Generator & SPF/DKIM Validator, Free Website Speed & Core Web Vitals Checker, or explore all 9 Free Webmaster & Cloud Engineering Tools.

This guide details how to deploy Linux NVMe-oF over TCP and RDMA in a private-cloud environment, covering target configuration, initiator multipathing, performance tuning, and operational runbooks.

Introduction

Private-cloud operators increasingly separate compute from persistent block storage to achieve elasticity and to reuse flash arrays across tenants. Linux NVMe-over-Fabrics (NVMe-oF) delivers sub-millisecond latency when the transport is tuned for the underlying network. This guide explains how to build a production-grade NVMe-oF fabric using both TCP and RDMA, how to configure the nvmet target with nvmetcli, and how to integrate the fabric with multipath-aware initiators.

Core Components

  • NVMe-oF target kernel – nvmet module, compiled into the kernel (RHEL 9 / AlmaLinux 9 default).

  • Target management – nvmetcli JSON interface for persistent configuration.

  • Initiator utilities – nvme-cli (v2.5+), multipathd (device-mapper multipath).

  • Transport stack – TCP (standard Linux socket) and RDMA (RoCEv2 or iWARP) drivers.

  • Observability stack – collectd + Prometheus exporter, nvme-cli health commands.

Target Configuration with nvmetcli

The target definition lives in /etc/nvmet/config.json. The JSON format is stable across kernel releases, which makes it suitable for GitOps pipelines.

{ "ports": [ { "addr": { "traddr": "10.0.1.10", "trsvcid": "4420", "adrfam": "ipv4", "trtype": "tcp" }, "referrals": [] }, { "addr": { "traddr": "10.0.1.10", "trsvcid": "4420", "adrfam": "ipv4", "trtype": "rdma", "traddr_family": "ipv4", "traddr": "10.0.1.10", "trsvcid": "4420", "subsysnqn": "nqn.2024-11.com.hostingpodium:storage" } } ], "subsystems": [ { "nqn": "nqn.2024-11.com.hostingpodium:storage", "serial": "HPOD-001", "model": "NVMe-OF-Target", "allow_any_host": false, "hosts": [ {"nqn": "nqn.2024-11.com.hostingpodium:host01"}, {"nqn": "nqn.2024-11.com.hostingpodium:host02"} ], "namespaces": [ { "nsid": 1, "device_path": "/dev/nvme0n1", "host_ns": 1, "enable": true } ] } ] }

After editing, apply the configuration with:

# nvmetcli restore

Persist the JSON file in /etc/nvmet/config.json and enable the service:

# systemctl enable --now nvmet.target

Initiator Provisioning

On each compute node install the same nvme-cli version and enable the kernel module:

# modprobe nvme-tcp # modprobe nvme-fabrics

Discover the subsystem:

# nvme discover -t tcp -a 10.0.1.10 -s 4420 # nvme discover -t rdma -a 10.0.1.10 -s 4420

Connect using a persistent host NQN that matches the target ACL:

# nvme connect -t tcp -n "nqn.2024-11.com.hostingpodium:storage" -a 10.0.1.10 -s 4420 # nvme connect -t rdma -n "nqn.2024-11.com.hostingpodium:storage" -a 10.0.1.10 -s 4420

Multipath Integration

NVMe devices appear as /dev/nvmeXnY. To expose them as a single block device for the OS, configure multipath.conf:

defaults { user_friendly_names yes find_multipaths yes } blacklist { devnode "^nvme" } multipaths { multipath { wwid 0x123456789abcdef0 alias storage01 path_selector "round-robin 0" path_grouping_policy multibus failover "failover" rr_min_io 8 } }

Reload the daemon and verify the map:

# systemctl restart multipathd # multipath -ll

Kernel and Network Tuning

Sub-millisecond latency requires low interrupt latency, high queue depth, and tuned TCP buffers. Table 1 lists the recommended sysctl values for a 100 GbE fabric.

Parameter Recommended Value Rationale
net.core.rmem_max 12582912 Large receive buffers for TCP-NVMe streams
net.core.wmem_max 12582912 Symmetric transmit buffers
net.ipv4.tcp_rmem 4096 87380 12582912 Auto-scale receive window
net.ipv4.tcp_wmem 4096 65536 12582912 Prevent head-of-line blocking
net.core.somaxconn 4096 Accommodate many concurrent connections per target
vm.nr_hugepages 1024 Allocate 2 MiB pages for NVMe submission/completion queues
fs.file-max 2097152 Support thousands of device nodes

Apply the values with sysctl -p /etc/sysctl.d/99-nvmeof.conf. For RDMA, ensure the NIC is configured for RoCEv2 with a dedicated VLAN and PFC enabled; the kernel module ib_uverbs must be loaded.

Operational Runbook

1. Deploy Target Nodes

  • Provision bare-metal or VM with at least 2 × Intel Xeon E5-2699 v4, 256 GiB RAM, and dual-port 100 GbE NICs supporting RoCEv2.

  • Install RHEL 9, enable the nvmet and nvmetcli packages from the AppStream repository.

  • Apply the kernel tuning from Table 1 and reboot.

  • Copy the immutable config.json from Git, run nvmetcli restore, and verify nvme list-subsys on the host.

  • Start nvmet.target and enable firewalld ports 4420/tcp and 4420/udp (RDMA).

2. Provision Initiator Nodes

  • Install nvme-cli ≥ 2.5, multipath-tools, and collectd.

  • Configure the host NQN in /etc/nvme/hostnqn and register it with the target ACL.

  • Run the discovery and connect commands shown earlier.

  • Update /etc/multipath.conf with the WWID reported by nvme id-ctrl and restart multipathd.

  • Validate I/O path redundancy by disabling one NIC and confirming multipath -ll shows an active path count of 2.

3. Monitoring & Alerting

  • Collectd plugin NVMe publishes nvme_io counters every 10 s.

  • Prometheus scrape /metrics from nvme_exporter (port 9100) on each target.

  • Alert on nvme_io_errors_total > 0 or latency nvme_latency_seconds > 0.001 for more than 5 min.

4. Upgrade Procedure

When moving from kernel 5.14 to 6.4, the nvmet ABI remains stable, but the JSON schema adds the optional max_qsize field. Perform a rolling upgrade:

  • Drain one target node (remove it from the load-balancer).

  • Backup /etc/nvmet/config.json and the /var/lib/nvmet state directory.

  • Install the new kernel, reboot, and run nvmetcli restore to re-apply the configuration.

  • Run nvme list-subsys on a random initiator to confirm connectivity.

  • Re-introduce the node to the fabric and repeat for the remaining nodes.

TCP vs. RDMA Trade-offs

Both transports expose the same NVMe command set, but they differ in CPU overhead, latency, and deployment complexity.

Metric TCP RDMA (RoCEv2)
Average latency (1 µs I/O) 0.75 ms 0.35 ms
CPU per GB/s 0.45 % core 0.12 % core
Network requirement Standard L3 routing, no PFC Lossless Ethernet, PFC + ETS
Operational complexity Low – works over existing VLANs Medium – requires DCB configuration
Compatibility All Linux kernels 5.0+ Requires RDMA NIC driver (mlx5, cx4)

For workloads that demand < 0.5 ms latency (e.g., high-frequency trading or real-time analytics), RDMA is the only viable path. For general-purpose private clouds where the network team prefers a simpler stack, TCP provides acceptable latency with zero-touch configuration.

Security Considerations

NVMe-oF does not encrypt data in flight. Two mitigation layers are recommended:

  • IPsec for TCP transports – use strongSwan with AES-256-GCM and a 5-second re-key interval.

  • TLS-offload for RDMA – deploy a hardware security module (HSM) that terminates TLS before the NIC, then forward clear traffic over RoCEv2 inside a dedicated VLAN.

Access control is enforced by the target ACL (the hosts array in the JSON). Rotate host NQNs every 90 days and audit the ACL with:

# jq '.subsystems[].hosts' /etc/nvmet/config.json

Enable auditd rule to log nvmet syscalls:

-w /sys/kernel/config/nvmet -p wa -k nvmet

Observability Stack

The following Prometheus scrape targets are required:

  • node_exporter – for CPU, memory, and network counters.

  • nvme_exporter – exposes nvme_io_read_bytes_total, nvme_io_write_bytes_total, and nvme_latency_seconds.

  • collectd – forwards nvme plugin metrics to InfluxDB for long-term retention.

Grafana dashboards can be built from the NVMe-oF.json dashboard template shipped with the exporter.

Comparison with Alternative Architectures

Table 3 positions NVMe-oF against iSCSI, NFS-Ganesha, and Ceph RBD for a private-cloud block-storage layer.

Feature NVMe-oF iSCSI NFS-Ganesha Ceph RBD
Latency (99th %) 0.35 ms (RDMA) 0.9 ms 1.2 ms 0.8 ms
IOPS per GB ~150 k ~70 k ~45 k ~100 k
CPU overhead 0.12 % core/GB (RDMA) 0.45 % core/GB 0.55 % core/GB 0.30 % core/GB
Management complexity Medium (ACL + transport) Low Low High (CRUSH maps)
Data protection Target-side RAID or hardware ECC Target RAID POSIX ACLs Erasure coding

NVMe-oF wins on raw performance and CPU efficiency, but it requires a lossless Ethernet fabric for the best numbers. Organizations that cannot guarantee DCB should start with TCP NVMe-oF and evaluate a migration path to RDMA as the network matures.

Summary

The guide presents a repeatable, version-controlled method to deliver sub-millisecond block storage across a private-cloud fabric. By separating target configuration (nvmetcli), initiator multipathing, and observability, operators can scale to thousands of hosts while maintaining deterministic latency and auditable security.

Free Webmaster, SRE & Cloud Diagnostic Tools (No Signup Required)

Verify and benchmark your production infrastructure directly from your browser using the HostingPodium Free Webmaster & Developer Suite:

Free Engineering Utility Primary Diagnostic Use Case
Global DNS Propagation & SSL Checker Verify live A, AAAA, CNAME, MX, TXT, NS & CAA records across 21+ global resolvers + TLS certificate audits
Website Speed Test & Core Web Vitals Auditor Measure server TTFB, LCP, INP, CLS, and HTTP/2 compression latency in real time
Rich Snippets & Schema.org JSON-LD Generator Generate & validate structured JSON-LD markup for TechArticles, FAQs, Products & Organizations
HTTP Redirect Chain & Status Code Tracer Trace 301/302 redirect hops, inspect response headers, and verify load balancer health checks
DMARC, SPF & DKIM Email Security Validator Audit domain email authentication policies and generate compliant DMARC records
Google SERP Simulator & Meta Tag Previewer Preview desktop/mobile Google snippets, OpenGraph cards, and pixel-width truncation
Free WebP Image Converter & Optimizer Client-side PNG/JPG to WebP compression for faster Core Web Vitals LCP

Related Enterprise Architecture & Benchmarks