InfiniBand setup and benchmarking
Goal
By the end of this tutorial you will have:
- Built a virtual cluster with InfiniBand
- Verified IB link status and measured bandwidth
- Run a multi-VM NCCL communication test
Prerequisites
- An ECI account with quota for GPU instances (at least two)
- A GPU instance type that supports the InfiniBand fabric
- An SSH client (built-in on macOS/Linux; WSL or PuTTY on Windows) — see Access methods
Step 1: Build the virtual cluster
- In the Compute > Virtual Cluster list, click Create Virtual Cluster to create a cluster. Select InfiniBand as the fabric type. (See Virtual cluster for the detailed setup.)
- Under Compute > Virtual Machine, click Create Virtual Machine to create the member VMs. Pick the Ubuntu 24.04 LTS (AI/GPU) image and the same GPU instance type as the cluster. The cluster needs at least two VMs to start.
- Add the VMs you created to the cluster. VMs can only be attached while in the Idle status, so either don't start them after creation, or stop them first if you already did.
- Click Start Cluster on the virtual cluster's detail page.
- Start each VM individually.
Each OS image ships a fixed NVIDIA driver version, and the compatible NCCL and CUDA (nvcc) versions depend on that driver. For ease of demonstration this tutorial is based on the Ubuntu 24.04 LTS image, but by following Step 3: Prepare the environment you can run it the same way on other operating systems.
Step 2: Verify InfiniBand
After connecting over SSH to a VM:
# HCA info
ibstat
ibv_devinfo
# IB port state (should be Active)
ibstat | grep -A5 "Port 1"
# List connected VMs
ibhosts
Step 3: Prepare the environment
The package installation in step 3 and the nccl-tests build in step 5 must be performed identically on every VM in the cluster. In particular, the libnccl version and the path of the nccl-tests build output (./build/all_reduce_perf) must match on every VM for mpirun to work.
In contrast, running the commands themselves (preparing the IPs and SSH keys, running the server and client in step 4, mpirun in step 5) is done only on the VM named in each instruction.
Install OpenMPI for communication between VMs, and perftest for testing.
sudo apt update
sudo apt install openmpi-bin libopenmpi-dev perftest -y
Also install the libnccl package for the NCCL test. libnccl must match the installed CUDA version, so check the CUDA version first. nvcc may not be on the PATH, so run it with its full path.
/usr/local/cuda/bin/nvcc --version
# nvcc: NVIDIA (R) Cuda compiler driver
# Copyright (c) 2005-2025 NVIDIA Corporation
# Built on Fri_Feb_21_20:23:50_PST_2025
# Cuda compilation tools, release 12.8, V12.8.93
# Build cuda_12.8.r12.8/compiler.35583870_0
Look up the libnccl package that matches the CUDA version shown by nvcc (e.g. cuda12.8) in the NVIDIA CUDA repository. Replace ubuntu2404 in the path with the OS version you are using.
Alternatively, the command below lists the installable versions in one go.
V=$(/usr/local/cuda/bin/nvcc --version | grep -oP 'release \K[0-9]+\.[0-9]+') && apt list -a libnccl2 2>/dev/null | grep "cuda$V"
# libnccl2/unknown 2.30.7-1+cuda13.3 amd64 [upgradable from: 2.26.2-1+cuda12.8]
# libnccl2/unknown,now 2.26.2-1+cuda12.8 amd64 [installed,upgradable to: 2.30.7-1+cuda13.3]
# libnccl2/unknown 2.25.1-1+cuda12.8 amd64
Install libnccl and its development package at the version you found. Both libnccl and libnccl-dev are required. Follow the command format below.
sudo apt install libnccl2=2.26.2-1+cuda12.8 libnccl-dev=2.26.2-1+cuda12.8 -y
Set the IP of each VM as an environment variable, since the following steps use them repeatedly. Use the IP address shown under Compute > Virtual Machine > (select the VM) > Network > Network Interface. All VMs must be on the same network. Fill in the real values, then export them on the VM where you will run the commands.
export VM_A_IP=<VM A IP> # server/master VM
export VM_B_IP=<VM B IP>
export VM_C_IP=<VM C IP>
Step 4: Bandwidth benchmark
Measure the RDMA write bandwidth between VMs with ib_write_bw. Start VM A (server) first, then have VM B (client) connect to VM A's IP.
# VM A (server)
ib_write_bw -d mlx5_0 --pkey_index 1
# VM B (client): use VM A's IP
ib_write_bw -d mlx5_0 ${VM_A_IP} --pkey_index 1
Set each value as follows.
| Value | Description |
|---|---|
-d mlx5_0 | The HCA device to use. It is one of the names listed by ibstat / ibv_devinfo in step 2; make the server and the client point at the same device. |
${VM_A_IP} | (Client only) The IP of the server VM. This is the value you exported in step 3. |
A healthy run should show bandwidth close to the cluster's IB fabric spec. If the measurement is significantly below spec, check cables, drivers, and cluster configuration.
Depending on your purpose, you can check link status and performance further with other ib_* commands.
| Command | Purpose |
|---|---|
ib_write_bw | RDMA Write bandwidth |
ib_write_lat | RDMA Write latency |
ibstat | IB link and port status between VMs |
Step 5: NCCL All-Reduce test
Build nccl-tests.
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests && make MPI=1 MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi
The nccl-tests build must be performed identically on every VM, and the libnccl version must also match on every VM for the test to run smoothly.
First, verify operation on a single VM with 8 GPUs.
# Single VM × 8 GPUs
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8
Before scaling out to multiple VMs, copy the master VM's SSH public key to the other VMs and log in once to register the host keys, so that mpirun can connect to each VM.
# Generate a key on the master VM if it has none
ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
# Copy the public key to each VM, then log in once
ssh-copy-id ${VM_B_IP} && ssh ${VM_B_IP} exit
ssh-copy-id ${VM_C_IP} && ssh ${VM_C_IP} exit
Now run the multi-VM benchmark from the master VM.
# All-Reduce test on 3 VMs × 8 GPUs
mpirun -np 24 \
-H ${VM_A_IP}:8,${VM_B_IP}:8,${VM_C_IP}:8 \
--map-by ppr:8:node --bind-to numa \
-x NCCL_IB_DISABLE=0 \
-x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
-x NCCL_SOCKET_IFNAME=enp0s3 \
-x NCCL_IB_PKEY=1 \
./build/all_reduce_perf -b 512M -e 8G -f 2 -g 1
The key options are:
| Option | Description |
|---|---|
-np 24 / -H ...:8 | Total number of processes and processes per VM (8 GPUs per VM → 3 VMs × 8 = 24) |
--map-by ppr:8:node --bind-to numa | Place 8 processes per VM, bound to NUMA boundaries |
NCCL_IB_DISABLE=0 | Force the use of InfiniBand |
NCCL_IB_HCA | List of HCAs to use (device names from ibstat in step 2) |
NCCL_SOCKET_IFNAME | Ethernet interface used for bootstrap (check with ip addr) |
Performance tuning options (NCCL_IB_GID_INDEX, NCCL_IB_QPS_PER_CONNECTION, NCCL_NET_GDR_LEVEL, NCCL_CROSS_NIC, and others) can be added selectively to suit your environment.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
ibstat not found | OFED not in the image | Recreate the VM with an OFED-enabled image |
| NCCL timeouts | VMs can't reach each other | Allow the traffic in the firewall and recheck the network subnet |
References
- NCCL environment variables: full list and meaning of performance tuning options such as
NCCL_IB_*andNCCL_NET_GDR_LEVEL - NCCL troubleshooting guide: diagnosing communication failures and performance degradation
- linux-rdma/perftest: IB bandwidth and latency benchmark tools such as
ib_write_bwandib_write_lat
Next steps
- Virtual cluster: build an InfiniBand-based HPC environment
- Parallel file system: shared storage for multi-VM training