Users Guide

13–NVMe-oF Configuration with RDMA
Optimizing Performance
243 AH0054602-00 M
In this example, the target NVMe device latency is 8µsec. The total latency
that results from the use of NVMe-oF is the difference between the initiator
device NVMe-oF latency (30µsec) and the target device NVMe-oF latency
(8µsec), or 22µsec.
4. Run FIO to measure bandwidth of the local NVMe device on the target
server. Issue the following command:
fio --verify=crc32 --do_verify=1 --bs=8k --numjobs=1
--iodepth=32 --loops=1 --ioengine=libaio --direct=1
--invalidate=1 --fsync_on_close=1 --randrepeat=1
--norandommap --time_based --runtime=60
--filename=/dev/nvme0n1 --name=Write-BW-to-NVMe-Device
--rw=randwrite
Where --rw can be randread for reads only, randwrite for writes only,
or
randrw for reads and writes.
Optimizing Performance
To optimize performance on both initiator and target servers:
1. Configure the following system BIOS settings:
 Power Profiles = 'Max Performance' or equivalent
 ALL C-States = Disabled
 Hyperthreading = Disabled
2. Configure the Linux kernel parameters by editing the
grub file
(
/etc/default/grub).
a. Add parameters to end of line
GRUB_CMDLINE_LINUX:
GRUB_CMDLINE_LINUX="nosoftlockup intel_idle.max_cstate=0
processor.max_cstate=1 mce=ignore_ce idle=poll"
b. Save the grub file.
c. Rebuild the
grub file.
 To rebuild the
grub file for a legacy BIOS boot, issue the
following command:
# grub2-mkconfig -o /boot/grub2/grub.cfg (Legacy BIOS boot)
 To rebuild the grub file for an EFI boot, issue the following
command:
# grub2-mkconfig -o /boot/efi/EFI/<os>/grub.cfg (EFI boot)
d. Reboot the server to implement the changes.