Emulex® NVMe over Fibre Channel User Guide Release 12.
Broadcom, the pulse logo, Connecting everything, Avago Technologies, Avago, the A logo, Emulex, ExpressLane, LightPulse, and OneCommand are among the trademarks of Broadcom and/or its affiliates in the United States, certain other countries, and/or the EU. Copyright © 2016–2018 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. For more information, please visit www.broadcom.com.
Emulex NVMe over Fibre Channel User Guide Table of Contents Chapter 1: Overview ........................................................................................................................... 4 1.1 Abbreviations ............................................................................................................................................................4 Chapter 2: Installing NVMe over FC .............................................................................................
Emulex NVMe over Fibre Channel User Guide Chapter 1: Overview This user guide provides instructions for installing and configuring NVMe over FC drivers on Emulex® LPe31000-series and LPe32000-series adapters in initiator and target systems, and on LPe35000-series adapters in target systems. NVMe over FC is a relatively new protocol for solid-state storage devices built with nonvolatile memory technologies.
Emulex NVMe over Fibre Channel User Guide Acronym/Abbreviation Description I/O input/output ID identifier IOPs input/output operations per second IOQ input/output queue LBA logical block address LUN logical unit number NQN NVMe qualified name NVMe Nonvolatile Memory Express NVMeT Nonvolatile Memory Express target SCSI Small Computer System Interface SLES SUSE Linux Enterprise Server WWNN World Wide Node Name WWPN World Wide Port Name XRI extensible resource indicator Broadcom
Emulex NVMe over Fibre Channel User Guide Chapter 2: Installing NVMe over FC For SLES 12 SP3 and SLES 15, this chapter describes how to install the out-of-box NVMe drivers, and it describes the NVMe-specific driver parameters. For Windows Server 2016, Windows Server 2012 SP2, and Windows Server 2012: The FC driver for Windows supports both SCSI and NVMe on initiator servers. Refer to the Emulex Drivers for Windows for LightPulse Adapters User Guide for installation instructions.
Emulex NVMe over Fibre Channel User Guide 2.3 NVMe-Specific Driver Parameters NVMe-specific driver parameters are displayed in the following table. For information on how to set the parameters, refer to the Emulex Drivers for Linux for LightPulse Adapters User Guide. sysfs Visible Activation Defines the number of I/O channels supported by the driver. The default value is 0. A value of 0 means the driver looks at the number of online CPUs in the system. The default value is strongly recommended.
Emulex NVMe over Fibre Channel User Guide Chapter 3: Configuring NVMe over FC on SLES 12 SP3 and SLES 15 To configure NVMe over FC, you must perform the following procedures in the indicated sequence: 1. Configure NVMe over FC on target systems (see Appendix A, Configuring NVMe over FC on a Target (SLES 12 SP3 Only)). 2. Configure NVMe over FC on initiator systems, as described in this chapter.
Emulex NVMe over Fibre Channel User Guide NVME RPORT WWPN x10000090fa942779 WWNN x20000090fa942779 DID x011400 TARGET DISCSRVC ONLINE NVME Statistics LS: Xmt 0000000000000019 Cmpl 0000000000000019 FCP: Rd 000000000000002e Wr 0000000000000011 IO 000000000000000c Cmpl 000000000000004b Outstanding 0000000000000000 5. If multipathing is used in SLES 15, skip to Step 6.
Emulex NVMe over Fibre Channel User Guide +- NVME4 FC TRADDR=NN-0X20000090FA942E16:PN-0X10000090FA942E16 HOST_TRADDR=NN-0X200000109B346F1F:PN0X100000109B346F1F +- NVME8 FC TRADDR=NN-0X20000090FA942E16:PN-0X10000090FA942E16 HOST_TRADDR=NN-0X200000109B346F20:PN0X100000109B346F20 NOTE: For additional information on multipathing, see Appendix B, Multipathing. 7.
Emulex NVMe over Fibre Channel User Guide Chapter 4: Enabling and Configuring NVMe over FC on Windows This chapter provides instructions for the following tasks: Enabling and configuring NVMe over FC on a Windows initiator using the OneCommand Manager GUI or the OneCommand Manager CLI Monitoring NVMe targets using the OneCommand Manager CLI NOTE: Windows initiator systems must connect to target devices that are configured for NVMe. 4.
Emulex NVMe over Fibre Channel User Guide Parameter Definitions Activation Requirement Notes NumNVMEIOQ NumNVMEIOQ specifies the maximum number of Reboot NVMe I/O queues supported by the driver within an NVMe subsystem. Value: 4 to 255 (decimal) or 0x4 to 0xFF (hexadecimal) The NumNVMEIOQ parameter caps the maximum number of IOQs supported by the driver. The driver allocates resources based on this setting.
Emulex NVMe over Fibre Channel User Guide Figure 1: Enabling NVMe over FC 4.1.1 Driver Limitations The following limitations apply to the Windows driver when it is enabled for NVMe over FC: NVMe subsystems are assigned SCSI path IDs and target IDs, and they do not persist across boots. Thus, persistent binding is not supported. NVMe namespaces are assigned LUN numbers in the order in which they are discovered, and they do not persist across boot.
Emulex NVMe over Fibre Channel User Guide 4.1.2 Error Codes The following NVMe error codes have been added to the driver for Windows. Byte 0x10 Interpretation 0x80 (severe) The NVMe over FC Keep Alive feature failed on the indicated path and target. 0x81 (command) The indicated IOQ is not available. 0x82 (malfunction) IOQ creation failure. Byte 0x11 indicates the number of IOQs created. Byte 0x12 indicates the expected number of IOQs to be created. 4.
Emulex NVMe over Fibre Channel User Guide Parameters local Displays only local adapters. m=model The model filter. Append * to the end of the model name for a wildcard match. For example, LPe32*. pt=type The port type filter. Valid types are FC and NVMe. down Not applicable. Example hbacmd listhbas pt=nvme Information similar to the following is displayed: Manageable HBA List Port WWN Node WWN Fabric Name Flags Host Name Mfg Serial No.
Emulex NVMe over Fibre Channel User Guide Parameters WWPN The WWPN of an FC function. Filter Type The type of filter to apply (optional). S = Display SCSI nodes only. N = Display NVMe nodes only.
Emulex NVMe over Fibre Channel User Guide Port WWN : 50:06:01:60:10:20:5C:38 OS Device Name : \\.\Scsi5:0:1 4.2.2 NVMe-Specific Commands The OneCommand Manager CLI commands in this section are specifically used for NVMe over FC targets. Support by NVMe targets for some of the commands in this section is optional and depends on whether namespace management is supported by the NVMe target. See the command descriptions for support requirements.
Emulex NVMe over Fibre Channel User Guide Example hbacmd nvme-set-sstable 10:00:00:90:fa:5d:05:a9 50:06:01:60:90:20:5C:38 nqn.201408.com.example:nvme.host.sys.xyz 0x0001 1 4096 Information similar to the following is displayed: NVMe Subsystem table successfully updated for port 10:00:00:90:fa:5d:05:a9 4.2.2.2 The nvme-get-sstable Command The nvme-get-sstable command allows you to retrieve the NVMe subsystem management table that is maintained for each FC+NVMe port.
Emulex NVMe over Fibre Channel User Guide Support Requirement Mandatory Example hbacmd nvme-list 10:00:00:90:fa:5d:05:a9 Information similar to the following is displayed: Discovered NVMe Subsystems for 10:00:00:90:fa:5d:05:a9 NVMe Qualified Name : nqn.2014-08.com.example:nvme.host.sys.xyz Port WWN : 50:06:01:60:90:20:5C:38 Controller ID : 0x0001 Model Number : NVMfxx999 Serial Number : 0123456789 Firmware Version : 1.2.0.
Emulex NVMe over Fibre Channel User Guide Information similar to the following is displayed: Controllers for subsystem nqn.2014-08.com.example:nvme.host.sys.xyz Number of Controllers: 3 Controller ID: 0x0000 Controller ID: 0x0001 Controller ID: 0x0002 4.2.2.5 The nvme-id-ctrl Command The nvme-id-ctrl command sends the NVMe identify command for a controller and displays the results.
Emulex NVMe over Fibre Channel User Guide Controller Features: Namespace Management Non-zero feature Select option : Supported : Not Supported This example shows the raw parameter specified: hbacmd nvme-id-ctrl 10:00:00:90:fa:5d:05:a9 50:06:01:60:90:20:5C:38 nqn.201408.com.example:nvme.host.sys.
Emulex NVMe over Fibre Channel User Guide Information similar to the following is displayed: Attributes for Controller 0x0001: IEEE OUI Identifier : 032957 Maximum data transfer size : 5120 NVMe version supported : 1.2.
Emulex NVMe over Fibre Channel User Guide FeatureID The feature identifier: 0x1 = Arbitration 0x2 = Power management 0x3 = LBA range type 0x4 = Temperature threshold 0x5 = Error recovery 0x6 = Volatile write cache 0x7 = Number of queues 0x8 = Interrupt coalescing (not supported) 0x9 = Interrupt vector configuration (not supported) 0xA = Write atomicity normal 0xB = Asynchronous event configuration 0xC = Autonomous power state transition 0xD = Host memory buffer 0xF = Keep alive timer 0x80 = Software progr
Emulex NVMe over Fibre Channel User Guide tempselect The temperature threshold value to be retrieved: 0x00 = Over-temperature threshold value for composite temperature 0x01 = Over-temperature threshold value for temperature sensor 1 0x02 = Over-temperature threshold value for temperature sensor 2 ...
Emulex NVMe over Fibre Channel User Guide Examples This example shows no optional parameters specified: hbacmd nvme-get-feature 10:00:00:90:fa:5d:05:a9 50:06:01:60:90:20:5C:38 nqn.201408.com.example:nvme.host.sys.
Emulex NVMe over Fibre Channel User Guide Information similar to the following is displayed: Feature: LBA Range Type Byte Dump (Saved Values) 0000: 03 04 07 00 00 00 00 0010: 04 02 00 0a 45 4d 55 0020: 20 20 20 20 00 00 17 0030: 35 4d 5a 2d 45 4c 58 0040: 00 3a 00 00 41 41 31 60 4c 6a 20 32 40 45 41 20 32 0c 58 46 20 31 70 20 42 20 4a 06 20 52 20 30 8c 20 2d 52 43 0a 20 35 03 32 00 20 37 00 50 00 20 46 48 20 This example shows the temperature threshold feature selected: hbacmd nvme-get-feature 10
Emulex NVMe over Fibre Channel User Guide Information similar to the following is displayed: Feature: Number of Queues Attribute Name Number of I/O Submission Queues Allocated Number of I/O Completion Queues Allocated Value(Current) 63 63 Note: The above values are zero based This example shows the interrupt coalescing feature selected: hbacmd nvme-get-feature 10:00:00:90:fa:5d:05:a9 50:06:01:60:90:20:5C:38 nqn.201408.com.example:nvme.host.sys.
Emulex NVMe over Fibre Channel User Guide This example shows the autonomous power state transition feature selected: hbacmd nvme-get-feature 10:00:00:90:fa:5d:05:a9 50:06:01:60:90:20:5C:38 nqn.201408.com.example:nvme.host.sys.
Emulex NVMe over Fibre Channel User Guide This example shows the host identifier feature selected: hbacmd nvme-get-feature 10:00:00:90:fa:5d:05:a9 50:06:01:60:90:20:5C:38 nqn.201408.com.example:nvme.host.sys.
Emulex NVMe over Fibre Channel User Guide 4.2.2.7 The nvme-list-ns Command The nvme-list-ns command lists the namespaces for a specified NVMe target.
Emulex NVMe over Fibre Channel User Guide This example shows allocated namespace identifiers selected: hbacmd nvme-list-ns 10:00:00:90:fa:5d:05:a9 50:06:01:60:90:20:5C:38 nqn.201408.com.example:nvme.host.sys.xyz 1 Information similar to the following is displayed: Allocated Namespaces: Namespace ID : 0x00000003 Namepsece ID : 0x00000004 4.2.2.8 The nvme-id-ns Command The nvme-id-ns command sends the NVMe identify command for a namespace and displays the results.
Emulex NVMe over Fibre Channel User Guide Logical Block Size Multi-path Access LBA Format Number of Supported LBA Formats End-to-end Data Protection Reservations : : : : : : 512 Bytes Yes 0 0 Disabled Not Supported Namespace Features: Thin Provisioning : Not Supported Unwritten Logical Block Error: Not Supported NAWUN, NAWUPF, NAWCWU : Not Supported End-to-end Protection Capabilities: Protection Information Type 1: Supported Protection Information Type 2: Supported Protection Information Type 3: Support
Emulex NVMe over Fibre Channel User Guide End-To-End Data Protection Capabilities First Eight Bytes : Supported Last Eight Bytes : Not Supported Namespace Multi-path I/O and Sharing Capabilities Namespace Sharing : Supported Reservation Capabilities Persist Through Power Lost Write Exclusive Exclusive Access Write Exclusive - Registrants Only Exclusive Access - Registrants Only Write Exclusive - All Registrants Exclusive Access - All Registrants Ignore Existing Key Supported LBA LBA Format LBA Format LBA F
Emulex NVMe over Fibre Channel User Guide Vendor-specific data bytes: Byte Dump 0180: 03 04 0190: 04 02 01A0: 20 20 01B0: 35 4d 01C0: 00 3a 01D0: 20 20 01E0: 00 00 01F0: 00 00 0200: 00 00 0210: 00 00 0220: 00 00 0230: 00 00 …………: 00 00 …………: 00 00 …………: 00 00 0FF0: 00 00 07 00 20 5a 00 20 00 00 00 00 00 00 00 00 00 00 00 0a 20 2d 00 20 00 00 00 00 00 00 00 00 00 00 00 45 00 45 41 31 00 00 00 00 00 00 00 00 00 00 00 4d 00 4c 41 32 00 00 00 00 00 00 00 00 00 00 00 55 17 58 31 30 00 00 00 00 00 00 00 00
Emulex NVMe over Fibre Channel User Guide Support Requirements Mandatory if the target supports namespace management. Not supported if namespace management is not supported. Examples This example shows no optional parameters specified: hbacmd nvme-create-ns 10:00:00:90:fa:e0:63:48 20:03:00:11:0d:a5:70:00 nqn.2014-08.org.
Emulex NVMe over Fibre Channel User Guide NQN The qualified name of the NVMe subsystem. NSID The namespace identifier to be deleted, in hexadecimal. Set to all to delete all namespaces in the subsystem. ctrlid The controller that processes the request, in hexadecimal (optional). Support Requirements Mandatory if the target supports namespace management. Not supported if namespace management is not supported.
Emulex NVMe over Fibre Channel User Guide Syntax hbacmd nvme-detach-ns Parameters WWPN The WWPN of the FC initiator port. Target WWPN The WWPN of an FC port on an NVMe-capable device. NQN The qualified name of the NVMe subsystem. NSID The namespace identifier to be detached, in hexadecimal. ctrlid1 ... ctrlidn One or more controller identifiers from which the namespace is to be detached, in hexadecimal.
Emulex NVMe over Fibre Channel User Guide Appendix A: Configuring NVMe over FC on a Target (SLES 12 SP3 Only) NOTE: Before configuring NVMe over FC using native NVMe CLI commands, ensure that you have installed the latest Emulex firmware for the LPe31000-series, LPe32000-series, and LPe35000-series adapters. NOTE: At this time, NVME over FC on target devices is supported on SLES 12 SP3 only. A.1 Configuring the lpfc.conf File To create and configure the lpfc.conf file, perform the following steps: 1.
Emulex NVMe over Fibre Channel User Guide └─3600000e00d2a0000002a017300350000 sdd └─3600000e00d2a0000002a017300360000 sde └─3600000e00d2a0000002a017300370000 sdf └─3600000e00d2a0000002a017300380000 254:5 0 3G 0 mpath 8:48 0 3G 0 disk 254:6 0 3G 0 mpath 8:64 0 3G 0 disk 254:7 0 3G 0 mpath 8:80 0 3G 0 disk 254:8 0 3G 0 mpath Record the block devices you want to use, for example /dev/sde and /dev/sdf. NOTE: Block devices are located in the /dev directory. 4.
Emulex NVMe over Fibre Channel User Guide 5. Navigate to the /subsystems/fctest1/namespaces/1 subdirectory, and set the device path to the backing storage device you recorded previously in Step 3 in Section A.1, Configuring the lpfc.conf File, by typing the following command: set device path=/dev/ For example: set device path=/dev/sde Information similar to the following is displayed: Parameter path is now ‘/dev/sde’.
Emulex NVMe over Fibre Channel User Guide For example: create fctest1 12. Save the .json file by typing the following command: saveconfig .json NOTE: If you want the configuration to be restored automatically after rebooting, this file must be named config.json, and it must be saved under the path /etc/nvmet/. Otherwise, can be any valid file name. 13.
Emulex NVMe over Fibre Channel User Guide 4. Reconfigure the target by following the instructions in Step 1 through Step 13 in this section. 5. Enable the initiator links. You can set the saved configuration to be restored automatically after a reboot by using systemctl to enable the nvmet.service. Type the following command: systemctl enable nvmet.service NOTE: The service requires that the .json file must be named config.
Emulex NVMe over Fibre Channel User Guide Appendix B: Multipathing SLES 15 includes support for native NVMe multipathing, This support changes the manner in which NVMe devices are presented, especially in cases where multiple paths to target namespaces exist. The additional detection and management of multipath NVME configurations has changed the manner in which NVMe devices are presented and related in /sys/class/nvme.
Emulex NVMe over Fibre Channel User Guide NOTE: This document covers are two similar commands: nvme list (with a space) is a native NVMe CLI command. nvme-list (with a hyphen) is an Emulex OneCommand Manager CLI command. This chapter refers to the nvme list command.
Emulex NVMe over Fibre Channel User Guide In the above example, five subsystems are displayed. Each subsystem contains multiple controllers representing multiple paths. Note that the nvme-subsys numbering is disassociated from the component controller identification, as demonstrated by nvme-subsys3, and controllers nvme5 and nvme9. In the preceding example configuration, each of the five subsystems contain four namespaces. The nvme list-subsys view shows two controllers.
Emulex NVMe over Fibre Channel User Guide 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 brw-rw---brw-rw---brw-rw---crw------brw-rw---brw-rw---brw-rw---brw-rw---crw------brw-rw---brw-rw---brw-rw---brw-rw---crw------brw-rw---brw-rw---brw-rw---brw-rw---crw------crw------crw------crw------crw------- 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 root root root root root root root root root root root root root root root root root root root root root root root disk disk disk root disk disk disk disk root disk
Emulex NVMe over Fibre Channel User Guide lrwxrwxrwx 1 root subsys0/nvme0n4 lrwxrwxrwx 1 root nvme1/nvme1c223n1 lrwxrwxrwx 1 root nvme1/nvme1c223n2 lrwxrwxrwx 1 root nvme1/nvme1c223n3 lrwxrwxrwx 1 root nvme1/nvme1c223n4 lrwxrwxrwx 1 root nvme6/nvme1c229n1 lrwxrwxrwx 1 root nvme6/nvme1c229n2 lrwxrwxrwx 1 root nvme6/nvme1c229n3 lrwxrwxrwx 1 root nvme6/nvme1c229n4 lrwxrwxrwx 1 root subsys1/nvme1n1 lrwxrwxrwx 1 root subsys1/nvme1n2 lrwxrwxrwx 1 root subsys1/nvme1n3 lrwxrwxrwx 1 root subsys1/nvme1n4 lrwxrwxrwx
Emulex NVMe over Fibre Channel User Guide lrwxrwxrwx 1 root nvme9/nvme3c232n1 lrwxrwxrwx 1 root nvme9/nvme3c232n2 lrwxrwxrwx 1 root nvme9/nvme3c232n3 lrwxrwxrwx 1 root nvme9/nvme3c232n4 lrwxrwxrwx 1 root subsys3/nvme3n1 lrwxrwxrwx 1 root subsys3/nvme3n2 lrwxrwxrwx 1 root subsys3/nvme3n3 lrwxrwxrwx 1 root subsys3/nvme3n4 lrwxrwxrwx 1 root nvme4/nvme4c226n1 lrwxrwxrwx 1 root nvme4/nvme4c226n2 lrwxrwxrwx 1 root nvme4/nvme4c226n3 lrwxrwxrwx 1 root nvme4/nvme4c226n4 lrwxrwxrwx 1 root nvme8/nvme4c231n1 lrwxrwxrw
