Technical white paper NVMe Hot-Plug on Dell EMC PowerEdge servers running VMware vSphere or vSAN Abstract This white paper describes the support for Non-Volatile Memory Express (NVMe) Hot-Plug on Dell EMC PowerEdge servers running VMware vSphere or vSAN.
Revisions Revisions Date Description June 2020 Initial release Acknowledgements Authors: Syama Poluri, Himanshu Shrivastava, Thiru Navukkarasu, Krishnaprasad K Support: Austin Bolen, Gurupreet Kaushik, Sherry Keller, IDD team The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Table of contents Table of contents Revisions.............................................................................................................................................................................2 Acknowledgements .............................................................................................................................................................2 Table of contents .................................................................................................
Executive summary Executive summary This white paper describes the use cases for NVMe devices on Dell EMC PowerEdge servers running VMware vSphere or VMware vSAN and supported hot-plug operations, the best practices for performing hotplug operations on NVMe devices through each use case.
Introduction 1 Introduction Non-Volatile Memory Express (NVMe) is an architecture that is optimized for solid-state devices (SSD) connected directly to a PCIe bus. NVMe devices consist of a PCIe host interface, SSD controller and a nonvolatile memory. Unlike SAS or SATA interfaces, NVMe does not make use of host bus adapters (HBA). 1.
Use cases for NVMe and hot-plug support on VMware ESXi 2 Use cases for NVMe and hot-plug support on VMware ESXi Listed below are the use cases for NVMe in VMware ESXi environment. • Virtual Machine File System (VMFS) datastore: Virtual Machine File System is used to store virtual machine disk files and associated files. Dell EMC and VMware support the usage of NVMe devices for hosting virtual machines with the creation of VMFS. For more information about VMFS, see Understanding VMFS Datastores.
Use cases for NVMe and hot-plug support on VMware ESXi 2.1 Support for NVMe hot-plug operations The following table describes the respective version of VMware ESXi and Dell EMC Generations of PowerEdge servers that support hot-plug operations on NVMe devices. Supported NVMe hot-plug operations Dell EMC PowerEdge generation Dell EMC customized VMware ESXi 6.x Dell EMC customized VMware ESXi 7.0 GA (A00 Build# 15843807) * Dell EMC customized VMware ESXi 7.
Support for NVMe hot-plug on VMware vSphere 6.x and VMware vSAN 6.x 3 Support for NVMe hot-plug on VMware vSphere 6.x and VMware vSAN 6.x VMware introduced support for surprise removal on NVMe devices from VMware ESXi 7.0 onwards. Surprise removal of NVMe is not supported in the previous versions of VMware ESXi when the NVMe device is in use. For more infomration, see VMware Knowledge Base article 2151404. Surprise removal of NVMe devices when they are in use is not supported by Dell EMC and VMware.
Support for NVMe hot-plug on VMware vSphere 6.x and VMware vSAN 6.x 7. Remove the NVMe device physically. Check VMFS Datastore is not connected to the ESXi 8. Make sure the VMFS Datastore is not connected to the ESXi using the command: esxcli storage filesystem list The NVMe device has been safely removed from VMware ESXi and the server. It is now safe to replace the NVMe device with a different device or hot insert the same NVMe device if required.
Support for NVMe hot-plug on VMware vSphere 6.x and VMware vSAN 6.x 3.3 Perform orderly removal of an NVMe device on a VSAN cluster VMware vSAN is a software-based storage solution for VMware hyperconverged solutions. Embedded in the hypervisor and leveraging industry-standard servers, vSAN delivers flash-optimized, high-performance computation and storage for hyperconverged infrastructures. vSAN can make use of NVMe devices in both cache and capacity tier.
Support for NVMe hot-plug on VMware vSphere 6.x and VMware vSAN 6.x Unmount the datastore 4. Perform hot removal of the NVMe device and check if the hot-plug slot is disabled. Use the lspci command to check if the NVMe device exists in ESXi. If there are no references to the specific NVMe device, then the NVMe device has been successfully removed. 3.
LED management of NVMe devices in VMware ESXi 4 LED management of NVMe devices in VMware ESXi Starting from ESXi 7.0 patch build# 16324942, Dell EMC and VMware support LED management of NVMe devices in VMware ESXi. Dell EMC and VMware provide an esxcli namespace to monitor the LED indications of NVMe devices. For more information, see Dell Knowledge Base article SLN321848.
Summary 5 Summary This white paper is an exclusive documentation on NVMe hot-plug support for Dell EMC Generations of PowerEdge servers in VMware vSphere and vSAN environments and describes use cases of NVMe device(s). This document will be updated progressively when there is more information on the support stances, known issues and best practices for hot plug insertion or hot plug removal techniques.
References 6 References • • • • • • 14 NVMe Hot-Plug Challenges and Industry Adoption Dell Express Flash NVMe PCIe SSD User’s Guide vSphere Support for hot-swap of local disks vSphere and vSAN support for Hot-plug of NVMe SSDs on AMD EPYC Processors Supported scenarios of NVME hot plug or removal in ESXi 7.0 Dell EMC VMware ESXi 7.
