Dell EMC PMem 200 Series User's Guide June 2021 Rev.
Notes, cautions, and warnings NOTE: A NOTE indicates important information that helps you make better use of your product. CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem. WARNING: A WARNING indicates a potential for property damage, personal injury, or death. © 2021 Dell Inc. or its subsidiaries. All rights reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries.
Contents Chapter 1: Introduction................................................................................................................. 5 System requirements..........................................................................................................................................................5 Terminology...........................................................................................................................................................................
Chapter 9: Windows.................................................................................................................... 34 PMem in App-direct mode.............................................................................................................................................. 34 PMem Disk management...........................................................................................................................................
1 Introduction Dell EMC offers Intel Optane persistent memory (PMem) 200 series, a nonvolatile memory that has the same form factor as a standard DDR4 DIMM. PMems come in three capacities: 128 GB, 256 GB, and 512 GB. PMems are installed in the memory slots of the server and are compatible with RDIMMs and LRDIMMs. PMem can be configured in two modes: ● Memory mode ● App-direct mode In Memory mode, PMems act as volatile system memory and any RDIMMs or LRDIMMs installed operate as cache for the PMems.
Table 1. System requirements (continued) Component Minimum version required OS Microsoft Windows Server 2019 VMware ESXi 7.0 Update 2 Red Hat Enterprise Linux 8.2 SUSE Linux Enterprise Server 15 SP2 Terminology Table 2. Terminology Terminology Description App-direct mode (AD) Persistent memory is accessed directly by applications as byte-addressable memory.
2 Change list Table 3.
3 Hardware Topics: • • • Server hardware configuration DIMM installation and removal PMem hardware configuration Server hardware configuration PMem is supported in R650, R750, R750xa, MX750c, XR11 and XR12 PowerEdge servers with third-generation Intel Xeon Scalable Processors of Gold and Platinum grade. See PMem configurations for a list of fully supported and validated PMem configurations in two-socket servers. Four-socket configurations are a direct scale up of two-socket configurations.
Figure 1. Memory layout for the PowerEdge R750 Table 4.
Table 4.
Table 6.
Table 8.
Table 10.
● ● ● ● ● ● ● ● ● ● 14 If only one DIMM is populated on a channel, it should always go to the first slot in that channel (white slot). If a PMem and a DDR4 DIMM are populated on the same channel, always install PMem on the second slot (black slot). PMem cannot be mixed with other PMem capacities. Mixing different capacities of RDIMMs and LRDIMMs are not allowed when PMem is installed. PMems of different capacities are not allowed.
4 BIOS Topics: • • • BIOS configuration setting for Intel PMem App-direct mode configuration Memory mode configuration BIOS configuration setting for Intel PMem DIMM discovery All installed PMems that the BIOS has discovered during system inventory is displayed in the BIOS Intel Persistent Memory tab: Memory Settings > Persistent Memory > Intel Persistent Memory > Persistent Memory DIMM Configuration. Figure 2.
NOTE: PMems are shown as DIMMs. There is one entry for each PMem that is installed and the current health and status information for each PMem is displayed as: Figure 3. Memory information NOTE: Data is always assumed to be in units of MiB/GiB/TiB even if labeled MB/GB/TB. User capacity overhead is up to 2% of capacity (GiB). Another overhead may be required for Regions, Namespace, and Filesystems.
Figure 4. Goal configuration Persistent [%]: ● No Change - Does not apply any changes to the current goal. ● 100 - Creates a goal of 100% Persistent memory across the selected PMems. ● 0 - Creates a goal of 0% Persistent memory across the selected PMems. This operation configures all the PMem as Memory mode. Persistent memory type: ● App-direct Interleaved - Persistent mode interleave across the PMems in a socket. The PMems are displayed one PMem device per socket in the operating system.
Figure 5. Region configuration The number of regions that are displayed depends on the number of processors in the system and not on the PMems interleaved. If the PMems are configured as interleaved, one Persistent Memory Region is listed per socket in the system that has PMems installed. If the PMems are configured as non-interleaved, one Persistent Memory Region is listed per PMem installed in the system. Region information can be accessed by clicking each Persistent Memory Region link in the BIOS.
Figure 6. Region info Memory mode configuration Create goal Goal is created in the BIOS. To create a goal in BIOS, go to: Memory Settings > Persistent Memory > Intel Persistent Memory > Region Configuration > Create Goal Config.
Figure 7. Create Goal Config Persistent [%]: ● No Change - Does not apply any changes to the current goal. ● 100 - Creates a goal of 100% Persistent memory across the selected PMems. ● 0 - Creates a goal of 0% Persistent memory across the selected PMems. This operation configures all the PMem as Memory mode.
5 PMem event reporting When system detects PMem-related event either during runtime or POST, system will log the events in Server System Event Log(SEL) and Life-Cycle Log(LCL). If an event is detected during boot time, system halts during the POST and user needs to press F1 to continue the boot process. NOTE: NVDIMM is frequently used in these messages. The term NVDIMM is generic to several different families of persistent memory including PMem and not meant to indicate NVDIMM-N modules.
Table 12. Message ID and Event Message (continued) Message ID Event Message Severity MEM5103 A diagnostic event occurred in the memory device at . Perform a warm reboot. (Debug Code: ). Info MEM5104 A diagnostic event occurred in the memory device at . Perform a cold reboot. (Debug Code: ). Info MEM5107 A diagnostic event occurred in the memory device at . Retry operation. (Debug Code: ).
Events during boot time ● UEFI0337 : Unable to update the firmware of NVDIMM located in the memory slot . Recommended Action : Disconnect the input power to the system, wait for 30 seconds, and reconnect the power, Power on the server and retry the operation. If the issue persists, replace the NVDIMM. For more information, see the product's owner's manual on the support site. ● UEFI0338 : The firmware of NVDIMM located in the memory slot is successfully updated.
Recommended Action : Update the Persistent Memory Passphrase to the correct passphrase or perform a Secure Erase operation on the Dual inline memory module (DIMM). Secure erase, erases all persistent data. ● UEFI0357 : The Cryptographic Erase operation on the Intel Persistent Memory DIMM with serial number in slot is successfully completed.
● UEFI0378 : The Non-Volatile Dual In-Line memory module (NVDIMM) in the memory slot has encountered a thermal shutdown event during the previous boot. Recommended Action : Do one of the following: 1) Power off the server. 2) Disconnect the input power, wait for 30 seconds, and then reconnect to the power source. 3) Power on the server. 4) If the issue persists, contact your service provider.
6 iDRAC Intel PMem management Topics: • iDRAC GUI iDRAC GUI PMem firmware version The PMem firmware version is displayed under Configuration > System BIOS Settings > Memory Settings > Persistent Memory > Intel Persistent Memory > Intel Persistent Memory DIMM Configuration > DIMM in Slot X. Figure 8. PMem firmware version PMem hardware status Select the Memory link on the Dashboard to get more information about memory health.
Figure 9. PMem hardware status PMem goal configuration using iDRAC GUI 1. Log on to iDRAC GUI interface. 2. Navigate to Configuration > BIOS settings > Memory settings > Persistent Memory Settings tab. Figure 10. PMem goal configuration using iDRAC GUI 3. Navigate to Intel Persistent Memory > Region Configuration > Create Goal Config.
Figure 11. PMem goal configuration 4. Change Persistent percentage as 100% to configure Intel PMem into 100% App-direct Mode and 0% to configure the DIMMs into 100% memory mode. NOTE: The Persistent memory type field is for configuring a new goal. It is not for reading the status of the current goal. Use instructions under PMem Hardware Status to confirm current configuration. 5. Click Apply and reset the system. 6.
7 PMem security Topics: • • • Memory mode App-direct Cryptographic erase and PMem sanitize Memory mode In Memory mode PMems operate as volatile system memory. User passphrase is not supported and this BIOS setting will be greyed out. App-direct Users have the option to enable Passphrase protection of PMem regions. The intent of the passphrase is to protect against unauthorized access to data stored on the PMem region.
Figure 13.
● Sanitize Both erase methods can be executed using BIOS setup options. User can choose to perform an erase on all or a subset of installed PMems. Crypto erase The Crypto Erase function erases the App-direct Region Key (PM-RK) forcing the system to reboot. Cryptographic erase option can be accessed by going to: System BIOS Settings > Persistent Memory > Intel Persistent Memory > DIMM Configuration Figure 14.
Figure 15. Sanitize Sanitize can take up to 15 minutes with fully loaded 128 GB DIMM configuration, 30 minutes with 256 GB and 1 hour with 512 GB. NOTE: When the Sanitize operation is running, a prompt appears in BIOS indicating an Overwrite. Overwrite is the name for the second firmware command that is conducted. The first command which happens quickly and will not be displayed on-screen is the Crypto Erase (firmware command name is "Secure Erase").
8 DIMM configuration changes The following PMem migration scenarios are supported: ● Replacement of System Board due to failure All DIMMs must be re-populated in the exact same slots. PMems and data content will be available for customer application access after the board has been restored to the same configuration as the original board. System Restore will automatically restore the BIOS configuration on the replacement board, including the PMem Passphrase, if it is set.
9 Windows Dell EMC supports Intel Optane PMem with Microsoft Windows 2019 in Memory mode and App-direct mode. NOTE: Keep Windows updated with the monthly cumulative updates.
Figure 17. Memory Devices in Device Manager PMem Disk management Windows currently supports only one namespace per interleave set (this is independent of the number of physical devices in the interleave set). The option to interleave PMems can be selected during goal creation as described in App Direct and Memory mode configurations. PMem disks have to be created with the help of "New-Pmemdisk" command by providing relevant region IDs.
○ Requires user confirmation, which can be overridden with Force. ○ This cmdlet is intended as a “big hammer” recovery mechanism. It is not recommended for normal use. List PMem physical disks and check their health status The following image displays command usage to list all usage to list all PMem physical devices and get their health. Physical location gives the location of the DIMM on the motherboard. Figure 18.
Remove PMem disks Figure 20. Remove PMem disks Windows 2019 does not support redundant volume creation on PMem disks by using Windows VDS (Virtual disk service). In order to create redundant volumes, use the storage spaces method. For information about storage spaces method refer to: https://docs.microsoft.com/en-us/windows-server/storage/storagespaces/deploy-standalone-storage-spaces.
Figure 22. PMem Disk creation using Region id PMem in memory Mode When Intel Optane PMem is configured in memory mode, operating system sees it as system memory. Persistent memory size is shown as zero and volatile memory size accounts for the entire size of the PMem. Figure 23. PMem in memory Mode Windows troubleshooting and event monitoring If any of the PMem physical devices or logical devices are not functioning properly, it is suggested to check Windows event logs.
Figure 24. Windows troubleshooting and event monitoring Windows Errata The following errata affects Windows OSes and are expected to be fixed in a future OS patch. ● When a namespace is created in Windows Server 2019 (WS2019), the logged message appears as "The driver for persistent memory disk encounters internal error". The error is expected and it may occur during testing of Storage Class Memory (SCM) device. This means the namespace is created, and is considered as WAD for WS2019. Workaround: None.
10 Linux Topics: • • • Identify and configure persistent memory device Management utility Linux errata Identify and configure persistent memory device Listing PMem devices To list all physical devices in the system, run command ndctl list -DHi Create namespace The configuration of namespaces will decide how much memory capacity user wants to expose to the OS.
To write data into the device, run the following command: cd /mnt/pmemX echo “Hello World” >>test.txt Reset the system and the data should be persistent over power cycle. Delete namespaces Namespace can be deleted using NDCTL command: ndctl destroy-namespace is namespaceX.
Workaround: Add "x-systemd.device-timeout=0" to the mount options for the PMem partition in fstab. Example: /dev/pmem5 /mnt/somedir ext4 defaults, x-systemd.device-timeout=0,dax 0 2 3. Dell BIOS does not support boot from PMem. Some Linux operating system (Red Hat Enterprise Linux 7.6, Red Hat Enterprise Linux 8.0) distros are offering early support of this feature, but Dell BIOS does not support this feature. Workaround: None.
11 VMware ESXi Dell EMC supports Intel Optane PMem from VMWare ESXi 7.0 Update 2 and above. The support is available for Intel PMem Memory Mode and App Direct modes. The persistent memory inventory details are available as part of Host client. See Managing Persistent Memory in the vSphere Single Host Management - VMware Host Client which detail the persistent memory data that are presented to the user.
Figure 27. Namespaces created on ESXi out of the interleave sets exposed After creating the Namespaces, ESXi automatically creates a PMem datastore and mounts it as a datastore for the users to consume it. Figure 28. PMem Datastore PMem in Memory Mode When Intel PMem is configured in Memory Mode, ESXi sees it as system memory. NOTE: The inventory details in the vSphere HTML client will not be available if Intel PMem is set to Memory Mode.
ESXi troubleshooting and maintenance NOTE: Go through/var/log/vmkernel.log to see if there are any errors or warnings reported related to persistent memory. The log entries related to persistent memory would be appended with NVD and or IntelNVDimm. ● Configuring Intel PMem in memory mode shows a reduction of system memory in system BIOS. ○ For example, a system configuration with 4 * 128 GB Intel PMems, configured in memory mode provides a system memory of 506.8 GB instead of 512 GB.
12 System diagnostics System diagnostics in the Lifecycle Controller does not test Intel PMem in App Direct mode to avoid destroying customer data. NOTE: System diagnostics are not recommended to investigate Intel PMem failures when in Memory mode due to extremely long test time.
13 Firmware update Topics: • Dell DUP update Dell DUP update Download PMem DUP from www.dell.com/support/drivers. NOTE: Make sure BIOS is up to date with the latest version before applying PMem DUP. 1. Boot to the operating system and execute DUP package. a. On a Windows system, double click the .exe DUP file. Reboot the system and DUP will be automatically executed. b. On Linux system: i. Navigate to the folder that contains DUP bin file ii.
