Technical white paper Best Practices When Deploying Linux on the HP ProLiant DL980 Table of contents Introduction 3 Linux hardware overview CPU mappings PCI slot mappings 4 4 4 Hardware best practices Memory configuration Boot adapter location Power consumption settings – max performance Power consumption settings – OS control 5 5 5 Non-Uniform Memory Access (NUMA) overview Displaying node distances Displaying task information Displaying memory allocation statistics Controlling tasks and memory objec
Test Red Hat Enterprise Linux 5.5 (or later) and SUSE Linux Enterprise Server 10 SP3/SP4 workloads with HyperThreading (HT) Disable x2APIC with Intel Xeon E-7 series processors Verify 40-bit mode before OS installation Use NOOP in the I/O scheduler Increase crash kernel size Known issues and workarounds Red Hat Enterprise Linux 6.X and SUSE Linux Enterprise Server 11 SPX best practices Test Red Hat Enterprise Linux 6.
Introduction The purpose of this white paper is to facilitate the successful deployment and configuration of Linux operating systems on HP ProLiant DL980 servers. It includes a description of additional recommended HP software components for the DL980, and best practices you can use when deploying Linux on the DL980.
Note The following pages contain numerous examples. The text displayed in the examples may differ from what you see in your specific environment, depending on the configuration of your system and the OS version used. Linux hardware overview CPU mappings Linux uses the /proc/cpuinfo file to enumerate logical processors. As an example, Table 1 shows the logical processor assignments for the first two processors of a 64-core DL980 (8 processors with 8 cores per processor) with Hyper-Threading enabled.
0b:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02) 0b:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02) Hardware best practices This section describes hardware recommendations for optimal performance when running Linux on the HP ProLiant DL980 server. Memory configuration Dual and quad rank DIMMs should be used in high performance implementations.
Figure 2: Enabling Maximum Performance Mode using RBSU Power consumption settings – OS control To take advantage of the Linux operating system’s ability to adjust server power consumption (and reduce costs) based on workload, follow these steps: 1. 2. 3. 4. 5. 6. During system boot, press F9 to run the ROM-Based Setup Utility (RBSU). Select Power Management Options. Select HP Power Regulator. Select OS Control Mode (see Figure 3). Exit all RBSU menu screens using the Esc key.
Figure 3: Enabling OS Control Mode using RBSU Non-Uniform Memory Access (NUMA) overview The latest Intel® Xeon® processors provide Socket Local Memory, which consists of embedded memory controllers that access memory DIMMS connected to the socket. A processor and the associated Socket Local Memory are considered a node. Each node can be thought of as a separate Symmetric Multi-Processing (SMP) system where accesses to the Socket Local Memory have the same latency.
Figure 4: DL980 server architecture IOH CPU 0 Slots 7-11 CPU 1 QPI Island Optional Slots 1-6 IOH CPU 2 IOH CPU 3 CPU 4 CPU 5 CPU 6 Optional LP Slots 12-16 CPU 7 QPI Links XNC XNC XNC XNC Upper processor board Socket Local Memory Lower processor board CPU 0 Processor IOH I/O Hub (Boxboro) XNC Node Controller Displaying node distances The distance between the local and remote node affects memory access latency. The greater the distance, the higher are the memory access latencies.
The following illustrates the relative distances between the nodes shown in Figure 4: Intra nodes Shortest distance (such as node 0 to node 0) Nodes sharing the same QPI island (such as nodes 0 and 1) Nodes on the same side of a node controller (XNC) (such as between nodes 0–1 and 2–3) All other nodes on the other side of the XNC (such as between nodes 0–1 and 4–7) Longest distance Displaying task information You can also use numactl command to display the memory allocation policy and pr
Where: numa_hit is the number of successful allocations for a node. numa_miss is the number of failed allocations for a node because of low memory (allocations are successful on another node). numa_foreign is the number of allocations that were intended for another node but ended up on the local node. local_node is incremented when a process running on the local node has allocated memory on the same node.
[root@tux ~]# echo 1 > mems [root@tud ~]# echo $$ > tasks For more information about the cpuset command, see the cpuset(7) manpage. Red Hat Enterprise Linux 6 and SUSE Linux Enterprise Server 11 introduced Linux Control Groups or cgroups, which use the cpuset subsystem. For more information, see the Red Hat Enterprise Linux 6 Resource Management Guide: access.redhat.com/knowledge/docs/enUS/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/index.
Table 2: HP ProLiant DL980 selected Linux OS support matrix Max CPUs Max Memory (RAM) HP-supported/ OS limit HP-supported/OS limit Red Hat Enterprise Linux 5.5 (or later) x86-64 160 / 255 1 / 1 TB No x2APIC support Red Hat Enterprise Linux 6.X (6.
BladeSystem SUSE Linux Enterprise Server Pack BladeSystem Microsoft Windows Server Pack ProLiant ML/DL/SL Red Hat Enterprise Linux Pack ProLiant ML/DL/SL SUSE Linux Enterprise Server Pack ProLiant ML/DL/SL Microsoft Windows Server Pack HP SPP benefits HP SPP simplifies software maintenance in several ways: Leverages the power of HP SUM for consolidated management and distribution to multiple servers.
the zip file containing the ISO for the target Linux distribution, and mount the ISO as a loop back device for access. The HP SUM utility used for installation is located in /hp/swpackages/. If a graphics terminal or a remote desktop program (like VNC) is available, HP SUM can be launched in graphic mode by running ./hpsum from this directory. Otherwise, the utility can be launched in console mode by running ./hpsum /s .
Figure 5 shows an example of the HP SMH interface. Figure 5: System Management Homepage (SMH) HP SMH features HP SMH provides streamlined operations for HP servers running Linux operating systems with the following security enhancements and features: Browser access using secure OS-based, Secure Sockets Layer (SSL) authentication Common HTTP and HTTPS service for HP Insight Management Agents and utilities, for reduced complexity and system resource requirements.
Disable x2APIC with Intel Xeon E-7 series processors (page 17) Verify 40-bit mode before OS installation (page 18) Use NOOP in the I/O scheduler (page 20) Increase crash kernel size (page 20) Known issues and workarounds (page 20) Test Red Hat Enterprise Linux 5.5 (or later) and SUSE Linux Enterprise Server 10 SP3/SP4 workloads with Hyper-Threading (HT) Hyper-Threading (HT) refers to Intel’s proprietary technology for increasing parallel computational power (processor multi-tasking).
Figure 6: Disabling Hyper-Threading (HT) using RBSU For more information about HP’s RBSU, see the HP ROM-Based Setup Utility User Guide: www.hp.com/support/rbsu Disable x2APIC with Intel Xeon E-7 series processors As noted in the “OS support matrix for the HP ProLiant DL980 server” section on page 11, Red Hat Enterprise Linux 5.X and SUSE Linux Enterprise Server 10 do not support the Intel x2 Advanced Programmable Interrupt Controller (x2APIC) architecture.
Figure 7: Disabling x2APIC using RBSU Verify 40-bit mode before OS installation The default RBSU method for addressing memory on the DL980 is 40-bit mode. However, this should always be verified before installing Red Hat Enterprise Linux 5.X or SUSE Linux Enterprise Server 10, because they do not support 44-bit address mode. During the boot process, confirm that the system is in the default 40-bit mode by looking at the RBSU splash screen as shown in Figure 8.
Figure 8: Confirming System Address Mode reported by RBSU If the System Address Mode is set to 44-bit, follow these steps to reset it to 40-bit mode: 1. 2. 3. 4. 5. 6. 7. During system boot, press F9 to run the ROM-Based Setup Utility (RBSU). Select Advanced Options. Select Advanced System ROM Options. Select Address Mode 44-bit (see Figure 9). Select Disabled to turn 44- bit memory addressing OFF. Exit all RBSU menu screens using the Esc key.
Figure 9: Disabling 44-bit System Address Mode using RBSU Use NOOP in the I/O scheduler The default I/O scheduler is Completely Fair Queuing (CFQ), which attempts to balance throughput and latency. However, today’s intelligent storage subsystems usually perform best by using the NOOP I/O scheduler, which disables all I/O scheduling operations in the operating system.
This issue has been fixed in SUSE Linux Enterprise Server 10 SP4. The BCM5720 network controller (NC332T) does not work on SUSE Linux Enterprise Server 10 SPR4 The tg3 driver on SUSE Linux Enterprise Server 10 SP4 does not recognize the BCM5720 Network interface controller (NC332T). The workaround is to download the driver provided by HP from the following location: h20000.www2.hp.com/bizsupport/TechSupport/DriverDownload.
3. 4. Select Advanced Options. Select Advanced System ROM Options. Figure 10: Confirming System Address Mode reported by RBSU 5. 22 Select Address Mode 44-bit (see Figure 11).
Figure 11: Enabling 44-bit System Address Mode using RBSU 6. Ignore the Warning message: Caution: Enabling 44-bit Address Mode may cause problems with an operating system if it does not support 44-bit memory addressing. 7. 8. 9. Select Enabled to turn 44- bit memory addressing ON. Exit all RBSU menu screens using the Esc key. Press the F10 key to confirm your changes and reboot the system. 10.
Test Hyper-Threading Online Transaction Processing (OLTP) type Oracle database workloads typically perform better with Hyper-Threading (HT) enabled. I/O-bound workloads have spare processing power and are not likely benefited by HT. HT is controlled by the system BIOS and is enabled by default on the DL980. HP recommends that you test your workload with HT to make sure HT actually helps overall performance.
The Linux kernel shared memory parameters should also be adjusted for the largest SGA: [root@tux ~]# grep shm /etc/sysctl.conf kernel.shmall = kernel.shmmax = More details on HugePages and Oracle can be found in the following Oracle articles (an Oracle account is required): Article ID 361468.1: support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=REFERENCE&id=361468.1 Article ID 361323.1: support.oracle.
For more information HP ProLiant DL980 Server overview: hp.com/servers/dl980 HP ProLiant DL980 Server with HP PREMA Architecture: h20195.www2.hp.com/V2/GetPDF.aspx/4AA3-0643ENW.pdf Download the recommended components described in this document, along with other drivers and software from hp.com/support/dl980g7 HP ProLiant DL980 G7 Server User Guide: h20000.www2.hp.com/bc/docs/support/SupportManual/c02510094/c02510094.
