Specifications

ManualsBrandsCompaq ManualsComputer equipmentProLiant DL980 - G7 Server

Best Practices When Deploying

Linux on HP ProLiant DL980 G7 Servers

Technical white paper

Table of contents

Introduction .................................................................................................................................. 2

Linux Hardware Overview ............................................................................................................. 3

CPU Mappings ......................................................................................................................... 3

PCI Slot Mappings .................................................................................................................... 3

Non-Uniform Memory Access (NUMA) Overview ............................................................................. 4

Displaying Node Distances ........................................................................................................ 4

Displaying Task Information ....................................................................................................... 5

Displaying Memory Allocation Statistics ....................................................................................... 5

Controlling Task and Memory Placement ..................................................................................... 6

Moving Tasks to Other NUMA Nodes ......................................................................................... 6

Controlling Task and Memory Object Placement ........................................................................... 6

Determining Task Memory Allocation Location .............................................................................. 7

OS Support Matrix for the HP ProLiant DL980 G7 Server .................................................................. 7

Additional Recommended Software Components When Running Linux on the DL980 ........................... 8

HP ProLiant Support Pack (PSP) ................................................................................................... 8

HP System Management Homepage (SMH) ............................................................................... 11

Hardware Best Practices .............................................................................................................. 12

Boot Adapter Location ............................................................................................................. 12

Power Consumption Settings .................................................................................................... 12

Virtualization Settings .............................................................................................................. 14

RHEL 5.6 and SLES 10 SP4 Best Practices ..................................................................................... 15

Disable Hyper-threading to Reduce Scheduler Overhead ............................................................. 15

Disable x2APIC with Intel Xeon E-7 Series Processors .................................................................. 16

Verify 40-bit Mode before OS Installation .................................................................................. 17

Use the NOOP I/O Scheduler .................................................................................................. 19

Increase Crash Kernel Size ...................................................................................................... 19

Known Issues and Workarounds ............................................................................................... 19

RHEL 6.X and SLES 11 SP1 Best Practices ...................................................................................... 20

Disable Hyper-threading to Reduce Scheduler Overhead ............................................................. 20

Enable 44-bit System Address Mode with 1 TB or more Memory .................................................. 21

Use the NOOP I/O Scheduler .................................................................................................. 23

Increase Crash Kernel Size ...................................................................................................... 23

Oracle Database Best Practices .................................................................................................... 24

Enable Hyper-threading for CPU-intensive Workloads.................................................................. 24

Use HugePages ...................................................................................................................... 24

Enable Oracle NUMA Optimizations ........................................................................................ 25

For More Information .................................................................................................................. 26

Documentation Feedback ......................................................................................................... 26

Summary of content (26 pages)

PAGE 1
Best Practices When Deploying Linux on HP ProLiant DL980 G7 Servers Technical white paper Table of contents Introduction ..................................................................................................................................2 Linux Hardware Overview .............................................................................................................3 CPU Mappings .............................................................................................................
PAGE 2
Introduction This purpose of this white paper is to facilitate the successful deployment and configuration of Linux operating systems on HP ProLiant DL980 G7 servers. It includes a description of additional recommended HP software components for the DL980, and best practices you can use when deploying Linux on the DL980.
PAGE 3
For a detailed discussion of the HP PREMA architecture, see the technical overview at: http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA3-0643ENW.
PAGE 4
Non-Uniform Memory Access (NUMA) Overview The latest Intel Xeon processors include embedded memory controllers which access memory DIMMS connected to the socket, also called Socket Local Memory. The processor cores (and threads, if Hyper-threading is enabled) plus the attached memory make up a NUMA node in a Linux system. Each node can be thought of as a separate SMP (Symmetric Multi-Processing) system where all accesses to the Socket Local Memory have the same latency, or Uniform Memory Access (UMA).
PAGE 5
node 7 size: 64640 MB node 7 free: 40784 MB node distances: node 0: 1: 2: 3: 4: 5: 6: 7: 0 10 14 23 23 27 27 27 27 1 14 10 23 23 27 27 27 27 2 23 23 10 14 27 27 27 27 3 23 23 14 10 27 27 27 4 27 27 27 27 10 14 23 27 5 27 27 27 27 14 10 23 23 6 27 27 27 27 23 23 10 23 7 27 27 27 27 23 23 14 14 10 Local accesses, such as node 0 to node 0, have the shortest distance; then nodes sharing the same QPI island, such as nodes 0 and 1; then nodes on the same side of the node controller (XNC), such a
PAGE 6
• numa_foreign — a task wanted to allocate on another node but ended up with memory from this one. • local_node — a task ran on this node and got memory from it. • • other_node — a task ran on this node and got memory from another node interleave_hit — interleaving wanted to allocate from this node and succeeded Controlling Task and Memory Placement Best performance is obtained when tasks and their associated memory objects are close together, preferably in the same node.
PAGE 7
Determining Task Memory Allocation Location Examine the /proc//numa_maps file to determine a task’s memory allocation location. This file displays each memory object for a particular task. Here are just the entries for a current shell’s heap and stack: [root@tux ~]# cat /proc/$$/numa_maps 0245f000 default heap anon=65 dirty=65 active=60 N1=65 7fff23318000 default stack anon=7 dirty=7 N1=7 The first column is the start of the Virtual Memory Address (VMA) range.
PAGE 8
Additional Recommended Software Components When Running Linux on the DL980 The HP Smart Start CD (with ProLiant Support Pack minimum recommended version 8.70) is an essential single-server deployment tool that simplifies ProLiant (DL, ML, and BL) server setup and provides a reliable and consistent method for deploying ProLiant servers.
PAGE 9
• Including installation logic and version control that automatically verifies hardware, software, and operating system dependencies. Only the correct software updates and latest drivers for optimal system configuration (excluding RPMs) are installed.
PAGE 10
• RHEL 6 — PSP version 8.72 • SLES 10 — PSP version 8.71 • SLES 11 — PSP version 8.70 PSP Installation Methods for Linux Installing PSP using autorun: You can install the PSP using autorun if it is configured on the target Linux server. Red Hat distributions include autorun. If autorun is not installed, install and configure it properly before beginning. Autorun typically mounts the CD-ROM on /media/cdrom for RHEL- and SUSE-based distributions.
PAGE 11
a. Select the components to be installed. b. Review failed dependencies before installation. c. Review the revision history of the components. 5. Click Install to start the installation. When finished, the Installation Results screen displays. 6. For any components that did not install successfully, complete the following steps: a. Review the installation logs for information about the failures. b. Exit HP SUM. c. Reinstall the PSP after addressing any issues described in the installation logs.
PAGE 12
HP SMH Features HP SMH provides streamlined operations for HP servers running Linux operating systems with the following security enhancements and features: • Browser access using secure OS-based, Secure Sockets Layer (SSL) authentication • Common HTTP and HTTPS service for HP Insight Management Agents and utilities, for reduced complexity and system resource requirements • Simplified architecture for implementing HTTP security and HP management updates • Access control through Network Interface Car
PAGE 13
Figure 4: Enabling HP Static High Performance Mode using RBSU 5. Exit all RBSU menu screens using the Esc key. 6. Press the F10 key to confirm your changes and reboot the system. To take advantage of the Linux operating system’s ability to adjust server power consumption (and reduce costs) based on workload, follow these steps: 1. During system boot, press F9 to run the ROM-Based Setup Utility (RBSU). 2. Select Power Management Options. 3. Select HP Power Regulator. 4.
PAGE 14
Figure 5: Enabling OS Control Mode using RBSU 5. Exit all RBSU menu screens using the Esc key. 6. Press the F10 key to confirm your changes and reboot the system. Virtualization Settings Note This section applies only if the system will not be deployed to host virtual guest machines. ® To maximize I/O performance by disabling Intel Virtualization Technology for Directed I/O, follow these steps: 14 1. During system boot, press F9 to run the ROM-Based Setup Utility (RBSU). 2.
PAGE 15
Figure 6: Disabling Intel Virtualization Technology for Directed I/O (VT-d) using RBSU 5. Select Disabled to turn Intel® Virtualization Technology for Directed I/O (VT-d) OFF. 6. Exit all RBSU menu screens using the Esc key. 7. Press the F10 key to confirm your changes and reboot the system. RHEL 5.6 and SLES 10 SP4 Best Practices When running RHEL 5.
PAGE 16
While RHEL 5.6 and SUSE LES 10 SP4 can support over 160 logical processors, they are not optimized for these high logical processor counts. This can result in increased kernel CPU consumption when attempting to schedule tasks across the available logical processors. Unless your workload is proven to benefit demonstrably from Hyper-threading, HT should be disabled in order to reduce this system overhead. HT can always be enabled later if additional CPU capacity is required.
PAGE 17
1. During system boot, press F9 to run the ROM-Based Setup Utility (RBSU). 2. Select Advanced Options. 3. Select System ROM Options. 4. Select x2APIC (see Figure 8). Figure 8: Disabling x2APIC using RBSU 5. If necessary, select Disabled to turn x2APIC OFF. 6. Exit all RBSU menu screens using the Esc key. 7. Press the F10 key to confirm your changes and reboot the system. Verify 40-bit Mode before OS Installation The default RBSU method for addressing memory on the DL980 G7 is 40-bit mode.
PAGE 18
Figure 9: Confirming System Address Mode reported by RBSU If the System Address Mode is set to 44-bit, follow these steps to reset it to 40-bit mode: 18 1. During system boot, press F9 to run the ROM-Based Setup Utility (RBSU). 2. Select Advanced Options. 3. Select Advanced System ROM Options. 4. Select Address Mode 44-bit (see Figure 10).
PAGE 19
Figure 10: Disabling 44-bit System Address Mode using RBSU 5. Select Disabled to turn 44- bit memory addressing OFF. 6. Exit all RBSU menu screens using the Esc key. 7. Press the F10 key to confirm your changes and reboot the system. 8. Confirm the new 40-bit System Address Mode by looking at the RBSU splash screen during system boot, as shown in Figure 9. Use the NOOP I/O Scheduler The default I/O scheduler is Completely Fair Queuing (CFQ), which attempts to balance throughput and latency.
PAGE 20
Installation Hangs with SLES 10 SP3 + Netxen Driver Installation of SLES10 SP3 with a Netxen driver sometimes hangs, due to a non-functional LAN-on-Motherboard (LOM). The workaround here is to add ”brokenmodules=netxen_nic” to the kernel boot parameters and then use the driver DVD to manually load the correct version of the driver. RHEL 6.X and SLES 11 SP1 Best Practices When running RHEL 6.
PAGE 21
Figure 11: Disabling Hyper-threading using RBSU 5. Select Disabled to turn Hyper-threading OFF. 6. Exit all RBSU menu screens using the Esc key. 7. Press the F10 key to confirm your changes and reboot the system. For more information about HP’s RBSU, see the HP ROM-Based Setup Utility User Guide: http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00191707/c00191707.
PAGE 22
Figure 12: Confirming System Address Mode reported by RBSU 22 2. Press F9 to run the ROM-Based Setup Utility (RBSU). 3. Select Advanced Options. 4. Select Advanced System ROM Options. 5. Select Address Mode 44-bit (see Figure 13).
PAGE 23
Figure 13: Enabling 44-bit System Address Mode using RBSU 6. 7. Ignore the Warning message: Caution: Enabling 44-bit Address Mode may cause problems with an operating system if it does not support 44-bit memory addressing. Select Enabled to turn 44- bit memory addressing ON. 8. Exit all RBSU menu screens using the Esc key. 9. Press the F10 key to confirm your changes and reboot the system. 10.
PAGE 24
Oracle Database Best Practices When running Oracle 11gR2 (11.2.0.2) on the HP ProLiant DL980 G7 Server, there are several recommended best practices that are explained in this section: • Enable Hyper-threading for CPU-intensive workloads • Use HugePages • Enable Oracle NUMA optimizations Enable Hyper-threading for CPU-intensive Workloads CPU-intensive Oracle database workloads typically perform better with Hyper-threading (HT) enabled, while I/Obound workloads may not.
PAGE 25
Oracle’s Automatic Memory Management (AMM) feature must be disabled for Oracle to make use of HugePages. Disabling AMM requires setting the Oracle initialization parameter MEMORY_TARGET to zero and SGA_TARGET to the desired Shared Global Area (SGA) size. An Oracle database instance can be configured to use HugePages at startup by setting the initialization parameter use_large_pages=ONLY.
PAGE 26
For More Information For an overview of the HP Proliant DL980 G7 server: http://www.hp.com/servers/dl980 For more information about the HP DL980 G7 server with HP PREMA Architecture: http://h10010.www1.hp.com/wwpc/us/en/sm/WF25a/15351-15351-3328412-241644-42225844231377.html To download the recommended components described in this document, along with other drivers and software, visit the HP DL980 G7 Support web page: http://www.hp.