Engineering white paper, 2nd Edition Configuring and using DDR3 memory with HP ProLiant Gen8 Servers Best Practice Guidelines for ProLiant servers with Intel® Xeon® processors Table of contents Introduction 3 Overview of DDR3 memory technology Basics of DDR3 memory technology Basics of DIMMs DDR3 DIMM types HP SmartMemory HP Advanced Memory Error Detection 3 3 4 5 6 6 ProLiant Gen8 memory architecture for servers with Intel® Xeon® E5-2600 series processors Overview ProLiant Gen8 servers using the Inte
General population guidelines 14 Optimizing memory configurations Optimizing for capacity Optimizing for performance Optimizing for lowest power consumption Optimizing for Resiliency 15 15 15 Understanding unbalanced memory configurations Memory configurations that are unbalanced across channels Memory configurations that are unbalanced across Processors 20 22 23 23 23 BIOS Settings for memory Controlling Memory Speed Setting Memory Interleave 24 24 25 For more information 26 Appendix A - Sample C
Introduction This paper provides an overview of the new DDR3 memory and its use in the 2 socket HP ProLiant Gen8 servers using the latest Intel® Xeon® E5-2600 series processor family. With the introduction of HP ProLiant Gen8 servers, DDR3 maximum operating speed is increasing and a new type of Load Reduced DIMM (LRDIMM) is being introduced. We are also introducing HP SmartMemory, which provides superior performance over 3rd party memory in certain configurations.
Table 1. DDR3 memory speeds DIMM Label JEDEC Name Data Transfer Rate Maximum DIMM Throughput PC3 – 14900 DDR3-1866 1866 MT/s 14.9 GB/s PC3 – 12800 DDR3-1600 1600 MT/s 12.8GB/s PC3 – 10600 DDR3-1333 1333 MT/s 10.6 GB/s PC3 – 8500 DDR3-1066 1066 MT/s 8.5 GB/s PC3 – 6400 DDR3- 800 800 MT/s 6.4 GB/s Basics of DIMMs Before exploring the new technologies in DDR3 DIMMs for ProLiant Gen8 servers, let’s quickly review some of the basics of DIMM technology.
DDR3 DIMM types ProLiant Gen8 servers support four different DIMM types – Unbuffered with ECC Memory (UDIMMs), Registered Memory (RDIMMs), Load Reduced Memory (LRDIMMs), and HyperCloud Memory (HDIMMs). UDIMMs and RDIMMs are familiar from their use in both ProLiant G6 and G7 servers. However, LRDIMMs are a new class of DIMMs that work solely with the ProLiant Gen8 server architecture. HyperCloud DIMMs are special purpose memory available only as a Factory Installed option.
HP SmartMemory ProLiant Gen8 servers introduce HP SmartMemory technology for DDR3 memory. HP SmartMemory enables authentication of installed memory. This verifies whether DIMMs have passed our qualification and testing processes and determines if the memory has been optimized to run on HP ProLiant Gen8 servers. Use of HP SmartMemory DIMMs enables extended performance and manageability features for the 2P ProLiant Gen8 servers.
Figure 1. ProLiant Gen8 memory architecture for servers using the E5-2600 family processor series ProLiant Gen8 servers using the Intel® Xeon® E5-2600 series processors As shown in Table 4, there are several models of 2P ProLiant Gen8 servers that use the Intel Xeon E5-2600 family of processors. Table 4.
Table 5. HP ProLiant Gen8 Intel Xeon E5-2600 Series Processor Family Processor Model Number CPU Frequency Level 3 Cache Size Maximum Memory Speed Maximum Memory Throughput (per channel) E5-2690 2.90 GHz 20MB 1600 MT/s 12.8GB/s E5-2680 2.70 GHz 20MB 1600 MT/s 12.8GB/s E5-2670 2.60 GHz 20MB 1600 MT/s 12.8GB/s E5-2667 2.90 GHz 15MB 1600 MT/s 12.8GB/s E5-2665 2.40 GHz 20MB 1600 MT/s 12.8GB/s E5-2660 2.20 GHz 20MB 1600 MT/s 12.8GB/s E5-2650 2.00 GHz 20MB 1600 MT/s 12.
Figure 2. ProLiant Gen8 memory architecture for servers using the E5-2400 series processors ProLiant Gen8 servers using Intel® Xeon® E5-2400 series processors There are several different models of the 2P ProLiant Gen8 servers that use the Intel Xeon E5-2400 family of processors. These are shown in Table 6. Table 6.
Table 7. ProLiant Gen8 E5-2400 series processors Processor Model Number CPU Frequency Level 3 Cache Size Maximum Memory Speed Maximum Memory Throughput (per channel) E5-2450 2.10 GHz 20MB 1600 MT/s 12.8GB/s E5-2450L 1.80 GHz 20MB 1600 MT/s 12.8GB/s E5-2430 2.20 GHz 15MB 1600 MT/s 12.8GB/s E5-2420 1.90 GHz 15MB 1600 MT/s 12.8GB/s E5-2407 2.20 GHz 10MB 1600 MT/s 12.8GB/s E5-2403 1.80 GHz 10MB 1600 MT/s 12.
Figure 3. HP DDR3 Memory Part Number Decoder Populating memory in ProLiant Gen8 servers ProLiant Gen8 memory slot configurations The ProLiant Gen8 servers feature three different memory slot configurations: • Either 24 or 16 memory slots total for servers using E5-2600 series processors. • 12 memory slots total for servers using E5-2400 series processors For ProLiant Gen8 servers, we recommend populating all memory channels whenever possible. This ensures the best memory performance.
Rules for DIMM types • Do not mix UDIMMs, RDIMMs, or LRDIMMs. • Quad rank RDIMMs are not supported in ProLiant Gen8 servers. • LRDIMMs are capable of up to three DIMMs per channel. • RDIMMs operating at either 1.35V or 1.5V may be mixed in any order, but the system will operate at the higher voltage. • DIMMs of different speeds may be mixed in any order. The server will select the lowest common speed.
For more information, you should consult the User Guide: hp.com > support & drivers > product support & troubleshooting > enter your product Figure 5 shows the memory slot configuration for 16 slot 2P ProLiant Gen8 servers. The configuration is similar to the to the 24 slot servers. However, 16 slot servers have only 2 DIMM slots per channel. Once again, the first memory slots for each channel on each processor are the white memory slots (A, B, C, and D).
Figure 6. DIMM slots and population order for 12 slot 2P ProLiant Gen8 servers. Memory system operating speeds All DDR3 DIMMs for ProLiant Gen8 servers operate natively at either 1600 MT/s or 1333 MT/s. However, the final operating speed of the memory system for the server depends on the type of DIMMs you install as well as the number of DIMMs you install per channel. Larger configurations using 2 or 3 DIMMs per memory channel may operate at a slower speed than the native speed of the DIMMs.
• Use the same HP SmartMemory part number in each memory channel. Optimizing memory configurations By taking advantage of the different DIMM types, sizes and speeds available for HP ProLiant Gen8 servers, you can optimize server memory configuration to meet different application or datacenter requirements. Optimizing for capacity You can maximize memory capacity on ProLiant Gen8 servers using the new 32 GB LRDIMMs.
• CAS latency. CAS (Column Address Strobe) latency represents the basic DRAM response time. It is specified as the number of clock cycles (e.g. 6, 7, 11) that the controller must wait after asserting the Column Address signal before data is available on the bus. CAS latency plays a larger role in determining the unloaded latency than loaded latency. Figure 7 shows both unloaded and loaded latency numbers for various DDR3 DIMMs when used in one DIMM per channel configurations.
Figure 8. System throughput with 1,2,4,8 channels populated in a 2 processor system. Maximum read throughput as a function of number of populated channels 16 GB 2R DDR3-1600 at 1 DIMM per channel Throughput (GB/s) 100.00 80.00 60.00 40.00 20.00 0.00 1 2 4 6 8 Memory speed and throughput Higher memory speeds increase throughput. Using a one DIMM per channel configuration, Figure 9 shows that system memory throughput at 1333 MT/s is 20% higher than at 1066 MT/s.
Figure 9. Throughput at different DIMM speeds with 1 DIMM per channel using 16GB 1600 RDIMMs. Memory throughput as a function of Memory Speed (Using 8 x 16GB 2R 1600 RDIMMs at 1 DIMM per channel) 100 90 80 70 60 GB/s Throughput 50 40 30 20 10 0 800 MT/s 1066 MT/s 1333 MT/s 1600 MT/s Number of DIMMs per channel and throughput Figure 10 shows the measured memory throughput for several one and two DIMM per channel configurations. Throughput actually decreases when a second DIMM is added to each channel.
Figure 10.
HP 1333 MT/s SmartMemory LRDIMMs are capable of operating at 1.35V at one and two DIMMs per channel and 1066 MT/s at three DIMMs per channel. Standard RDIMMs require 1.5V operation to maintain 1333 MT/s speed at one and two DIMMs per channel. As Table 13 shows, using HP SmartMemory saves about 20% on power consumption while providing the same performance as standard DIMMs in 2 DIMM per channel configurations. Table 13. Lower power consumption with two 32 GB LRDIMMs per channel at 1.35V vs. 1.
Figure 11. Power by DIMM capacity.
Figure 12. Power by memory speed using 8 x 8GB DIMMs installed at 1DIMM per channel. Total memory power consumption 8GB DIMMs installed at 1 DIMM per channel 50 45 40 35 30 25 20 15 10 5 0 8 GB 2R UDIMMs 1333 8 GB 2R RDIMMs 1333 Idle Power (W) 8 GB 1R RDIMMs 1600 Loaded Power (W) General guidelines when optimizing for power consumption When optimizing for lowest power consumption, you can use the following general rules.
Understanding unbalanced memory configurations Unbalanced memory configurations are those in which the installed memory is not distributed evenly across the memory channels and/or the processors. ISS discourages unbalanced configurations because they will always have lower performance than similar balanced configurations. There are two types of unbalanced configurations, each with its own performance implications. • Unbalanced across channels.
Figure 14. A memory configuration that is unbalanced across processors. BIOS Settings for memory The HP server BIOS provides control over several memory configuration settings for ProLiant Gen8 servers. You can access and change these settings using the ROM Based Setup Utility (RBSU), which is part of all HP ProLiant servers. To launch RBSU, press the F9 key during the server boot sequence.
Setting Memory Interleave Diasabling Memory Interleaving This option is available from the Advanced Power Management menu in RBSU. Disabling memory interleaving saves some power per DIMM, but also decreases memory system performance. Setting Node Interleaving This option is available from the RBSU Advanced Options menu and controls how the server maps the system memory across the processors.
For more information Visit the URLs listed below if you need additional information. Resource description Web address Online DDR3 Memory Configuration Tool www.hp.com/go/ddr3memory-configurator DDR3 memory technology Technology brief, 2nd edition http://h20000.www2.hp.com/bc/docs/support/SupportManual/c0212649 9/c02126499.pdf HP Advanced Memory Error Detection Technology Technology brief http://h20000.www2.hp.com/bc/docs/support/SupportManual/c0287859 8/c02878598.pdf Get connected hp.
Appendix A - Sample Configurations for 2P ProLiant Gen8 servers 24 DIMM slot servers using Intel® Xeon® E5-2600 processor series Total Memory (GB) Number of DIMMs DIMM Size DIMM Ranks DIMM Type (UDIMM, RDIMM, LRDIMM) Data Rate DIMMs per Channel Unloaded Latency (ns) Loaded Latency (ns) Throughput (GB/s) Idle Power (W) Loaded Power (W) 8 x 2GB 1R 1333 U 16 8 2GB 1R UDIMM 1333 1 65.0 136.3 74.3 0.6 11.8 8 x 4GB 2R 1333 U 32 8 4GB 2R UDIMM 1333 1 64.6 140.3 76.1 1.1 18.
16 DIMM Slot Servers using Intel® Xeon® E5-2600 series processors Total Memory (GB) Number of DIMMs DIMM Size DIMM Rank DIMM Type (UDIMM, RDIMM, LRDIMM) Data Rate DIMMs per Channel Unloaded Latency (ns) Loaded Latency (ns) Throughput (GB/s) Idle Power (W) Loaded Power (W) 8 x 2GB 1R 1333 U 16 8 2GB 1R UDIMM 1333 1 65.0 136.3 74.3 0.56 11.8 8 x 4GB 2R 1333 U 32 8 4GB 2R UDIMM 1333 1 64.6 140.3 76.1 1.13 18.7 8 x 4GB 1R 1333 R 32 8 4GB 1R RDIMM 1333 1 65.3 136.
12 DIMM Slot Servers using Intel® Xeon® E5-2400 series processors DIMM Rank DIMM Type (UDIMM, RDIMM, LRDIMM) Data Rate DIMMs per Channel Unloaded Latency (ns) Loaded Latency (ns) Throughput (GB/s) Idle Power (W) 2GB 1R UDIMM 1333 1 71.4 104.4 52.1 0.5 8.0 4GB 2R UDIMM 1333 1 71.0 104.4 48.2 0.8 12.9 6 4GB 1R RDIMM 1333 1 71.4 104.8 47.8 1.4 14.8 24 6 4GB 1R RDIMM 1600 1 66.7 94.1 57.5 1.6 25.2 12 x 2GB 1R 1333 U 24 12 2GB 1R UDIMM 1333 2 71.0 108.
