Front cover Draft Document for Review May 28, 2009 2:58 pm REDP-4405-00 IBM Power 570 Technical Overview and Introduction Expandable modular design supporting advanced mainframe class continuous availability PowerVM virtualization including the optional Enterprise edition POWER6 processor efficiency operating at state-of-the-art throughput levels Giuliano Anselmi YoungHoon Cho Gregor Linzmeier Marcos Quezada John T Schmidt Guido Somers ibm.
Draft Document for Review September 2, 2008 5:05 pm 4405edno.
405edno.fm Draft Document for Review September 2, 2008 5:05 pm Note: Before using this information and the product it supports, read the information in “Notices” on page vii. First Edition (October 2008) This edition applies to the IBM Power 570 (9117-MMA). © Copyright International Business Machines Corporation 2008. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Draft Document for Review March 26, 2009 5:10 pm 4405TOC.fm Contents Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4405TOC.fm iv Draft Document for Review March 26, 2009 5:10 pm 2.7.3 IVE subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 PCI adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.1 LAN adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.2 SCSI and SAS adapters . . . . . . . . . . . . . . . . . . . . .
Draft Document for Review March 26, 2009 5:10 pm 4405TOC.fm 3.4 System Planning Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Chapter 4. Continuous availability and manageability . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.1 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.1.1 Designed for reliability. . . . . . . . . . . . . . . . . . . . . . . .
4405TOC.
Draft Document for Review March 26, 2009 5:10 pm 4405spec.fm Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area.
4405spec.
Draft Document for Review March 26, 2009 5:10 pm 4405pref.fm Preface This IBM® Redpaper is a comprehensive guide covering the IBM Power™ 570 server supporting AIX, IBM i, and Linux for Power operating systems. The goal of this paper is to introduce the major innovative Power 570 offerings and their prominent functions, including the following. Unique modular server packaging New POWER6™ processors available at frequencies of 4.2, 4.4, and 5.0 GHz. The POWER6 processor available at frequencies of 3.
4405pref.fm Draft Document for Review March 26, 2009 5:10 pm System p products. He is an IBM Certified Specialist in System p and AIX® 5L™. He provides second line support to field engineers with technical support on System p, and system management.
Draft Document for Review March 26, 2009 5:10 pm 4405pref.fm will have the opportunity to team with IBM technical professionals, Business Partners, and Clients. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you will develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.
4405pref.
Draft Document for Review May 22, 2009 3:10 pm 4405ch01 General description.fm 1 Chapter 1. General description The innovative IBM Power 570 mid-range server with POWER6 and available POWER6+ f processor cards delivers outstanding price/performance, mainframe-inspired reliability and availability features, flexible capacity upgrades, and innovative virtualization technologies to enable management of growth, complexity, and risk.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm processor also includes an AltiVec SIMD accelerator, which helps to improve the performance of high performance computing (HPC) workloads. All Power Systems servers can utilize logical partitioning (LPAR) technology implemented using System p virtualization technologies, the operating system (OS), and a hardware management console (HMC).
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm 1.1 System specifications Table 1-1 lists the general system specifications of a single Central Electronics Complex (CEC) enclosure. Table 1-1 System specifications Description Range (operating) Operating temperature 5 to 35 degrees C (41 to 95 F) Relative humidity 8% to 80% Maximum wet bulb 23 degrees C (73 F) Noise level with 3.5 GHz processors FC 5620: 7.1 bels with 3.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm Figure 1-1 shows system views. Figure 1-1 Views of the system 1.3 System features The full system configuration is made of four CEC building blocks. It features: 2-, 4-, 8-, 12-, 16-, and 32-core configurations utilizing the POWER6 chip on up to eight dual core processor cards, or eight dual-core POWER6 dual-chip processor cards. Up to 192 GB DDR2 memory per enclosure, 768 GB DDR2 max per system.
Draft Document for Review May 22, 2009 3:10 pm 4405ch01 General description.fm 6 PCI slots per enclosure: 4 PCIe, 2 PCI-X; 24 PCI per system: 16 PCIe, 8 PCI-X. Up to 2 GX+ adapters per enclosure; 8 per system One hot-plug slim-line media bay per enclosure, 4 max per system. The external processor fabric bus in this system is modular. For a multiple-drawer server configuration, a processor fabric cable or cables, and a service interface cable are required.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm Utility CoD autonomically provides additional processor performance on a temporary basis within the shared processor pool in one minute increments. It adds additional cores to allow greater parellel operation, and can increase the effective L2 cache of the shared processor pool On/Off CoD enables processors or memory to be temporarily activated in full-day increments as needed.
Draft Document for Review May 22, 2009 3:10 pm 4405ch01 General description.fm Table 1-4 Memory feature codes Feature code Description 5692 0/2 GB DDR2 Memory (4X0.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm FC 5757 IBM 4.7 GB IDE Slimline DVD-RAM Drive 1.3.4 I/O drawers The system has seven I/O expansion slots per enclosure, including one dedicated GX+ slot. The other 6 slots support PCI adapters. There are 3 PCIe 8X long slots and 1 PCIe 8X short slot. The short PCIe slot may also be used for a second GX+ adapter. The remaining 2 slots PCI-X long slots.
Draft Document for Review May 22, 2009 3:10 pm 4405ch01 General description.fm optional feature (FC 6268). The 7311 Model D20 I/O drawer offers a modular growth path for systems with increasing I/O requirements. When fully configured with 20 attached 7311 Model D20 drawers, the combined system supports up to 148 PCI-X adapters, 16 PCIe adapters, and 264 hot-swappable disks. In a full configuration, Remote I/O expansion cards (FC 1800 - GX Dual Port RIO-2) are required.
4405ch01 General description.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm Dual-Port 12X Channel Interface adapter options: – short run: cables between this adapter and a host system may not exceed 3.0 Meters in length. Cables between two I/O drawers may not exceed 1.5 Meters if both I/O drawers include this short run adapter. Cables between two I/O drawers may not exceed 3.0 Meters if either of the I/O drawers includes this short run adapter.
4405ch01 General description.fm System drawers/cores Max RIO-2 drawersa Draft Document for Review May 22, 2009 3:10 pm Max 12X Ch drawersa Total number of slots D11 D20 G30 PCI-X PCIe PCI-X PCIe PCI-X PCIe 3 drawers / 12-core 16 24 102 11b 118 11b 150 9c 4 drawers / 16-core 20 32 128 15b 148 15b 200 12c a. Up to four I/O drawers are supported in a loop b. One PCIe slot is reserved for the Remote I/O expansion card c.
Draft Document for Review May 22, 2009 3:10 pm 4405ch01 General description.fm racks. The client’s input is collected and verified through the marketing configurator (eConfig). The client’s request is reviewed by eConfig for safe handling by checking the weight distribution within the rack. The manufacturing plant provides the final approval for the configuration. This information is then used by IBM Manufacturing to assemble the system components (drawers) in the rack according to the client’s request.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm 1.4.2 IBM 7014 Model T42 rack The 2.0-meter (79.3-inch) Model T42 addresses the client requirement for a tall enclosure to house the maximum amount of equipment in the smallest possible floor space. The features that differ in the Model T42 rack from the Model T00 include: 42 EIA units (42 U) of usable space (6 U of additional space). The Model T42 supports AC only. Weights: – T42 base empty rack: 261 kg (575 lb.
Draft Document for Review May 22, 2009 3:10 pm 4405ch01 General description.fm PDUs can be installed horizontally in the T00 rack and three in the T42 rack. The four vertical mounting locations will be filled first in the T00 and T42 racks. Mounting PDUs horizontally consumes 1 U per PDU and reduces the space available for other racked components.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm The Universal PDUs are compatible with previous models. Note: Each system drawer to be mounted in the rack requires two power cords, which are not included in the base order. For maximum availability it is highly recommended to connect power cords from the same system to two separate PDUs in the rack. And to connect each PDU to independent power sources. 1.4.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm RS232 serial RJ-45 LAN connector Individual outlet Outlet group Input connector RJ-45 console connector Figure 1-4 Intelligent Power Distribution Unit In this panel, outlet names and outlet group names are shown. Each iPDU node will contain either node group outlets or individual outlets. For further information of integration and IBM Director functionality, see: http://www.ibm.com/systems/management/director/about 1.4.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm When a system is installed in an 7014-T00 or -T42 rack that has no front door, a Thin Profile Front Trim Kit must be ordered for the rack. The required trim kit for the 7014-T00 rack is FC 6246. The required trim kit for the 7014-T42 rack is FC 6247. The design of the 570 is optimized for use in a 7014-T00 or -T42 rack.
Draft Document for Review May 22, 2009 3:10 pm 4405ch01 General description.fm IBM PS/2 Travel Keyboards are supported on the 7316-TF3 for use in configurations where only PS/2 keyboard ports are available. The IBM 7316-TF3 Flat Panel Console Kit provides an option for the USB Travel Keyboards with UltraNav. The keyboard enables the 7316-TF3 to be connected to systems that do not have PS/2 keyboard ports.
4405ch01 General description.fm Draft Document for Review May 22, 2009 3:10 pm Figure 1-5 Top view of non-IBM rack specification dimensions The vertical distance between the mounting holes must consist of sets of three holes spaced (from bottom to top) 15.9 mm (0.625 in.), 15.9 mm (0.625 in.), and 12.67 mm (0.5 in.) on center, making each three-hole set of vertical hole spacing 44.45 mm (1.75 in.) apart on center. Rail-mounting holes must be 7.1 mm + 0.1 mm (0.28 in. + 0.004 in.) in diameter.
Draft Document for Review May 22, 2009 3:10 pm 4405ch01 General description.fm Figure 1-7 Rack specification dimensions, bottom front view It might be necessary to supply additional hardware, such as fasteners, for use in some manufacturer’s racks. The system rack or cabinet must be capable of supporting an average load of 15.9 kg (35 lb.) of product weight per EIA unit.
4405ch01 General description.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm 2 Chapter 2. Architecture and technical overview This chapter discusses the overall system architecture represented by Figure 2-1, with its major components described in the following sections. The bandwidths that are provided throughout the section are theoretical maximums used for reference. You should always obtain real-world performance measurements using production workloads.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm 2.1 The POWER6 processor The POWER6 processor capitalizes on all the enhancements brought by the POWER5 chip. Two of the enhancements of the POWER6 processor is the ability to do processor instruction retry and alternate processor recovery. This significantly reduces exposure to both hard (logic) and soft (transient) errors in the processor core.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm POWER5 Alti POWER6 core Vec 1.9 4.2 GHz 32 MB L3 Ctrl POWER6 Alti core Vec 4.2 1.9 GHz L2 4 MB L2 4 MB Fabric bus controller L3 cache Memory Controller GX Bus Controller Memory+ GX+ Bridge Figure 2-2 POWER6 processor The CMOS 11S0 lithography technology in the POWER6 processor uses a 65 nm fabrication process, which enables: Performance gains through faster clock rates from 3.5 GHz, 4.2 GHz up to 4.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm The POWER6 processor implements the 64-bit IBM Power Architecture® technology.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm 2.1.2 AltiVec and Single Instruction, Multiple Data IBM Semiconductor’s advanced Single Instruction, Multiple Data (SIMD) technology based on the AltiVec instruction set is designed to enable exceptional general-purpose processing power for high-performance POWER processors.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm power in the data center. The margined power is the amount of extra power that is allocated to a server during its installation in a datacenter. It’s based on the server environmental specifications that usually are never reached. Since server specifications are always based on maximum configurations and worst case scenarios.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Table 2-2 EnergyScale function’s software interfaces EnergyScale functions ASMI HMC Active Energy Manager Power Trending N N Y Power Saver Mode Y Y Y Schedule Power Saver Mode Operation N Y Y Power Capping N N Y Schedule Power Capping Operation N N Y 2.3 Processor cards In the 570, the POWER6 processors, associated L3 cache chip, and memory DIMMs are packaged in processor cards.
05ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm sees a single shared memory resource. They are interfaced to 12 memory slots, where as each memory DIMM has its own memory buffer chip and are interfaced in a point-to-point connection. I/O connects to the 570 processor module using the GX+ bus. The processor module provides a single GX+ bus. The GX+ bus provides an interface to I/O devices through the RIO-2 connections or a 12X Channel attach connections. 2.3.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm Note: Any system made of more than one processor card must have all processor cards running at the same speed. To verify the processor characteristics on a system running at 4.2 GHz, use one of the following commands: lsattr -El procX Where X is the number of the processor, for example, proc0 is the first processor in the system.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm between the memory controller and the buffer is changed from a shared parallel interface to a point-to-point serial interface (see Figure 2-5). POWER6 chip Memory controller DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM Memory buffer chip Memory buffer chip Memory buffer chip DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM ...
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm If the initial p5 570 server does not configure supported DDR2 memory, then the target 570 can be configured with FC 5622 processor card and the desired amount of memory must be included in the model migration order.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm Figure 2-8 IBM memory certification label 2.4.5 Memory throughput The memory subsystem throughput is based on the speed of the memory. On processor, there are four memory channels, each with single 2 byte read and 1 byte write. Memory channels of POWER6 memory controller are connected to Memory buffers. The processor chip has two POWER6 processors. The DDR2 bus allows double reads or writes per clock cycle.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Two 64-bit PCI-X 2.0 buses, one 64-bit PCI-X 1.0 bus, and one 32-bit PCI-X 1.0 bus Four 8x PCI Express links Two 10 Gbps Ethernet ports: Each port is individually configurable to function as two 1 Gb/s port In a fully populated 570, there are two GX+ buses, one from each processor. Each 570 has 2 GX+ slots with a single GX+ bus.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Figure 2-9 SP Flex™ cables FC 5657: 2-drawer SP cable FC 5658: 3-drawer SP cable FC 5660: 4-drawer SP cable. 2.6 Internal I/O subsystem The internal I/O subsystem resides on the system planar which supports a mixture of both PCIe and PCI-X slots. All PCIe or PCI-X slots are hot pluggable and Enhanced Error Handling (EEH) enabled.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Slot# Description Location code PHB Max Card Size Slot 6 PCIe x8 GX+ P1-C6 P1-C8 PCIe PHB3 Short Slot 7 GX+ P1-C9 Adapter slots P1-C6 and P1-C8 share the same physical space in a system enclosure. When a GX+ adapter is installed in GX slot P1-C8, PCIe slot P1-C6 cannot be used. The 570 uses generation 3, blind swap cassettes to manage the installation and removal of adapters.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Virtual Ethernet resources without the Virtual I/O Server Designed to operate at media speeds The IVE is a physical Ethernet adapter that is connected directly to the GX+ bus instead of being connected to a PCIe or PCI-X bus, either as an optional or integrated PCI adapter. This provides IVE high throughput, and low latency.
4405ch02 Architecture and technical overview.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm Figure 2-12 Integrated Virtual Ethernet adapter connection on System p 570 I/O system board 2.7.2 Feature code port and cable support All the IVE feature codes have different connectivity options and different cable support (see Figure 2-13).
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm FC 5637 supports: Only 10 Gbps SR connectivity 62.5 micron multi-mode fiber cable type – LC physical connector type – 33 meters maximum range 2.7.3 IVE subsystem Figure 2-14 shows a high level-logical diagram of the IVE.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm or feature on top of the IVE card assembly itself. Another key goal is the support of all the state-of-art NIC functionality provided by leading Ethernet NIC vendors. IVE offers the following functions with respect to virtualization: Up to 32 logical ports identified by MAC address Sixteen MAC addresses are assigned to each IVE port group.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Before adding or rearranging adapters, use the System Planning Toll to validate the new adapter configuration. See the System Planning Tool Web site at : http://www-03.ibm.com/servers/eserver/support/tools/systemplanningtool/ If you are installing a new feature, ensure that you have the software required to support the new feature and determine whether there are any existing PTF prerequisites to install.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Table 2-8 on page 45 shows comparing Parallel SCSI to SAS.
4405ch02 Architecture and technical overview.fm Software Draft Document for Review May 28, 2009 1:59 pm Supported on standard Gigabit adapters, additional software (see “IBM iSCSI software Host Support Kit” on page 46) must be installed. The main processor is utilized for processing related to the iSCSI protocol. Initial iSCSI implementations are targeted at small to medium-sized businesses and departments or branch offices of larger enterprises that have not deployed Fibre Channel SANs.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm 2.8.4 Fibre Channel adapter The 570 servers support direct or SAN connection to devices using Fibre Channel adapters. Table 2-10 provides a summary of the available Fibre Channel adapters. All of these adapters have LC connectors. If you are attaching a device or switch with an SC type fibre connector, then an LC-SC 50 Micron Fiber Converter Cable (FC 2456) or an LC-SC 62.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Table 2-12 Asynchronous PCI-X adapters Feature code Description Slot Size 2943 8-Port Asynchronous Adapter EIA-232/RS-422 PCI-X Short 5723 2-Port Asynchronous IEA-232 PCI Adapter PCI-X Short In many cases, the FC 5723 asynchronous adapter is configured to supply a backup HACMP heartbeat. In these cases, a serial cable (FC 3927 or FC 3928) must be also configured.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Feature Description 3651 + 5900 or 5909 A hardware feature that occupies PCIe slot P1C3 provides a mini SAS 4x connector to the rear bulkhead and allows all of the internal SAS drives to be controlled using an external SAS controller. 2.9.2 Split backplane As described in paragraph 2.9.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm 2.10 External I/O subsystems This section describes the external I/O subsystems, which include the I/O drawers, the IBM System Storage™ EXP 12S SAS drawer, as well as the 7311-D11, 7311-D20, 7311-G30, 7031-D24, and 7031-T24 deskside tower. Table 2-15 provided an overview of all the supported I/O drawers.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm slots 1 to 6 are PCI-X slots that operate at 133 MHz and 3.3 V signaling. Figure 2-15 on page 51 shows a conceptual diagram of the 7311 Model D11 I/O drawer.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm 2.10.3 7311 Model D20 I/O drawer The 7311 Model D20 I/O drawer must have the RIO-2 loop adapter (FC 6417) to be connected to the 570 system. The PCI-X host bridge inside the I/O drawer provides two primary 64-bit PCI-X buses running at 133 MHz. Therefore, a maximum bandwidth of 1 GBps is provided by each of the buses. Figure 2-16 shows a conceptual diagram of the 7311 Model D20 I/O drawer subsystem.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm 2.10.4 7311 Model D11 and Model D20 I/O drawers and RIO-2 cabling As described in 2.10, “External I/O subsystems” on page 50, we can connect up to four I/O drawers in the same loop, and up to 20 I/O drawers to the p5-570 system. Each RIO-2 port can operate at 1 GHz in bidirectional mode and is capable of passing data in each direction on each cycle of the port.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm 2.10.5 7311 I/O drawer and SPCN cabling SPCN4 is used to control and monitor the status of power and cooling within the I/O drawer. SPCN is a loop: Cabling starts from SPCN port 0 on the 570 to SPCN port 0 on the first I/O drawer. The loop is closed, connecting the SPCN port 1 of the I/O drawer back to port 1 of the 570 system.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm paging space. Specific client requirements can be satisfied with the several external disk possibilities that the 570 supports. 2.11.1 IBM System Storage EXP 12S (FC 5886) The IBM System Storage EXP 12S is a high density rack-mountable disk drive enclosure for supporting a total of twelve 3.5-inch disk drives on POWER6 systems only. Using hot-swappable 300 GB SAS disk drives the EXP 12S drawer can provide 3.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Feature Description 3691 SAS cable (YO) adapter to SAS enclosure 1.5 meter 3692 SAS cable (YO) adapter to SAS enclosure 3 meter 3693 SAS cable (YO) adapter to SAS enclosure 6 meter A typical base configuration is using a server machine and a single attached drawer as shown in Figure 2-21.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm storage solutions or simply high capacity storage for a single server installation, the unit provides a cost-effective solution. It provides 24 hot-swappable disk bays, 12 accessible from the front and 12 from the rear. Disk options that can be accommodated in any of the four six-packs disk drive enclosure are 73.4 GB, 146.8 GB, or 300 GB 10 K rpm or 36.4 GB, 73.4 GB, or 146.8 GB 15 K rpm drives.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm Micro-Partitioning technology. System Storage and the IBM Power servers are usually connected together to a storage area network. For further information about ESS, refer to the following Web site: http://www.ibm.com/servers/storage/disk/enterprise/ds_family.html 2.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Figure 2-23 HMC to service processor and LPARs network connection The default mechanism for allocation of the IP addresses for the service processor HMC ports is dynamic. The HMC can be configured as a DHCP server, providing the IP address at the time the managed server is powered on.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Detecting, reporting, and storing changes in hardware conditions Powering managed systems on and off Acting as a service focal point Generating or importing System Plans The HMC provides both graphical and command line interface for all management tasks.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Redundant Service Processor connectivity Redundant Service Processor function for managing the service processors when one fails is supported on all systems that are operating with system firmware level FM320_xxx_xxx, or later. This support is available for configurations with two or more CEC enclosures.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm IBM AIX 5L If installing AIX 5L on the 570, the following minimum requirements must be met: AIX 5L for POWER V5.3 with 5300-07 Technology Level (APAR IY99422), CD# LCD4-7463-09, DVD# LCD4-7544-05 or later AIX 5L for POWER V5.3 with 5300-06 Technology Level with Service Pack 4 (APAR IZ06992) AIX 5L for POWER V5.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Interface Tool (SMIT) and the IBM Systems Director Console for AIX Linux for System p systems Linux is an open source operating system that runs on numerous platforms from embedded systems to mainframe computers. It provides a UNIX-like implementation across many computer architectures. This section discusses two brands of Linux to be run in partitions.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm Note: IBM only supports the Linux systems of clients with a SupportLine contract covering Linux. Otherwise, contact the Linux distributor for support. i5/OS At the time of writing i5/OS® is not supported. 2.13 Service information The 570 is not a client setup server (CSU). Therefore, the IBM service representative completes the system installation. 2.13.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Figure 2-26 Operator control panel physical details and button Note: For servers managed by the HMC, use it to perform control panel functions. Primary control panel functions The primary control panel functions are defined as functions 01 to 20, including options to view and manipulate IPL modes, server operating modes, IPL speed, and IPL type.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm Function 30 – CEC SP IP address and location Function 30 is one of the Extended control panel functions and is only available when Manual mode is selected. This function can be used to display the central electronic complex (CEC) Service Processor IP address and location segment. The Table 2-21 shows an example of how to use the Function 03.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm The following examples are the output of the lsmcode command for AIX and Linux, showing the firmware levels as they are displayed in the outputs. AIX: The current permanent system firmware image is SF220_005. The current temporary system firmware image is SF220_006. The system is currently booted from the temporary image.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm Figure 2-27 IBM Microcode downloads site Receive server firmware fixes using an HMC If you use an HMC to manage your server and you periodically configure several partitions on the server, you need to download and install fixes for your server and power subsystem firmware. How you get the fix depends on whether the HMC or server is connected to the Internet: The HMC or server is connected to the Internet.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm For a detailed description of each task, select System p information, support, and troubleshooting → Fixes and upgrades → Getting fixes and upgrades from the IBM Systems Hardware Information Center Web site at: http://publib.boulder.ibm.com/infocenter/eserver/v1r3s/index.jsp?lang=en Receive server firmware fixes without an HMC Periodically, you need to install fixes for your server firmware.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm options from a client Web browser, using the HTTP server integrated into the service processor internal operating system. Both Ethernet ports have a default IP address: – Service processor Eth0 or HMC1 port is configured as 169.254.2.147. – Service processor Eth1 or HMC2 port is configured as 169.254.3.147. 2.14.
4405ch02 Architecture and technical overview.fm Draft Document for Review May 28, 2009 1:59 pm Opera 7.23 running on a PC or mobile computer connected to the service processor. The Web interface is available during all phases of system operation, including the initial program load and runtime. However, some of the menu options in the Web interface are unavailable during IPL or runtime to prevent usage or ownership conflicts if the system resources are in use during that phase.
Draft Document for Review May 28, 2009 1:59 pm 4405ch02 Architecture and technical overview.fm HMC Ethernet IP addresses have been assigned, it is possible to download the Web-based System Manager Remote Client from a web browser, using the following URL: http://HMC_IP_address/remote_client.html You can then use the PC to access other HMCs remotely. Web-based System Manager Remote Clients can be present in private and open networks.
4405ch02 Architecture and technical overview.
Draft Document for Review September 2, 2008 5:05 pm 4405ch03 Virtualization.fm 3 Chapter 3. Virtualization As you look for ways to maximize the return on your IT infrastructure investments, consolidating workloads becomes an attractive proposition. IBM Power Systems combined with PowerVM technology are designed to help you consolidate and simplify your IT environment.
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm 3.1 POWER Hypervisor Combined with features designed into the POWER6 processors, the POWER Hypervisor delivers functions that enable other system technologies, including logical partitioning technology, virtualized processors, IEEE VLAN compatible virtual switch, virtual SCSI adapters, and virtual consoles.
Draft Document for Review September 2, 2008 5:05 pm 4405ch03 Virtualization.fm Virtual Ethernet Virtual (TTY) console Virtual SCSI The POWER Hypervisor provides a virtual SCSI mechanism for virtualization of storage devices (a special logical partition to install the Virtual I/O Server is required to use this feature, as described in 3.3.2, “Virtual I/O Server” on page 81).
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm adapter and a set of Hypervisor calls to operate on them. Virtual TTY does not require the purchase of any additional features or software such as the PowerVM Edition features. Depending on the system configuration, the operating system console can be provided by the Hardware Management Console virtual TTY, IVM virtual TTY, or from a terminal emulator that is connected to a system port. 3.
Draft Document for Review September 2, 2008 5:05 pm 4405ch03 Virtualization.fm Select whether the partition will be able or not to access extra processing power to “fill up” its virtual processors above its capacity entitlement. Selecting either to capp or uncapp your partition.
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm Figure 3-1 Logical partitioning concepts Dedicated mode In dedicated mode, physical processors are assigned as a whole to partitions. The simultaneous multithreading feature in the POWER6 processor core allows the core to execute instructions from two independent software threads simultaneously. To support this feature we use the concept of logical processors.
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm 3.3 PowerVM The PowerVM platform is the family of technologies, capabilities and offerings that deliver industry-leading virtualization on this 570. It is the new umbrella branding term for Power Systems Virtualization (Logical Partitioning, Micro-Partitioning, Hypervisor, Virtual I/O Server, Advanced Power Virtualization, Live Partition Mobility, Workload Partitions, etc.).
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm network services and IP addresses. Figure 3-2 shows an overview of a Virtual I/O Server configuration.
Draft Document for Review September 2, 2008 5:05 pm 4405ch03 Virtualization.fm Figure 3-3 Architectural view of a Shared Ethernet Adapter A single SEA setup can have up to 16 Virtual Ethernet trunk adapters and each virtual Ethernet trunk adapter can support up to 20 VLAN networks. Therefore, it is possible for a single physical Ethernet to be shared between 320 internal VLAN.
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm Virtual SCSI Virtual SCSI is used to refer to a virtualized implementation of the SCSI protocol. Virtual SCSI is based on a client/server relationship. The Virtual I/O Server logical partition owns the physical resources and acts as server or, in SCSI terms, target device. The client logical partitions access the virtual SCSI backing storage devices provided by the Virtual I/O Server as clients.
Draft Document for Review September 2, 2008 5:05 pm 4405ch03 Virtualization.fm At the time of writing, virtual SCSI supports Fibre Channel, parallel SCSI, iSCSI, SAS, SCSI RAID devices and optical devices, including DVD-RAM and DVD-ROM. Other protocols such as SSA and tape devices are not supported. For more information about the specific storage devices supported for Virtual I/O Server, see: http://www14.software.ibm.com/webapp/set2/sas/f/vios/documentation/datasheet.
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm Figure 3-5 shows the diagram of the Linux x86 application environment. Figure 3-5 Diagram of the Linux x86 Application Environment Supported Operating Systems PowerVM Lx86 version 1.1 will support the following Linux on POWER operating systems: Red Hat Enterprise Linux 4 (RHEL 4) for POWER version 4.4 and 4.5. Also, x86 Linux applications running on RHEL 4.3 are supported.
Draft Document for Review September 2, 2008 5:05 pm 4405ch03 Virtualization.fm Do not run correctly on Red Hat Enterprise Linux 4 starting with version 4.3 or Novell SUSE Linux Enterprise Server (SLES) 9 starting with version SP3 or Novell SLES 10; Are x86 Linux specific system administration or configuration tools. For more information about PowerVM Lx86 please refer to Getting started with PowerVM Lx86, REDP-4298. 3.3.
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm The hypervisor must support the Partition Mobility functionality also called migration process. POWER 6 processor-based hypervisors have this capability and firmware must be at firmware level eFW3.2 or later. Source and destination systems could have different firmware levels, but they must be compatible with each other.
Draft Document for Review September 2, 2008 5:05 pm 4405ch03 Virtualization.fm same workload partition. The systems administrator accesses the workload partition via the administrator console or via regular network tools such as telnet or ssh. The system workload partition is removed only when requested.
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm Inventory and automated discovery Complete inventory of WPAR instances deployed on all servers with WPAR Manager Agents installed whether created by the WPAR Manager or through the command line interface (CLI) on the local system console Note: The IBM PowerVM Workload Partition Manager for AIX 6 is a separate, optional product as part of the IBM PowerVM suite (5765-WPM). 3.3.
Draft Document for Review September 2, 2008 5:05 pm 4405ch03 Virtualization.fm 3.4 System Planning Tool The IBM System Planning Tool (SPT) helps you design a system or systems to be partitioned with logical partitions. You can also plan for and design non-partitioned systems using the SPT. The resulting output of your design is called a system plan, which is stored in a .sysplan file. This file can contain plans for a single system or multiple systems. The .
4405ch03 Virtualization.fm Draft Document for Review September 2, 2008 5:05 pm Figure 3-6 IBM System Planning Tool window showing Hypervisor memory requirements Note: In previous releases of the SPT, you could view an HMC or IVM system plan, but you could not edit the plan. The SPT now allows you to convert an HMC or IVM system plan into a format that will allow you to edit the plan in the SPT. Also note that SPT 2.0 will be the last release that will support .lvt and .xml files.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm 4 Chapter 4. Continuous availability and manageability This chapter provides information about IBM Power Systems design features that help lower the total cost of ownership (TCO). The advanced IBM RAS (Reliability, Availability, and Service ability) technology allows the possibility to improve your architecture’s TCO by reducing unplanned down time.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm single-CPU-per-processor design. Not only does this reduce the total number of system components, it reduces the total amount of heat generated in the design, resulting in an additional reduction in required power and cooling components. Parts selection also plays a critical role in overall system reliability.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm The use of redundant part allows the system to remain operational: Redundant service processor Redundant service processor function for managing service processors when one fails, is available for system configurations with two or more CEC enclosures. Redundant Service Processor function requires that the HMC be attached to the Service Interface Card in both CEC enclosure 1 and CEC enclosure 2.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm action to deconfigure the faulty hardware, helping to avoid a potential system outage and to enhance system availability. Persistent deallocation To enhance system availability, a component that is identified for deallocation or deconfiguration on a POWER6 processor-based system is flagged for persistent deallocation.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm unrecoverable system error, which might otherwise result in an unscheduled server outage. Dynamic processor deallocation relies upon the service processor’s ability to use FFDC-generated recoverable error information to notify the POWER Hypervisor when a processor core reaches its predefined error limit.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm The deallocation event is not successful if the POWER Hypervisor and OS cannot create a full core equivalent. This result in an error message and the requirement for a system administrator to take corrective action. In all cases, a log entry is made for each partition that could use the physical core in question.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm Memory protection Memory and cache arrays comprise data bit lines that feed into a memory word. A memory word is addressed by the system as a single element. Depending on the size and addressability of the memory element, each data bit line may include thousands of individual bits or memory cells.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm Figure 4-5 Memory protection capabilities in action Memory page deallocation While coincident single cell errors in separate memory chips is a statistic rarity, POWER6 processor-based systems can contain these errors using a memory page deallocation scheme for partitions running AIX and for memory pages owned by the POWER Hypervisor.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm Figure 4-6 Memory control hierarchy Memory deconfiguration Defective memory discovered at boot time is automatically switched off, unless it is already the minimum amount required to boot. If the Service Processor detects a memory fault at boot time, it marks the affected memory as bad so it is not to be used on subsequent reboots (Memory Persistent Deallocation).
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm In cases where the data cannot be recovered from another source, a technique called Special Uncorrectable Error (SUE) handling is used to determine whether the corruption is truly a threat to the system. If, as may sometimes be the case, the data is never actually used but is simply over-written, then the error condition can safely be voided and the system will continue to operate normally.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm L3 Array Protection In addition to protection through ECC and Special Uncorrectable Error handling, the L3 cache also incorporates technology to handle memory cell errors via a special cache line delete algorithm. During system run-time, a correctable error is reported as a recoverable error to the Service Processor.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm Figure 4-7 PCI error recovery 4.3 Serviceability The IBM POWER6 Serviceability strategy evolves from, and improves upon, the service architecture deployed on the POWER5 processor-based systems.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm 4.3.1 Detecting errors The first and most crucial component of a solid serviceability strategy is the ability to accurately and effectively detect errors when they occur. While not all errors are a guaranteed threat to system availability, those that go undetected can cause problems because the system does not have the opportunity to evaluate and act if necessary.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm non-critical error is detected or if the error occurs in a resource that can be removed from the system configuration, the booting process is designed to proceed to completion. The errors are logged in the system nonvolatile random access memory (NVRAM).
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm FFDC “check stations” are carefully positioned within the server logic and data paths to ensure that potential errors can be quickly identified and accurately tracked to a Field Replaceable Unit (FRU). This proactive diagnostic strategy is a significant improvement over the classic, less accurate “reboot and diagnose” service approaches.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm Boot-time When an IBM POWER6 processor-based system powers up, the Service Processor initializes system hardware. Boot-time diagnostic testing uses a multi-tier approach for system validation, starting with managed low-level diagnostics supplemented with system firmware initialization and configuration of I/O hardware, followed by OS-initiated software test routines.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm 4.3.3 Reporting problems In the unlikely event of a system hardware or environmentally induced failure is diagnosed, POWER6 processor-based systems report the error through a number of mechanisms. This ensures that appropriate entities are aware that the system may be operating in an error state.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm initiates a call home for service. This methodology insures that all platform errors will be reported through at least one functional path, ultimately resulting in a single notification for a single problem. Extended Error Data (EED) Extended error data (EED) is additional data that is collected either automatically at the time of a failure or manually at a later time.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm in order for the appropriate service action to begin, automatically opening a problem report and in some cases also dispatching field support. This automated reporting provides faster and potentially more accurate transmittal of error information.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm Concurrent maintenance The IBM POWER6 processor-based systems are designed with the understanding that certain components have higher intrinsic failure rates than others. The movement of fans, power supplies, and physical storage devices naturally make them more susceptible to wear down or burnout, while other devices such as I/O adapters may begin to wear from repeated plugging or unplugging.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm Additional capability is being added to the POWER6 firmware to be able to view the status of a system power control network background firmware update. This subsystem will update as necessary as migrated nodes or I/O drawers are added to the configuration.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm Table 4-1 Operating system support for selected RAS features RAS feature AIX V5.3 AIX V6.1 RHEL V5.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm RAS feature AIX V5.3 AIX V6.1 RHEL V5.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm 4.5 Manageability Several functions and tools help manageability, and can allow you to efficiently and effectively manage your system. 4.5.1 Service processor The service processor is a controller running its own operating system. It is a component of the service interface card. The service processor operating system has specific programs and device drivers for the service processor hardware.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm mode, or run concurrently (concurrent mode) with other applications. They have access to the AIX error log and the AIX configuration data. – Service mode, which requires a service mode boot of the system, enables the checking of system devices and features. Service mode provides the most complete checkout of the system resources.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm 2. Select your country 3. Click “IBM Electronic Service Agent Connectivity Guide” Note: To receive maximum coverage, activate Electronic Service Agent on every platform, partition, and Hardware Management Console (HMC) in your network.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm Premium Search: A search service using information gathered by Service Agents (this is a paid service that requires a special contract). For more information on how to utilize the power of IBM Electronic Services, visit the following Web site or contact an IBM Systems Services Representative. https://www-304.ibm.com/jct03004c/support/electronic/portal 4.5.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm 3. From the System Management tasks list, select Operations. 4. From the Operations task list, select Advanced System Management Interface (ASMI). Accessing the ASMI using a Web browser The Web interface to the ASMI is accessible through Microsoft Internet Explorer 6.0, Microsoft Internet Explorer 7, Netscape 7.1, Mozilla Firefox, or Opera 7.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm Accepted level – This is the backup level of server or power subsystem firmware. You can return to this level of server or power subsystem firmware if you decide to remove the installed level. It is installed on the p side of system firmware. IBM provide the Concurrent Firmware Maintenance (CFM) function on System p systems.
4405ch04 Continuous availability and manageability.fm Draft Document for Review September 2, 2008 5:05 pm take resolution actions directly from the Tivoli Enterprise Portal or set up automated actions. Users have the ability to visualize the monitoring data in the Tivoli Enterprise Portal determine the current state of the AIX, LPAR, CEC, HMC and VIOS resources.
Draft Document for Review September 2, 2008 5:05 pm 4405ch04 Continuous availability and manageability.fm Centralized system management The POWER processor-based AIX and Linux cluster target scientific and technical computing, large-scale databases, and workload consolidation.
4405ch04 Continuous availability and manageability.
Draft Document for Review September 2, 2008 5:05 pm 4405bibl.fm Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this Redpaper. IBM Redbooks For information about ordering these publications, see “How to get Redbooks” on page 126. Note that some of the documents referenced here may be available in softcopy only.
4405bibl.fm Draft Document for Review September 2, 2008 5:05 pm Managing Devices, SA76-0107 Managing PCI Devices, SA76-0092 SAS RAID Controller Reference Guide, SA76-0112 Service Guide for HMC Models 7042-Cr4 and 7042-C06, SA76-0120 Online resources These Web sites are also relevant as further information sources: IBM Systems Information Center http://publib.boulder.ibm.com/infocenter/systems Support for IBM System p http://www.ibm.com/systems/support/p IBM System Planning Tool http://www.
Draft Document for Review May 28, 2009 2:58 pm Back cover IBM Power 570 Technical Overview and Introduction Expandable modular design supporting advanced mainframe class continuous availability enhancements PowerVM virtualization including the optional Enterprise edition POWER6 processor efficiency operating at state-of-the-art throughput levels This IBM® Redpaper is a comprehensive guide covering the IBM System p 570 UNIX® server.
