EXPRESS5800/A1160 Technical Overview
Proprietary Notice and Liability Disclaimer The information disclosed in this document, including all designs and related materials, is the valuable property of NEC Corporation of America, Inc. and/or its licensors. NEC Corporation of America and/or its licensors, as appropriate, reserve all patent, copyright and other proprietary rights to this document, including all design, manufacturing, reproduction, use, and sales rights thereto, except to the extent said rights are expressly granted to others.
Contents Section 1 Introduction ..................................................................1-1 Documentation Updates .................................................................................. 1-1 Scalable Symmetric Multiprocessing Servers .................................................. 1-1 Operating Systems .......................................................................................... 1-2 Virtual Machine Monitors ................................................................
Contents System Components........................................................................................ 4-1 Network Components ...................................................................................... 4-2 Management Architecture ................................................................................ 4-4 Management Board ......................................................................................... 4-5 Remote Console Interfaces ....................................
Figures Figures Figure 1-1 Express5800/A1160 Cell ............................................................................. 1-1 Figure 2-1 Basic Cell..................................................................................................... 2-2 Figure 2-2 Processor Socket Location on Processor Board ......................................... 2-3 Figure 2-3 Node Controller ........................................................................................... 2-5 Figure 2-4 Memory Boards ..
Tables Tables Table 2-1 DIMM Capacity and Cell Memory ................................................................. 2-7 Table 2-2 Device Names on the EXPRESSSCOPE® Monitor LCD ........................... 2-18 Table 3-1 Relation of Interconnect Port and Cable End to Cell ID ................................ 3-3 Table 3-2 Minimum and Maximum Configurations ........................................................ 3-7 Table A-1 Component on the Front of the Cell .............................................
Using This Guide Using This Guide This guide contains information how to operate and administrate Express5800/A1160 server. Who Should Use This Guide This guide is intended for system administrators and operation personnel who are using Express5800/A1160 server. Symbols and Conventions This guide uses the following text conventions and graphic symbols.
Related Documents The NECCare Guide contains information about NEC’s warranty and server registration.
Safety Notices Safety Notices WARNING To avoid a risk of injuries, maintenance procedures require trained technical personnel. In maintenance procedures with voltages of 42.4V peak or 60Vdc or more, take safety measures, such as wearing insulated rubber gloves. Performing work without these measures may cause electric shock. In an emergency, such as a dangerous event that requires turning off the power supply, turn off the breaker at the rear of the server.
Safety Notices temperature of 89.6°F. Reduced air Flow – Installation of the equipment in a rack should be such that the amount of air flow required for safe operation of the equipment is not compromised. To prevent fires, and damage to rack equipment and supply wiring, make sure that the rated load of the power branch circuit is not exceeded. Equipment nameplate ratings should be used when addressing this concern.
WARNING Take care not to short live components with conductive tools, such as an adjustable wrench. To prevent shock, take care not to drop or leave conductive parts, such as a screw, in the server when servicing the system. Be careful when accessing a fan or rotating parts to avoid cutting your hand or fingers. Safety inspections – When servicing the system, check equipment that can cause harm due to deterioration, and if necessary, replace the part. Safety Notices for Users Outside of the U.S.A.
Section 1 Introduction This overview provides a detailed introduction to Express5800/A1160 systems and the technologies on which they are based. It describes features that might be available to customers in a series of hardware and software releases. For information about the availability of any feature, contact your sales representative. Documentation Updates This document contains all the information that was available at the time of publication.
Operating Systems dual inline memory module (DIMM) sockets that support 2-GB or 4-GB fully buffered DIMMs, and all the other components it needs to function as a self-contained computer system. Modular Expansion The cell-based building-block design allows Express5800/A1160 systems to grow as business needs evolve. By connecting cells together, you can expand an Express5800 Model 7600R system from 4 to 16 processor sockets. Memory expands from 2 to 128 DIMM sockets.
Virtual Machine Monitors y Windows Server 2008, Enterprise x64 Edition y Windows Server 2008, Datacenter x64 Edition Note: Partitions running Windows Server 2003, 32-bit Enterprise Edition, or Windows Server 2003, 32-bit Datacenter Edition, are restricted to a maximum of two cells. Linux Operating Systems Express5800/A1160 systems support the following Linux operating systems: y Red Hat Enterprise Linux Advanced Platform 5.
Section 2 Cell Component Express5800/A1160 Cell In Express5800/A1160 systems, hardware components are organized into cells. The main components of a cell are • Main logic module that hosts the processors, processor voltage regulator modules, and node controller. (The node controller interconnects the components of a cell and provides the connection among the cells in a multiple-cell system.) • Memory board that contains the DIMM sockets for the fully buffered DIMMs.
Processors Figure 2-1 Basic Cell Processors Within each cell, the processor board in the main logic module hosts the processors. The processor board contains the processor sockets, processor voltage regulator modules, node controller, and the scalability ports, which are the ports that enable you to connect up to four cells. Processor Sockets Each cell contains four processor sockets. Four direct front-side buses provide the connections for the four processor sockets.
Processors Figure 2-2 shows the location of the processor sockets (along with the processor number) on the processor board. Figure 2-2 Processor Socket Location on Processor Board Supported Processors Express5800/A1160 systems support the following Intel Xeon processor 7400 series (Dunnington) processors: • Model E7440 (2.4 GHz with 2 x 3-MB L2 cache, 16-MB L3 cache, quad core, 1066 MHz front-side bus) • Model X7460 (2.
Processors steppings. A stepping is a version of the processor that is associated with a set of fixes and errata. When mixing processors or processor steppings, observe the following rules: • Each processor board (or each cell) supports only one processor family. • All processors in a partition must be the same frequency and have the same amount of cache.
Memory Figure 2-3 Node Controller Memory Express5800/A1160 systems use a directory-based cache-coherent memory system. The directory-based memory tracks ownership of data. The cache coherency protocols force the processor cache components and main storage components to y Keep track of all copies of a cache line. y Determine which processor has permission to update an instance of a cache line. y Mark other copies of the cache line as "invalid" when an update occurs. Memory Board.
Memory Each channel pair operates in lockstep; that is, a single cache line is stored across two DIMMs with each DIMM on a different channel of the lockstep pair. y Eight fully buffered DIMM slots for each channel. A cell with two memory boards contains 32 DIMM slots. y Double Data Rate 2 533-MHz and 667-MHz fully buffered DIMMs (single, dual, or quad rank). y DIMM capacities of 2 GB or 4 GB. y Hot-add memory (requires operating system support).
Memory Figure 2-5 DIMM Numbering DIMM Capacity and Cell Memory Table 2-1 lists the minimum and maximum memory in the cell if DIMMs of a specific capacity are used. Table 2-1 DIMM Capacity and Cell Memory DIMM Capacity Minimum Memory in Cell Maximum Memory in Cell 2 GB 4 GB 64 GB 4 GB 8 GB 128 GB Check with your sales representative for the availability of specific DIMMs. Rules for Memory Expansion Note the following rules when expanding system memory: y DIMMs must be installed in pairs.
I/O Subsystem Indicators on the Memory Board The memory board contains a green LED to indicate that the board is in service and an amber LED to indicate that the board needs attention. Do not remove a memory board while the green LED is lit. Dynamic DIMM Sparing Dynamic DIMM sparing automatically substitutes a spare DIMM in place of an operational DIMM before a high rate of correctable errors leads to an uncorrectable error and loss of data.
I/O Subsystem Each slot has its own power and attention indicators (LEDs). y Internal slot for PCIe Serial Attached SCSI (SAS) RAID controller Figure 2-6 shows the I/O riser board. Figure 2-6 I/O Riser Board PCIe Card Carrier The PCIe card carrier enables an administrator or technician to insert or remove the PCIe cards from the rear of the cell. Typically, a PCIe card plugs into a motherboard vertically. Vertical insertion requires access to the cards from the top of the cell.
I/O Subsystem Figure 2-7 PCIe Card Carrier Figure 2-8 shows the location of the external card slots on the back of the cell.
I/O Subsystem y Partition universal serial bus (USB) ports — Two USB 2.0 ports on the front and two USB 2.0 ports on the back of the cell y Partition serial port y Partition LAN ports — Two 1-Gigabit Ethernet (LAN) ports In addition, the cell includes the following ports on the management board: y Maintenance LAN port (10/100 Ethernet port) y USB management port y Management serial port Figure 2-9 shows the location of the ports on the back of the cell.
Power Power Each cell is independently powered. The power system is fully distributed and consists of the following components. AC Connections The AC input for each cell is two 15-ampere line cords. The line cords are connected to the power strips mounted in the rear of the cell. AC/DC Power Supplies The AC/DC power supplies take the AC input power and produce the DC power used by the cell components. Each cell contains two hot-pluggable AC/DC power supplies.
Cooling The main logic module contains the single master power controller for the cell. The master power controller y Controls and monitors the power supplies, and monitors the voltages on all cell components y Responds when someone presses the power button (to turn the power on or off) y Monitors and controls the fans y Monitors thermal sensors y Communicates with the management system 200-Volt Redundancy and AC Source Options A second AC/DC 12-volt power supply provides n+1 DC redundancy.
Cooling Cooling Each cell is cooled by two fan modules. Each fan module contains two 12-volt high-performance fans that provide the airflow required to cool the cell components and processors. The fan modules are mounted in the front of the cell. Note: The power supplies within the cell are self-cooled. The fans have four speeds. The system automatically adjusts fan speed based on the ambient and system component temperatures or under certain failure or maintenance conditions.
Peripherals Figure 2-11 Fan Modules Peripherals The cell supports the following peripherals through the drive tray: y Six 2.5-inch Serial Attached SCSI (SAS) disk drives with one of the following capacities: y 73 GB (15000 RPM) y 146 GB (10000 RPM) y DVD drive (DVD-RW) The drive tray also hosts the two USB ports on the front of the cell. Figure 2-12 illustrates the peripherals on the drive tray.
Peripherals Figure 2-12 Peripherals on the Drive Tray Floppy disk drive support, if required, can be supplied through a floppy disk drive attached to the USB port. The BIOS also supports booting the system from any device attached to the USB port. A keyboard and mouse can also be attached to the USB ports. In a system with multiple partitions, each partition can contain only one keyboard and mouse.
System Clocks System Clocks A clock synthesizer on the main logic module is the master clock source. This source provides a number of output clocks that are further distributed to the system components as needed. The clock distribution system is self-contained within each cell. Systems with multiple cells do not provide redundant system clocks. EXPRESSSCOPE® Monitor Each cell contains an EXPRESSSCOPE® Monitor on the front of the cell.
Fault Messages on the EXPRESSSCOPE® Monitor LCD Figure 2-13 EXPRESSSCOPE® Monitor Fault Messages on the EXPRESSSCOPE® Monitor LCD If a hardware problem occurs, a message is displayed on the EXPRESSSCOPE® Monitor LCD on the front of the cell. The display identifies the failed unit, sometimes with abbreviated language.
Fault Messages on the EXPRESSSCOPE® Monitor LCD Full Name fan module EXPRESSSCOPE® Monitor LCD Name Fan_Modx Additional Display Details x= Unit Type (FRU/CRU) CRU 0 (left module) 1 (right module) interconnect cable Intercon_Cablex x= FRU 0 (right) 1 (middle) 2 (left) I/O riser board IOR_Brd PCI expansion card IO_Slotx_y_z FRU x = cell card slot 0 through 5 CRU y = expansion rack 1 or 2 z = expansion rack slot 1 through 5 voltage regulator module IPVRMx main logic module MainLogic_Module m
Inbuilt Virtualization Full Name PCIe carrier EXPRESSSCOPE® Monitor LCD Name Additional Display Details n/a Unit Type (FRU/CRU) CRU Inbuilt Virtualization Inbuilt virtualization is an optional capability for Express5800/A1160 systems. This capability enables a partition to boot with VMware Infrastructure 3. The partition contains a USB drive with a preloaded copy of the embedded version of VMware ESX 3.5. The USB drive is attached to the compatibility hardware of the partition.
Section 3 System Configuration This section describes the packaging of Express5800/A1160 systems and the different configuration options. Cell Interconnect and Multiple Cell Systems A four-processor socket cell is capable of containing all the hardware it needs to boot as a server. To create a larger system, a cell can be connected together with up to three other cells using interconnect cables. Interconnect Cables. The interconnect cables connect to the ports on the rear of the cell.
Cell Interconnect and Multiple Cell Systems A cell can be connected together with up to three other cells to create a multiple-cell system. > shows a system made up of two cells (eight processor sockets). The two cells are connected together with two interconnect cables. > shows a system made up of four cells (16 processor sockets). Systems with four cells require six interconnect cables. Systems with three cells are also possible.
Cell Identification Cell Identification Each cell in a system has a unique physical cell identifier-cell 0, cell 1, cell 2, or cell 3. The management firmware uses the cell identifier to enable the cells within the system to communicate with each other. The cell identifier is established by the interconnect cables. The cell identifier is based on which ends of the interconnect cables are connected to the interconnect ports on the back of the cell. Each interconnect cable has an "A" and a "B" end.
Cabinets Figure 3-4 System Serial Number Label Cabinets Express5800/A1160 systems can be installed in any standard 19-inch-wide cabinet. You can either order a cabinet from NEC or provide your own. See the System Planning Guide for the requirements for customer-provided cabinets. Optionally, each system can have a directly connected keyboard, monitor, and mouse to serve as a system console. Figure 3-5 shows the configuration of cells and other components in a cabinet.
Partitions Figure 3-5 Cabinet Configuration Partitions You can configure one or more cells as a partition to support an instance of an operating system or a virtual machine monitor. A partition is a logical computer in the system. You use partitions to run user applications that perform work to support your business. Each cell in the system can support one partition. Systems with four cells can support one to four partitions. See the User's Guide for the partitioning rules.
Redundant Configurations hardware resources among cluster nodes in response to planned and unplanned events. This reallocation is virtually impossible to accomplish in a traditional cluster. Clustering Software Express5800 systems support various clustering paradigms, including shared disk clusters, and shared nothing (only interconnected) clusters. Redundant Configurations Redundant configurations ensure that Express5800 systems do not have a single point of failure.
Express5800/A1160 Configuration Summary See the User's Guide for additional information on the customer-replaceable components. Express5800/A1160 Configuration Summary Table 3-2 summarizes the configuration options for Express5800/A1160 systems.
Section 4 Server Management This section provides an overview of the hardware and software components used to manage Express5800/A1160 systems. System Components The following illustration shows how the components are connected in a typical system environment. Your system can include the following components.
Components on the Front of a Cell Cells A cell contains the physical resources of the system: instruction processors, memory, input/output devices, peripheral storage devices, management board, fans, power supplies, EXPRESSSCOPE® Monitor, and so on. Each system comprises one to four cells. Maintenance LAN (MLAN) The maintenance LAN (MLAN) connects the internal components of the system.
Network Components Windows Domain Controller The domain controller typically acts as the Domain Name System (DNS), Windows Internet Name Service (WINS), and Dynamic Host Configuration Protocol (DHCP) server. Note: DNS and WINS servers must be made secure in accordance with local security policy guidelines. Microsoft recommends that production applications such as Server Management software not be installed on domain controllers.
Components on the Front of a Cell operate the partition. Management Architecture Management of Express5800/A1160 systems is based on a modular, distributed, and scalable architecture. This architecture distributes the processing load among various processing engines and grows as the size of the system increases. Intelligent Platform Management Interface Express5800/A1160 systems implement the message-based Intelligent Platform Management Interface (IPMI) version 2.
Management Board Figure 4-1 Management Architecture Management Board The management board in each cell is the main hardware component of server management. Figure 4-2 illustrates the management board.
Components on the Front of a Cell Figure 4-2 Management Board The following paragraphs describe the main components on the management board in more detail. Service Processor The Service Processor is a management controller.
Remote Console Interfaces The firmware hubs in each cell store the BIOS. The firmware hub and the TPM are attached to the low pin count (LPC) bus, an internal bus that connects low-bandwidth devices to the CPU. Trusted Platform Module (TPM) "Trusted Platform Module (TPM)" in Section 2 describes the TPM.
Components on the Front of a Cell See the User's Guide for more information about the remote console interfaces. BIOS The BIOS is bootstrapping software that initializes and configures the hardware (processors, memory, I/O, and so on) in the partition. Running in the partition, the BIOS provides many configuration, setup, diagnostic, and recovery functions. In Express5800/A1160 systems, the BIOS is stored in the firmware hub in each cell.
Management Software Components Figure 4-3 Server Management Hardware Components Management Software Components The Server Management software includes the following components. ESMPRO Manager and Agent Server Management software includes ESMPRO Manager and ESMPRO Agent software. ESMPRO Manager enables a system administrator to manage a network by monitoring system hardware and software configurations, system failures, and system performance.
Components on the Front of a Cell long-term and short-term performance, monitor usage, create graphs to record trends, and check failure rates. System administrators can use the information collected to create more efficient data routing procedures and optimize server and partition usage. ESMPRO Manager is installed on the management server along with other Server Management software. ESMPRO Agent is installed on servers and partitions running Windows, Linux, and VMware ESX Server.
Appendix A Cell Components The following text identifies the main components, ports, and connectors that are visible on the front and back of a cell. Components on the Front of a Cell Figure A-1 identifies the components that are visible on the front of the cell. Table A-1 describes each component in more detail. Figure A-1 Front of the Cell Table A-1 Component on the Front of the Cell Component Description Fan module Provides the airflow needed to cool the cell.
Components on the Front of a Cell Component Description EXPRESSSCOPE® Monitor A control and display interface for the cell and the partition containing the cell. LED fan status indicators Provide fan status with one LED light on each fan module. A green light indicates normal operations. LCD Displays status information and provides a menu of management tasks for the cell on a 4-line by 20-character liquid crystal display (LCD) panel. Refer to the User’s Guide for information on using the LCD.
Components on the Rear of a Cell Figure A-1 Rear of the Cell Table A-2 Component on the Rear of the Cell Component Description PCIe carrier/cards Six hot-plug switched PCIe 8x card slots. Slots 1 and 4 can support full-length cards. Each PCIe card is housed in an individual carrier. Cell ID An LED that is lit when the ID button on the EXPRESSSCOPE® Monitor is pressed. Press the cell ID button on the rear of the cell to turn off the light. Partition USB ports Two universal serial bus (USB) 2.0 ports.
Components on the Front of a Cell Component A-4 Description LED status indicators Provides status for the cell. The top light indicates power status. The bottom light indicates cell status. See the EXPRESSSCOPE® Monitor LCD or the remote console interface for additional information on the power or cell status. Power supplies Two power supplies that provide n+1 redundancy when used in high-voltage configurations. Each power supply has its own AC power cord.
Note: Consumers are cautioned that Product performance is affected by system configuration, software, the application, Customer data, and operator control of the system, among other factors. While NEC Corporation of America products is considered to be compatible with many systems, the specific functional implementation by the Customers of the product may vary.
456-01810-000
