Sun™ Dual 10GbE SFP+ PCIe ExpressModule User’s Guide Sun Microsystems, Inc. www.sun.com Part No. 820-4979-10 August 2008, Revision A Submit comments about this document at: http://www.sun.
Copyright © 2008 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved. This distribution may include materials developed by third parties. Sun, Sun Microsystems, the Sun logo, SunSolve, Sun Fire, and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc., or its subsidiaries, in the U.S. and other countries. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd.
Contents Preface 1. xv Sun Dual 10GbE SFP+ PCIe ExpressModule Overview Shipping Kit Contents 1 ExpressModule Hardware Overview ExpressModule Features 1 2 Indicator Lights on the ExpressModule Hardware and Software Requirements Patch Requirements Patches and Updates 2.
Installing an Optical Transceiver ▼ To Install an Optical Transceiver Installing the ExpressModule ▼ 13 16 To Install the ExpressModule With the Power Off Verifying the Installation 4. 13 16 19 ▼ To Verify the Installation in a Linux System ▼ To Verify the Installation in a Microsoft Windows System Network Configuration 19 19 21 Configuring the Network Host Files for Booting Over the Gigabit Ethernet Network for Linux Systems 21 ▼ 5.
Physical Characteristics 37 Power and Environmental Requirements Index 38 39 Contents v
vi Sun Dual 10GbE SFP+ PCIe ExpressModule User’s Guide • August 2008
Regulatory Compliance Statements Your Sun product is marked to indicate its compliance class: • Federal Communications Commission (FCC) — USA • Industry Canada Equipment Standard for Digital Equipment (ICES-003) — Canada • Voluntary Control Council for Interference (VCCI) — Japan • Bureau of Standards Metrology and Inspection (BSMI) — Taiwan Please read the appropriate section that corresponds to the marking on your Sun product before attempting to install the product.
BSMI Class A Notice The following statement is applicable to products shipped to Taiwan and marked as Class A on the product compliance label. T33012 CCC Class A Notice The following statement is applicable to products shipped to China and marked with “Class A” on the product’s compliance label.
Safety Agency Compliance Statements Read this section before beginning any procedure. The following text provides safety precautions to follow when installing a Sun Microsystems product. Depending on the type of power switch your device has, one of the following symbols may be used: On – Applies AC power to the system. Off – Removes AC power from the system.
Conformité aux normes de sécurité Veuillez lire attentivement cette section avant de commencer. Ce texte traite des mesures de sécurité qu’il convient de prendre pour l’installation d’un produit Sun Microsystems. Mesures de sécurité Pour votre sécurité, nous vous recommandons de suivre scrupuleusement les mesures de sécurité ci-dessous lorsque vous installez votre matériel: ■ Suivez tous les avertissements et toutes les instructions inscrites sur le matériel.
Einhaltung sicherheitsbehördlicher Vorschriften Lesen Sie vor dem Ausführen von Arbeiten diesen Abschnitt. Im folgenden Text werden Sicherheitsvorkehrungen beschrieben, die Sie bei der Installation eines Sun Microsystems-Produkts beachten müssen. Aus– Unterbricht die Wechselstromzufuhr zum Gerät. Wartezustand – Der Ein-/Standby-Netzschalter befindet sich in der Standby-Position.
Normativas de seguridad Lea esta sección antes de realizar cualquier operación. En ella se explican las medidas de seguridad que debe tomar al instalar un producto de Sun Microsystems. Medidas de seguridad Apagado – Corta la alimentación de CA del sistema. Espera – El interruptor de encendido/espera está en la posición de espera. Para su protección, tome las medidas de seguridad siguientes durante la instalación del equipo: ■ Siga todos los avisos e instrucciones indicados en el equipo.
Declaration of Conformity Compliance Model Number: Product Family Name: 5945532 Sun Dual 10GbE SFP+ PCIe ExpressModule EMC USA—FCC Class A This equipment complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: 1. This equipment may not cause harmful interference. 2. This equipment must accept any interference that may cause undesired operation. Canada This Class A digital apparatus complies with Canadian ICES-003.
xiv Sun Dual 10GbE SFP+ PCIe ExpressModule User’s Guide • August 2008
Preface This user’s guide provides hardware and software installation instructions for the Sun™ Dual 10GbE SFP+ PCIe Express Module. This document also describes how to configure the driver software for the ixgbe driver for Linux operating systems and Microsoft Windows Server 2003. These instructions are designed for enterprise system administrators with experience installing network hardware and software.
Chapter 6 explains virtual local area networks (VLANs), and provides configuration instructions and examples. Appendix A lists the specifications for the Sun 10GbE XFP SR PCI Express Card. Typographic Conventions Typeface* Meaning Examples AaBbCc123 The names of commands, files, and directories; on-screen computer output Edit your.login file. Use ls -a to list all files. % You have mail.
Related Documentation The documents listed as online are available at: http://docs.sun.com/app/docs/prod/dual.sfp.
Sun Welcomes Your Comments Sun is interested in improving its documentation and welcomes your comments and suggestions. You can submit your comments by going to: http://www.sun.
CHAPTER 1 Sun Dual 10GbE SFP+ PCIe ExpressModule Overview This chapter describes the Sun 10GbE XFP SR PCI Express Card hardware and software, and includes the following sections: ■ “Shipping Kit Contents” on page 1 ■ “ExpressModule Hardware Overview” on page 1 ■ “Hardware and Software Requirements” on page 5 ■ “Patch Requirements” on page 5 ■ “Patches and Updates” on page 6 Shipping Kit Contents The carton in which your Sun 10GbE XFP SR PCI Express Card was shipped should contain the following i
The ExpressModule is optimized for Intel I/O Acceleration Technology (I/OAT), which is designed to optimize network I/O. The ExpressModule is a highperformance, highly integrated 10 Gigabit Ethernet LAN card with PCIe host interface and fiber LAN connectors on the optical modules. The product conforms to the IEEE 802.3 standard and supports standards for system manageability and power management.
■ Compatible with x8 standard and low-profile PCI Express slots ■ Support for most network operating systems (NOS) ■ Remote management support ■ Support for SFP+ form factor transceivers ■ RoHS compliant, lead-free technology ■ Intel PROSet Utility for Windows Device Manager Indicator Lights on the ExpressModule On the front panel of the ExpressModule (shown in FIGURE 1-2) next to each port, there are viewing holes for two lights dedicated to that port.
TABLE 1-1 Indicator Lights for Each Port on the ExpressModule Label Color Meaning ACT/LINK Green Activity 10G/1G Green 10GbE link 10G/1G Amber 1GbE link Two LEDs are on the ExpressModule next to the Attention switch. One LED emits green light, the LED other emits amber light. Each light can be on, off, or blinking. TABLE 1-2 explains the meaning of these lights.
Hardware and Software Requirements Before using the Sun 10GbE XFP SR PCI Express Card, ensure that your system meets the hardware and software requirements in TABLE 1-3. TABLE 1-3 Hardware and Software Requirements Requirements Supported Products Hardware Sun Sun Sun Sun Operating system Red Hat Enterprise Linux 5.1 (32-bit Red Hat Enterprise Linux 4.
Patches and Updates Check the Sun Update Connection to ensure that you have the latest recommended patch clusters and security patches. You can download the latest recommended patch clusters and security patches at: http://sunsolve.sun.com/pub-cgi/show.
CHAPTER 2 Installing and Setting Up the Device Driver Software This chapter explains how to install and remove the ixgbe driver on Linux and Microsoft Windows systems.
4. Select this option: Download drivers and software 5. Select Linux as the operating system. 6. Locate the following driver and select Download: Network Adapter Driver for 10 GbE PCI-E Based Network Connections for Linux 7. Review and accept the software license agreement. 8. Select this option: Download Network Adapter Driver for 10 GbE PCI-E Based Network Connections for Linux The download begins. The file named ixgbe-1.3.16.1.tar.gz is saved in the ~/Desktop directory of your system.
6. Verify that the ixgbe driver has been successfully installed with this lsmod command: # lsmod | grep ixgbe The output should be similar to the following: ixgbe 118052 0 7. Check the ixgbe driver version with this modinfo command: # modinfo ixgbe | grep ver For example, the output might be the following: filename: version: description: srcversion: vermagic: ▼ /lib/modules/2.6.18-53.el5/kernel/drivers/net/ixgbe/ixgbe.ko 1.3.16.
Downloading and Installing the Driver on a Microsoft Windows Platform If your system uses the Microsoft Windows Server 2003 operating system, perform the following procedures to download and install the device driver. ▼ To Download and Install the Driver on a Microsoft Windows Platform 1. Log in to your system. 2. With a browser, go to this location: http://support.intel.com/support/network/adapter/10gbe/srdualserverxpr/ 3. Select the following product: Intel® 10 Gigabit XF SR Dual Port ExpressModule 4.
9. Click on the following exe files to install the driver: ■ For a 64-bit driver: PROEM64T.exe ■ For a 32-bit driver: PRO2KXP.exe 10. Follow the instructions in the installation wizard. 11. If the Found New Hardware Wizard screen is displayed, click Cancel. The autorun automatically runs after you have extracted the files. ▼ To Remove the Driver From a Microsoft Windows Platform 1. From the Control Panel, double-click Add/Remove Programs. 2. Select Intel PRO Network Connections Drivers. 3.
12 Sun Dual 10GbE SFP+ PCIe ExpressModule User’s Guide • August 2008
CHAPTER 3 Installing the ExpressModule This chapter describes how to install the ExpressModule in your system and verify that it is recognized by the operating system.
2. Holding the optical transceiver by the edges, align the transceiver with the slot in the ExpressModule and slide the transceiver into the opening. 3. Applying even pressure at both corners of the transceiver, push the transceiver until it is firmly seated in the slot. 4. Push the handle closed to lock the optical transceiver in place.
5. Repeat Step 1 through Step 4 to install the second optical transceiver. Caution – If you pull the locking handle down when the optical transceiver is installed, remove the optical transceiver entirely and reinstall it. The handle operates an internal lock. Pulling the handle down can disconnect the optical transceiver, even though it might appear to be connected.
Installing the ExpressModule The following instructions describe the basic tasks required to install the ExpressModule. Refer to your system installation or service manual for detailed ExpressModule installation instructions. Note – To maintain proper cooling for the ExpressModule in your chassis, all ExpressModule slots must be filled with either operating ExpressModules or filler panels. ▼ To Install the ExpressModule With the Power Off 1. Halt and power off your system. 2.
6. Align the ExpressModule with the vacant ExpressModule slot (1 in the following figure). Ensure that the ExpressModule’s indicator lights on the front panel are facing toward you and that the ExpressModule ejector lever on the bottom is fully opened.
7. Slide the ExpressModule into the vacant ExpressModule chassis slot until the ejector lever engages and starts to close (2 in the preceding figure). Failure to align the ExpressModule correctly can result in damage with the ExpressModule’s internal connection to the chassis midplane. 8. Complete the installation by closing the ejector lever until the latch snaps into place (3 in the preceding figure). Caution – Do not use excessive force when installing the ExpressModule into the slot.
Verifying the Installation After you have installed the Sun 10GbE XFP SR PCI Express Card, perform the following tasks to verify the installation. ▼ To Verify the Installation in a Linux System ● Verify the new network interface instances corresponding to the Sun 10GbE XFP SR PCI Express Card: # ifconfig -a | grep eth eth3 eth4 ▼ Link encap:Ethernet Link encap:Ethernet HWaddr 00:1B:21:17:67:B0 HWaddr 00:1B:21:17:67:9B To Verify the Installation in a Microsoft Windows System 1. Click on Control Panel.
2. Click on Network Connection. The Ethernet adapter interfaces labeled as "Intel(R) 82598EB 10 Gigabit AF Dual Port Network Connection" will be displayed at the Network Connection window screen, if the driver is installed successfully. 3. To check the driver version, use the Administration Tool. The minimum Windows Server 2003 driver version is 1.2.22.0. 4. In the Administration Tool click Computer Management, Device Manager, and Network Adapter.
CHAPTER 4 Network Configuration This chapter describes how to edit the network host files after the card has been installed on your system. This chapter contains the following section: ■ “Configuring the Network Host Files for Booting Over the Gigabit Ethernet Network for Linux Systems” on page 21 Note – To do a PXE boot (or netboot) on a dual-port card, you must use the topmost port. That port is the logical Port 0, and it has the lowest MAC address.
4. Plug the Ethernet cable into the card port. 5. Power on the system. 6. Press the F2 key or the Control-E keys to go to the BIOS. 7. Check and ensure that the boot order of the network devices is higher than the hard drive. 8. Press the F10 key to save the boot configuration changes and exit. The system should reboot after saving the boot configuration. 9. Press the F12 key to install the OS from the network.
CHAPTER 5 Configuring the Driver Parameters The ixgbe device driver controls the Sun 10GbE SFP+ PCIe ExpressModule interfaces. You can manually set the ixgbe device driver parameters to customize each device in your system. This chapter lists the available device driver parameters and describes how you can set these parameters.
Driver Parameters for Linux Systems TABLE 5-1 lists the tunable ixgbe driver parameters for Linux operating systems and describes their function. TABLE 5-1 Tunable ixgbe Driver Parameters for Linux Operating Systems Keyword Valid Range Default Value Description FlowControl 0 to 3 (0=none, 1=Rx only, 2=Tx only, 3=Rx and Tx) Read from the EEPROM. If EEPROM is not detected, default is 3. This parameter controls the automatic generation (Tx) and response (Rx) to Ethernet PAUSE frames.
Setting ixgbe Driver Parameters in Linux Systems ▼ To Configure Jumbo Frames Jumbo Frames can support up to 15000 MTU. The default value is 1500 MTU. ● Use the ifconfig command to increase MTUs to allow transmission of Jumbo Frames. For example, where the IP address for eth7 is 192.1.1.200, the following command increases MTUs to the maximum: # ifconfig eth7 192.1.1.
26 Sun Dual 10GbE SFP+ PCIe ExpressModule User’s Guide • August 2008
CHAPTER 6 Configuring VLANs This chapter describes how to configure virtual local area networks (VLANs). This chapter contains the following sections: ■ “VLAN Overview” on page 27 ■ “Configuring VLANs” on page 30 ■ “Configuring Bonding for Multiple Interfaces” on page 32 Note – If you change any of the VLAN configuration parameters, you must reboot the system before the changes take effect. If you make changes and do not reboot, you might experience configuration problems.
VLANs are commonly used to separate groups of network users into manageable broadcast domains, to create logical segmentation of workgroups, and to enforce security policies among each logical segment. Each defined VLAN behaves as its own separate network, with its traffic and broadcasts isolated from the others, increasing the bandwidth efficiency within each logical group.
The physical LAN network consists of a switch, two servers, and five clients. The LAN is logically organized into three different VLANs, each representing a different IP subnet. ■ VLAN 1 is an IP subnet consisting of the Main Server, Client 3, and Client 5. This represents an engineering group. ■ VLAN 2 includes the Main Server, Clients 1 and 2 by means of a shared media segment, and Client 5. This is a software development group. ■ VLAN 3 includes the Main Server, the Accounting Server, and Client 4.
Configuring VLANs VLANs can be created according to various criteria, but each VLAN must be assigned a VLAN tag or VLAN ID (VID). The VID is a 12-bit identifier between 1 and 4094 that identifies a unique VLAN. For each network interface (ixgbe0, ixgbe1, ixgbe2, and so on), 4094 possible VLAN IDs can be selected for each port. Tagging an Ethernet frame requires the addition of a tag header to the frame. The header is inserted immediately following the destination MAC address and the source MAC address.
2. Plumb the Sun 10GbE XFP SR PCI Express Card interface: # ifconfig eth6 xxx.xxx.xx.xxx up where xxx.xxx.xx.xxx = the IP address of the interface. 3. Add the VLAN instance (VID). For example: # vconfig add eth6 5 where eth6 is the interface and 5 is the VID. Note – In Linux systems, you can use any single digit as the VID. 4. Configure the ixgbe VLAN (eth2 in this example): # ifconfig eth6.5 xxx.xxx.xx.xxx up where xxx.xxx.xx.xxx = the IP address of the interface.
11. Click Subnet Mask. The value 255.255.255.0 is displayed. 12. Click OK. 13. Repeat Step 3 through Step 10 until all the network ports are VLAN configured. Note – Ensure that the firewall is configured to allow VLAN traffic. Otherwise, the VLAN might not operate properly. Configuring Bonding for Multiple Interfaces ▼ To Configure Bonding for Multiple ixgbe Interfaces 1.
In this example, two bonds (bond0 and bond1) are configured: # # # # # modprobe bonding mode=balance-rr miimon=100 max_bonds=2 ifconfig bond0 192.2.2.15 netmask 255.255.255.0 broadcast 192.2.2.255 ifenslave bond0 eth4 eth5 ifconfig bond1 193.2.2.15 netmask 255.255.255.0 broadcast 193.2.2.255 ifenslave bond1 eth6 eth7 Refer to Linux documentation for more information.
34 Sun Dual 10GbE SFP+ PCIe ExpressModule User’s Guide • August 2008
APPENDIX A Sun Dual 10GbE SFP+ PCIe ExpressModule Specifications This appendix lists the specifications for the Sun 10GbE XFP SR PCI Express Card.
Connectors FIGURE A-1 shows the connectors for the Sun 10GbE XFP SR PCI Express Card.
Technical Features TABLE A-1 Performance Specifications Feature Specification Data rate supported per port 10 Gigabit (Gb) Bus type PCI Express 2.0 Bus width x8 lane PCI Express Conforms to Ethernet Standard 802.3 Boot ROM 2 Mbit SPI Flash Electromagnetic Interference (EMI) FCC Class A Physical Characteristics TABLE A-2 Physical Characteristics Dimension Measurement Length 6.62 in. (168.2 mm) Width 4.25 in.
Power and Environmental Requirements TABLE A-3 38 Card Power Requirements Specification Measurement Typical power consumption 14W (1.
Index B J bonding for multiple interfaces, 32 booting over the network on Linux systems, 21 Jumbo Frames, 25 N networking interfaces, 23 C configuring bonding, 32 configuring the network host files, 21 configuring VLANs, 30 in Linux environments, 30 connectors, 36 P patches recommended patch clusters, 6 recommended security, 6 patches and updates, 6 power requirements, 38 D driver parameters tunable for Linux OS, 24 R E removing the ixgbe driver from Linux platforms, 9 from Windows platforms, 11
40 Sun Dual 10GbE SFP+ PCIe ExpressModule User’s Guide • August 2008
