User Guide NetXtreme II September 2013 Broadcom NetXtreme II ® N e t wo r k A d a p t e r U s e r G ui de • Introduction • Functionality and Features • Teaming • Virtual LANs (VLANs) • Manageability • Installing the Hardware • Installing the Driver Software • Broadcom Boot Agent Driver Software • NDIS2 Driver Software • Linux Driver Software • Solaris Driver Software • VMware Driver Software • Windows Driver Software • Installing Management Applications • Using iSCSI • Advanc
NetXtreme II User Guide September 2013 This document is protected by copyright and is distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this document may be reproduced in any form by any means without prior written authorization of Broadcom Corporation.
User Guide NetXtreme II September 2013 F un cti on al i t y an d Fe atu res : B ro adc om Ne tXtr eme II ® Network Adapter Us er Guide • Functional Description • Features FUNCTIONAL DESCRIPTION The Broadcom NetXtreme II adapter is a new class of Gigabit Ethernet (GbE) and 10 GbE converged network interface controller (C-NIC) that can simultaneously perform accelerated data networking and storage networking on a standard Ethernet network.
NetXtreme II User Guide September 2013 FEATURES The following is a list of the Broadcom NetXtreme II adapter features. Some features may not be available on all adapters. • TCP Offload Engine (TOE) • Internet Small Computer Systems Interface (iSCSI) offload • Fibre Channel over Ethernet (FCoE) • NIC Partitioning • Data Center Bridging (DCB) • • • • • Enhanced Transmission Selection (ETS; IEEE 802.1Qaz) • Priority-based Flow Control (PFC; IEEE 802.
User Guide NetXtreme II September 2013 • Logical Link Control (IEEE Std 802.2) • Layer-2 Priority Encoding (IEEE Std 802.
NetXtreme II User Guide September 2013 The Broadcom NetXtreme II adapter targets best-system performance, maintains system flexibility to changes, and supports current and future OS convergence and integration. Therefore, the adapter's iSCSI offload architecture is unique as evident by the split between hardware and host processing. NOTES: The iSCSI offload feature is not available for all Broadcom network adapters.
User Guide NetXtreme II September 2013 BROADCOM ADVANCED CONTROL SUITE Broadcom Advanced Control Suite (BACS) is an integrated utility that provides useful information about each network adapter that is installed in your system. The BACS utility also enables you to perform detailed tests, diagnostics, and analyses on each adapter, as well as to modify property values and view traffic statistics for each adapter.
NetXtreme II User Guide September 2013 C on f ig u r i ng Te a m i ng in Wi n dow s S e r v e r : Broadcom NetXtreme II ® Ne twork Adapter Us er G ui de • Broadcom Advanced Server Program Overview • Load Balancing and Fault Tolerance NOTE: This chapter describes teaming for adapters in Windows Server systems. For more information on a similar technology on Linux operating systems (called “Channel Bonding”), refer to your operating system documentation.
User Guide NetXtreme II September 2013 LOAD BALANCING AND FAULT TOLERANCE Teaming provides traffic load balancing and fault tolerance (redundant adapter operation in the event that a network connection fails). When multiple Gigabit Ethernet network adapters are installed in the same system, they can be grouped into teams, creating a virtual adapter.
NetXtreme II User Guide September 2013 LINK AGGREGATION (802.3AD) This mode supports link aggregation and conforms to the IEEE 802.3ad (LACP) specification. Configuration software allows you to dynamically configure which adapters you want to participate in a given team. If the link partner is not correctly configured for 802.3ad link configuration, errors are detected and noted. With this mode, all adapters in the team are configured to receive packets for the same MAC address.
User Guide NetXtreme II September 2013 SLB (AUTO-FALLBACK DISABLE) The SLB (Auto-Fallback Disable) type of team is identical to the Smart Load Balancing and Failover type of team, with the following exception—when the standby member is active, if a primary member comes back on line, the team continues using the standby member, rather than switching back to the primary member. All primary interfaces in a team participate in load-balancing operations by sending and receiving a portion of the total traffic.
NetXtreme II User Guide September 2013 determination when failing over. LiveLink mitigates such issues. TEAMING AND LARGE SEND OFFLOAD/CHECKSUM OFFLOAD SUPPORT Large Send Offload (LSO) and Checksum Offload are enabled for a team only when all of the members support and are configured for the feature.
User Guide NetXtreme II September 2013 Broadcom Te aming Ser vices: Broadcom Ne tXtr eme II ® Network Adapter Us er Guide NOTE: This chapter describes teaming for adapters in Windows Server systems. For more information on a similar technologies on other operating systems (for example, Linux Channel Bonding), refer to your operating system documentation.
NetXtreme II User Guide September 2013 GLOSSARY Table 1: Glossary Item Definition ARP Address Resolution Protocol BACS Broadcom Advanced Control Suite BASP Broadcom Advanced Server Program (intermediate driver) DNS domain name service G-ARP Gratuitous Address Resolution Protocol Generic Trunking (FEC/GEC)/802.3ad-Draft Static Switch-dependent load balancing and failover type of team in which the intermediate driver manages outgoing traffic and the switch manages incoming traffic.
User Guide NetXtreme II September 2013 Table 1: Glossary (Cont.
NetXtreme II User Guide September 2013 TEAMING CONCEPTS • Network Addressing • Teaming and Network Addresses • Description of Teaming Types • TOE Teaming The concept of grouping multiple physical devices to provide fault tolerance and load balancing is not new. It has been around for years. Storage devices use RAID technology to group individual hard drives. Switch ports can be grouped together using technologies such as Cisco Gigabit EtherChannel, IEEE 802.
User Guide NetXtreme II September 2013 team must use a unique MAC address to be IEEE compliant. It is important to note that ARP cache entries are not learned from received frames, but only from ARP requests and ARP replies. Description of Teaming Types • Smart Load Balancing and Failover • Generic Trunking • Link Aggregation (IEEE 802.
NetXtreme II User Guide September 2013 Smart Load Balancing enables both transmit and receive load balancing based on the Layer 3/Layer 4 IP address and TCP/ UDP port number. In other words, the load balancing is not done at a byte or frame level but on a TCP/UDP session basis. This methodology is required to maintain in-order delivery of frames that belong to the same socket conversation. Load balancing is supported on 2 to 8 ports.
User Guide NetXtreme II September 2013 different default gateways on the host nodes) across multiple routers in HSRP groups, it always points to the primary MAC address of the team. Generic Trunking Generic Trunking is a switch-assisted teaming mode and requires configuring ports at both ends of the link: server interfaces and switch ports. This is often referred to as Cisco Fast EtherChannel or Gigabit EtherChannel.
NetXtreme II User Guide September 2013 If any primary adapter assigned to a team is disabled, the team functions as a Smart Load Balancing and Failover type of team in which auto-fallback occurs. TOE Teaming All four basic teaming modes support failover of traffic from a failed adapter to other working adapters. All four teaming modes also support bidirectional load-balancing of TCP/IP traffic.
User Guide NetXtreme II September 2013 interrupt processing. The intermediate driver fits between the miniport driver and the protocol layer multiplexing several miniport driver instances, and creating a virtual adapter that looks like a single adapter to the NDIS layer. NDIS provides a set of library functions to enable the communications between either miniport drivers or intermediate drivers and the protocol stack. The protocol stack implements IP, IPX and ARP.
NetXtreme II User Guide September 2013 HARDWARE REQUIREMENTS • Repeater Hub • Switching Hub • Router The various teaming modes described in this document place certain restrictions on the networking equipment used to connect clients to teamed systems. Each type of network interconnect technology has an effect on teaming as described in the following sections. Repeater Hub A Repeater Hub allows a network administrator to extend an Ethernet network beyond the limits of an individual segment.
User Guide NetXtreme II September 2013 TEAMING SUPPORT BY PROCESSOR All team types are supported by the IA-32, AMD-64, and EM64T processors. CONFIGURING TEAMING The Broadcom Advanced Control Suite utility is used to configure teaming in the supported operating system environments. The Broadcom Advanced Control Suite (BACS) utility is designed to run on 32-bit and 64-bit Windows family of operating systems. BACS is used to configure load balancing and fault tolerance teaming, and VLANs.
NetXtreme II User Guide September 2013 Table 4: Comparison of Team Types (Cont.) Switch-Independent Dynamic Link Aggregation (IEEE 802.
User Guide NetXtreme II September 2013 SELECTING A TEAM TYPE The following flow chart provides the decision flow when planning for Layer 2 teaming. For TOE teaming, only Smart Load Balancing™ and Failover type team is supported. The primary rationale for teaming is the need for additional network bandwidth and fault tolerance. Teaming offers link aggregation and fault tolerance to meet both of these requirements.
NetXtreme II User Guide September 2013 TEAMING MECHANISMS • Architecture • Types of Teams • Attributes of the Features Associated with Each Type of Team • Speeds Supported for Each Type of Team ARCHITECTURE The Broadcom Advanced Server Program is implemented as an NDIS intermediate driver (see Figure 2). It operates below protocol stacks such as TCP/IP and IPX and appears as a virtual adapter. This virtual adapter inherits the MAC Address of the first port initialized in the team.
User Guide NetXtreme II September 2013 Figure 2: Intermediate Driver Outbound Traffic Flow The Broadcom Intermediate Driver manages the outbound traffic flow for all teaming modes. For outbound traffic, every packet is first classified into a flow, and then distributed to the selected physical adapter for transmission. The flow classification involves an efficient hash computation over known protocol fields. The resulting hash value is used to index into an Outbound Flow Hash Table.
NetXtreme II User Guide September 2013 The outbound TCP and UDP packets are classified using Layer 3 and Layer 4 header information. This scheme improves the load distributions for popular Internet protocol services using well-known ports such as HTTP and FTP. Therefore, BASP performs load balancing on a TCP session basis and not on a packet-by-packet basis. In the Outbound Flow Hash Entries, statistics counters are also updated after classification.
User Guide NetXtreme II September 2013 ICMP ------> PhysAdapter1 ------> 10.0.0.1 If the server also sends an TCP and UDP flow to the same 10.0.0.1 address, they can be on the same physical adapter as IGMP and ICMP, or on completely different physical adapters from ICMP and IGMP. The stream may look like this: IGMP ------> PhysAdapter1 ------> 10.0.0.1 ICMP ------> PhysAdapter1 ------> 10.0.0.1 TCP------> PhysAdapter1 ------> 10.0.0.1 UDP------> PhysAdatper1 ------> 10.0.0.
NetXtreme II User Guide September 2013 The BASP intermediate driver continually monitors the physical ports in a team for link loss. In the event of link loss on any port, traffic is automatically diverted to other ports in the team. The SLB teaming mode supports switch fault tolerance by allowing teaming across different switches- provided the switches are on the same physical network or broadcast domain.
User Guide NetXtreme II September 2013 Switch-Dependent Generic Static Trunking This mode supports a variety of environments where the adapter link partners are statically configured to support a proprietary trunking mechanism. This mode could be used to support Lucent’s Open Trunk, Cisco’s Fast EtherChannel (FEC), and Cisco’s Gigabit EtherChannel (GEC).
NetXtreme II User Guide September 2013 Dynamic Trunking (IEEE 802.3ad Link Aggregation) This mode supports link aggregation through static and dynamic configuration via the Link Aggregation Control Protocol (LACP). With this mode, all adapters in the team are configured to receive packets for the same MAC address. The MAC address of the first adapter in the team is used and cannot be substituted for a different MAC address.
User Guide NetXtreme II September 2013 ATTRIBUTES OF THE FEATURES ASSOCIATED WITH EACH TYPE OF TEAM The attributes of the features associated with each type of team are summarized in Table 5.
NetXtreme II User Guide September 2013 Table 5: Attributes (Cont.) Feature Attribute Hot replace Yes Hot add Yes Hot remove Yes Link speed support Different speeds Frame protocol All Incoming packet management Switch Outgoing packet management BASP Failover event Loss of link only Failover time <500 ms Fallback time 1.
User Guide NetXtreme II September 2013 TEAMING AND OTHER ADVANCED NETWORKING PROPERTIES • Checksum Offload • IEEE 802.1p QoS Tagging • Large Send Offload • TCP Offload Engine (TOE) • Jumbo Frames • IEEE 802.1Q VLANs • Wake On LAN • Preboot Execution Environment Before creating a team, adding or removing team members, or changing advanced settings of a team member, make sure each team member has been configured similarly.
NetXtreme II User Guide September 2013 CHECKSUM OFFLOAD Checksum Offload is a property of the Broadcom network adapters that allows the TCP/IP/UDP checksums for send and receive traffic to be calculated by the adapter hardware rather than by the host CPU. In high-traffic situations, this can allow a system to handle more connections more efficiently than if the host CPU were forced to calculate the checksums.
User Guide NetXtreme II September 2013 insertion of a tag into an Ethernet frame to identify the VLAN to which a frame belongs. If present, the 4-byte VLAN tag is inserted into the Ethernet frame between the source MAC address and the length/type field. The first 2-bytes of the VLAN tag consist of the IEEE 802.1Q tag type, whereas the second 2 bytes include a user priority field and the VLAN identifier (VID). Virtual LANs (VLANs) allow the user to split the physical LAN into logical subparts.
NetXtreme II User Guide September 2013 GENERAL NETWORK CONSIDERATIONS • Teaming with Microsoft Virtual Server 2005 • Teaming Across Switches • Spanning Tree Algorithm • Layer 3 Routing/Switching • Teaming with Hubs (for troubleshooting purposes only) • Teaming with Microsoft NLB TEAMING WITH MICROSOFT VIRTUAL SERVER 2005 The only supported BASP team configuration when using Microsoft Virtual Server 2005 is with a Smart Load Balancing (TM) team-type consisting of a single primary Broadcom adapt
User Guide NetXtreme II September 2013 types of teaming supported by Broadcom. Therefore, an interconnect link must be provided between the switches that connect to ports in the same team. In the configuration without the interconnect, an ICMP Request from Blue to Gray goes out port 82:83 destined for Gray port 5E:CA, but the Top Switch has no way to send it there because it cannot go along the 5E:C9 port on Gray. A similar scenario occurs when Gray attempts to ping Blue.
NetXtreme II User Guide September 2013 The addition of a link between the switches allows traffic from/to Blue and Gray to reach each other without any problems. Note the additional entries in the CAM table for both switches. The link interconnect is critical for the proper operation of the team. As a result, it is highly advisable to have a link aggregation trunk to interconnect the two switches to ensure high availability for the connection.
User Guide NetXtreme II September 2013 Figure 5 represents a failover event in which the cable is unplugged on the Top Switch port 4. This is a successful failover with all stations pinging each other without loss of connectivity.
NetXtreme II User Guide September 2013 SPANNING TREE ALGORITHM • Topology Change Notice (TCN) • Port Fast/Edge Port In Ethernet networks, only one active path may exist between any two bridges or switches. Multiple active paths between switches can cause loops in the network. When loops occur, some switches recognize stations on both sides of the switch. This situation causes the forwarding algorithm to malfunction allowing duplicate frames to be forwarded.
User Guide NetXtreme II September 2013 Topology Change Notice BPDUs are sent when a port that was forwarding changes to blocking or transitions to forwarding. A TCN BPDU does not initiate an STP recalculation. It only affects the aging time of the forwarding table entries in the switch.It will not change the topology of the network or create loops. End nodes such as servers or clients trigger a topology change when they power off and then power back on.
NetXtreme II User Guide September 2013 SLB Team Connected to a Single Hub SLB teams configured as shown in Figure 6 maintain their fault tolerance properties. Either server connection could potentially fail, and network functionality is maintained. Clients could be connected directly to the hub, and fault tolerance would still be maintained; server performance, however, would be degraded. Figure 6: Team Connected to a Single Hub Generic and Dynamic Trunking (FEC/GEC/IEEE 802.3ad) FEC/GEC and IEEE 802.
User Guide NetXtreme II September 2013 APPLICATION CONSIDERATIONS • Teaming and Clustering • Teaming and Network Backup TEAMING AND CLUSTERING • Microsoft Cluster Software • High-Performance Computing Cluster • Oracle Microsoft Cluster Software In each cluster node, it is strongly recommended that customers install at least two network adapters (on-board adapters are acceptable). These interfaces serve two purposes. One adapter is used exclusively for intra-cluster heartbeat communications.
NetXtreme II User Guide September 2013 Figure 7 shows a 2-node Fibre-Channel cluster with three network interfaces per cluster node: one private and two public. On each node, the two public adapters are teamed, and the private adapter is not. Teaming is supported across the same switch or across two switches. Figure 8 shows the same 2-node Fibre-Channel cluster in this configuration.
User Guide NetXtreme II September 2013 High-Performance Computing Cluster Gigabit Ethernet is typically used for the following three purposes in high-performance computing cluster (HPCC) applications: • Inter-Process Communications (IPC): For applications that do not require low-latency, high-bandwidth interconnects (such as Myrinet, InfiniBand), Gigabit Ethernet can be used for communication between the compute nodes.
NetXtreme II User Guide September 2013 Oracle In our Oracle Solution Stacks, we support adapter teaming in both the private network (interconnect between RAC nodes) and public network with clients or the application layer above the database layer.
User Guide NetXtreme II September 2013 TEAMING AND NETWORK BACKUP • Load Balancing and Failover • Fault Tolerance When you perform network backups in a nonteamed environment, overall throughput on a backup server adapter can be easily impacted due to excessive traffic and adapter overloading.
NetXtreme II User Guide September 2013 Because there are four client servers, the backup server can simultaneously stream four backup jobs (one per client) to a multidrive autoloader. Because of the single link between the switch and the backup server; however, a 4-stream backup can easily saturate the adapter and link.
User Guide NetXtreme II September 2013 The same algorithm applies if a third and fourth backup operation is initiated from the backup server. The teamed interface on the backup server transmits a unicast G-ARP to backup clients to inform them to update their ARP cache. Each client then transmits backup data along a route to the target MAC address on the backup server.
NetXtreme II User Guide September 2013 TROUBLESHOOTING TEAMING PROBLEMS • Teaming Configuration Tips • Troubleshooting Guidelines When running a protocol analyzer over a virtual adapter teamed interface, the MAC address shown in the transmitted frames may not be correct. The analyzer does not show the frames as constructed by BASP and shows the MAC address of the team and not the MAC address of the interface transmitting the frame.
User Guide NetXtreme II September 2013 and fallback. 15. Test the performance behavior of the team before placing into a production environment. 16. Network teaming is not supported when running iSCSI traffic via Microsoft iSCSI initiator or iSCSI offload. MPIO should be used instead of Broadcom network teaming for these ports. 17. For information on iSCSI boot and iSCSI offload restrictions, see iSCSI Protocol.
NetXtreme II User Guide September 2013 Question: Can I connect the teamed adapters to a hub? Answer: Teamed ports can be connected to a hub for troubleshooting purposes only. However, this practice is not recommended for normal operation because the performance would be degraded due to hub limitations. Connect the teamed ports to a switch instead. Question: Can I connect the teamed adapters to ports in a router? Answer: No.
User Guide NetXtreme II September 2013 Question: During backup operations, does the adapter teaming algorithm load balance data at a byte-level or a sessionlevel? Answer: When using adapter teaming, data is only load balanced at a session level and not a byte level to prevent out-oforder frames. Adapter teaming load balancing does not work the same way as other storage load balancing mechanisms such as EMC PowerPath.
NetXtreme II User Guide September 2013 APPENDIX A: EVENT LOG MESSAGES • Windows System Event Log Messages • Base Driver (Physical Adapter/Miniport) • Intermediate Driver (Virtual Adapter/Team) • Virtual Bus Driver (VBD) WINDOWS SYSTEM EVENT LOG MESSAGES The known base and intermediate Windows System Event Log status messages for the Broadcom NetXtreme II adapters are listed in Table 8 and Table 9. As a Broadcom adapter driver loads, Windows places a status code in the system event viewer.
User Guide NetXtreme II September 2013 Table 8: Base Driver Event Log Messages (Cont.) Message Severity Number Message Cause Corrective Action 6 Informational Network controller configured for 10Mb halfduplex link. The adapter has been manually configured for the selected line speed and duplex settings. No action is required. 7 Informational Network controller configured for 10Mb fullduplex link. The adapter has been manually configured for the selected line speed and duplex settings.
NetXtreme II User Guide September 2013 Table 8: Base Driver Event Log Messages (Cont.) Message Severity Number Message Cause Corrective Action 19 Error This driver does not support this device. Upgrade to the latest driver. The driver does not recognize the installed adapter. Upgrade to a driver version that supports this adapter. 20 Error Driver initialization failed. Unspecified failure during driver initialization.
User Guide NetXtreme II September 2013 INTERMEDIATE DRIVER (VIRTUAL ADAPTER/TEAM) The intermediate driver is identified by source BLFM, regardless of the base driver revision. Table 9 lists the event log messages supported by the intermediate driver, explains the cause for the message, and provides the recommended action.
NetXtreme II User Guide September 2013 Table 9: Intermediate Driver Event Log Messages (Cont.) System Event Message Number Severity Message Cause Corrective Action 14 Informational Network adapter does not support Advanced Failover. The physical adapter does not support the Broadcom NIC Extension (NICE). Replace the adapter with one that does support NICE. 15 Informational Network adapter is enabled via management interface.
User Guide NetXtreme II September 2013 VIRTUAL BUS DRIVER (VBD) Table 10: Virtual Bus Driver (VBD) Event Log Messages Message Number Severity Message Cause Corrective Action 1 Error Failed to allocate memory for the device block. Check system memory resource usage. The driver cannot allocate memory from the operating system. Close running applications to free memory. 2 Informational The network link is down. Check to make sure the network cable is properly connected.
NetXtreme II User Guide September 2013 Table 10: Virtual Bus Driver (VBD) Event Log Messages Message Number Severity Message Cause Corrective Action 13 Error Driver initialization failed. Unspecified failure during driver initialization. Reinstall the driver, update to a newer driver, run Broadcom Advanced Control Suite diagnostics, or replace the adapter. 14 Error This driver does not support this device. Upgrade to the latest driver. The driver does not recognize the installed adapter.
User Guide NetXtreme II September 2013 V i r t u a l LAN s i n W in d ow s: Broadcom NetXtreme II ® Ne twork Adapter Us er Guide • VLAN Overview • Adding VLANs to Teams VLAN OVERVIEW Virtual LANs (VLANs) allow you to split your physical LAN into logical parts, to create logical segmentation of workgroups, and to enforce security policies for each logical segment.
NetXtreme II User Guide September 2013 Table 1: Example VLAN Network Topology Component Description VLAN #1 An IP subnet consisting of the Main Server, PC #3, and PC #5. This subnet represents an engineering group. VLAN #2 Includes the Main Server, PCs #1 and #2 via shared media segment, and PC #5. This VLAN is a software development group. VLAN #3 Includes the Main Server, the Accounting Server and PC #4. This VLAN is an accounting group.
User Guide NetXtreme II September 2013 ADDING VLANS TO TEAMS Each team supports up to 64 VLANs (63 tagged and 1 untagged). Note that only Broadcom adapters and Alteon® AceNIC adapters can be part of a team with VLANs. With multiple VLANs on an adapter, a server with a single adapter can have a logical presence on multiple IP subnets. With multiple VLANs in a team, a server can have a logical presence on multiple IP subnets and benefit from load balancing and failover.
NetXtreme II User Guide September 2013 M a n ag e a b i li t y : B ro a d c o m N e t X t r e m e I I ® Network Adapter Us er Guide • CIM • SNMP • HBA API CIM The Common Information Model (CIM) is an industry standard defined by the Distributed Management Task Force (DMTF). Microsoft implements CIM on Windows server platforms. Broadcom support CIM on Windows Server and Linux platforms.
User Guide NetXtreme II September 2013 For detailed information about these events, see the CIM documentation at http://www.dmtf.org/sites/default/files/standards/ documents/DSP0004V2.3_final.pdf. Broadcom also implements the Storage Management Initiative-Specification (SMI-S), which defines CIM management profiles for storage systems.
NetXtreme II User Guide September 2013 SNMP BASP SUBAGENT The BASP subagent, baspmgnt.dll, is designed for the Windows Server 2008 and Windows Server 2008 R2 SNMP service. It is required to install the SNMP service before installing the BASP subagent. The BASP subagent allows an SNMP manager software to actively monitor the configurations and performance of the Broadcom Advanced Server features.
User Guide NetXtreme II September 2013 HBA API Broadcom supports the Storage Networking Industry Association (SNIA) Common HBA API on Windows and Linux operating systems. The Common HBA API is an application program interface for the management of Fibre Channel Host Bus Adapters. BASP EXTENSIBLE-AGENT The Broadcom NetXtreme II Gigabit Ethernet Controller Extended Information SNMP extensible-agent (bcmif.dll) is designed for Windows Server 2008 SNMP service.
NetXtreme II User Guide September 2013 Insta ll in g t h e Hardwa r e : B ro a d c o m N e t X t r e m e II ® Ne twork Adapter Us er Guide • Overview • System Requirements • Safety Precautions • Preinstallation Checklist • Installation of the Add-In NIC OVERVIEW This section applies to Broadcom NetXtreme II add-in network interface cards.
User Guide NetXtreme II September 2013 Linux Although the adapter driver should work with many Linux kernel versions and distributions, it has only been tested on 2.4x kernels (starting from 2.4.24) and 2.6.x kernels. The driver may not compile on kernels older than 2.4.24. Testing is concentrated on i386 and x86_64 architectures. Only limited testing has been done on other architectures. Minor changes to some source files and Makefile may be needed on some kernels. NOTE: Support for the 2.4.
NetXtreme II User Guide September 2013 PREINSTALLATION CHECKLIST 1. Verify that your system meets the hardware and software requirements listed under System Requirements. 2. Verify that your system is using the latest BIOS. NOTE: If you acquired the adapter software on a disk, verify the path to the adapter driver files. 1. If your system is active, shut it down. 2. When system shutdown is complete, turn off the power and unplug the power cord. 3.
User Guide NetXtreme II September 2013 INSTALLATION OF THE ADD-IN NIC The following instructions apply to installing the Broadcom NetXtreme II adapter (add-in NIC) in most systems. Refer to the manuals that were supplied with your system for details about performing these tasks on your particular system. INSTALLING THE ADD-IN NIC 1. Review Safety Precautions and Preinstallation Checklist.
NetXtreme II User Guide September 2013 Table 1: 10/100/1000BASE-T and 10GBASE-T Cable Specifications Port Type Connector Media Maximum Distance 10BASE-T RJ-45 Category 3, 4, or 5 unshielded twisted pairs (UTP) 100m (328 ft) 100/1000BASE-T1 RJ-45 Category 52 UTP 100m (328 ft) 10GBASE-T RJ-45 Category 63 UTP 50m (164 ft) 100m (328 ft) Category 6A3 UTP 1 1000BASE-T signaling requires four twisted pairs of Category 5 balanced cabling, as specified in ISO/IEC 11801:2002 and ANSI/EIA/TIA-568-B.
User Guide NetXtreme II September 2013 B ro adc om B oo t A g en t D ri ver S o f t wa re: Broadcom NetXtreme II ® Ne twork Adapter Us er G ui de • Overview • Setting Up MBA in a Client Environment • Setting Up MBA in a Server Environment OVERVIEW Broadcom NetXtreme II adapters support Preboot Execution Environment (PXE), Remote Program Load (RPL), iSCSI, and Bootstrap Protocol (BootP).
NetXtreme II User Guide September 2013 SETTING UP MBA IN A CLIENT ENVIRONMENT Setting up MBA in a client environment involves the following steps: 1. Enabling the MBA driver. 2. Configuring the MBA driver. 3. Setting up the BIOS for the boot order. ENABLING THE MBA DRIVER To enable or disable the MBA driver: 1. Insert the installation CD in the CD-ROM drive and boot up in DOS mode. NOTE: The uxdiag.exe file is on the installation CD. 2.
User Guide NetXtreme II September 2013 3. Use the UP ARROW and DOWN ARROW keys to move to the Boot Protocol menu item. Then use the RIGHT ARROW or LEFT ARROW key to select the boot protocol of choice if other boot protocols besides Preboot Execution Environment (PXE) are available. If available, other boot protocols include Remote Program Load (RPL), iSCSI, and Bootstrap Protocol (BOOTP). NOTE: For iSCSI boot-capable LOMs, the boot protocol is set via the BIOS.
NetXtreme II User Guide September 2013 NOTE: For iSCSI boot-capable LOMs, the boot protocol is set via the BIOS. See your system documentation for more information. 6. Use the UP ARROW, DOWN ARROW, LEFT ARROW, and RIGHT ARROW keys to move to and change the values for other menu items, as desired. 7. Select Back to go to Main menu 8. Select Finish to save and exit. SETTING UP THE BIOS To boot from the network with the MBA, make the MBA enabled adapter the first bootable device under the BIOS.
User Guide NetXtreme II September 2013 iSC S I P ro t oc ol : B ro a d c o m N e t X t r e m e I I ® Network Adapter Us er Guide • iSCSI Boot • iSCSI Crash Dump • iSCSI Offload in Windows Server ISCSI BOOT Broadcom NetXtreme II Gigabit Ethernet adapters support iSCSI boot to enable network boot of operating systems to diskless systems. iSCSI boot allows a Windows, Linux, or VMware operating system boot from an iSCSI target machine located remotely over a standard IP network.
NetXtreme II User Guide September 2013 ISCSI BOOT SETUP The iSCSI boot setup consists of: • Configuring the iSCSI Target • Configuring iSCSI Boot Parameters • Preparing the iSCSI Boot Image • Booting Configuring the iSCSI Target Configuring the iSCSI target varies by target vendors. For information on configuring the iSCSI target, refer to the documentation provided by the vendor. The general steps include: 1. Create an iSCSI target. 2. Create a virtual disk. 3.
User Guide NetXtreme II September 2013 Table 1: Configuration Options Option Description iSCSI parameters via DHCP Controls whether the iSCSI boot host software acquires its iSCSI target parameters using DHCP (Enabled) or through a static configuration (Disabled). The static information is entered through the iSCSI Initiator Parameters Configuration screen. CHAP Authentication Controls whether the iSCSI boot host software uses CHAP authentication when connecting to the iSCSI target.
NetXtreme II User Guide September 2013 Static iSCSI Boot Configuration In a static configuration, you must enter data for the system’s IP address, the system’s initiator IQN, and the target parameters obtained in Configuring the iSCSI Target. For information on configuration options, see Table 1. To configure the iSCSI boot parameters using static configuration 1. From the General Parameters Menu screen, set the following: • TCP/IP parameters via DHCP: Disabled. (For IPv4.
User Guide NetXtreme II September 2013 Dynamic iSCSI Boot Configuration In a dynamic configuration, you only need to specify that the system’s IP address and target/initiator information are provided by a DHCP server (see IPv4 and IPv6 configurations in Configuring the DHCP Server to Support iSCSI Boot). For IPv4, with the exception of the initiator iSCSI name, any settings on the Initiator Parameters, 1st Target Parameters, or 2nd Target Parameters screens are ignored and do not need to be cleared.
NetXtreme II User Guide September 2013 Enabling CHAP Authentication Ensure that CHAP authentication is enabled on the target. To enable CHAP authentication 1. From the General Parameters screen, set CHAP Authentication to Enabled. 2. From the Initiator Parameters screen, type values for the following: • CHAP ID (up to 128 bytes) • CHAP Secret (if authentication is required, and must be 12 characters in length or longer) 3. Select ESC to return to the Main menu. 4.
User Guide NetXtreme II September 2013 Table 2: DHCP Option 17 Parameter Definition Parameter Definition The IP address or FQDN of the iSCSI target ":" Separator The IP protocol used to access the iSCSI target. Currently, only TCP is supported so the protocol is 6. The port number associated with the protocol. The standard port number for iSCSI is 3260. The Logical Unit Number to use on the iSCSI target.
NetXtreme II User Guide September 2013 DHCP iSCSI Boot Configuration for IPv6 The DHCPv6 server can provide a number of options, including stateless or stateful IP configuration, as well s information to the DHCPv6 client. For iSCSI boot, Broadcom adapters support the following DHCP configurations: • DHCPv6 Option 16, Vendor Class Option • DHCPv6 Option 17, Vendor-Specific Information NOTE: The DHCPv6 standard Root Path option is not yet available.
User Guide NetXtreme II September 2013 Preparing the iSCSI Boot Image • Windows Server 2008 R2 and SP2 iSCSI Boot Setup • Windows Server 2012 iSCSI Boot Setup • Linux iSCSI Boot Setup • Injecting (Slipstreaming) Broadcom Drivers into Windows Image Files Windows Server 2008 R2 and SP2 iSCSI Boot Setup Windows Server 2008 R2 and Windows Server 2008 SP2 support booting as well as installing in either the offload or nonoffload paths.
NetXtreme II User Guide September 2013 12. Select Next to proceed with Windows Server 2008 R2 installation. A few minutes after the Windows Server 2008 R2 DVD installation process starts, a system reboot will follow. After the reboot, the Windows Server 2008 R2 installation routine should resume and complete the installation. 13. Following another system restart, check and verify that the remote system is able to boot to the desktop. 14.
User Guide NetXtreme II September 2013 14. After Windows Server 2012 boots to the OS, Broadcom recommends running the driver installer to complete the Broadcom drivers and application installation. Linux iSCSI Boot Setup Linux iSCSI boot is supported on Red Hat Enterprise Linux 5.5 and later and SUSE Linux Enterprise Server 11 SP1 and later in both the offload and non-offload paths. Note that SLES 10.x and SLES 11 have support only for the non-offload path. 1.
NetXtreme II User Guide September 2013 22. Remove ifcfg-eth*. 23. Reboot. 24. For SUSE 11.1, follow the remote DVD installation workaround shown below. 25. After the system reboots, log in, change to the /opt/bcm/bibt folder, and run iscsi_setup.sh script to create the offload and/or the non-offload initrd image. 26. Copy the initrd image(s), offload and/or non-offload, to the /boot folder. 27. Change the grub menu to point to the new initrd image. 28. To enable CHAP, you need to modify iscsid.
User Guide NetXtreme II September 2013 # service is stopped. # iscsi_load_iscsiuio() { TRANSPORT=`$ISCSIADM -m session 2> /dev/null | grep "bnx2i"` if [ "$TRANSPORT" ] ; then echo -n "Launch iscsiuio " startproc $ISCSIUIO fi } iscsi_mark_root_nodes() { $ISCSIADM -m session 2> /dev/null | while read t num i target ; do ip=${i%%:*} STARTUP=`$ISCSIADM -m node -p $ip -T $target 2> /dev/null | grep "node.conn\[0\].
NetXtreme II User Guide September 2013 exit 1 ;; esac rc_exit Injecting (Slipstreaming) Broadcom Drivers into Windows Image Files To inject Broadcom drivers into the Windows image files, you must obtain the following correct Broadcom driver packages for the applicable Windows Server version (2008 R2, 2008 SP2, 2012, or 2012 R2). • bxvbd • evbd • bxfcoe • bxnd • b57nd60a (when available) • bxois Then, you place these driver packages to a working directory.
User Guide NetXtreme II September 2013 Note that you will use this command prompt window in all subsequent steps. 6. Enter the following commands: attrib -r .\src\sources\boot.wim attrib -r .\src\sources\install.wim 7. Enter run the following command to mount the boot.wim image: dism /mount-wim /wimfile:.\src\sources\boot.wim /index:2 /mountdir:.\mnt Note that you must always use “2” for the index value. 8.
NetXtreme II User Guide September 2013 Booting After that the system has been prepared for an iSCSI boot and the operating system is present on the iSCSI target, the last step is to perform the actual boot. The system will boot to Windows or Linux over the network and operate as if it were a local disk drive. 1. Reboot the server. 2. Select CTRL+S. 3. To boot through an offload path, set the HBA Boot Mode to Enabled. To boot through a non-offload path, set the HBA Boot Mode to Disabled.
User Guide NetXtreme II September 2013 only on SuSE 10 as offload is not supported on SuSE 10. 14. Mount the /boot partition on the iSCSI Target. 15. Copy the initrd images you created in step 13 from your local hard drive to the partition mounted in step 14. 16. On the partition mounted in step 14, edit the grub menu to point to the new initrd images. 17. Unmount the /boot partition on the iSCSI Target. 18. (Red Hat Only) To enable CHAP, you need to modify the CHAP section of the iscsid.
NetXtreme II User Guide September 2013 TROUBLESHOOTING ISCSI BOOT The following troubleshooting tips are useful for iSCSI boot. Problem: A system blue screen occurs when iSCSI boots Windows Server 2008 R2 through the adapter’s NDIS path with the initiator configured using a link-local IPv6 address and the target configured using a router-configured IPv6 address. Solution: This is a known Windows TCP/IP stack issue.
User Guide NetXtreme II September 2013 Problem: In Windows Server 2012, toggling between iSCSI HBA offload mode and iSCSI software initiator boot can leave the machine in a state where the HBA offload miniport bxois will not load. Solution: Manually edit [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bxois\StartOverride] from 3 to 0. Modify the registry key before toggling back from NDIS to HBA path in CCM. NOTE: Microsoft recommends against this method.
NetXtreme II User Guide September 2013 ISCSI CRASH DUMP If you will use the Broadcom iSCSI Crash Dump utility, it is important to follow the installation procedure to install the iSCSI Crash Dump driver. See Using the Installer for more information.
User Guide NetXtreme II September 2013 CONFIGURING ISCSI OFFLOAD With the proper iSCSI offload licensing, you can configure your iSCSI-capable NetXtreme II network adapter to offload iSCSI processing from the host processor. The following process enables your system to take advantage of Broadcom’s iSCSI offload feature.
NetXtreme II User Guide September 2013 Configure Microsoft Initiator to Use Broadcom’s iSCSI Offload Now that the IP address has been configured for the iSCSI adapter, you need to use Microsoft Initiator to configure and add a connection to the iSCSI target using Broadcom iSCSI adapter. See Microsoft’s user guide for more details on Microsoft Initiator. 1. Open Microsoft Initiator. 2. Configure the initiator IQN name according to your setup. To change, click on Change. 3. Enter the initiator IQN name.
User Guide NetXtreme II September 2013 4. Select the Discovery tab and click Add to add a target portal. 5. Enter the IP address of the target and click Advanced.
NetXtreme II User Guide September 2013 6. From the General tab, select Broadcom NetXtreme II C-NIC iSCSI Adapter from Local adapter.
User Guide NetXtreme II September 2013 7. Select the IP address for the adapter from Source IP.
NetXtreme II User Guide September 2013 8. Click OK to close Advanced setting and then OK to add the target portal.
User Guide NetXtreme II September 2013 9. From the Targets tab, select the target and click Log On to log into your iSCSI target using the Broadcom iSCSI adapter. 10. Click on Advanced.
NetXtreme II User Guide September 2013 11. On the General tab, select the Broadcom NetXtreme Ii C-NIC iSCSI adapter from Local adapter. 12. Click OK to close Advanced settings. 13. Click OK to close the Microsoft Initiator. 14. To format your iSCSI partition, use Disk Manager. NOTES: • Teaming does not support iSCSI adapters. • Teaming does not support NDIS adapters that are in the boot path. • Teaming supports NDIS adapters that are not in the iSCSI boot path, but only for the SLB team type.
User Guide NetXtreme II September 2013 ISCSI OFFLOAD FAQS Q: How do I assign an IP address for iSCSI offload? A: Use the Configurations tab in Broadcom Advanced Control Suite (BACS). Q: What tools should be used to create the connection to the target? A: Use Microsoft iSCSI Software Initiator (version 2.08 or later). Q: How do I know that the connection is offloaded? A: Use Microsoft iSCSI Software Initiator. From a command line, type iscsicli sessionlist.
NetXtreme II User Guide September 2013 EVENT LOG MESSAGES Table 5 lists the offload iSCSI driver event log messages. OFFLOAD ISCSI (OIS) DRIVER Table 5: Offload iSCSI (OIS) Driver Event Log Messages Message Number Severity Message 1 Error Initiator failed to connect to the target. Target IP address and TCP Port number are given in dump data. 2 Error The initiator could not allocate resources for an iSCSI session.
User Guide NetXtreme II September 2013 Table 5: Offload iSCSI (OIS) Driver Event Log Messages Message Number Severity Message 24 Error Target sent an iSCSI PDU with an invalid opcode. Dump data contains the entire iSCSI header. 25 Error Data digest error was detected. Dump data contains the calculated checksum followed by the given checksum. 26 Error Target trying to send more data than requested by the initiator.
NetXtreme II User Guide September 2013 Table 5: Offload iSCSI (OIS) Driver Event Log Messages Message Number Severity Message 53 Error Initiator Service failed to respond in time to a request to release IPSec resources allocated for an iSCSI connection. 54 Error Initiator Service failed to respond in time to a request to encrypt or decrypt data. 55 Error Initiator failed to allocate resources to send data to target.
User Guide NetXtreme II September 2013 In s t a ll i ng M a nag e m e n t A p pl ic a t i on s : B road c o m Ne tXtr eme II ® Network Adapter Us er Guide • Overview • Installation Tasks • Detailed Procedures • Installing the Broadcom Advanced Control Suite and Related Management Applications • Managing Management Applications (Windows) OVERVIEW The Broadcom Advanced Control Suite version 4 (BACS4) is a management application for configuring the NetXtreme II family of adapters, also known as Co
NetXtreme II User Guide September 2013 If the client uses: And the managed host uses: BACS can use these communication protocols: • WMI = Windows Management Instrumentation. • WS-MAN = Web Service-Management. WinRM is a Windows-based implementation and OpenPegasus is an opensource implementation of the that operates on Linux. • CIM-XML = An XML-based version of OpenPegasus.
User Guide NetXtreme II September 2013 Installing WS-MAN on Linux Server On Linux server, use the following steps to install OpenPegasus. 1. Install OpenPegasus From Source (Red Hat and SuSE). 2. Start CIM Server on the Server. 3. Configure OpenPegasus on the Server. 4. Install Broadcom CMPI Provider. 5. Perform additional configuration, if required, such as firewall configuration. See Perform Linux Firewall Configuration, If Required. 6.
NetXtreme II User Guide September 2013 DETAILED PROCEDURES This section provides the step-by-step instructions for each installation task. The required tasks for each communication protocol differ, as listed in Installation Tasks. Refer to the appropriate task list to ensure you complete all necessary tasks for the chosen protocol. WS-MAN ON WINDOWS SERVER Install the WinRM Software Component on Server On the following operating systems, WinRM 2.
User Guide NetXtreme II September 2013 Perform HTTP Configuration on the Server To use the BACS GUI, you must configure the HTTP protocol, as follows: NOTE: The default HTTP port is 5985 for WinRM 2.0. 1. Click Start (or press the Windows logo key) and select Run. 2. Enter gpedit.msc to open the local Group Policy editor. 3. Under Computer Configuration, open the Administrative Templates folder and then open the Windows Components folder. 4. Select Windows Remote Management (WinRM). 5.
NetXtreme II User Guide September 2013 4. Select Windows Remote Management (WinRM). 5. Under Windows Remote Management (WinRM), select WinRm Client. 6. Under WinRM Client, double-click Trusted Hosts. 7. In the TrustedHostsList, enter the host names of the clients. If all clients are trusted then enter an asterisk (*) only. 8. Select WinRM Service. 9. Enable Allow Basic Authentication.
User Guide NetXtreme II September 2013 Signature ok subject=/C=US/ST=California/L=Irvine/O=Broadcom Corporation/OU=Engineering/CN=MGMTAPPLAB3/emailAddress= Getting Private key 6. Enter the following command to verify the generated self-signed certificate. openssl verify server.crt The following output displays: server.
NetXtreme II User Guide September 2013 12. Browse to select hostname.pfx. 13. When you are prompted for the password for the private key, enter the same password you created in Generate a SelfSigned Certificate for Windows/Linux Server. 14. Follow the instructions, select the defaults, and continue. The certificate is shown as installed on the right side of the window. The name will be the name you specified while creating a self-signed certificate. 15.
User Guide NetXtreme II September 2013 e. Click the Details tab. f. Scroll down and select the Thumbprint field. g. Select and copy the thumbprint in the Details window so you can insert it in the next step. h. Return to the command prompt. i. Enter the following command: winrm create winrm/config/Listener?Address=*+Transport= HTTPS @{Hostname=""; CertificateThumbprint=""} NOTES: j.
NetXtreme II User Guide September 2013 Windows XP 1. Click Start > Control Panel, and then double-click Windows Firewall. 2. Click the Exceptions tab 3. Click Add Port. 4. Enter a meaningful Name, for example “WinRM rule” and port number, for example, 5985 for HTTP or 5986 for HTTPS. 5. Click OK.
User Guide NetXtreme II September 2013 WS-MAN—WINDOWS CLIENT Perform HTTP Configuration (if you plan to use HTTP) 1. Click Start (or press the Windows logo key) and select Run. 2. Enter gpedit.msc to open the local Group Policy editor. 3. Under Computer Configuration, open the Administrative Templates folder and then open the Windows Components folder. 4. Select Windows Remote Management (WinRM). 5. Under Windows Remote Management (WinRM), select WinRm Client. 6.
NetXtreme II User Guide September 2013 1. To retrieve the server operating system information, enter the following command. winrm e wmi/root/cimv2/Win32_OperatingSystem -r:https://yourservername -u:username -p:password -skipCAcheck 2. To retrieve the server WinRM identity information, enter the following command. winrm id -r:https://yourservername -u:username -p:password -skipCAcheck 3. To enumerate Windows services on the server, enter the following command.
User Guide NetXtreme II September 2013 NOTE: If not already installed, download and install the openssl and libopenssl-devel rpm. This step is optional and required only if you are planning to use HTTPS to connect the client to the managed host. Set the Environment Variable Set the environment variables for building OpenPegasus as follows. Environment Variable Description PEGASUS_ROOT The location of the Pegasus source tree PEGASUS_HOME The location for the built executable, repository; e.g.
NetXtreme II User Guide September 2013 NOTE: You can add these exports at the end of the .bash_profile. This file is located in the /root directory. • The environment variables will be set when a user logs in using PuTTY. • On the Linux system itself, for each terminal where the environment variables are not set, run the following command: source /root/.bash_profile • When you logout and login, the environment variables will be set.
User Guide NetXtreme II September 2013 Enable Authentication The following OpenPegasus properties have to be set as described in this section. Otherwise, the Broadcom CIM Provider will not work properly. Ensure the following are set before launching BACS and connecting to the provider. Start CIM server if it is not already started.
NetXtreme II User Guide September 2013 Install Enter following command to install Broadcom CMPI Provider. % rpm -i BRCM_CMPIProvider-{version}.{arch}.rpm Uninstall Enter following command to uninstall Broadcom CMPI Provider: % rpm -e BRCM_CMPIProvider Perform Linux Firewall Configuration, If Required Follow these procedures to open the appropriate ports in the firewall: RedHat 1. Click System, select Administration, and then select Firewall. 2. Select Other Ports. 3.
User Guide NetXtreme II September 2013 • Source Port: Leave blank.
NetXtreme II User Guide September 2013 WS-MAN AND CIM-XML—LINUX CLIENT No special software components are required on the Linux client system to use the HTTP except installing the BACS management application. However, for WS-MAN installations, you can optionally configure the HTTPS protocol for use with BACS. Configure HTTPS on Linux Client Follow these steps if you want to use HTTPS rather than HTTP (WS-MAN only): Follow these steps if you want to use HTTPS rather than HTTP (WS-MAN only): 1.
User Guide NetXtreme II September 2013 WMI—WINDOWS Perform the steps in the following two sections only to configure WMI on the Windows server. Set up Namespace Security Using WMI Control The WMI Control provides one way to manage namespace security. You can start the WMI Control from the command prompt using this command: wmimgmt On Windows 9x or Windows NT4 computers that have WMI installed, use this command instead: wbemcntl.
NetXtreme II User Guide September 2013 3. The Component Services dialogue box displays. 4. Open Component Services and then open Computers. 5. Right-click My Computer and click Properties. 6. In My Computer Properties, click the COM Security tab. 7. Under Launch and Activation Permissions, click Edit Limits. 8. Follow these steps if your name or your group does not appear in the Groups or user names list. a. In the Launch Permission dialog box, click Add. b.
User Guide NetXtreme II September 2013 INSTALLING THE BROADCOM ADVANCED CONTROL SUITE AND RELATED MANAGEMENT APPLICATIONS • Installing on a Windows System • Installing on a Linux System INSTALLING ON A WINDOWS SYSTEM The Broadcom Advanced Control Suite (BACS) software and related management applications can be installed from the installation CD or by using the silent install option. The following are installed when running the installer: • Control Suite. Broadcom Advanced Control Suite (BACS).
NetXtreme II User Guide September 2013 Using Silent Installation NOTES: • All commands are case sensitive. • User must “Run as Administrator” for Vista when using “msiexec” for “silent” install/uninstall(s). • For detailed instructions and information about unattended installs, refer to the Silent.txt file in the MgmtApps folder.
User Guide NetXtreme II September 2013 To perform a silent install from within a batch file To perform a silent install from within a batch file and wait for the install to complete before continuing with the next command line, type the following: start /wait setup /s /w /v/qn INSTALLING ON A LINUX SYSTEM The Broadcom Advanced Control Suite (BACS) software can be installed on a Linux system using the Linux RPM package. This installation includes a BACS GUI and a CLI client.
NetXtreme II User Guide September 2013 MANAGING MANAGEMENT APPLICATIONS (WINDOWS) MODIFYING THE MANAGEMENT APPLICATION To modify the management applications: 1. In Control Panel, double-click Add or Remove Programs. 2. Click Broadcom Management Programs and then click Change. 3. Click Next to continue. 4. Click Modify to change program features. 5. Click Next to continue. 6. Click on an icon to change how a feature is installed. 7. Click Next. 8. Click Install. 9. Click Finish to close the wizard. 10.
User Guide NetXtreme II September 2013 N DIS2 D ri ver So f t w are : Broadcom NetXtreme II® Network Adapter Us er Guide • Overview • Preinstallation Requirements • Installing the NDIS2 Driver Software for Use on MS-DOS Platforms • Using Keywords for the Drivers OVERVIEW Two drivers are discussed in this section: • BXND20X: Broadcom NetXtreme II Gigabit Ethernet driver • BNX2EV: Broadcom NetXtreme II 10 Gigabit Ethernet driver The examples used in this section refer to the BXND20X driver, but
NetXtreme II User Guide September 2013 • Windows NT Server 4.0 users. When running Setup for Microsoft Network Client v3.0 for MS-DOS, click any network card from the list (NE2000 Compatible, for example) to create the startup disk. • After creating the startup disk, follow the instructions in Modifying the Startup Disk. To create a startup disk 1. Create a folder called NCADMIN in the root of the C drive. 2. Copy the NCADMIN.CN_, NCADMIN.EX_, and NCADMIN.
User Guide NetXtreme II September 2013 Example Protocol.ini file for IPX [network.setup] version=0x3110 netcard=ms$ne2clone,1,MS$NE2CLONE,1 transport=ms$ndishlp,MS$NDISHLP transport=ms$nwlink,MS$NWLINK lana0=ms$ne2clone,1,ms$nwlink lana1=ms$ne2clone,1,ms$ndishlp [MS$NE2CLONE] DriverName=BXND20X$ [protman] DriverName=PROTMAN$ PRIORITY=MS$NDISHLP [MS$NDISHLP] DriverName=ndishlp$ BINDINGS=ms$ne2clone [ms$nwlink] DriverName=nwlink$ FRAME=Ethernet_802.2 BINDINGS=MS$NE2CLONE LANABASE=0 Example Protocol.
NetXtreme II User Guide September 2013 2. Edit A:\Net\System.ini. a. Change netcard= to netcard=BXND20X.dos. b. Check for references to C:\NET and change C:\NET to A:\NET if necessary. Example System.ini file [network] sizworkbuf=1498 filesharing=no printsharing=no autologon=yes computername=MYPC lanroot=A:\NET username=USER1 workgroup=WORKGROUP reconnect=yes dospophotkey=N lmlogon=0 logondomain= preferredredir=basic autostart=basic maxconnections=8 [network drivers] netcard=BXND20X.dos transport=ndishlp.
User Guide NetXtreme II September 2013 lastdrive=z INSTALLING THE DOS NDIS2 DRIVER SOFTWARE ON THE HARD DISK To install the DOS NDIS2 Driver Software on the hard disk 1. Verify that the system has Microsoft LAN Manager 2.2 installed, with a protocol such as NetBEUI configured. 2. Create a folder on your hard disk to store the NDIS 2.01 driver. Example: C:\LANMAN 3. Copy the BXND20X.dos file to this folder. Edit the Config.sys file by adding the following lines: DEVICE = C:\LANMAN\PROTMAN.
NetXtreme II User Guide September 2013 USING KEYWORDS FOR THE DRIVERS The Protocol.ini file contains certain keywords that are used by the BXND20X.dos AND BXND20X.dos drivers. These keywords are listed below: BusNum. Specifies the number of the PCI bus on which the network adapter is located. Requires a decimal number having a value ranging from 0 to 255. DevNum. Specifies the device number assigned to the network adapter when it is configured by the PCI BIOS.
User Guide NetXtreme II September 2013 AcceptAllMC. Informs the driver to deliver all multicast packets to the upper protocol.
NetXtreme II User Guide September 2013 Linu x Driver Software: Broadcom NetXtreme II ® Network Adapter Us er Guide • Introduction • Limitations • Packaging • Installing Linux Driver Software • Unloading/Removing the Linux Driver • Patching PCI Files (Optional) • Network Installations • Setting Values for Optional Properties • Driver Defaults • Driver Messages • Teaming with Channel Bonding • Statistics • Linux iSCSI Offload Bro adco m C orp or atio n Page 142 Linux Driver Softwa
User Guide NetXtreme II September 2013 INTRODUCTION This section discusses the Linux drivers for the Broadcom NetXtreme II network adapters. Table 1: Broadcom NetXtreme II Linux Drivers Linux Driver Description bnx2 Linux driver for the NetXtreme II 1 Gb network adapters. bnx2x Linux driver for the NetXtreme II 10 Gb network adapters. This driver directly controls the hardware and is responsible for sending and receiving Ethernet packets on behalf of the Linux host networking stack.
NetXtreme II User Guide September 2013 LIMITATIONS • bnx2 Driver • bnx2x Driver • bnx2i Driver BNX2 DRIVER The current version of the driver has been tested on 2.4.x kernels (starting from 2.4.24) and all 2.6.x kernels. The driver may not compile on kernels older than 2.4.24. NOTE: Support for the 2.4.21 kernels is provided in Red Hat Enterprise Linux 3. Testing is concentrated on i386 and x86_64 architectures. Only limited testing has been done on other architectures.
User Guide NetXtreme II September 2013 PACKAGING The Linux drivers are released in the following packaging formats: DKMS Packages • netxtreme2-version.dkms.noarch.rpm • netxtreme2-version.dkms.src.rpm KMP Packages • • SLES • broadcom-netxtreme2-kmp-[kernel]-version.i586.rpm • broadcom-netxtreme2-kmp-[kernel]-version.x86_64.rpm Red Hat • kmod-kmp-netxtreme2-{kernel]-version.i686.rpm • kmod-kmp-netxtreme2-{kernel]-version.x86_64.
NetXtreme II User Guide September 2013 INSTALLING LINUX DRIVER SOFTWARE • Installing the Source RPM Package • Building the Driver from the Source TAR File NOTE: If a bnx2/bnx2x/bnx2i driver is loaded and the Linux kernel is updated, the driver module must be recompiled if the driver module was installed using the source RPM or the TAR package. INSTALLING THE SOURCE RPM PACKAGE The following are guidelines for installing the driver source RPM Package.
User Guide NetXtreme II September 2013 5. For SLES, turn on the fcoe and lldpad services. For SLES11 SP1: chkconfig lldpad on chkconfig fcoe on For SLES11 SP2: chkconfig boot.lldpad on chkconfig boot.fcoe on 6. Inbox drivers are included with all of the supported operating systems. The simplest means to ensure the newly installed drivers are loaded is to reboot. 7.
NetXtreme II User Guide September 2013 rcfcoe restart INSTALLING THE KMP PACKAGE NOTE: The examples in this procedure refer to the bnx2x driver, but also apply to the bnx2 and bnx2i drivers. 1. Install the KMP package: rpm -ivh rmmod bnx2x 2.
User Guide NetXtreme II September 2013 BUILDING THE DRIVER FROM THE SOURCE TAR FILE NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver. 1. Create a directory and extract the TAR files to the directory: tar xvzf netxtreme2-version.tar.gz 2. Build the driver bnx2.ko (or bnx2.o) as a loadable module for the running kernel: cd netxtreme2-version make 3. Test the driver by loading it (first unload the existing driver, if necessary): rmmod bnx2 insmod bnx2.
NetXtreme II User Guide September 2013 LOAD AND RUN NECESSARY ISCSI SOFTWARE COMPONENTS The Broadcom iSCSI Offload software suite consists of three kernel modules and a user daemon. Required software components can be loaded either manually or through system services. 1. Unload the existing driver, if necessary: Manual: rmmod bnx2i 2. Load the iSCSI driver: Manual: insmod bnx2i.
User Guide NetXtreme II September 2013 UNLOADING/REMOVING THE LINUX DRIVER • Unloading/Removing the Driver from an RPM Installation • Removing the Driver from a TAR Installation UNLOADING/REMOVING THE DRIVER FROM AN RPM INSTALLATION NOTES: • The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver. • On 2.6 kernels, it is not necessary to bring down the eth# interfaces before unloading the driver module.
NetXtreme II User Guide September 2013 PATCHING PCI FILES (OPTIONAL) NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x and bnx2i drivers. For hardware detection utilities such as Red Hat kudzu to properly identify bnx2 supported devices, a number of files containing PCI vendor and device information may need to be updated. Apply the updates by running the scripts provided in the supplemental tar file.
User Guide NetXtreme II September 2013 persist across a reboot or module reload. The ethtool commands can be put in a startup script such as /etc/rc.local to preserve the settings across a reboot. NOTE: Some combinations of property values may conflict and result in failures. The driver cannot detect all such conflicting combinations. This property is used to disable Message Signal Interrupts (MSI), and the property is valid only on 2.6 kernels that support MSI. On 2.
NetXtreme II User Guide September 2013 dropless_fc The dropless_fc parameter can be used to enable a complementary flow control mechanism on BCM57711/BCM57712 adapters. The default flow control mechanism is to send pause frames when the on-chip buffer (BRB) is reaching a certain level of occupancy. This is a performance targeted flow control mechanism.
User Guide NetXtreme II September 2013 The optional parameter qs_per_cos is used to specify how many queues will share the same CoS. This parameter is evaluated by the driver up to 3 values of 8 bits each. Each byte sets the desired number of queues for that CoS. The total number of queues is limited by the hardware limit. For example, set qs_per_cos to 0x10101 to create a total of three queues, one per CoS.
NetXtreme II User Guide September 2013 insmod bnx2i.ko en_tcp_dack=0 or modprobe bnx2i en_tcp_dack=0 time_stamps “Enable TCP TimeStamps”, enables/disables TCP time stamp feature on offloaded iSCSI connections. Defaults: TCP time stamp option is DISABLED. For example: insmod bnx2i.ko time_stamps=1 or modprobe bnx2i time_stamps=1 sq_size "Configure SQ size", used to choose send queue size for offloaded connections and SQ size determines the maximum SCSI commands that can be queued.
User Guide NetXtreme II September 2013 “Last active TCP port used”, status parameter used to indicate the last TCP port number used in the iSCSI offload connection. Defaults: N/A Valid values: N/A Note: This is a read-only parameter. ooo_enable “Enable TCP out-of-order feature”, enables/disables TCP out-of-order rx handling feature on offloaded iSCSI connections. Defaults: TCP out-of-order feature is ENABLED. For example: insmod bnx2i.
NetXtreme II User Guide September 2013 Coalesce TX Microseconds IRQ: 80 (range is 0–1023) Coalesce TX Frames: 20 (range is 0–255) Coalesce TX Frames IRQ: 20 (range is 0–255) Coalesce Statistics Microseconds: 999936 (approximately 1 second) (range is 0–16776960 in increments of 256) MSI: Enabled (if supported by the 2.6 kernel and the interrupt test passes) TSO: Enabled (on 2.
User Guide NetXtreme II September 2013 BNX2 AND BNX2X DRIVER NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver. Driver Sign on Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.6.3c (July 23, 2007) CNIC Driver Sign on (bnx2 only) Broadcom NetXtreme II cnic v1.1.
NetXtreme II User Guide September 2013 bnx2i: iSCSI not supported, dev=eth3 bnx2i: bnx2i: LOM is not enabled to offload iSCSI connections, dev=eth0 bnx2i: dev eth0 does not support iSCSI Exceeds maximum allowed iSCSI connection offload limit bnx2i: alloc_ep: unable to allocate iscsi cid bnx2i: unable to allocate iSCSI context resources Network route to target node and transport name binding are two different devices bnx2i: conn bind, ep=0x...
User Guide NetXtreme II September 2013 bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error - invalid DataSN burst len violation buf offset violation invalid LUN field invalid R2TSN field invalid cmd len1 invalid cmd
NetXtreme II User Guide September 2013 STATISTICS Detailed statistics and configuration information can be viewed using the ethtool utility. See the ethtool man page for more information.
User Guide NetXtreme II September 2013 LINUX ISCSI OFFLOAD • Open iSCSI User Applications • User Application - brcm_iscsiuio • Bind iSCSI Target to Broadcom NX2 iSCSI Transport Name • Making Connections to iSCSI Targets • Maximum Offload iSCSI Connections • Linux iSCSI Offload FAQ OPEN ISCSI USER APPLICATIONS Install and run the inbox open-iscsi initiator programs from the DVD. Refer to Packaging for details.
NetXtreme II User Guide September 2013 If you wish to switch back to use the software initiator, use the following: iscsiadm -m iface -I -n iface.transport_name -v tcp -o update where the iface file includes the following information: iface.net_ifacename = ethX iface.iscsi_ifacename = iface.transport_name = tcp MAKING CONNECTIONS TO ISCSI TARGETS Refer to open-iscsi documentation for a comprehensive list of iscsiadm commands.
User Guide NetXtreme II September 2013 solution, ensure that the iscsi node under /etc/iscsi/nodes for the boot target is bound to the NIC that is used for booting.
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 166 Linux iSCSI Offload Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 S ol ar is D ri ver S o f t wa re : Broa dc om Ne tXtrem e II ® Ne twork Adapter Us er Guide • Overview • Installing the Driver • Upgrading the Driver • Uninstalling Driver • Configuring the Driver • Memory Usage • Interrupt Management • FCoE Support OVERVIEW This file describes how to install the Solaris driver for Broadcom's NetXtreme II 10 Gigabit Ethernet network adapters.
NetXtreme II User Guide September 2013 INSTALLING THE DRIVER 1. Change directory to where BRCMbnxe-version.pkg resides. 2. pkgadd -d BRCMbnxe-version.pkg or 1. Copy BRCMbnxe-X.Y.Z.tar.Z to /tmp. 2. cd /tmp uncompress BRCMbnxe-version.tar.Z tar -xvf BRCMbnxe-version.tar pkgadd -d /tmp 3. Execute prtconf to determine instance numbers of the NIC. 4. ifconfig bnxe[instance_number] plumb 5. ifconfig bnxe[instance_number] ip_address netmask .... up To make these changes permanent: 1.
User Guide NetXtreme II September 2013 CONFIGURING THE DRIVER The bnxe driver can be configured via the bnxe.conf file installed under /kernel/drv. When this config file is modified, the system must be either rebooted or the driver unloaded and reconfigured using the update_drv admin command. All configuration can be specified per-instance.
NetXtreme II User Guide September 2013 DMA allocations at a minimum. After installation, it is suggested these descriptor counts be increased until stability is guaranteed and the desired performance is reached.
User Guide NetXtreme II September 2013 INTERRUPT MANAGEMENT If you have a system with many interfaces, it is possible to reach the allocation limit of MSIX interrupts. By default, Solaris limits each driver to 2 MSIX allocations, and there is an issue with the pcplusmp module where only a maximum of 31 MSIX interrupts are available per interrupt priority level. If your system has four Broadcom BCM57711 network adapter ports, each running in multifunction mode, Solaris will enumerate 16 bnxe interfaces.
NetXtreme II User Guide September 2013 FCOE SUPPORT OVERVIEW FCoE is supported on Solaris 11 with limited support on Solaris 10, Update 9. The following features are the differences in Solaris 10, Update 9 when compared to Solaris 11: • Support does not exist for NPIV in the Solaris 10 Update 9. • Some of the fcinfo(1M) options, which are available in Solaris 11, are not be available in Solaris 10, Update 9. For more information, read the man page fcinfo(1M).
User Guide NetXtreme II September 2013 The details of the configurations parameters are detailed in the bnxef(7D) man page. The default parameters should work for all conditions.
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 174 FCoE Support Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 V M w a r e D r i v e r S o f tw a r e: Broadcom NetXtreme II ® Ne twork Adapter Us er Guide • Packaging • Networking Support • Drivers • FCoE Support PACKAGING The VMware driver is released in the packaging formats shown in Table 1. Table 1: VMware Driver Packaging Format Drivers Compressed tar bnx2x-version.tar.gz VMware VIB vmware-esx-drivers-net-bnx2x-version.x86_64.
NetXtreme II User Guide September 2013 DRIVERS DOWNLOAD, INSTALL, AND UPDATE DRIVERS To download, install, or update the VMware ESX/ESXi driver for NetXtreme II 10 GbE network adapters, see http:// www.vmware.com/support. DRIVER PARAMETERS Several optional parameters can be supplied as a command line argument to the vmkload_mod command. These parameters can also be set via the esxcfg-module command. See the man page for more information.
User Guide NetXtreme II September 2013 The optional parameter num_tx_queues may be used to set the number of Tx queues on kernels starting from 2.6.27 when multi_mode is set to 1 and interrupt mode is MSI-X. The number of Rx queues must be equal to or greater than the number of Tx queues (see num_rx_queues parameter). If the interrupt mode is different than MSI-X (see int_mode parameter), then the number of Tx queues will be set to 1, discarding the value of this parameter.
NetXtreme II User Guide September 2013 Flow Control: Autonegotiation with rx and tx advertised MTU: 1500 (range 46–9000) Rx Ring Size: 4078 (range 0–4078) Tx Ring Size: 4078 (range (MAX_SKB_FRAGS+4) - 4078). MAX_SKB_FRAGS varies on different kernels and different architectures. On a 2.6 kernel for x86, MAX_SKB_FRAGS is 18. Coalesce RX Microseconds: 25 (range 0–3000) Coalesce TX Microseconds: 50 (range 0–12288) MSI-X: Enabled (if supported by 2.
User Guide NetXtreme II September 2013 If you see messages in the log file that look like the following, then the ESX host is severely strained. To relieve this, disable NetQueue. Dec 2 18:24:20 ESX4 vmkernel: 0:00:00:32.342 cpu2:4142)WARNING: Heap: 1435: Heap bnx2x already at its maximumSize. Cannot expand. Dec 2 18:24:20 ESX4 vmkernel: 0:00:00:32.342 cpu2:4142)WARNING: Heap: 1645: Heap_Align(bnx2x, 4096/4096 bytes, 4096 align) failed. caller: 0x41800187d654 Dec 2 18:24:20 ESX4 vmkernel: 0:00:00:32.
NetXtreme II User Guide September 2013 FCOE SUPPORT This section describes the contents and procedures associated with installation of the VMware software package for supporting Broadcom FCoE C-NICs. DRIVERS Table 2: Broadcom NetXtreme II FCoE Drivers Driver Description bnx2x This driver manages all PCI device resources (registers, host interface queues, etc.) and also acts as the Layer 2 VMware low-level network driver for Broadcom's NetXtreme II 10G device.
User Guide NetXtreme II September 2013 ENABLING FCOE To enable FCoE hardware offload on the C-NIC 1. Determine the ports that are FCoE-capable: # esxcli fcoe nic list Output example: vmnic4 User Priority: 3 Source MAC: FF:FF:FF:FF:FF:FF Active: false Priority Settable: false Source MAC Settable: false VLAN Range Settable: false 2. Enable the FCoE interface: # esxcli fcoe nic discover -n vmnicX Where X is the interface number gained from esxcli fcoe nic list. 3.
NetXtreme II User Guide September 2013 To verify the correct installation of the driver 1. Verify the host port shows up in the switch FLOGI database using the "show flogi database" command for the case of a Cisco FCF and "fcoe -loginshow" command for the case of a Brocade FCF. 2. If the Host WWPN does not appear in the FLOGI database, then provide driver log messages for review. LIMITATIONS • NPIV is not currently supported with this release on ESX, due to lack of supporting inbox components.
User Guide NetXtreme II September 2013 Wi n dows D r i v e r S of t w a r e : B ro a d c o m N e t X tr e m e II ® Ne twork Adapter Us er Guide • Installing the Driver Software • Removing the Device Drivers • Using the NetXtreme II Monolithic Driver • Inserting the NetXtreme II Monolithic Driver in a WinPE 2.0 or 3.
NetXtreme II User Guide September 2013 To install the Broadcom NetXtreme II drivers 1. When the Found New Hardware Wizard appears, click Cancel. 2. Insert the installation CD into the CD or DVD drive. 3. On the installation CD, open the folder for your operating system, open the DrvInst folder, and then double-click Setup.exe to open the InstallShield Wizard. 4. Click Next to continue. 5.
User Guide NetXtreme II September 2013 with the Broadcom Advanced Control Suite 3 (BACS) application in HKLM\Software\Mictrosoft\Windows\CurrentVersion\Uninstall for the correct hexidecimal number. To perform a silent reinstall of the same installer Type the following: setup /s /v"/qn REINSTALL=ALL" NOTE: The REINSTALL switch should only be used if the same installer is already installed on the system. If upgrading an earlier version of the installer, use setup /s /v/qn as listed above.
NetXtreme II User Guide September 2013 USING THE NETXTREME II MONOLITHIC DRIVER The NetXtreme II, based on its advanced functionalities, uses a software architecture that includes a Virtual Bus Device (VBD) to extend functionalities beyond basic network connectivity.
User Guide NetXtreme II September 2013 INSERTING THE NETXTREME II MONOLITHIC DRIVER IN A WINPE 2.0 OR 3.1 IMAGE Follow these procedures for inserting the NeXtreme II monolithic driver into WinPE images. The instructions differ depending on the WinPE version and the Windows Server OS version system being used. WINPE 2.0 The Microsoft Windows Server 2008 method of inserting the NetXtreme II monolithic driver in a WinPe 2.0 image is different from the Windows Server 2008 R2 method, as discussed below.
NetXtreme II User Guide September 2013 To add a device driver to an offline Windows PE image (Windows Server 2008 R2) This procedure demonstrates how to use the Deployment Image Servicing and Management (DISM) tool to add a device driver (.inf) to an offline Windows PE image. Before running a DISM command, first mount the Windows PE image. 1. Mount the base image by using the DISM tool to a local Windows PE directory. For example: Dism /Mount-WIM /WimFile:c:\winpe_x86\winpe.
User Guide NetXtreme II September 2013 CONFIGURING THE SPEED/DUPLEX SETTING FOR THE NETXTREME II MONOLITHIC DRIVER Since the typical environment where the NetXtreme II monolithic driver is used does not provide the means to configure advanced network adapter properties, the driver file (b06nd.inf) was modified to include a section that allows it to be configured for a specific speed and/or duplex.
NetXtreme II User Guide September 2013 VIEWING OR CHANGING THE PROPERTIES OF THE ADAPTER To view or change the properties of the Broadcom network adapter 1. In Control Panel, click Broadcom Control Suite 4. 2. Click the Advanced section of the Configurations tab. SETTING POWER MANAGEMENT OPTIONS You can set power management options to allow the operating system to turn off the controller to save power or to allow the controller to wake up the computer.
User Guide NetXtreme II September 2013 NOTES: • The Power Management tab is available only for servers that support power management. • To enable Wake on LAN (WOL) when the computer is on standby, click Allow the device to bring the computer out of standby box. • If you select Only allow management stations to bring the computer out of standby, the computer can be brought out of standby only by Magic Packet.
NetXtreme II User Guide September 2013 N IC Par t i t i on in g : Broa dc om Ne tXtrem e II ® Network Adapter Us er Guide • Overview • Configuring for NIC Partitioning OVERVIEW NIC partitioning (NPAR) divides a Broadcom NetXtreme II 10 Gigabit Ethernet NIC into multiple virtual NICs by having multiple PCI physical functions per port. Each PCI function is associated with a different virtual NIC. To the OS and the network, each physical function appears as a separate NIC port.
User Guide NetXtreme II September 2013 NOTE: 32-bit Linux operating systems have a limited amount of memory space available for Kernel data structures. Therefore, it is recommended that only 64-bit Linux be used when configuring NPAR.
NetXtreme II User Guide September 2013 CONFIGURING FOR NIC PARTITIONING When NIC partitioning is enabled on an adapter, by default, only TCP Offload Engine (TOE) offloads are configured on each physical function (PF). You must explicitly configure storage offloads on a PF to use FCoE and iSCSI offload functionality on an adapter. NIC partitioning can be configured using Broadcom’s Comprehensive Configuration Management (CCM) utility.
User Guide NetXtreme II September 2013 Table 2: Function Description Function Description Option Bandwidth Weight Configures the weight or importance of a particular function. There are four functions per port and the weight is used to arbitrate between the functions in case of congestion. The sum of all weights for the four functions are either 0 or 100. Maximum Bandwidth Configures the maximum bandwidth (in percentage) of the physical port link.
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 196 Configuring for NIC Partitioning Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 F ib r e C h a n n e l O v e r Ethe rne t: B ro adc om Ne tXtr eme II ® Network Adapter Us er Guide • Overview • FCoE Boot from SAN • Configuring FCoE OVERVIEW In today’s data center, multiple networks, including network attached storage (NAS), management, IPC, and storage, are used to achieve the desired performance and versatility. In addition to iSCSI for storage solutions, Fibre Channel over Ethernet (FCoE) can now be used with capable Broadcom C-NICs.
NetXtreme II User Guide September 2013 FCOE BOOT FROM SAN This section describes the install and boot procedures for the Windows, Linux, ESX, and Solaris operating systems. NOTE: FCoE Boot from SAN is not supported on ESXi 5.0. ESX Boot from SAN is supported on ESXi 5.1 and above. The following section details the BIOS setup and configuration of the boot environment prior to the OS install.
User Guide NetXtreme II September 2013 PREPARE BROADCOM MULTIPLE BOOT AGENT FOR FCOE BOOT 1. During POST, press CTRL+S at the Broadcom NetXtreme Ethernet Boot Agent banner to invoke the CCM utility. 2. Select the device through which boot is to be configured. NOTE: IMPORTANT: When running in NIC Partitioning (NPAR) mode, FCoE boot is supported only when the first function on each port is assigned an FCoE personality.
NetXtreme II User Guide September 2013 4. On some platforms, you may need to set the boot protocol through system BIOS configuration in the integrated devices pane as described above. For all other devices, set the Boot Protocol field to FCoE in the MBA Configuration Menu through CCM. 5. Configure the desired boot target and LUN. From the Target Information Menu, select the first available path.
User Guide NetXtreme II September 2013 6. Enable the Connect field. Enter the target WWPN and Boot LUN information for the target to be used for boot.
NetXtreme II User Guide September 2013 7. Press ESC until prompted to exit and save changes. 8. Proceed to OS installation once storage access has been provisioned in the SAN. PROVISIONING STORAGE ACCESS IN THE SAN Storage access consists of zone provisioning and storage selective LUN presentation, each of which is commonly provisioned per initiator WWPN.
User Guide NetXtreme II September 2013 present the option to press CTRL+R. 4. Once CTRL+R has been activated, the boot initiator will maintain a link in whatever phase has most recently succeeded and allow you time to make the necessary provisioning corrections to proceed to the next phase. 5. If the initiator logs into the fabric, but is unable to log into the target, a CTRL+R will pause the boot process and allow you to configure fabric zoning. 6.
NetXtreme II User Guide September 2013 WINDOWS SERVER 2008 R2 AND WINDOWS SERVER 2008 SP2 FCOE BOOT INSTALLATION Ensure that no USB flash drive is attached before starting the OS installer. The EVBD and OFC/BXFOE drivers need to be loaded during installation. Go through the normal procedures for OS installation. When no disk devices are found, Windows will prompt you to load additional drivers.
User Guide NetXtreme II September 2013 2. Then load the bxfcoe (OFC) driver. 3.
NetXtreme II User Guide September 2013 4. Continue with the rest of the installation. After installation is complete and booted to SAN, execute the provided Windows driver installer and reboot. Installation is now complete. NOTE: The boot initiator must be configured to point at the desired installation LUN, and the boot initiator must have successfully logged and determined the readiness of the LUN prior to starting installation.
User Guide NetXtreme II September 2013 LINUX FCOE BOOT INSTALLATION Configure the adapter boot parameters and Target Information (press CTRL+S and enter the CCM utility) as detailed in Preparing System BIOS for FCoE Build and Boot. Then, use the guidelines in the following sections for FCoE boot installation with the appropriate Linux version. • SLES11 SP2 Installation • RHEL6 Installation SLES11 SP2 Installation 1.
NetXtreme II User Guide September 2013 3. Once the driver update is complete, select Next to continue with OS installation. 4. When requested, click Configure FCoE Interfaces.
User Guide NetXtreme II September 2013 5. Ensure FCoE Enable is set to yes on the 10GbE Broadcom initiator ports you wish to use as the SAN boot path(s). 6.
NetXtreme II User Guide September 2013 are set to yes and DCB required is set to no. 7. For each interface to be enabled for FCoE boot, click on Create FCoE VLAN Interface. The VLAN interface creation dialog will launch. Click Yes to confirm. This will trigger automatic FIP VLAN discovery. If successful, the VLAN will be displayed under FCoE VLAN Interface. If no VLAN is visible, check your connectivity and switch configuration. 8. Once complete with configuration of all interface, click OK to proceed.
User Guide NetXtreme II September 2013 9. Click Next to continue installation. YaST2 will prompt to activate multipath. Answer as appropriate. 10. Continue installation as usual. 11. Under the Expert tab on the Installation Settings screen, select Booting.
NetXtreme II User Guide September 2013 12. Select the Boot Loader Installation tab and then select Boot Loader Installation Details, make sure you have one boot loader entry here. Delete all redundant entries. 13. Click OK to proceed and complete installation.
User Guide NetXtreme II September 2013 RHEL6 Installation 1. Boot from the installation medium. 2. For RHEL6.3, an updated Anaconda image is required for FCoE BFS. That updated image is provided by Red Hat at the following URL http://rvykydal.fedorapeople.org/updates.823086-fcoe.img. 3. For RHEL6.3, on the installation splash screen, press Tab and add the options dd updates=< URL_TO_ANACONDA_UPDATE_IMAGE > to the boot command line. Please refer to the RedHat Installation Guide Section 28.1.
NetXtreme II User Guide September 2013 6. Once drivers are loaded, proceed with installation. 7. Select Specialized Storage Devices when prompted.
User Guide NetXtreme II September 2013 8. Click Add Advanced Target.
NetXtreme II User Guide September 2013 9. Select Add FCoE SAN. and select Add drive. 10. For each interface intended for FCoE boot, select the interface, deselect Use DCB, select Use auto vlan, and then click Add FCoE Disk(s).
User Guide NetXtreme II September 2013 11. Repeat steps 8 through 10 for all initiator ports. 12. Confirm all FCoE visible disks are visible under Multipath Devices and/or Other SAN Devices.
NetXtreme II User Guide September 2013 13. Click Next to proceed. 14. Click Next and complete installation as usual. Upon completion of installation, the system will reboot. 15. Once booted, ensure all boot path devices are set to start on boot. Set onboot=yes under each network interface config file in /etc/sysconfig/network-scripts. 16. On RHEL 6.4 only, edit /boot/grub/menu.lst. a. Delete all "fcoe=:nodcb" parameters from the "kernel /vmlinuz … " line.
User Guide NetXtreme II September 2013 3. Create a /etc/fcoe/cfg- file for each new FCoE initiator by duplicating the /etc/fcoe/cfg file that was already configured during initial installation. 4. Execute nm-connection-editor. a. Open Network Connection and choose each new interface. b. Configure each interface as desired, including DHCP settings. c. Click Apply to save. 5.
NetXtreme II User Guide September 2013 VMWARE ESXI 5.1 FCOE BOOT INSTALLATION FCoE Boot from SAN requires that the latest Broadcom NetXtreme II async drivers be included into the ESXi 5.1 install image. Refer to Image_builder_doc.pdf from VMware on how to slipstream drivers. 1. Boot from the updated ESXi 5.1 installation image and select ESXi 5.1 installer when prompted. 2. Press Enter to continue. 3. Press F11 to accept the agreement and continue. 4.
User Guide NetXtreme II September 2013 5. Select the desired installation method. 6. Select the keyboard layout. 7. Enter a password.
NetXtreme II User Guide September 2013 8. Press F11 to confirm the install. 9. Press Enter to reboot after installation. 10. On 57800 and 57810 boards, the management network is not vmnic0. After booting, open the GUI console and display the configure management network > network adapters screen to select the NIC to be used as the management network device.
User Guide NetXtreme II September 2013 11. For BCM57800 and BCM57810 boards, the FCoE boot devices need to have a separate vSwitch other than vSwith0. This allows DHCP to assign the IP address to the management network rather than to the FCoE boot device. To create a vSwitch for the FCoE boot devices, add the boot device vmnics in vSphere Client under Networking. NOTE: ESXi 5.1 has a limitation in that a VLAN ID change for a boot device is not possible. It works only for nonboot devices.
NetXtreme II User Guide September 2013 BOOTING FROM SAN AFTER INSTALLATION Now that boot configuration and OS installation are complete, you can reboot and test the installation. On this and all future reboots, no other user interactivity is required. Ignore the CTRL+D prompt and allow the system to boot through to the FCoE SAN LUN.
User Guide NetXtreme II September 2013 the ramdisk with the default original name. Also, verify that your appropriate entry for the boot from SAN setup uses the correct or updated intrd name in /boot/grub/ menu.lst. 5. To complete your driver upgrade, reboot the system and select the modified grub boot entry that contains the updated initrd.
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 226 Configuring FCoE Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 U s in g D a t a C e n t e r B r id g in g ( D C B ) : B ro a dc om Ne tXtr eme II ® Network Adapter Us er Guide • Overview • DCB Capabilities • Configuring DCB • DCB Conditions • Data Center Bridging in Windows Server 2012 OVERVIEW Data Center Bridging (DCB) is a collection of IEEE specified standard extensions to Ethernet to provide lossless data delivery, low latency, and standards-based bandwidth sharing of data center physical links.
NetXtreme II User Guide September 2013 DCB CAPABILITIES ENHANCED TRANSMISSION SELECTION (ETS) Enhanced Transmission Selection (ETS) provides a common management framework for assignment of bandwidth to traffic classes. Each traffic class or priority can be grouped in a Priority Group (PG), and it can be considered as a virtual link or virtual interface queue. The transmission scheduler in the peer is responsible for maintaining the allocated bandwidth for each PG.
User Guide NetXtreme II September 2013 CONFIGURING DCB By default, DCB is enabled on Broadcom NetXtreme II DCB-compatible C-NICs. DCB configuration is rarely required, as the default configuration should satisfy most scenarios. DCB parameters can be configured through BACS. See Using Broadcom Advanced Control Suite for more information on BACS. DCB CONDITIONS The following is a list of conditions that allow DCB technology to function on the network.
NetXtreme II User Guide September 2013 DATA CENTER BRIDGING IN WINDOWS SERVER 2012 Windows Server 2012 introduces a new way of managing Quality Of Service (QoS) at the OS level. There are two main aspects of Windows QoS: • A vendor-independent method for managing DCB settings on NICs, both individually and across an entire domain. The management interface is provided by Windows PowerShell Cmdlets.
User Guide NetXtreme II September 2013 SR -IOV: Broa dco m Ne tXtr eme I I ® Network A d a p t e r U s e r G ui de • Overview • Enabling SR-IOV OVERVIEW Virtualization of network controllers allows users to consolidate their networking hardware resources and run multiple virtual machines concurrently on consolidated hardware.
NetXtreme II User Guide September 2013 If Multi-Function Mode is set to NPAR, then display the Main Menu and select NIC Partition Configuration. Then, select the NPAR Function to configure and enter the appropriate value in the Number of VFs per PF field. 2. In Virtual Switch Manager, create a virtual NIC. Ensure that Allow Management operating system to share the network adapter is selected as you create the NIC. 3.
User Guide NetXtreme II September 2013 U s i n g B ro a d c o m A dvan c e d C o nt rol S u i t e 4 : Broadcom NetXtreme II ® Ne twork Adapter Us er G ui de • Broadcom Advanced Control Suite Overview • Managing Ethernet Controller (Port) • Starting Broadcom Advanced Control Suite • Analyzing Cables in Windows • BACS Interface • Managing the LAN Device • Configuring Preferences in Windows • Viewing Resource Information • Connecting to a Host • Configuring Teaming • Managing the Host
NetXtreme II User Guide September 2013 STARTING BROADCOM ADVANCED CONTROL SUITE In Control Panel, click Broadcom Control Suite 4, or click the BACS icon in the taskbar located at the bottom of the Windows or Windows Server desktop. On Linux systems, you can double-click the BACS4 desktop icon, or access the BACS program from the task bar under System Tools. (If you are having difficulty launching BACS on a Linux system, see the related topic in Troubleshooting BACS.
User Guide NetXtreme II September 2013 BACS INTERFACE The BACS interface is comprised of the following regions: • Explorer View pane • Context View selector • Context View pane • Menu bar • Description pane By default, the Explorer View pane is docked and pinned on the left side of the main window, the Context View pane on the right, the Context View selector below the menu bar, and the Description pane below the Context View pane.
NetXtreme II User Guide September 2013 EXPLORER VIEW PANE You can dock and pin the Explorer View pane on the left side, right side, top, or bottom of the main window. The Explorer View pane lists the objects that can be viewed, analyzed, tested, or configured by BACS. When an item is selected in the Explorer View pane, the tabs showing the information and options that are available for the item appear in the Context View pane.
User Guide NetXtreme II September 2013 MENU BAR The following appear on the menu bar, but because the menu items are context-sensitive, not all items will be available at all times: File menu • Team Save As: Saves the current team configurations to a file. • Team Restore: Restores any saved team configuration from a file. Action menu • Remove Host: Removes the selected host. • Refresh Host: Refreshes the selected host. View menu • Explorer View: Displays/hides the Explorer View pane.
NetXtreme II User Guide September 2013 Manage Discovery Portals The Manage Discovery Portals window is available from the iSCSI menu. From this window, you can add or remove iSCSI discovery portals. Boot Configuration Wizard The Boot Configuration Wizard is available by right-clicking a port. Follow the prompts in the wizard to configure the iSCSI boot parameters.
User Guide NetXtreme II September 2013 CONFIGURING PREFERENCES IN WINDOWS To enable or disable the BACS tray icon in Windows On Windows systems, BACS places an icon in the Windows taskbar when the program is installed. Use the Options window to turn this icon on or off. 1. From the Tools menu, select Options. 2. Select or clear Enable BACSTray (the option is enabled by default). 3. Click OK. Setting the teaming mode in Windows 1. From the Tools menu, select Options. 2.
NetXtreme II User Guide September 2013 CONNECTING TO A HOST You can add one or more Windows or Linux hosts to manage from BACS. To add a local host 1. From the Action menu, click Add Host. 2. For both Windows and Linux hosts, do not change the default settings. The User name and Password are not required while connecting to the local host. 3. Select Persist if you want BACS to save the information for this host. 4. Click Ok. BACS can now be used to view information and manage the host.
User Guide NetXtreme II September 2013 MANAGING THE HOST At the host level, you can view host information and configure parameters from the following tabs: • Information • Configuration To view host information Select the host in the Explorer View pane, and then select the Information tab to view host-level information. INFORMATION TAB: HOST INFORMATION Host Name. Displays the name of the host. OS Version Info. Displays the operating system, including the version. Platform.
NetXtreme II User Guide September 2013 MANAGING THE NETWORK ADAPTER The installed network adapters appear one level below the host in the hierarchical tree in the Explorer View pane. At the adapter level, you can view information and configure parameters from the following tabs: • Information • Configuration VIEWING ADAPTER INFORMATION Select the network adapter in the Explorer View pane, and then select the Information tab to view adapter-level information.
User Guide NetXtreme II September 2013 Interrupt Request. The interrupt line number that is associated with the adapter. Valid numbers range from 2 to 25. Memory Address. The memory mapped address that is assigned to the adapter. This value can never be 0. MSI Version. This is the Message Signaled Interrupts (MSI) version being used. The option MSI corresponds to the PCI 2.2 specification that supports 32 messages and a single MSI address value. The option MSI-X corresponds to the PCI 3.
NetXtreme II User Guide September 2013 VIEWING HARDWARE INFORMATION The Hardware section of the Information tab displays information about the hardware settings for the selected network adapter. NOTE: Some information may not be available for all Broadcom network adapters. Information Tab: Hardware ASIC Version. The chip version of the Broadcom adapter (this information is not available for adapters made by others). Bootcode Version. The version of the boot code.
User Guide NetXtreme II September 2013 CONFIGURE SR-IOV (SINGLE ROOT I/O VIRTUALIZATION) SR-IOV Global Enable. Enable Single Root I/O Virtualization (SR-IOV). SR-IOV VFs per PF. Configure the number of SR-IOV Virtual Functions (VF) per a PCIe Physical Function (PF). The range is 0 to 64 in increments of 8 with a default of 16. SR-IOV Max Chains per VF. Enter the maximum number of transmit and receive queues (such as receive side scaling (RSS) queues) that can be used for each virtual function.
NetXtreme II User Guide September 2013 • It is possible for the sum of all maximum bandwidth values across the four functions of a single port to exceed the physical link speed value of either 10 Gbps or 1 Gbps. This case is considered as oversubscription. In a case where oversubscription congestion occurs on transmit, the Relative Bandwidth Weight value comes into effect. • The Maximum Bandwidth setting is only valid in the context of Tx, but not Rx. Relative Bandwidth Weight (%).
User Guide NetXtreme II September 2013 To view Vital Signs information for any installed network adapter, select the name of the adapter listed in the Explorer View pane, then click the Information tab. NOTES: • Information about Broadcom network adapters may be more comprehensive than information about network adapters made by others. • Some information may not be available for all Broadcom network adapters. MAC Address.
NetXtreme II User Guide September 2013 series of smaller packets with headers appended to them. • CO. Checksum Offload (CO) allows the TCP/IP/UDP checksums for send and receive traffic to be calculated by the adapter hardware rather than by the host CPU. LiveLink IP Address. The network address of the LiveLink enabled adapter. Local Connection. Identifies the module to which the blade server is attached. • Chassis SW. Chassis switch module • Chassis PHY. Pass-through module • None.
User Guide NetXtreme II September 2013 • A value for a function between 1 and 100 represent a percentage of the physical link speed and is used by an internal arbitration logic as a input value (weight). A higher value will cause this function to transmit relatively more data, compared to a function (on the same port) that has defined a lower value. Maximum Bandwidth (%) • The maximum bandwidth setting defines an upper threshold value, ensuring that this limit will not be exceeded during transmission.
NetXtreme II User Guide September 2013 BACScli -t NDIS -f mac -i 0010181a1b1c "networkdiag -p 192.168.1.5" 2. The following command runs the network test for the current selected virtual adapter. Since there is no IP address specified, BACScli will use gateway address for the test. BACScli -t VNIC -f mac -i 0010181a1b1c "networkdiag" In Interactive mode, use the list and select commands to select the desired target device.
User Guide NetXtreme II September 2013 RUNNING DIAGNOSTIC TESTS IN WINDOWS The Diagnostic Tests option on the Diagnostics tab lets you check the state of the physical components on a Broadcom network adapter. You can trigger the tests manually, or choose to have BACS continuously perform them. If the test are performed continuously, then the number of passes and fails in the Result field for each test increments every time the tests are performed.
NetXtreme II User Guide September 2013 Test LED. This test causes all of the port LEDs to blink 5 times for the purpose of identifying the adapter. To run the diagnostic tests using the BACS CLI You can use the following CLI command to run diagnostics tests on a specified target.
User Guide NetXtreme II September 2013 NOTES: • This feature can be used with Windows Server managed hosts only. It is not available for hosts operating on Linux or other OSes. You can, however use BACS on a Linux client to connect to a Windows Server host and run the cable analysis utility. • You must have administrator privileges to run the cable analysis test. • The network connection is temporarily lost during an analysis. • This option is not available for NetXtreme II 10 GbE network adapters.
NetXtreme II User Guide September 2013 BACScli -t -f -i cablediag Example: 1. The following command runs the cable diagnostics test for the current selected target. BACScli -t PHYPORTS -f bdf -i 01:00.00 "cablediag" In Interactive mode, use the list and select commands to select the desired target device. Use the cablediag command to run the cable analysis test for the selected target. Example: 1.
User Guide NetXtreme II September 2013 Driver Status. The status of the adapter driver. • Loaded. Normal operating mode. The adapter driver has been loaded by the OS and is functioning. • Not Loaded. The driver associated with the adapter has not been loaded by the OS. • Information Not Available. The value is not obtainable from the driver that is associated with the adapter. Driver Name. The file name of the adapter driver. Driver Version. The current version of the adapter driver. Driver Date.
NetXtreme II User Guide September 2013 CONFIGURING THE NDIS DRIVER Select the NDIS driver in the Explorer View pane, and then select the Configuration tab to configure NDIS-level parameters. After making changes, click Apply to confirm the changes to all properties. Click Reset to return the properties to their original values. Click Defaults to restore all settings to their default values. NOTES: • Clicking Reset after clicking Defaults, but before clicking Apply, will purge all values.
User Guide NetXtreme II September 2013 IPv4 Large Send Offload. Normally, the TCP segmentation is done by the protocol stack. When you enable the Large Send Offload property, the TCP segmentation can be done by the network adapter. The default setting for this property is Enabled. This property is only available for Broadcom NetXtreme II adapters. IPv6 Checksum Offload. Normally, the checksum function is computed by the protocol stack.
NetXtreme II User Guide September 2013 Switch Configuration. Allows configuring of the connected switch for the network adapters. NOTE: Switch Configuration only applies to blade configurations. • SW_Config_10G (default). Sets the switch speed to 10 Gbit/s. • SW_Config_1G. Sets the switch speed to 1 Gbit/s. Speed & Duplex. The Speed & Duplex property sets the connection speed and mode to that of the network.
User Guide NetXtreme II September 2013 • Tx Enabled. Enables transmit TCP/IP/UDP checksum offload. • TX & Rx Enabled (default). Enables transmit and receive TCP/IP/UDP checksum offload. Priority & VLAN. Allows enabling both the prioritization of network traffic and VLAN tagging. VLAN tagging only occurs when the VLAN ID setting is configured with a value other than 0 (zero). • Priority & VLAN Enabled (default). Allows for packet prioritization and VLAN tagging. • Priority & VLAN Disabled.
NetXtreme II User Guide September 2013 Receive Buffers. The number of receive buffers. Receive buffers are data segments that allow the network adapter to allocate receive packets to memory. For 1 Gbps adapters, the range of valid receive buffers is 50 to 5000 in increments of 1 with 750 receive buffers as the default value. Receive Buffers (0=Auto). The number of receive buffers. Receive buffers are data segments that allow the network adapter to allocate receive packets to memory.
User Guide NetXtreme II September 2013 VIEWING RESOURCE INFORMATION The Resources section of the Information tab displays information about connections and other essential functions for the selected network adapter. NOTE: Some information may not be available for all Broadcom network adapters. Information Tab: Resources Bus Type. The type of input/output (I/O) interconnect used by the adapter.
NetXtreme II User Guide September 2013 VIEWING STATISTICS The information provided on the Statistics tab allows you to view traffic statistics for both Broadcom network adapters and network adapters made by others. Statistical information and coverage are more comprehensive for Broadcom adapters. To view Statistics information for any installed network adapter, click the name of the adapter listed in the Explorer View pane, then click the Statistics tab.
User Guide NetXtreme II September 2013 General Statistics General Statistics show the transmitted and received statistics to and from the adapter. Frames Tx OK. A count of the frames that were successfully transmitted. This counter is incremented when the transmit status is reported as Transmit OK. Frames Rx OK. A count of the frames that were successfully received.
NetXtreme II User Guide September 2013 Local Initiator login failures. The number of login failures likely caused by the initiator. Initiator Instance Statistics. The statistics in this area pertain to all sessions. Session digest errors. The number of sessions with errors due to an invalid payload or header. Session connection timeout error. The number of sessions that were terminated due to any of the many timeout errors. Session format error.
User Guide NetXtreme II September 2013 Frames Tx with more than one Collision. A count of the frames that were involved in more than one collision and were subsequently transmitted successfully. This counter is incremented when the transmit status is reported as Transmit OK, and the value of the attempts variable is greater than 2 and less than or equal to the attempt limit. Frames Tx after Deferral.
NetXtreme II User Guide September 2013 • Internet Small Computer Systems Interface (iSCSI) offload for accelerating network storage access featuring centralized boot functionality (iSCSI boot). You can also view the number of unlicensed resources and unallocated resources. TOE and iSCSI can only be configured on certain adapters and require a license key. License keys are preprogrammed in the hardware. To view resource reservations 1.
User Guide NetXtreme II September 2013 CONFIGURING TEAMING NOTE: BACS does not support teaming on Linux systems. Linux provides a similar built-in functionality called Channel Bonding. Refer to the Linux OS documentation for more information. The teaming function allows you to group any available network adapters together to function as a team. Teaming is a method of creating a virtual NIC (a group of multiple adapters that functions as a single adapter).
NetXtreme II User Guide September 2013 TEAM TYPES You can create four types of load balance teams: • Smart Load Balance and Failover • Link Aggregation (802.3ad) (TOE is not applicable) • Generic Trunking (FEC/GEC)/802.3ad-Draft Static (TOE is not applicable) • SLB (Auto-Fallback Disable) – The Auto-Fallback Disable feature is configured for Smart Load Balance and Failover type teams in the Teaming Wizard.
User Guide NetXtreme II September 2013 Device Manager or Hot-Plug PCI. If any primary adapter assigned to a team is disabled, the team functions as a Smart Load Balancing and Failover type of team in which auto-fallback occurs. The LiveLink feature is supported for this type of team. STANDBY TEAM MEMBER AND AUTO-FALLBACK DISABLE MODE You can designate one team member in an SLB type of team to be the standby member.
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 270 Configuring Teaming Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 3. Type the team name and then click Next. If you want to review or change any of your settings, click Back. Click Cancel to discard your settings and exit the wizard.
NetXtreme II User Guide September 2013 4. Select the type of team you want to create. 5. Select Enable Hyper-V Mode if you want to enable Windows virtualization services. See “Microsoft Virtualization with Hyper-V” in the Troubleshooting topic in the NetXtreme II Network Adapter User Guide for more information about this feature. 6. If the team type is an SLB type team, click Next. If the team type is not an SLB type team, then a dialog box appears.
User Guide NetXtreme II September 2013 7. From the Available Adapters list, click the adapter you want to add to the team and then click Add. Remove team members from the Team Members list by clicking the adapter and then clicking Remove. Click Next. NOTE: There must be at least one Broadcom network adapter assigned to the team.
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 274 Configuring Teaming Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 10. If you want to configure LiveLink, select Yes, otherwise select No, then click Next. 11. Select the probe interval (the number of seconds between each retransmission of a link packet to the probe target) and the maximum number of probe retries (the number of consecutively missed responses from a probe target before a failover is triggered). 12. Set the Probe VLAN ID to allow for connectivity with probe targets residing on a tagged VLAN.
NetXtreme II User Guide September 2013 VLANs enable you to add multiple virtual adapters that are on different subnets. The benefit of this is that your system can have one network adapter that can belong to multiple subnets. NOTE: VLANs can only be created when all team members are Broadcom adapters.
User Guide NetXtreme II September 2013 16. Type the VLAN name and then click Next.
NetXtreme II User Guide September 2013 17. To tag the VLAN, select Tagged and then click Next. Otherwise, click Untagged, click Next, and continue with the wizard to add additional VLANs (see Step 19. of this procedure).
User Guide NetXtreme II September 2013 18. Type the VLAN tag value and then click Next. The value must be between 1 and 4094.
NetXtreme II User Guide September 2013 19. Select Yes to add or manage another VLAN and then click Next. Repeat until you do not want to add or manage any additional VLANs. NOTE: You can define up to 64 VLANs per team (63 VLANs that are tagged and 1 VLAN that is not tagged). Adding several VLANS may slow down the reaction time of the Windows interface due to memory and processor time usage for each VLAN. The degree to which Windows performance may suffer depends on system configuration.
User Guide NetXtreme II September 2013 20. To apply and commit the changes to the team, select Commit changes to system and Exit the wizard. To apply your changes but continue using the wizard, select Save changes and continue to manage more teams. Click Finish.
NetXtreme II User Guide September 2013 NOTE: At any point in the Broadcom Teaming Wizard procedure, click Preview to get a visual representation of what the team will look like before committing any changes.
User Guide NetXtreme II September 2013 21. Click the team name in the Team Management pane to view the team's properties in the Information tab, transfer and receive data in the Statistics tab, and team customization options in the Configurations tab. USING EXPERT MODE Use Expert Mode to create a team, modify a team, add a VLAN, and configure LiveLink for a Smart Load Balance and Failover and SLB (Auto-Fallback Disable) team. To create a team using the wizard, see Using the Broadcom Teaming Wizard.
NetXtreme II User Guide September 2013 NOTE: The Create Team tab appears only if there are teamable adapters available. 4. Click the Team Name field to enter a team name. 5. Click the Team Type field to select a team type. 6. Click Hyper-V Mode if you want to enable Windows virtualization services. See “Microsoft Virtualization with Hyper-V” in the “Troubleshooting” topic in the NetXtreme II Network Adapter User Guide for more information about this feature. 7.
User Guide NetXtreme II September 2013 NOTE: Adding a network adapter to a team where its driver is disabled may negatively affect the offloading capabilities of the team. This may have an impact on the team’s performance. Therefore, it is recommended that only driver-enabled network adapters be added as members to a team. 9. Type the value for Team MTU. 10. Click Create to save the team information. 11. Repeat steps 4. through 10. to define additional teams.
NetXtreme II User Guide September 2013 1. From the Team menu, click Edit Team, or right-click one of the teams in the list and select Edit Team. This option is only available if a team has already been created and is listed in the Team Management pane. 2. The wizard Welcome screen appears. Click Next to continue modifying a team using the wizard or click Expert Mode to work in Expert Mode. NOTE: The Edit Team tab in Expert Mode appears only if there are teams configured on the system. 3.
User Guide NetXtreme II September 2013 You can define up to 64 VLANs per team (63 VLANs that are tagged and 1 VLAN that is not tagged). VLANs can only be created when all teams members are Broadcom adapters. If you try to create a VLAN with a non-Broadcom adapter, an error message is displayed. To configure a team with a VLAN 1. From the Teams menu, select Add VLAN. 2. The Welcome screen appears. 3. Click Expert Mode. 4. On the Create Team tab of the Manage Teams window, click Manage VLAN(s). 5.
NetXtreme II User Guide September 2013 Read the following notes before you attempt to configure LiveLink. NOTES: • Before you begin configuring LiveLink™, review the description of LiveLink. Also verify that each probe target you plan to specify is available and working. If the IP address of the probe target changes for any reason, LiveLink must be reconfigured.
User Guide NetXtreme II September 2013 Saving and Restoring a Team Configuration To save a configuration 1. From the File menu, select Team Save As. 2. Type the path and file name of the new configuration file, and then click Save. The configuration file is a text file that can be viewed by any text editor. The file contains information about both the adapter and the team configuration. To restore a configuration 1. From the File menu, select Team Restore. 2.
NetXtreme II User Guide September 2013 VIEWING BASP STATISTICS The Statistics section shows performance information about the network adapters that are on a team. To view BASP Statistics information for any team member adapter or the team as a whole, click the name of the adapter or team listed in the Team Management pane, then click the Statistics tab. Click Refresh to get the most recent values for each statistic. Click Reset to change all values to zero.
User Guide NetXtreme II September 2013 CONFIGURING WITH THE COMMAND LINE INTERFACE UTILITY An alternate method to BACS for configuring Broadcom network adapters is with BACSCLI, which is a Broadcom utility that allows you to view information and configure network adapters using a console in either a non-interactive command line interface (CLI) mode or an interactive mode.
NetXtreme II User Guide September 2013 S pe c i f ic a t i on s : B ro ad c o m N e t X t r e m e I I ® Network Adapter Us er Guide • 10/100/1000BASE-T and 10GBASE-T Cable Specifications • 1000/2500BASE-X Fiber Optic Specifications • Interface Specifications • NIC Physical Characteristics • NIC Power Requirements • Wake On LAN Power Requirements • Environmental Specifications 10/100/1000BASE-T AND 10GBASE-T CABLE SPECIFICATIONS Table 1: 10/100/1000BASE-T Cable Specifications Port Type Conne
User Guide NetXtreme II September 2013 1000/2500BASE-X FIBER OPTIC SPECIFICATIONS Table 3: 1000/2500BASE-X Fiber Optic Specifications Port Type Connector Media Maximum Distance 1000BASE-X Small form-factor pluggable (SFP) transceiver with LC™ connection system (Infineon p/n V23818-K305L57) Multimode fiber (MMF) System optimized for 62.
NetXtreme II User Guide September 2013 NIC PHYSICAL CHARACTERISTICS PCI Table 6: NIC Physical Characteristics NIC Type NIC Length NIC Width BCM5708 PCI Express 14.7 cm (5.79 inches) 6.4 cm (2.52 inches) BCM5709/BCM5716 PCI Express x4 low-profile 11.9 cm (4.7 inches) 6.9 cm (2.7 inches) BCM57710/BCM57711/BCM57712 PCI Express x8 low profile 16.8 cm (6.6 inches) 5.1 cm (2.0 inches) NIC POWER REQUIREMENTS Table 7: BCM5708C NIC Power Requirements Link NIC 3.
User Guide NetXtreme II September 2013 WAKE ON LAN POWER REQUIREMENTS The tables below show the Wake On LAN power requirements for 1G adapters. Table 10: BCM5708C Wake On LAN Power Requirements (Nominal Conditions) 100 Mbit Link NIC 3.3V Current (mA) 10 Mbit Link NIC Power (W) 236 NIC 3.3V Current (mA) NIC Power (W) 150 0.5 0.78 Table 11: BCM5709C and BCM5716 Wake On LAN Power Requirements (Nominal Conditions) 100 Mbit Link NIC 3.3V Current (mA) 10 Mbit Link NIC Power (W) 0 NIC 3.
NetXtreme II User Guide September 2013 R e gu l a t o r y I n for m a t i on : Broadcom NetXtreme II ® Ne twork Adapter Us er Guide • FCC Notice • VCCI Notice • CE Notice • Canadian Regulatory Information (Canada Only) • Korea Communications Commission (KCC) Notice (Republic of Korea Only) FCC NOTICE FCC, CLASS B Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA The equipment complies with Part 15 of the FCC Rules.
User Guide NetXtreme II September 2013 FCC, CLASS A Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom NetXtreme II 10 Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA This device complies with Part 15 of the FCC Rules. Operations is subject to the following two conditions: 1) This device may not cause harmful interference, and 2) This device must accept any interference received, including interference that may cause undesired operation.
NetXtreme II User Guide September 2013 VCCI NOTICE CLASS B Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA The equipment is a Class B product based on the standard of the Voluntary Control Council for Interference from Information Technology Equipment (VCCI). If used near a radio or television receiver in a domestic environment, it may cause radio interference. Install and use the equipment according to the instruction manual.
User Guide NetXtreme II September 2013 CLASS A Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom NetXtreme II 10 Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA This equipment is a Class A product based on the standard of the Voluntary Control Council for interference by Information Technology Equipment (VCCI). If used in a domestic environment, radio disturbance may arise. Install and use the equipment according to the instruction manual.
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 300 CE Notice Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 Bro adco m Co rp or atio n Document INGSRVT78-CDUM100-R CE Notice Page 301
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 302 CE Notice Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 Bro adco m Co rp or atio n Document INGSRVT78-CDUM100-R CE Notice Page 303
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 304 CE Notice Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 CANADIAN REGULATORY INFORMATION (CANADA ONLY) INDUSTRY CANADA, CLASS B Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA This Class B digital apparatus complies with Canadian ICES-003. Notice: The Industry Canada regulations provide that changes or modifications not expressly approved by Broadcom could void your authority to operate this equipment.
NetXtreme II User Guide September 2013 INDUSTRY CANADA, CLASSE B Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA Cet appareil numérique de la classe B est conforme à la norme canadienne ICES-003. Avis : Dans le cadre des réglementations d'Industry Canada, vos droits d'utilisation de cet équipement peuvent être annulés si des changements ou modifications non expressément approuvés par Broadcom y sont apportés.
User Guide NetXtreme II September 2013 KOREA COMMUNICATIONS COMMISSION (KCC) NOTICE (REPUBLIC OF KOREA ONLY) B CLASS DEVICE Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA Note that this device has been approved for non-business purposes and may be used in any environment, including residential areas.
NetXtreme II User Guide September 2013 A CLASS DEVICE Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom NetXtreme II 10 Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA Bro adco m C orp or atio n Page 308 Korea Communications Commission (KCC) Notice (Republic of Korea Only) Document INGSRVT78-CDUM100-R
User Guide NetXtreme II September 2013 U ser Diag nostics in DOS : Broa dc om N etXtrem e II ® Ne twork Adapter Us er Guide • Introduction • System Requirements • Performing Diagnostics • Diagnostic Test Descriptions INTRODUCTION Broadcom NetXtreme II User Diagnostics is an MS-DOS based application that runs a series of diagnostic tests (see Table 3) on the Broadcom NetXtreme II network adapters in your system.
NetXtreme II User Guide September 2013 PERFORMING DIAGNOSTICS At the MS-DOS prompt, type uxdiag (for BCM5706/BCM5708/BCM5709 network adapters) or uediag (for BCM577XX and BCM578XX network adapters) followed by the command options. The uxdiag command options are shown in Table 1 and the uediag command options are shown in Table 2.
User Guide NetXtreme II September 2013 Table 1: uxdiag Command Options (Cont.) Command Options Description uxdiag -mbas Sets the MBA/PXE speed. 0 = Auto 1 = 10H 2 = 10F 3 = 100H 4 = 100F 6 = 1000F uxdiag -mbav <1|0> Enables/disables MBA VLAN. 1 = Enable 0 = Disable uxdiag -mbavval Sets MBA VLAN (<65536). uxdiag -mfw <1/0> Enables/disables management firmware. 1 = Enable 0 = Disable uxdiag -t Disables certain groups/tests.
NetXtreme II User Guide September 2013 Table 2: uediag Command Options Command Options Description uediag Performs all tests on all Broadcom NetXtreme II adapters in your system. uediag -c Specifies the adapter (device#) to test. Similar to -dev (for backward compatibility). uediag -cof Allows tests to continue after detecting a failure. uediag -dev Specifies the adapter (device#) to test. uediag -F Forces an upgrade of the image without checking the version.
User Guide NetXtreme II September 2013 Table 2: uediag Command Options (Cont.) Command Options Description uediag -wol <1/0> Enables/disable Magic Packet WOL.
NetXtreme II User Guide September 2013 DIAGNOSTIC TEST DESCRIPTIONS The diagnostic tests are divided into four groups: Basic Functional Tests (Group A), Memory Tests (Group B), Block Tests (Group C), and Ethernet Traffic Tests (Group D). The diagnostic tests are listed and described in Table 3.
User Guide NetXtreme II September 2013 Table 3: Diagnostic Tests (Cont.) Test Description Number Name Group C: Block Tests C1 CPU Logic and DMA Interface Verifies the basic logic functionality of all the on-chip CPUs. It also exercises the DMA interface exposed to those CPUs. The internal CPU tries to initiate DMA activities (both read and write) to system memory and then compares the values to confirm that the DMA operation completed successfully.
NetXtreme II User Guide September 2013 Table 3: Diagnostic Tests (Cont.) Test Description Number Name D2 PHY Loopback Enables PHY loopback mode in the adapter and transmits 5000 Layer 2 packets of various sizes. As the packets are received back by Broadcom NetXtreme II User Diagnostics, they are checked for errors. Packets are returned through the PHY receive path and never reach the wire. The adapter should not be connected to a network.
User Guide NetXtreme II September 2013 Tro uble sh oo t in g : Broa dc om Ne tXtrem e II ® Network Adapter Us er Guide • Hardware Diagnostics • Checking Port LEDs • Troubleshooting Checklist • Checking if Current Drivers are Loaded • Running a Cable Length Test • Testing Network Connectivity • Microsoft Virtualization with Hyper-V • Removing the Broadcom NetXtreme II Device Drivers • Upgrading Windows Operating Systems • Broadcom Boot Agent • Broadcom Advanced Server Program (BASP)
NetXtreme II User Guide September 2013 TROUBLESHOOTING CHECKLIST CAUTION! Before you open the cabinet of your server to add or remove the adapter, review Safety Precautions. The following checklist provides recommended actions to take to resolve problems installing the Broadcom NetXtreme II adapter or running it in your system. • Inspect all cables and connections. Verify that the cable connections at the network adapter and the switch are attached properly.
User Guide NetXtreme II September 2013 CHECKING IF CURRENT DRIVERS ARE LOADED WINDOWS See Viewing Vital Signs to view vital information about the adapter, link status, and network connectivity. LINUX To verify that the bnx2.o driver is loaded properly, run: lsmod | grep -i If the driver is loaded, the output of this command shows the size of the driver in bytes and the number of adapters configured and their names.
NetXtreme II User Guide September 2013 RUNNING A CABLE LENGTH TEST For Windows operating systems, see Analyzing Cables in Windows for information on running a cable length test. Cable analysis is not available for NetXtreme II 10 GbE network adapters. TESTING NETWORK CONNECTIVITY NOTE: When using forced link speeds, verify that both the adapter and the switch are forced to the same speed. WINDOWS Network connectivity can be tested using the Testing the Network feature in Broadcom Advanced Control Suite.
User Guide NetXtreme II September 2013 MICROSOFT VIRTUALIZATION WITH HYPER-V Microsoft Virtualization is a hypervisor virtualization system for Windows Server 2008 and Windows Server 2008 R2. This section is intended for those who are familiar with Hyper-V, and it addresses issues that affect the configuration of NetXtreme II network adapters and teamed network adapters when Hyper-V is used. For more information on Hyper-V, see http://www.microsoft.com/windowsserver2008/en/us/hyperv.aspx.
NetXtreme II User Guide September 2013 SINGLE NETWORK ADAPTER Windows Server 2008 When configuring a NetXtreme II network adapter on a Hyper-V system, be aware of the following: • An adapter that is to be bound to a virtual network should not be configured for VLAN tagging through the driver’s advanced properties. Instead, Hyper-V should manage VLAN tagging exclusively.
User Guide NetXtreme II September 2013 TEAMED NETWORK ADAPTERS Table 2 identifies Hyper-V supported features that are configurable for NetXtreme II teamed network adapters. This table is not an all-inclusive list of Hyper-V features. Table 2: Configurable Teamed Network Adapter Hyper-V Features Supported in Windows Server Version Feature Comments/Limitation 2008 2008 R2 2012 Smart Load Balancing and Failover (SLB) team type Yes Yes Yes Multi-member SLB team allowed with latest BASP6 version.
NetXtreme II User Guide September 2013 Windows Server 2008 When configuring a team of NetXtreme II network adapters on a Hyper-V system, be aware of the following: • Create the team prior to binding the team to the Hyper-V virtual network. • Create a team only with an adapter that is not already assigned to a Hyper-V virtual network. • A TOE-enabled team that is bound to a Hyper-V virtual network will report TOE as an offload capability in BACS; however, TOE will not work.
User Guide NetXtreme II September 2013 REMOVING THE BROADCOM NETXTREME II DEVICE DRIVERS Uninstall the Broadcom NetXtreme II device drivers from your system only through the InstallShield wizard. Uninstalling the device drivers with Device Manager or any other means may not provide a clean uninstall and may cause the system to become unstable. For information on uninstalling Broadcom NetXtreme II device drivers, see Removing the Device Drivers.
NetXtreme II User Guide September 2013 BROADCOM ADVANCED SERVER PROGRAM (BASP) Problem: After physically removing a NIC that was part of a team and then rebooting, the team did not perform as expected. Solution: To physically remove a teamed NIC from a system, you must first delete the NIC from the team. Not doing this before shutting down could result in breaking the team on a subsequent reboot, which may result in unexpected team behavior.
User Guide NetXtreme II September 2013 Problem: The advanced properties of a team do not change after changing the advanced properties of an adapter that is a member of the team. Solution: If an adapter is included as a member of a team and you change any advanced property, then you must rebuild the team to ensure that the team’s advanced properties are properly set.
NetXtreme II User Guide September 2013 LINUX Problem: BCM5771x devices with SFP+ Flow Control default to Off rather than Rx/Tx Enable. Solution: The Flow Control default setting for revision 1.6.x and newer has been changed to Rx Off and Tx Off because SFP+ devices do not support Autonegotiation for Flow Control. Problem: On kernels older than 2.6.
User Guide NetXtreme II September 2013 MISCELLANEOUS Problem: The BCM57810 10 GbE NIC does not support 10 Gbps or 1 Gbps WOL link speed. Solution: The BCM57810 10 GbE NIC can only support 100 Mbps WOL link speed due to power consumption limitations. Problem: When setting the Jumbo MTU property to 5000 bytes or greater and forcing Flow Control on network adapters that support a link speed of 10 Gbps, the system performance performs at less than optimal levels.
NetXtreme II User Guide September 2013 Solution: Ensure that the system has at least 2 GB of main memory when using up to four network adapters and 4 GB of main memory when using four or more network adapters. Problem: Remote installation of Windows Server 2008 to an iSCSI target via iSCSI offload fails to complete, and the computer restarts, repeatedly. Solution: This is a Microsoft issue.
User Guide NetXtreme II September 2013 Bro adco m Co rp or atio n Document INGSRVT78-CDUM100-R Miscellaneous Page 331
NetXtreme II User Guide September 2013 Bro adco m C orp or atio n Page 332 Miscellaneous Document INGSRVT78-CDUM100-R
