NetXtreme-E User Guide for Dell Platforms User Guide Broadcom NetXtreme-E-UG304-2CS March 23, 2021
NetXtreme-E User Guide User Guide for Dell Platforms Broadcom, the pulse logo, NetXtreme, TruTrust, Connecting everything, Avago Technologies, Avago, and the A logo are among the trademarks of Broadcom and/or its affiliates in the United States, certain other countries, and/or the EU. Copyright © 2020 – 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. For more information, please visit www.broadcom.com.
NetXtreme-E User Guide User Guide for Dell Platforms Table of Contents 1 Regulatory and Safety Approvals...............................................................................................................................7 1.1 Class A Warning Statements ...............................................................................................................................8 1.2 Class B Warning Statements ......................................................................................
NetXtreme-E User Guide User Guide for Dell Platforms 4.10 Security ............................................................................................................................................................37 4.11 RDMA over Converged Ethernet – RoCE........................................................................................................39 4.12 VMWare Enhanced Networking Stack (ENS) .............................................................................................
NetXtreme-E User Guide User Guide for Dell Platforms 9.1.5 NIC Partitioning Configuration Menu .......................................................................................................58 9.2 BIOS Comprehensive Configuration Management............................................................................................60 9.2.1 Main Menu ...............................................................................................................................................61 9.2.
NetXtreme-E User Guide User Guide for Dell Platforms 13.1 Network Diagram ............................................................................................................................................. 99 13.2 VEB and VEPA Modes ..................................................................................................................................100 13.2.1 VLAN Configuration ...........................................................................................................
NetXtreme-E User Guide User Guide for Dell Platforms 1 Regulatory and Safety Approvals The following sections detail the regulatory approvals, safety approvals, and warning statements for the NetXtreme®-E network interface cards. See the individual data sheets for the product classification and referenced standard dates.
NetXtreme-E User Guide User Guide for Dell Platforms 1.1 Class A Warning Statements FCC Warning Statement This device complies with part 15 of the FCC Rules. Operation is subject to the following two conditions: This device may not cause harmful interference. This device must accept any interference received, including interference that may cause undesired operation.
NetXtreme-E User Guide User Guide for Dell Platforms 1.2 Class B Warning Statements FCC Warning Statement This equipment has been tested and found to comply with the limits for a Class B digital device, pursuant to part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference in a residential installation.
NetXtreme-E User Guide User Guide for Dell Platforms 2 Functional Description The Broadcom NetXtreme-E (BCM9574XX and BCM9575XX) family of Ethernet controllers are highly-integrated, fullfeatured Ethernet LAN controllers optimized for data center and cloud infrastructures. These controllers support 200G/100G/ 50G/40G/25G/10G/1G in single or dual-port configurations. All of these controllers support sixteen lanes of PCIe 3. The BCM9575XX family additionally supports PCIe 4.
NetXtreme-E User Guide User Guide for Dell Platforms Table 3: Functional Description (Continued) Network Interface Card Description UEFI Name Broadcom Dual 10GBASE-T Ethernet BCM957504-425D Speed PCIe Interface Device NDIS Name UEFI Name Quad-Port 25Gb/s Ethernet Gen 4 x16 SFP 28 for 25 Gb/s Broadcom BCM57504 50G Gb/s MAC controller with integrated quad-channel 50 Gb/s SFI transceiver. Broadcom NetXtreme-E Series Quad-Port 25Gb SFP28 OCP 3.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 2: BCM957414A4141D Network Interface Card NOTE: Broadcom Figure 2 shows the low-profile bracket installed. The surface markings of the component may not reflect the product upon receive. Broadcom reserves the right to change any component on the printed circuit board with the same functionality.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 3: BCM957416A4160D Network Interface Card NOTE: Broadcom Figure 3 shows the low-profile bracket installed. The surface markings of the component may not reflect the product upon receive. Broadcom reserves the right to change any component on the printed circuit board with the same functionality.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 4: BCM957414M4140D Network Daughtercard (rNDC) Figure 5: BCM957412M4120D Network Daughtercard (rNDC) Broadcom NetXtreme-E-UG304-2CS 14
NetXtreme-E User Guide User Guide for Dell Platforms Figure 6: BCM957416M4160 Network Daughtercard (rNDC) Figure 7: BCM957412N4120 Small-Form-Factor Network Adapter Broadcom NetXtreme-E-UG304-2CS 15
NetXtreme-E User Guide User Guide for Dell Platforms Figure 8: BCM957414N4140 Small-Form-Factor Network Adapter Figure 9: BCM957416N4160 Small-Form-Factor Network Adapter Broadcom NetXtreme-E-UG304-2CS 16
NetXtreme-E User Guide User Guide for Dell Platforms Figure 10: BCM957504-NGM250D Mezzanine Card Broadcom NetXtreme-E-UG304-2CS 17
NetXtreme-E User Guide User Guide for Dell Platforms Figure 11: BCM957504-N425D OCP 3.0 SFF Card 3 Network Link and Activity Indication Ethernet connections, the state of the network link, and activity are indicated by the LEDs on the rear connector as shown in Table 4. See the individual board data sheets for specific media design.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 12: BCM957412AXXXX Activity and Link LED Locations Port 1 SFP+ Cage Port 1 Activity LED Port 1 Link LED Port 2 SFP+ Cage Port 2 Activity LED NOTE: Port 2 Link LED Figure 12 shows the low-profile bracket installed. The surface markings of the component may not reflect the product upon receipt. Broadcom reserves the right to change any component on the printed circuit board with the same functionality.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 13: BCM957414AXXXX Activity and Link LED Locations Port 1 SFP28 Cage Port 1 Activity LED Port 1 Link LED Port 2 SFP28 Cage Port 2 Activity LED NOTE: Port 2 Link LED Figure 13 shows the low-profile bracket installed. The surface markings of the component may not reflect the product upon receipt. Broadcom reserves the right to change any component on the printed circuit board with the same functionality.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 14: BCM957416AXXXX Activity and Link LED Locations Port 1 Link LED Port 1 Activity LED Port 2 Link LED Port 2 Activity LED Table 7: BCM957416AXXXX Activity and Link LED Locations LED Type Color/Behavior Notes Activity Off No Activity Green blinking Traffic Flowing Activity Off No Link Green Linked at 10 Gb/s Amber Linked at 1 Gb/s Link 3.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 15: BCM957414M4140D Network Daughtercard (rNDC) Activity and Link LED Locations Port 1 Link LED Port 1 Port 2 Link LED Port 2 Port 1 Activity LED Port 2 Activity LED Table 8: BCM957414M4140D Network Daughtercard (rNDC) Activity and Link LED Locations LED Type Color/Behavior Notes Activity Off No Activity Green blinking Traffic Flowing Activity Off No Link Green Linked at 25 Gb/s Yellow Linked at 10 Gb/s Link NOTE: When a XTY
NetXtreme-E User Guide User Guide for Dell Platforms Figure 16: BCM957412M4120D Network Daughtercard (rNDC) Activity and Link LED Locations Port 1 Link LED Port 1 Activity LED Port 2 Link LED Port 3 Link LED Port 2 Activity LED Port 4 Link LED Port 3 Activity LED Port 4 Activity LED Table 9: BCM957412M4120D Network Daughtercard (rNDC) Activity and Link LED Locations SFP+ Port 1 and 2 LED Type Color/Behavior Notes Activity Off No Activity Green blinking Traffic Flowing Activity Link Off
NetXtreme-E User Guide User Guide for Dell Platforms Figure 17: BCM957416M4160 Network Daughtercard (rNDC) Activity and Link LED Locations Port 1 Link LED Port 2 Link LED Port 1 Activity LED Port 3 Link LED Port 2 Activity LED Port 4 Link LED Port 3 Activity LED Port 4 Activity LED Table 11: BCM957416M4160 Network Daughtercard (rNDC) Activity and Link LED Locations 10GBASET Port 1 and 2 LED Type Color/Behavior Notes Activity Off No Activity Green blinking Traffic Flowing Activity Link Of
NetXtreme-E User Guide User Guide for Dell Platforms Table 12: BCM957412N4120DC Network Adapter Activity and Link LED Locations LED Type Color/Behavior Notes Activity Off No Activity Green blinking Traffic Flowing Activity Off No Link Green Linked at 10 Gb/s Link NOTE: When a XTY28 transceiver is connected to a 10 Gb/s controller and if the link establishes at 1 Gb/s, the link LED is off and the activity LED blinks. 3.
NetXtreme-E User Guide User Guide for Dell Platforms 3.9 BCM957416N4160DC Each Ethernet interface has a link LED to indicate Link status and an activity LED to indicate data traffic. The LEDs are shown in Figure 20 and described Table 14. Its locations and form factors conform to the OCP 3.0 Design Specification.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 21: Activity and Link LED Locations Port 2 Port 2 Link LED Activity LED Port 1 Link LED Port 4 Port 4 Link LED Activity LED Port 3 Port 3 Link LED Activity LED Port 1 Activity LED Table 15: BCM957504-N425D LED Functions LED Type Color/Behavior Note Activity Off No Link Green (blinking) Link up (traffic flowing) Off No Link Green Linked at 25 Gb/s Amber Linked at lower speed Link 4 Features See the following sections for device
NetXtreme-E User Guide User Guide for Dell Platforms Table 16: Host Interface Features (Continued) Feature Details MSI-X Interrupt vector per queue 1 per RSS queue, 1 per NetQueue, 1 per Virtual Machine Queue (VMQ). IP Checksum Offload Support for transmit and receive side. TCP Checksum Offload Support for transmit and receive side. UDP Checksum Offload Support for transmit and receive side. NDIS TCP Large Send Offload Support for LSOV1 and LSOV2.
NetXtreme-E User Guide User Guide for Dell Platforms Table 17: Virtualization Features (Continued) Feature Details Supported. Virtual eXtensible LAN (VXLAN) – Aware stateless offloads (IP/ UDP/TCP checksum offloads, VLAN insertion/removal, NetQueue, VMQ, RSS, TCP segmentation offload, Large Send Offload, Generic Receive Offload).
NetXtreme-E User Guide User Guide for Dell Platforms 4.3 VXLAN A Virtual eXtensible Local Area Network (VXLAN), defined in IETF RFC 7348, is used to address the need for overlay networks within virtualized data centers accommodating multiple tenants. VXLAN is a Layer 2 overlay or tunneling scheme over a Layer 3 network. Only VMs within the same VXLAN segment can communicate with each other. 4.
NetXtreme-E User Guide User Guide for Dell Platforms 4.5.5 Header and Data Split Header-payload split is a feature that enables the software TCP/IP stack to receive TCP/IP packets with header and payload data split into separate buffers. The support for this feature is available in both Windows and Linux environments. The following are potential benefits of header-payload split: The header-payload split enables compact and efficient caching of packet headers into host CPU caches.
NetXtreme-E User Guide User Guide for Dell Platforms 4.6 Priority Flow Control Priority Flow Control (PFC) is a standard-compliant backpressure mechanism implemented in the NetXtreme-E controllers. The goal of PFC is to backpressure congested priority traffic flow without affecting the traffic flows of uncongested priorities. PFC can be used in a network with real-time or time-sensitive traffic because of its capability to provide differential treatment to Traffic Classes.
NetXtreme-E User Guide User Guide for Dell Platforms The NetXtreme-E family of Ethernet controllers support Network Overlays or Tunneling, specifically VXLAN, variants of GRE, and IP-in-IP. Both VXLAN and NVGRE are defined to support larger scale than basic IEEE 802.1Q VLANs, using a 24-bit label space rather than 12-bit VID. Both are L2-in-L3 tunneling methods with NVGRE using GRE to carry the tunnel label and VXLAN using UDP to identify the tunnel label.
NetXtreme-E User Guide User Guide for Dell Platforms NVGRE is a method for network virtualization, similar to VXLAN in purpose. NVGRE uses a GRE header. The key field defined in RFC 2890 is used to carry a virtual subnet identifier. NVGRE is identified using EtherType = 0x6558, the EtherType for Transparent Ethernet Bridging (carrying a full Ethernet frame as the payload). 4.7.4.
NetXtreme-E User Guide User Guide for Dell Platforms The device supports NetQueue selection based on any combination of the fields above. NOTE: For GRE/IP-in-IP frames, the VMQ selection is performed using the Ethernet header of the encapsulated packet (inner packet) that includes inner destination MAC address and inner 802.1Q Tag (optional). 4.7.4.
NetXtreme-E User Guide User Guide for Dell Platforms SR-IOV uses IOMMU functionality to translate PCIe virtual addresses to physical addresses by using a translation table. The number of Physical Functions (PFs) and Virtual Functions (VFs) are managed through the UEFI HII menu, the CCM, and through NVRAM configurations. SR-IOV can be supported in combination with NPAR mode.
NetXtreme-E User Guide User Guide for Dell Platforms 4.8.1 SR-IOV Configuration Support Matrix Table 18 provides a SR-IOV support matrix. Table 18: SR-IOV Support Matrix SR-IOV Support Host OS Guest OS – VF Win2k16 Win2k19 RH7.3+ RH8.x SLES12.2+ SLES15.x Windows 2016 Yes Yes Yes Yes Yes Yes Windows 2019 Yes Yes Yes Yes Yes Yes RH7.8+ Yes Yes Yes Yes Yes Yes RH8.x Yes Yes Yes Yes Yes Yes SLES15.x Yes Yes Yes Yes Yes Yes ESX6.7+ Yes Yes Yes Yes Yes Yes ESX7.
NetXtreme-E User Guide User Guide for Dell Platforms Customer signed images do not run on other customers secure devices. This prevents piracy of customer images. However, it does not relieve the customer of the responsibility for protecting unsecured binaries from reverse engineering. Device or customer specific encrypted execution images can be generated. This prevents piracy of customer images and cloning of devices.
NetXtreme-E User Guide User Guide for Dell Platforms 4.11 RDMA over Converged Ethernet – RoCE Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) is a complete hardware offload feature in the NetXtreme-E that allows RDMA functionality over an Ethernet network. RoCE functionality is available in user mode and kernel mode application. RoCE Physical Functions (PF) and SR-IOV Virtual Functions (VF) are available in single function mode and in multi-function mode (NIC Partitioning mode).
NetXtreme-E User Guide User Guide for Dell Platforms 4.12.3 ENS Performance This section contains ENS performance information: 3-5x improvement in packet rate over the existing vSphere networking stack performance scales with the number of system threads. Acceptable packet loss. Low jitter and latency 1x 3-5x Guest: DPDK API + App Host: Default vSwitch Guest: DPDK API + App Host: ENS vSwitch.
NetXtreme-E User Guide User Guide for Dell Platforms Table 19: NPAR, SR-IOV, and RoCE (Continued) SW Feature Notes SR-IOV Up to 128 VFs (total per chip) RoCE on PFs Up to 4 PFs for the BCM5741X devices and 16 PFs for BCM5751X devices RoCE on VFs Valid for VFs attached to RoCE-enabled PFs Host OS Linux, Windows, ESXi (no vRDMA support) Guest OS Linux and Windows DCB Up to two COS per port with non-shared reserved memory NOTE: Certain 4 port BCM9575XX adapters support up to 32 VF and 4 NPAR p
NetXtreme-E User Guide User Guide for Dell Platforms 5.2.1 Hardware Requirements See the following list of hardware requirements: One open PCIe Gen 3 or Gen 4 slot in x8, x16, or x32 mode. – 574XXA41XX – PCIe Gen 3 slot in X8 mode. – 574XXM41XX – OCP 2.0 or rNDC slots. – 57XXXNXXXX – OCP 3.0 slot 16 GB memory or more (32 GB or more is recommended for virtualization applications and nominal network throughput performance). 5.2.
NetXtreme-E User Guide User Guide for Dell Platforms 5.3 Installing the Adapter The following instructions apply to installing the Broadcom NetXtreme-E Ethernet adapter (add-in NIC) into most servers. See the manuals that are supplied with the server for details about performing these tasks on this particular server. 1. Review the Safety Precautions and Preinstallation Checklist before installing the adapter.
NetXtreme-E User Guide User Guide for Dell Platforms 5.4.1.2 SFP+ The BCM957412AXXXX and BCM957412MXXXX adapters have two SFP+ connectors used for attaching the system to a 10 Gb/s Ethernet switch. 5.4.1.3 SFP28 The BCM957414XXXX, BCM957414AXXXX, BCM95750X adapters have two SFP28 connectors used for attaching the system to a 100 Gb/s Ethernet switch. 5.4.1.4 QSFP28/QSFP56 The BCM9575XX adapters have two QSFP28 connectors or a QSFP56 connector used for attaching the system to a 100 Gb/s Ethernet switch.
NetXtreme-E User Guide User Guide for Dell Platforms 6 Software Packages and Installation See the following sections for information on software packages and installation. 6.1 Supported Operating Systems Table 24 provides a list of supported operating systems. Table 24: Supported Operating Systems OS Flavor Distribution Windows Windows 2016 or 2019 Linux Redhat 7.9 or Redhat 8.3 SLES 15 SP 2 VMware ESXi 6.7 U3 or ESXi 7.0 U1 6.
NetXtreme-E User Guide User Guide for Dell Platforms Table 25: Linux Ethtool Commands (Continued) Command Description ethtool --set-eee ethX eee off Disable EEE. ethtool -L ethX combined 1 rx 0 tx 0 Disable RSS. Set the combined channels to 1. ethtool -K ethX ntuple on Enable Accelerated RFS. echo 32768 > /proc/sys/net/core/ rps_sock_flow_entries echo 2048 > /sys/class/net/ethX/queues/rx-X/ rps_flow_cnt Enable RFS for Ring X. ethtool --set-eee ethX eee on tx-lpi off Enable EEE, but disable LPI.
NetXtreme-E User Guide User Guide for Dell Platforms Table 26: VMware Commands (Continued) Command Description esxcfg-module –g bnxtnet Print module parameters. vmkload_mod –u bnxtnet Unload bnxtnet module. esxcli network nic set –n vmnicX –D full –S 25000 Set the speed and duplex of vmnicX. esxcli network nic up –n vmnic6 Enable vmnicX. esxcfg-module –s ‘multi_rx_filters=2 disable_tap=0 max_vfs=0,0 RSS=0’ Set the module parameters. vmkload_mod bnxtnet Load bnxtnet module.
NetXtreme-E User Guide User Guide for Dell Platforms 7 Updating the Firmware This section provides information for updating the adapter firmware. 7.1 Dell Update Package DUP packages can be downloaded from http://support.dell.com. See the following sections to use the Dell Update Package (DUP): 7.1.1 Windows Broadcom NetXtreme-E series controller firmware can be upgraded using the Dell DUP package. The executable is provided in standard Windows x64 executable format.
NetXtreme-E User Guide User Guide for Dell Platforms 8 Link Aggregation The following sections provide information on link aggregation. 8.1 Windows The Broadcom NetXtreme-E devices can aggregate network links using the Microsoft teaming feature. For more information on the NIC teaming functionality, see the Microsoft public documentation on Microsoft.com. Microsoft LBFO is a native teaming driver that can be used in the Windows OS. The teaming driver also provides VLAN tagging capabilities. 8.
NetXtreme-E User Guide User Guide for Dell Platforms 9 System-Level Configuration The following sections provide information on system-level NIC configuration. 9.1 UEFI HII Menu The Broadcom NetXtreme-E series controllers can be configured using the HII (Human Interface Infrastructure) menu at boot time. This menu system allows configuration of all persistent settings such as PXE, SR-IOV, NPAR, and so on. To enter the HII configuration menu, follow boot-time prompts to BIOS, then device configuration.
NetXtreme-E User Guide User Guide for Dell Platforms – Disabled – Turn off RDMA Blink LEDs – This setting allows the user to configure the duration for which the LEDs on the physical network port should blink to assist with port identification. This is a numeric setting. The value must be specified in the range 0 to 15 seconds. Link Status – This field displays the physical link status of the network port as reported by the controller. This is a readonly field.
NetXtreme-E User Guide User Guide for Dell Platforms 9.1.2 Firmware Image Menu This menu presents the various component versions present in the current firmware build. Depending on the adapter type, some components may not be available. All fields in this menu are read-only. Figure 24: Firmware Image Menu This page displays the following information (see Figure 24): Family Firmware Version – This field displays the family firmware version.
NetXtreme-E User Guide User Guide for Dell Platforms 9.1.3 Device Configuration Menu This menu presents adapter specific parameters for configuration. Depending on the adapter type, some settings may not be available. Figure 25: Device Configuration Menu This page allows the user to configure the following items (see Figure 25): Virtualization Mode – This setting configures the type of virtualization to be used by the controller on all ports.
NetXtreme-E User Guide User Guide for Dell Platforms NOTE: The maximum number of VFs supported by software is 128 VFs per device. Number of MSI-X Vectors per VF This setting allows the user to configure the MSI-X Vectors per VF. Message Signaled Interrupts (MSI) are an alternative in-band method of signaling an interrupt, using special in-band messages to replace traditional out-of-band assertion of dedicated interrupt lines. This is a numeric setting.
NetXtreme-E User Guide User Guide for Dell Platforms Auto-negotiation Protocol – This setting configures the Auto-negotiation protocol for the adapter. Auto-negotiation is a feature that allows a port on a switch, router, server, or other device to communicate with the device on the other end of the link to determine the optimal duplex mode and speed for the connection. This setting is not available on 10GBASE-T controllers.
NetXtreme-E User Guide User Guide for Dell Platforms 9.1.4 NIC Configuration This menu presents legacy boot related parameters for configuration. Depending on the adapter type, some settings may not be available. Figure 26: NIC Configuration The NIC Configuration menu consists of the following items (see Figure 26): Option ROM – This setting allows the user to control whether the Broadcom legacy option ROM driver should be advertised or not.
NetXtreme-E User Guide User Guide for Dell Platforms Setup Key Stroke – This setting allows users to select the hotkey combination for invoking the Broadcom legacy configuration setup menus (CCM). This setting is available only if legacy configuration menus are supported on the adapter. – Ctrl-S – Pressing the CTRL key along with S key will load CCM. – Ctrl-B – Pressing the CTRL key along with B key will load CCM.
NetXtreme-E User Guide User Guide for Dell Platforms 9.1.5 NIC Partitioning Configuration Menu Figure 27: NIC Partitioning Configuration Menu The NIC Partitioning Configuration screen has the following sub-menus (see Figure 27): Number of Partitions Per Port – This field displays the number of PCI Physical functions currently enabled on the current network port when MF mode is set to NPAR. This is a read-only field.
NetXtreme-E User Guide User Guide for Dell Platforms 9.1.5.1 Partition Configuration Menu This menu presents partition related parameters for configuration. Depending on the adapter type, some settings may not be available (see Figure 28).
NetXtreme-E User Guide User Guide for Dell Platforms BW Reservation – This setting configures the percentage of total available bandwidth that should be reserved for this partition. The total Bandwidth Reservation assigned for all active partitions cannot exceed 100. A value of 0 on all partitions indicates equal division of bandwidth between all partitions. This setting is available only on adapters that support the Bandwidth Reservation feature. This is a numeric setting.
NetXtreme-E User Guide User Guide for Dell Platforms 9.2.1 Main Menu Figure 29: CCM Main Menu Comprehensive Configuration Management v216.0.47.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms 9.2.2 Firmware Image Menu Figure 30: Firmware Image Menu Comprehensive Configuration Management v216.0.49.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved. Firmware Image Menu Bootcode Version MBA Version UEFI Version NC-SI Version CCM Version RDMA Version : : : : : : 214.4.60.0 214.0 237.0 214.0.288.0 214.4.27.0 216.0.49.0 214.0.193.
NetXtreme-E User Guide User Guide for Dell Platforms 9.2.3 Device Hardware Configuration Menu This menu presents adapter specific parameters for configuration. Depending on the adapter type, some settings are not available. Figure 31: Device Hardware Configuration Menu Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms This setting allows the user to configure the MSI-X Vectors per VF. Message Signaled Interrupts (MSI) are an alternative in-band method of signaling an interrupt, using special in-band messages to replace traditional out-of-band assertion of dedicated interrupt lines. This is a numeric setting. The value must be specified in the range 0 to 128.
NetXtreme-E User Guide User Guide for Dell Platforms – Disabled – Turn off LLDP nearest non-TPMRbridge Auto-negotiation Protocol – This setting configures the Auto-negotiation protocol for the adapter. Auto-negotiation is a feature that allows a port on a switch, router, server, or other device to communicate with the device on the other end of the link to determine the optimal duplex mode and speed for the connection. This setting is not available on 10GBASE-T controllers.
NetXtreme-E User Guide User Guide for Dell Platforms 9.2.4 MBA Configuration Menu Figure 32: MBA Configuration Menu Comprehensive Configuration Management v218.0.14.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms Banner Message Timeout – This setting configures the duration for which the Broadcom legacy option ROM banner is to be displayed on the screen during POST. This is a numeric setting. The value must be specified in the range 0 to 15 seconds. Wake on LAN – This setting configures Wake on LAN (WoL) which is the ability to remotely power on a server or to wake it up from sleep mode.
NetXtreme-E User Guide User Guide for Dell Platforms 9.2.5 NIC Partitioning Configuration Menu Figure 33: NIC Partitioning Configuration Menu Comprehensive Configuration Management v218.0.14.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 34: Partition Configuration Comprehensive Configuration Management v214.0.221.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved. PF# 0 Bandwidth Reservation Bandwith Limit Bandwith Reservation Valid Bandwith Limit Valid RDMA Support : : : : : 10 60 False False Disabled Configure bandwith Reservation in Percentage (All Partition Sum <= 100)(0..
NetXtreme-E User Guide User Guide for Dell Platforms 9.2.6 Exiting CCM To exit CCM, press ESC. Upon an attempt to exit CCM, if changes to any parameter values are detected, Save is prompted (see Figure 34). Figure 35: Exiting CCM Comprehensive Configuration Management v216.0.26.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms 9.3 Auto-Negotiation Configuration NOTE: In NPAR (NIC partitioning) devices where one port is shared by multiple PCI functions, the port speed is preconfigured and cannot be changed by the driver.
NetXtreme-E User Guide User Guide for Dell Platforms The supported combination of link speed settings for two port NetXtreme-E network controllers are shown in Table 27.
NetXtreme-E User Guide User Guide for Dell Platforms The expected link speeds based on the local and link partner settings are shown in Table 28 and Table 29.
NetXtreme-E User Guide User Guide for Dell Platforms 9.3.1 Operational Link Speed This option configures the link speed used by the OS driver and firmware. This setting is overridden by the driver setting in the OS present state. 9.3.2 Firmware Link Speed This option configures the link speed used by the firmware when the device is in D3. 9.3.3 Auto-Negotiation Protocol This is the supported auto-negotiation protocol used to negotiate the link speed with the link partner.
NetXtreme-E User Guide User Guide for Dell Platforms 9.3.6 ESXi Driver Settings NOTE: For 10GBASE-T NetXtreme-E network adapters, auto-negotiation must be enabled. Using forced speed on a 10GBASE-T adapter results in ESXCLI command failure. $ esxcli network nic get -n – This command shows the current speed, duplex, driver version, firmware version and link status. $ esxcli network nic set -S 10000 -D full -n – This command sets the forced speed to 10 Gb/s.
NetXtreme-E User Guide User Guide for Dell Platforms 10 PXE Boot To serve PXE requests from PXE clients, a PXE must be configured. The PXE server can be configured to run regular PXE or iPXE. Regular PXE is a network boot program that downloads config files over TFTP from the PXE server. iPXE is an enhanced implementation of the PXE client firmware and a network boot program which uses iPXE scripts rather than config files and can download scripts and images with HTTP.
NetXtreme-E User Guide User Guide for Dell Platforms 10.1.2 PXE The boot loader file is downloaded via TFTP from the PXE server automatically. The menu items from the grub.cfg file are listed. Additional files are downloaded from the PXE server over TFTP based on the menu selection and the PXE boot continues. 10.1.3 Secure Boot Secure boot is a UEFI based feature developed by UEFI forum to increase security in the pre-boot environment.
NetXtreme-E User Guide User Guide for Dell Platforms This feature halts the execution of unsigned code before the operating system boots. Any unsigned firmware running on the adapter is not allowed to run. Only adapters running digitally signed firmware images are listed in preboot configuration menus and in network boot options. In order to enable this security feature, select Enable Secure Boot in the System Setup Menu. This option is only available on server platforms that support secure boot.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 37: Enable Secure Boot Example Broadcom NetXtreme-E-UG304-2CS 79
NetXtreme-E User Guide User Guide for Dell Platforms 10.2 Legacy BIOS Mode To configure PXE for legacy BIOS mode: NOTE: For Legacy BIOS boot mode, only TFTP-based PXE over IPv4 is supported. 1. Set the Boot Protocol to PXE in the MBA Configuration menu. Comprehensive Configuration Management v218.0.14.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms 10.2.1 iPXE Download ipxe.pxe the boot loader file using TFTP from the PXE server. 10.2.2 PXE The boot loader file is downloaded using TFTP from the PXE server.
NetXtreme-E User Guide User Guide for Dell Platforms The menu items from the pxelinux.cfg/default file are listed. Additional files are downloaded from the PXE server over TFTP based on the menu selection and the PXE boot continues.
NetXtreme-E User Guide User Guide for Dell Platforms 10.3 PXE Boot with VLAN To configure PXE boot with VLAN: 1. Set Virtual LAN Mode to Enabled and assign a valid Virtual LAN ID in the MBA Configuration menu. Comprehensive Configuration Management v218.0.14.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms Once the boot loader is downloaded, press Ctrl+B to enter the iPXE menu Once in the iPXE shell, run the following commands to add a VLAN tag on the adapter interface. – Ifstat [to identify the adapter interface by its MAC address] – vcreate --tag – autoboot This starts the PXE boot on the test interface with VLAN.
NetXtreme-E User Guide User Guide for Dell Platforms 10.4.1 DHCP Configuration for PXE/iPXE To add the DHCP feature on the PXE server, use the following command: yum install dhcpd NOTE: All IP addresses mentioned in this section are for illustration purposes only. They can be modified to match the subnet that the PXE server is configured in. 10.4.1.
NetXtreme-E User Guide User Guide for Dell Platforms 10.4.1.1.2 PXE To run PXE, in the /etc/dhcp/dhcpd.conf file, make the following changes: ddns-update-style none; default-lease-time 600; option space PXE; option PXE.mtftp-ipcode 1 = ip-address; option PXE.mtftp-cport code 2 = unsigned integer option PXE.mtftp-sport code 3 = unsigned integer option PXE.mtftp-tmout code 4 = unsigned integer option PXE.
NetXtreme-E User Guide User Guide for Dell Platforms 10.4.1.2 IPv6 DHCP Configuration In /etc/sysconfig/network, make the following changes: NETWORKING_IPV6=yes IPV6FORWARDING=no IPV6_AUTOCONF=no IPV6_AUTOTUNNEL=no In /etc/sysconfig/network-scripts/ifcfg- make the following changes: IPV6_AUTOCONF=no IPV6INIT=yes IPV6ADDR=2015:9:19:ffff::10/64 # Replace with your static address 10.4.1.2.1 iPXE In /etc/dhcpd/dhcpd6.
NetXtreme-E User Guide User Guide for Dell Platforms option dhcp6.client-arch-type code 61 = array of unsigned integer 16; option dhcp6.name-servers 2015:9:19:ffff::10; subnet6 2015:9:19:ffff::/64 { range6 2015:9:19:ffff::11 2015:9:19:ffff::500; if exists dhcp6.client-arch-type and option dhcp6.client-arch-type = 00:07 { option dhcp6.bootfile-url "tftp://[2015:9:19:ffff::10]/BOOTX64.EFI "; } elsif exists dhcp6.client-arch-type and option dhcp6.client-arch-type = 00:00 { option dhcp6.
NetXtreme-E User Guide User Guide for Dell Platforms } The base path for the TFTP server is /var/lib/tftpboot/. 10.4.2.1 iPXE Under the TFTP root directory, create a new folder called ipxe and add the ipxe.efi and ipxe.pxe files to it. Also add the menu.ipxe file to it, which can chain more iPXE files to list the menu based on the boot mode. #!ipxe iseq ${platform} efi && goto uefibios || goto legacybios :uefibios echo Loading the UEFI Menu chain --replace --autofree ${menu-url}efimenu.
NetXtreme-E User Guide User Guide for Dell Platforms LABEL RHEL75 MENU LABEL ^RedHat 7u5 KERNEL images/rhel75/vmlinuz APPEND initrd=images/rhel75/initrd.img ramdisk_size=200000 ip=dhcp inst.xdriver=vesa nomodeset inst.repo=http://174.30.10.10/images/RHEL75 LABEL SLES12SP4 MENU LABEL ^SLES 12 SP 4 KERNEL images/sles12sp4/vmlinuz APPEND initrd=images/sles12sp4/initrd.img ramdisk_size=200000 ip=dhcp inst.xdriver=vesa nomodeset inst.repo=http://174.30.10.
NetXtreme-E User Guide User Guide for Dell Platforms a. SR-IOV in the NIC card can be enabled using the HII menu. During system boot, access the system BIOS→ NetXtreme-E NIC→Device Level Configuration menu. b. Set the Virtualization mode to SR-IOV. c. Set the number of virtual functions per physical function. d. Set the number of MSI-X vectors per the VF and Max number of physical function MSI-X vectors. If the VF is running out of resources, balance the number of MSI-X vectors per VM using CCM. 2.
NetXtreme-E User Guide User Guide for Dell Platforms Use "virsh start " Optional: virsh vcpupin 0 can be used to run the VM on a specific CPU. 11.3 Windows SR-IOV Use Case Example 1. Install the latest KB update for your Windows 2012 R2 or Windows 2016 OS. 2. Install the appropriate Virtualization (Hyper-V) options. For more detail requirements and steps on setting up Hyper-V, Virtual Switch, and Virtual Machine, visit Microsoft.com. 3.
NetXtreme-E User Guide User Guide for Dell Platforms esxcfg-module -s 'max_vfs=4,2' bnxtnet (reboot required) The required VFs of each supported PF are enabled in order during the PF bring up. See the VMware documentation for information on how to map a VF to a VM. NOTE: Broadcom When using NPAR + SR-IOV, every NPAR function (PF) is assigned a maximum of eight VFs.
NetXtreme-E User Guide User Guide for Dell Platforms 12 NPAR – Configuration and Use Case Example 12.1 Features and Requirements OS/BIOS Agnostic – The partitions are presented to the operating system as real network interfaces so no special BIOS or OS support is required like SR-IOV. Additional NIC functions without requiring additional switch ports, cabling, PCIe expansion slots.
NetXtreme-E User Guide User Guide for Dell Platforms 12.3 Configuration NPAR can be configured using BIOS configuration HII menus or by using the Broadcom CCM utility on legacy boot systems. Some vendors also expose the configuration via additional proprietary interfaces. To enable NPAR: 1. Select the target NIC from the BIOS HII Menu or CCM interface and set the Multi-Function Mode or Virtualization Mode option. The choice of options affects the whole NIC instead of the individual port.
NetXtreme-E User Guide User Guide for Dell Platforms 2. Once NPAR is enabled, the NIC Partitioning Main Configuration menu option is available from the main NIC Configuration Menu associated with each physical port. Comprehensive Configuration Management v216.0.47.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms for each partition are also accessible from this menu. For some OEM systems, the HII menu also includes a Global Bandwidth Allocation page where the minimum (reserved) and maximum (limit) TX bandwidth for all partitions can be configured. Comprehensive Configuration Management v218.0.14.0 Copyright (C) 2000-2019 Broadcom Limited All rights reserved.
NetXtreme-E User Guide User Guide for Dell Platforms Table 30: NPAR Parameters Parameter Description Valid Options BW Limit Maximum percentage of available bandwidth this partition is allowed. Value 0 to 100 BW Limit Valid Functions as an on/off switch for the BW Limit setting. True/False RDMA Support Functions as an on/off switch for RDMA support on this partition. NOTE: Only two partitions per physical port can support RDMA. For a dual-port device, up to 4 NPAR partitions can support RDMA.
NetXtreme-E User Guide User Guide for Dell Platforms 12.4 Reducing NIC Memory Consumption with NPAR The default value of receive buffers was selected to work well for typical configurations. If you have many NICs in a system, have enabled NPAR on multiple NICs, or if you have only a small amount of RAM, you may see a Code 12 yellow bang in the Device Manager for some of the NICs. Code 12 means that the driver failed to load because there were not enough resources.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 38: Network Diagram Virtual Machine Linux Tenanat 1 DEMO-VM1 Virtual Machine Linux Tenanat 2 DEMO-VM3 SR-IOV-VF0 5F:02:0 Virtual Machine Linux Tenanat 1 DEMO-VM2 SR-IOV-VF1 5F:02:1 SR-IOV-VF0 5F:03:0 5F:00:1 5F:00:0 Server-DEM02 RHEL 7.5 PF1 SFP28 PF0 SFP28 Z9100-ON : 25 Gb/s QSFP28 Ethernet Switch 13.2 VEB and VEPA Modes VEB (Virtual Ethernet Bridging) mode generates an internal bridge within the NIC for VM-to-VM communication.
NetXtreme-E User Guide User Guide for Dell Platforms 13.4 GRE Tunnelling An IP GRE is an IP inside an IP tunnel which can carry private network traffic between two heterogeneous networks. On the VM, use the following commands: modprobe ip_gre; ip tunnel add gre45 mode gre local remove ip link set dev gre45 up; ip addr add /mask broadcast dev gre45 In this example, gre45 is the interface name. 13.
NetXtreme-E User Guide User Guide for Dell Platforms Table 32: VXLAN Command and Configuration Examples (Continued) System 1 System 2 ping 192.168.1.10 – 14 RoCE – Configuration and Use Case Examples This section provides configuration and use case examples for Remote Direct Memory Access over Converged Ethernet (RoCE). NOTE: If using NPAR + SR-IOV mode, only two VFs from each parent physical port can enable RDMA support, or a total of four VFs + RDMA per physical device.
NetXtreme-E User Guide User Guide for Dell Platforms 14.2.1.2 BNXT_RE Driver Dependencies The RoCE driver (bnxt_re) depends on the L2 Linux driver (bnxt_en) networking counterpart. The bnxt_re driver depends on the IB stack available with the Linux kernel or the OFED IB stack. 14.2.1.
NetXtreme-E User Guide User Guide for Dell Platforms # yast2 --install libibverbs libipathverbs compat-dapl dapl libamso-rdmav2 libcxgb3-rdmav2 libcxgb4 -rdmav2 libmlx4-rdmav2 libmthca-rdmav2 libnes-rdmav2 librdmacm ib-bonding ibsim ibutils ibvexdmtools infiniband-diags libibcm libibcommon1 libibmad5 libibumad3 libsdp ofed ofed-doc ofedkmp-default ofed-kmp-pae ofed-kmp-trace opensm mstflint rds-tools ibutils libibcommon libibcommon1 srptools 14.2.2.
NetXtreme-E User Guide User Guide for Dell Platforms # lsmod | grep ib_ ib_isert 50818 0 iscsi_target_mod 291713 1 ib_isert ib_iser 47861 0 libiscsi 57233 1 ib_iser scsi_transport_iscsi 99909 2 ib_iser,libiscsi ib_srpt 43624 0 target_core_mod 340809 3 iscsi_target_mod,ib_srpt,ib_isert ib_srp 52589 0 scsi_transport_srp 20993 1 ib_srp ib_ipoib 115042 0 ib_ucm 22636 0 ib_uverbs 78543 2 ib_ucm,rdma_ucm ib_umad 22119 0 rdma_cm 59529 4 rpcrdma,ib_iser,rdma_ucm,ib_isert ib_cm 51741 5 rdma_cm,ib_srp,ib_ucm,ib_srp
NetXtreme-E User Guide node_guid: sys_image_guid: vendor_id: vendor_part_id: hw_ver: phys_port_cnt: port: User Guide for Dell Platforms 1 state: max_mtu: active_mtu: sm_lid: port_lid: port_lmc: link_layer: 0210:18ff: fead:1fa1 0210:18ff: fead:1fa1 0x14e4 5847 0x1405 1 PORT_ACTIVE (4) 4096 (5) 1024 (3) 0 0 0x00 Ethernet 14.2.4 RoCE Connectivity Tests This section provides information on how to test basic RoCE connectivity using rping. The test utilities use a server and client model during RoCE testing.
NetXtreme-E User Guide User Guide for Dell Platforms server ping data: rdma-ping-1051: XYZ[\]^_'abcdefghijklmnopqrstuvwxyzABCDEFGHIK server ping data: rdma-ping-1051: YZ[\]^_'abcdefghijklmnopqrstuvwxyzABCDEFGHIKM server ping data: rdma-ping-1051: Z[\]^_' 3. Basic ping pong test: a.
NetXtreme-E User Guide User Guide for Dell Platforms a. Server Side: ib_send_bw -d --report_gbits b. Client Side: ib_send_bw -d --report_gbits NOTE: With dual-port NICs, if both ports are on the same subnet, RDMA perftest commands may fail. The is caused by an arp flux issue in Linux. Use multiple subnets for testing or bringing the other interface down. 14.2.
NetXtreme-E User Guide User Guide for Dell Platforms 14.2.5.1 Using the Section This section is intended to provide information on the process of configuring a new server for RoCE use with RoCEv2 enabled, Congestion Control enabled, and optionally, PFC enabled and configured. Any installation of other vendor's proprietary Infiniband software stack should be removed before proceeding with this section. 14.2.5.2 Understanding Congestion Control 14.2.5.2.
NetXtreme-E User Guide User Guide for Dell Platforms 2. As the Egress Queue of the switch reaches the congestion marking threshold as defined by the user, the switch starts marking the ECN bits to CE(11b) to notify to the notification point that congestion has been detected for that port. 3. Once the CE bit is detected, the notification point transmits Congestion Notification Packet (CNP) packet(s) to the reaction points(s) to signify congestion. 4.
NetXtreme-E User Guide User Guide for Dell Platforms 14.2.5.4.2 Red Hat/CentOS To install the LLDP agent using RedHat/CentOS, use the following commands: sudo yum install lldpad sudo systemctl enable lldpad sudo systemctl start lldpad 14.2.5.5 BnxtQoS Similar to the LLDP Agent, the Broadcom QoS configuration utility is used to set QoS mappings, priority flow control, and to configure Enhanced Transmission Selection (ETS).
NetXtreme-E User Guide User Guide for Dell Platforms 14.2.5.6 DSCP vs. VLAN Mode NCC has two operational modes: VLAN and DSCP. If your network uses VLAN tagging for traffic, the VLAN header is used to track traffic priority values, necessary to separate RoCE traffic from other traffic for congestion control. If your network uses DSCP tagging for traffic, the proper DSCP mapping is required to be set.
NetXtreme-E User Guide User Guide for Dell Platforms bnxtqos -dev=$IFACE set_apptlv set_ets tsa=0:ets,1:ets,2:strict,3:strict,4:strict,5:strict,6:strict,7:strict up2tc=0:0,1:1,2:0,3:0,4:0,5:0,6:0,7:0 tcbw=10,90 bnxtqos -dev=$IFACE set_pfc enabled=1 #Uncomment if using LLDP Agent #/usr/sbin/lldptool -T -i $IFACE -V APP app=1,5,34* #/usr/sbin/lldptool -T -i $IFACE -V APP app=1,1,35093 #/usr/sbin/lldptool -T -i $IFACE -V APP app=1,3,4791 #/usr/sbin/lldptool -T -i $IFACE -V ETS-CFG tsa=0:ets,1:ets,2:strict,3:
NetXtreme-E User Guide User Guide for Dell Platforms #!/bin/sh export IFACE=$1 if [ -x /etc/ifup-roce-${IFACE} ]; then . /etc/ifup-roce-${IFACE} fi 2. Set the script to be executable with the following commands: sudo chmod 755 /sbin/ifup-local 3. Reconfigure the interface with the following commands: sudo ifdown sudo ifup 14.2.5.7 Congestion Control Settings Different topologies may require different tunings. See Performance for additional information. 14.2.5.7.
NetXtreme-E User Guide User Guide for Dell Platforms mkdir -p /sys/kernel/config/rdma_cm/bnxt_re0 echo RoCE v2 > /sys/kernel/config/rdma_cm/bnxt_re0/ports/1/default_roce_mode mkdir -p /sys/kernel/config/bnxt_re/bnxt_re0 cd /sys/kernel/config/bnxt_re/bnxt_re0/ports/1/cc/ # DSCP configuration echo -n 0x1 > roce_dscp echo -n 0x1 > roce_prio echo -n 0x1 > vlan_tx_disable # Set CC parameters for and enable echo -n 0x1 > ecn_enable echo -n 0x1 > ecn_marking echo -n 0xc8 > tcp_cp echo -n 0x3 > nph_per_state echo
NetXtreme-E User Guide User Guide for Dell Platforms Link Speed ECN Min./Max. Threshold (kilobytes) 25 Gb/s 16/16 50 Gb/s 22/22 100 Gb/s 33/33 Each switch port participating in NCC must be configured with the above ECN threshold for the class of service associated with RoCEv2 traffic. Different vendors have different naming conventions that specifies the minimum and maximum marking threshold as well as the marking percentage.
NetXtreme-E User Guide User Guide for Dell Platforms For example, the perftest tools can be run as: # ib_write_bw -d bnxt_re0 -x 3 -F --report_gbits -p 1800 -s -q -D 192.168.30.3 # ib_write_lat -d bnxt_re0 -x 3 -F --report_gbits -p 1800 -s 2 192.168.30.3 For a complete explenation of the parameters used by ib_write_bw/lat and other IB utilities, see github as previously described. 14.2.5.8.
NetXtreme-E User Guide User Guide for Dell Platforms Table 36: Active Resources Counters Description Active QP Number of active QPs. Active CQ Number of active CQs. Active MW Number of active Memory Windows. Active SRQ Number of active SRQs. Active MR Number of active Memory Regions. Table 37: Bytes and Packet Counters Counters Description RX Pkts Number of RoCE (v1/v2) packets received. TX Pkts Number of RoCE (v1/v2) packets transmitted. RX Bytes Number of RoCE (v1/v2) bytes received.
NetXtreme-E User Guide User Guide for Dell Platforms Table 40: Fatal Errors (Continued) Counters Description remote_invalid_req_err Number of invalid request received from the remote rdma initiator. remote_access_err remote_op_err Number of times H/W received a REMOTE ACCESS ERROR NAK from the peer. Number of times HW received a REMOTE OPERATIONAL ERROR NAK from the peer.
NetXtreme-E User Guide User Guide for Dell Platforms Table 42: PCI Errors Counters Description res_tx_pci_err Number of responder transmit PCI errors detected. res_rx_pci_err Number of responder receive PCI errors detected. 14.2.5.10 Applications 14.2.5.10.1 OpenMPI If OpenMPI is used, the OpenMPI source and configuration must be patched using the following commands: $ cd openmpi-x.y.z $ cat > openmpi.patch << EOF diff --git a/opal/mca/common/verbs/common_verbs_port.
NetXtreme-E User Guide User Guide for Dell Platforms Congestion control performance is measured by these network traffic performance metrics during periods of congestion: Fairness of bandwidth allocation between qp Link utilization Latency Under heavy congestion, NCC can enforce fairness at a per-QP level, with low variation between streams. Streams not crossing the point of network contention are not affected.
NetXtreme-E User Guide User Guide for Dell Platforms bnxt_en.conf to override to use updated version. Users can also erase the current NetXtreme-E Linux kernel driver using the rpm -e kmod-bnxt_en command. RHEL 7.3/SLES 12 Sp2 has bnxt_en inbox driver (older than v1.7.x). This driver must be removed and the latest bnxt_en be added before applying the bnxt_re (RoCE drivers). 14.3 Windows and Use Case Examples 14.3.
NetXtreme-E User Guide User Guide for Dell Platforms Get-NetOffLoadGlobalSetting Get-NetAdapterRDMA 14.3.1.2 User Mode Before running a user mode application written to NDSPI, copy and install the bxndspi.dll user mode driver. To copy and install the user mode driver: 1. Copy bxndspi.dll to C:\Windows\System32. 2. Install the driver by running the following command: rundll32.exe .\bxndspi.dll,Config install|more 14.3.1.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 41: Server/Target Configuration 2. Create seven folders named Share1 to Share7 and individually share them. Figure 42: Shared Folders 3. Configure the client by creating seven or more IP Address with different subnets that are accessible by the server system. For example: – 172.1.54.1 – 172.2.54.2 – 172.3.54.3 – 172.4.54.4 – 172.5.54.5 – 172.6.54.6 – 172.7.54.7 4. Map a network drive to the server as follows: – Drive letter:\\172.1.55.1\Share1
NetXtreme-E User Guide User Guide for Dell Platforms Figure 43: Drive Mapping Each of the network drives generate 16 RDMA connections for a total of 112 with seven drives. 14.3.1.6 Traffic Generation To exercise all 16 RDMA connections simultaneously using a different traffic generation tool: If MLTT (License Software) is being used: – Change the number of thread counts from 1 to 16. This ensures that all 16 channels have traffic running.
NetXtreme-E User Guide User Guide for Dell Platforms By default, the driver sets up two RDMA connections for each network share per IP address (on a unique subnet). Scale up the number of RDMA connections by adding multiple IP addresses, each with different a subnet, for the same physical port under test. Multiple network shares can be created and mapped to each link partner using the unique IP addresses created. Example: – On Server 1, create the following IP addresses for Network Port1: 172.1.10.
NetXtreme-E User Guide User Guide for Dell Platforms Figure 44: Perfmon for Verification (Part 1) Broadcom NetXtreme-E-UG304-2CS 127
NetXtreme-E User Guide User Guide for Dell Platforms Figure 45: Perfmon for Verification (Part 2) Broadcom NetXtreme-E-UG304-2CS 128
NetXtreme-E User Guide User Guide for Dell Platforms Figure 46: Perfmon for Verification (Part 3) 14.3.1.8.3 Powershell PowerShell can be used for verification. Use the following commands at the Powershell prompt. There are connections listed if using RDMA. Otherwise, it defaults to TCP. Netstat -xan – Shows RDMA Connections Netstat -an – Shows TCP Connections (Get-NetAdapterStatistics).
NetXtreme-E User Guide User Guide for Dell Platforms Figure 47: Powershell for Verification 14.4 VMware ESX and Use Case Examples 14.4.1 Limitations The current version of the RoCE supported driver requires ESXi-6.5.0 GA build 4564106 or above. 14.4.2 BNXT RoCE Driver Requirements The BNXTNET L2 driver must be installed with the disable_roce=0 module parameter before installing the driver. To set the module parameter, use the following command: esxcfg-module -s "disable_roce=0" bnxtnet Use ESX6.
NetXtreme-E User Guide User Guide for Dell Platforms To disable ECN (enabled by default) for RoCE traffic use the tos_ecn=0 module parameter for bnxtroce. 14.4.4 Configuring Paravirtualized RDMA Network Adapters See Vmware.com for additional information on setting up and using Paravirtualized RDMA (PVRDMA) network adapters. 14.4.4.1 Configuring a Virtual Center for PVRDMA To configure a Virtual Center for PVRDMA: 1. Create DVS (requires a Distributed Virtual Switch for PVRDMA). 2.
NetXtreme-E User Guide User Guide for Dell Platforms 14.4.5 Configuring the VM on Linux Guest OS NOTE: The user must install the appropriate development tools including git before proceeding with the configuration steps below. 1. Download the PVRDMA driver and library using the following commands: git clone git://git.openfabrics.org/~aditr/pvrdma_driver.git git clone git://git.openfabrics.org/~aditr/libpvrdma.git 2. Compile and install the PVRDMA guest driver and library. 3.
NetXtreme-E User Guide User Guide for Dell Platforms 15 DCBX – Data Center Bridging Broadcom NetXtreme-E controllers support IEEE 802.1Qaz DCBX as well as the older CEE DCBX specification. DCB configuration is obtained by exchanging the locally configured settings with the link peer. Since the two ends of a link may be configured differently, DCBX uses a concept of willing to indicate which end of the link is ready to accept parameters from the other end.
NetXtreme-E User Guide User Guide for Dell Platforms 15.2 DCBX Mode – Enable (IEEE only) This option allows a user to enable/disable DCBX with the indicated specification. IEEE only indicates that IEEE 802.1Qaz DCBX is selected. Windows Driver setting: After enabling the indicated options in the UEFI HII menu to set firmware level settings, perform the follow selection in the Windows driver advanced properties.
NetXtreme-E User Guide User Guide for Dell Platforms Use the following in modifying the Traffic Class: PS C:\> Set-NetQoSTrafficClass -Name "SMB class" -BandwidthPercentage 40 PS C:\> get-NetQosTrafficClass Name Algorithm Bandwidth(%) Priority -----------------------------------------------------------------[Default] ETS 60 0-3,5-7 SMB class ETS 40 4 Use the following to remove the Traffic Class: PS C:\> Remove-NetQosTrafficClass -Name "SMB class" PS C:\> Get-NetQosTrafficClass Name Algorithm Bandwidth(%
NetXtreme-E User Guide User Guide for Dell Platforms Use the following to create QoS Policy based on the Source/Destination Address: PS C:\> New-NetQosPolicy "Network Management" -IPDstPrefixMatchCondition 10.240.1.0/24 IPProtocolMatchCondition both -NetworkProfile all -PriorityValue8021Action 7 Name : Network Management Owner : Group Policy (Machine) Network Profile : All Precedence : 127 IPProtocol : Both IPDstPrefix : 10.240.1.0/24 PriorityValue : 7 Use the following to display QoS Policy: PS C:\> Get
NetXtreme-E User Guide User Guide for Dell Platforms 16 DPDK – Configuration and Use Case Examples The testpmd application can be used to test the DPDK in a packet forwarding mode and also to access NIC hardware features such as Flow Director. It also serves as an example of how to build a more fully-featured application using the DPDK SDK. The below chapters shows how to build and run the testpmd application and how to configure the application from the command line and the run-time environment. 16.
NetXtreme-E User Guide User Guide for Dell Platforms 16.3 Testpmd Runtime Functions When the testpmd application is started in interactive mode, (-i|--interactive), it displays a prompt that can be used to start and stop forwarding, configure the application, display statistics (including the extended NIC statistics aka xstats), set the Flow Director, and other tasks. testpmd> 16.4 Control Functions This section contains the control functions.
NetXtreme-E User Guide User Guide for Dell Platforms 16.6 Configuration Functions The testpmd application can be configured from the runtime as well as from the command line. This section describes the available configuration functions that are available. csum set – Selects the hardware or software calculation of the checksum when transmitting a packet using the csum forwarding engine: testpmd> csum set (ip|udp|tcp|sctp|outer-ip) (hw|sw) (port_id).
NetXtreme-E User Guide User Guide for Dell Platforms Revision History NetXtreme-E-UG304-2CS; March 23, 2021 Updated: Figure 25, Device Configuration Menu NetXtreme-E-UG303-2CS; February 10, 2021 Updated: Supported Operating Systems Added BCM957504-NGM250 to the functional description table.
