This document and related products are distributed under licenses restricting their use, copying, distribution, and reverse-engineering. No part of this document may be reproduced in any form or by any means without prior written permission by Chelsio Communications. All third party trademarks are copyright of their respective owners.
Document History Version 1.0.0 1.0.1 1.0.2 1.0.3 1.0.4 1.0.5 1.0.6 1.0.7 1.0.8 Copyright ©2013.Chelsio Communications. All Rights Reserved.
TABLE OF CONTENTS I. 1. 2. 3. 4. 5. 6. CHELSIO UNIFIED WIRE Introduction 1.1. Features 1.2. Hardware Requirements 1.3. Software Requirements 1.4. Package Contents Hardware Installation Software/Driver Installation 3.1. Pre-requisites 3.2. Installing Chelsio Unified Wire from source 3.3. Installing Chelsio Unified Wire from RPM 3.4. Firmware update Software/Driver Uninstallation 4.1. Uninstalling Chelsio Unified Wire from source 4.2.
4. 3.1. Unloading the driver Software/Driver Configuration and Fine-tuning 4.1. Instantiate Virtual Functions IV. IWARP (RDMA) 1. Introduction 1.1. Hardware Requirements 1.2. Software Requirements 2. Software/Driver Loading 2.1. Installing OFED software 2.2. Compiling and Loading iWARP driver 3. Software/Driver Unloading 4. Software/Driver Configuration and Fine-tuning 4.1. Testing connectivity with ping and rping 4.2. Enabling various MPIs 4.3. Setting up NFS-RDMA V. WD-UDP 1. Introduction 1.1.
4.1. 4.2. 4.3. 4.4. 4.5. 4.6. 4.7. 4.8. 4.9. 4.10. 4.11. Command Line Tools iSCSI Configuration File A Quick Start Guide for Target The iSCSI Configuration File Challenge-Handshake Authenticate Protocol (CHAP) Target Access Control List (ACL) Configuration Target Storage Device Configuration Target Redirection Support The command line interface tools “iscsictl” & “chisns” Rules of Target Reload (i.e. “on the fly” changes) System Wide Parameters VIII. ISCSI PDU OFFLOAD INITIATOR 1. Introduction 1.1.
1. Introduction 1.1. Hardware Requirements 1.2. Software Requirements 2. Software/Driver Loading 3. Software/Driver Unloading 4. Software/Driver Configuration and Fine-tuning 4.1. Modifying the application 4.2. Configuring UDP Pacing XII. OFFLOAD IPV6 DRIVER 1. Introduction 1.1. Hardware Requirements 1.2. Software Requirements 2. Software/Driver Loading 3. Software/Driver Unloading 4. Software/Driver Configuration and Fine-tuning 4.1. Offloading IPv6 traffic 4.2. Network Device Configuration XIII.
2.3. 2.4. Listing Filter Rules Removing Filter Rules XVI. TRAFFIC MANAGEMENT 1. 2. 3. 4. 5. Introduction 1.1. Hardware Requirements 1.2. Software Requirements Software/Driver Loading Software/Driver Unloading Software/Driver Configuration and Fine-tuning 4.1. Traffic Management Rules 4.2. Configuring Traffic Management Usage 5.1. Non-Offloaded Connections 5.2. Offloaded Connections 5.3. Offloaded Connections with Modified Application XVII. UNIFIED WIRE MANAGER (UM) 1. 2. 3. 4. 5. 6. 7. 8.
8.3. Layout and Navigation 8.4. Home page 8.5. System page 8.6. Network page 8.7. Storage (Linux and Windows) 8.8. Hardware Features 9. Uninstalling Unified Wire Manager 9.1. Uninstalling Management Agent 9.2. Uninstalling Management Client 9.3. Uninstalling Management Station XVIII. UNIFIED BOOT 1. 2. 3. 4. 5. 6. 7. Introduction 1.1. Hardware Requirements 1.2. Software Requirements Flashing configuration file, firmware & option ROM Configuring PXE Server PXE boot process 4.1. For Legacy PXE boot 4.2.
Chapter I. Chelsio Unified Wire I. Chelsio Unified Wire Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter I. Chelsio Unified Wire 1. Introduction Thank you for choosing Chelsio T4 Unified Wire adapters. These high speed, single chip, single firmware cards provide enterprises and data centers with high performance solutions for various Network and Storage related requirements. The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and Filtering.
Chapter I. Chelsio Unified Wire Offload IPv6 driver Bypass driver Classification and Filtering feature Traffic Management feature (TM) Unified Wire Manager (UM) Unified Boot Software Utility Tools(cop,cxgbtool,t4_perftune,benchmark tools, sniffer & tracer) libs (iWARP and WD-UDP libraries) For detailed instructions on loading, unloading and configuring the drivers/tools please refer to their respective sections. 1.2.
Chapter I. Chelsio Unified Wire library that accelerates UDP Socket communications transparently and without recompilation of the user application. Makefile: The Makefile for building and installing from the source. sample_machinefile: Sample file used during iWARP installation on cluster nodes. scripts: Directory containing scripts used by installer for iWARP driver installation on cluster nodes. specs: The packaging specification files required for building RPM packages.
Chapter I. Chelsio Unified Wire 1.4.2. RPM package The Chelsio Unified Wire RPM package consists of the following: docs: The docs directory contains support documents i.e. README, Release Notes and User‟s Guide (this document) for the software. DRIVER-RPMS: RPM packages of Chelsio drivers. OFED-RPMS: OFED RPM packages required to install iWARP driver. install.py: Python script that installs the RPM package. Refer to the Software/Driver Installation section for more information. uninstall.
Chapter I. Chelsio Unified Wire 2. Hardware Installation 1. 2. 3. 4. 5. 6. Shutdown/power off your system. Power off all remaining peripherals attached to your system. Unpack the Chelsio adapter and place it on an anti-static surface. Remove the system case cover according to the system manufacturer‟s instructions. Remove the PCI filler plate from the slot where you will install the 10Gb Ethernet adapter. For maximum performance, it is highly recommended to install the adapter into a PCIE x8 slot. 7.
Chapter I. Chelsio Unified Wire Physical function 5: for iSCSI Physical function 6: for FCoE Physical function 7: Currently not assigned Once Unified Wire package is installed and loaded, examine the output of dmesg to see if the card is discovered eth0: Chelsio T420-CR rev 2 10GBASE-SFP RNIC PCIe x8 MSI-X 0000:04:00.4: S/N: PT18111226, P/N: 110112140D0 This output indicates the hardware configuration of the card as well as the Serial number of the card.
Chapter I. Chelsio Unified Wire 3. Software/Driver Installation There are two main methods to install the Chelsio Unified Wire package: from source and RPM. If you decide to use source, you can install the package using CLI or GUI mode. If you decide to use RPM, you can install the package using Menu or CLI mode. Irrespective of the method chosen for installation, the machine needs to be rebooted for changes to take effect. 3.1.
Chapter I. Chelsio Unified Wire v. Select “View EULA” to read the Chelsio End User License Agreement: vi. Select “Agree” to continue: Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter I. Chelsio Unified Wire vii. Select “install” under “Choose an action” viii. Select the required T4 configuration tuning option: a. Unified Wire: Configures T4 adapters to run multiple protocols like NIC/TOE, iWARP, iSCSI and FCoE Initiator simultaneously. b. Low latency Networking: Configures T4 adapters to run NIC/TOE and iWARP traffic with low latency specially needed for financial applications. c. High capacity RDMA: Configures T4 adapters to establish a large number of RDMA connections. d.
Chapter I. Chelsio Unified Wire ix. Under “Choose install components”, select “all” to install all the related components for the option chosen in step (viii) or select “custom” to install specific components. Important To install Bypass or WD-TOE drivers, please select Unified Wire with FCoE Initiator in step (viii). Then select “custom” option. x. Select the required performance tuning option. a. Disable-affinity: Disable binding IRQs to CPUs. b.
Chapter I. Chelsio Unified Wire xi. The selected components will now be installed: xii. After successful installation, summary of installed components will be displayed. xiii. Select “View log” to view the installation log or “Exit” to continue. xiv. Select “Yes” to exit the installer or “No” to go back. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter I. Chelsio Unified Wire xv. Reboot your machine for changes to take effect. Note Press Esc or Ctrl+C to exit the installer at any point of time. 3.2.1.1. Installation on updated kernels If the kernel version on your Linux distribution is updated, follow the steps mentioned below to install the Unified Wire package: i. Run the following script to start the GUI installer: [root@host]# ./install.py ii. Select “Yes” to continue with the installation on the updated kernel or “No” to exit. iii.
Chapter I. Chelsio Unified Wire iv. Follow steps (v) to (xv) mentioned in the previous section. 3.2.2. CLI mode (without Dialog utility) If your system does not have Dialog or you choose not to install it, follow the steps mentioned below to install the Unified Wire package: i. Download the tarball ChelsioUwire-x.x.x.x.tar.gz from Chelsio Download Center, http://service.chelsio.com/ ii. Untar the tarball using the following command: [root@host]# tar -zxvfm ChelsioUwire-x.x.x.x.tar.gz iii.
Chapter I. Chelsio Unified Wire Important To install Bypass or WD-TOE driver, run [root@host]# ./install.py -c and follow steps (iv) - (vi) mentioned above. To customize the installation, view the help by typing [root@host]#./install.py –h viii. Reboot your machine for changes to take effect. 3.2.2.1. iWARP driver installation on Cluster nodes Chelsio‟s Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a single command.
Chapter I. Chelsio Unified Wire iv. Install the drivers, tools and libraries using the following command: [root@host]# make install v. The default T4 configuration tuning option is Unified Wire (includes FCoE Initiator). The configuration tuning can be selected using the following commands: [root@host]# make CONF= [root@host]# make CONF= install Important Note Steps (iii), (iv)and (v) mentioned above will NOT install WD-TOE and Bypass drivers.
Chapter I. Chelsio Unified Wire To view the list of the supported kernel versions, run the following command: [root@host]# make list_kernels Reboot your machine for changes to take effect. 3.2.4. CLI mode (individual drivers) You can also choose to install drivers individually. Provided here are steps to build and install NIC, TOE, iWARP, Bypass, WD-TOE and UDP Segmentation Offload drivers. To know about other drivers, access help by running make help.
Chapter I. Chelsio Unified Wire To build and install all drivers without IPv6 support: [root@host]# make ipv6_disable=1 [root@host]# make ipv6_disable=1 install The above step will not install Bypass and WD-TOE drivers.
Chapter I. Chelsio Unified Wire 3.3. Installing Chelsio Unified Wire from RPM 3.3.1. Menu Mode i. Download the tarball specific to your operating system and architecture from Chelsio Download Center, http://service.chelsio.com/ ii. Untar the tarball: E.g. For RHEL 6.0, untar using the following command: [root@host]# tar -zxvfm ChelsioUwire-x.x.x.x-RHEL6.0_x86_64.tar.gz iii. Navigate to „ChelsioUwire-x.x.x.x‟ directory. Run the following command: [root@host]# ./install.py iv.
Chapter I. Chelsio Unified Wire Note The Installation options may vary depending on the T4 Configuration tuning option selected. ix. The selected components will now be installed. x. Reboot your machine for changes to take effect. Note If the installation aborts with the message "Resolve the errors/dependencies manually and restart the installation", please go through the install.log to resolve errors/dependencies and then start the installation again. 3.3.2. CLI mode i.
Chapter I. Chelsio Unified Wire Note To view the different T4 configuration tuning options, view the help by typing [root@host]# ./install.py –h v. To install OFED and Chelsio drivers built against OFED, run the above command with -o option. [root@host]# ./install.py –i -c -o vi. Reboot your machine for changes to take effect. 3.4.
Chapter I. Chelsio Unified Wire 4. Software/Driver Uninstallation Similar to installation, the Chelsio Unified Wire package can be uninstalled using two main methods: from the source and RPM, based on the method used for installation. If you decide to use source, you can uninstall the package using CLI or GUI mode. 4.1. Uninstalling Chelsio Unified Wire from source 4.1.1. GUI mode (with Dialog utility) i. Run the following script to start the GUI installer: [root@host]# ./install.py ii.
Chapter I. Chelsio Unified Wire iii. Select “Agree” to continue: iv. Select “uninstall” , Under “Choose an action” Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter I. Chelsio Unified Wire v. Select “all” to uninstall all the installed drivers, libraries and tools or select “custom” to remove specific components. vi. The selected components will now be uninstalled. vii. After successful uninstalltion, summary of the uninstalled components will be displayed. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter I. Chelsio Unified Wire viii. Select “View log” to view uninstallation log or “Exit” to continue. ix. Select “Yes” to exit the installer or “No” to go back. Note Press Esc or Ctrl+C to exit the installer at any point of time. 4.1.2. CLI mode (without Dialog utility) Run the following script with –u option to uninstall the Unified Wire Package: [root@host]# ./install.py –u Note View help by typing [root@host]# ./install.py –h for more information 4.1.3.
Chapter I. Chelsio Unified Wire 4.1.4. CLI mode (individual drivers) You can also choose to uninstall drivers individually. Provided here are steps to uninstall NIC, TOE, iWARP, Bypass, WD-TOE and UDP Segmentation Offload drivers.
Chapter I. Chelsio Unified Wire 4.2. Uninstalling Chelsio Unified Wire from RPM Navigate to the „ChelsioUwire-x.x.x.x‟ directory. Run the following command: [root@host]# ./uninstall.py inbox ofed Note : for removing all Chelsio drivers. : for removing OFED and Chelsio drivers. The uninstallation options may vary depending on Linux distribution. View help by typing [root@host]# ./uninstall.py –h for more information. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter I. Chelsio Unified Wire 5. Configuring T4 interfaces In order to test T4‟s features it is required to use two machines both with Chelsio‟s T4 network card installed. These two machines can be connected directly without a switch (back-to-back), or both connected to a 10Gb switch. The interfaces have to be declared and configured. The configuration files for network interfaces on Red Hat Enterprise Linux (RHEL) distributions are kept under /etc/sysconfig/network-scripts. 5.1.
Chapter I. Chelsio Unified Wire [root@host]# ifconfig -a | grep HWaddr eth0 Link encap:Ethernet HWaddr 00:30:48:32:6A:AA Then load the driver using the modprobe cxgb4 command (for the moment it does not make any difference whether we are using NIC-only or the TOE-enabling driver).
Chapter I. Chelsio Unified Wire 6. Software/Driver Update For any distribution specific problems, please check README and Release Notes included in the release for possible workaround. Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various software/drivers. You can also subscribe to our newsletter for the latest software updates. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter II. Network (NIC/TOE) II. Network (NIC/TOE) Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter II. Network (NIC/TOE) 1. Introduction Chelsio‟s T4 series of Unified Wire Adapters provide extensive support for NIC operation, including all stateless offload mechanisms for both IPv4 and IPv6 (IP, TCP and UDP checksum offload, LSO - Large Send Offload aka TSO - TCP Segmentation Offload, and assist mechanisms for accelerating LRO - Large Receive Offload). A high performance fully offloaded and fully featured TCP/IP stack meets or exceeds software implementations in RFC compliance.
Chapter II.
Chapter II. Network (NIC/TOE) 2. Software/Driver Loading The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. 2.1. Loading in NIC mode (without offload support) To load the Network driver without offload support, run the following command: [root@host]# modprobe cxgb4 2.2.
Chapter II. Network (NIC/TOE) 3. Software/Driver Unloading 3.1. Unloading the NIC driver To unload the NIC driver, run the following command: [root@host]# rmmod cxgb4 3.2. Unloading the TOE driver Please reboot the system to unload the TOE driver. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter II. Network (NIC/TOE) 4. Software/Driver Configuration and Fine-tuning 4.1. Instantiate Virtual Functions (SR-IOV) To instantiate the Virtual functions, load the cxgb4 driver with num_vf parameter with a non-zero value. For example: [root@host]# modprobe cxgb4 num_vf=1,0,0,0 The number(s) provided for num_vf parameter specifies the number of Virtual Functions to be instantiated per Physical Function. The Virtual Functions can be assigned to Virtual Machines (Guests).
Chapter II. Network (NIC/TOE) Receiver Side Scaling (RSS) Receiver Side Scaling enables the receiving network traffic to scale with the available number of processors on a modern networked computer. RSS enables parallel receive processing and dynamically balances the load among multiple processors. Chelsio‟s T4 network controller fully supports Receiver Side Scaling for IPv4 and IPv6.
Chapter II. Network (NIC/TOE) Then on the receiver host, look at interrupt rate at /proc/interrupts: [root@receiver_host]# cat /proc/interrupts | grep eth6 Id CPU0 CPU1 CPU2 CPU3 type interface 36: 115229 0 0 1 PCI-MSI-edge eth6 (queue 0) 37: 0 121083 1 0 PCI-MSI-edge eth6 (queue 1) 38: 0 0 105423 1 PCI-MSI-edge eth6 (queue 2) 39: 0 0 0 115724 PCI-MSI-edge eth6 (queue 3) Now interrupts from eth6 are evenly distributed among the 4 CPUs.
Chapter II. Network (NIC/TOE) Interrupt Coalescing The idea behind Interrupt Coalescing (IC) is to avoid flooding the host CPUs with too many interrupts. Instead of throwing one interrupt per incoming packet, IC waits for „n‟ packets to be available in the Rx queues and placed into the host memory through DMA operations before an interrupt is thrown, reducing the CPU load and thus improving latency.
Chapter II. Network (NIC/TOE) Then T4‟s hardware GRO/LRO implementation is enabled. If you would like to use the Linux GRO/LRO for any reason, first the t4_tom kernel module needs to be removed from kernel module list. Please note you might need to reboot your system.
Chapter II. Network (NIC/TOE) ethtool can also be used to switch off the GRO/LRO options when necessary: [root@host]# ethtool -K eth6 gro off [root@host]# ethtool -k eth6 Offload parameters for eth6: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off The output above shows a disabled GRO. 4.3.
Chapter III. Virtual Function Network (vNIC) III. Virtual Function Network (vNIC) Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter III. Virtual Function Network (vNIC) 1. Introduction The ever increasing network infrastructure of IT enterprises has lead to a phenomenal increase in maintenance and operational costs. IT managers are forced to acquire more physical servers and other data center resources to satisfy storage and network demands.
Chapter III.
Chapter III. Virtual Function Network (vNIC) 2. Software/Driver Loading The vNIC driver must be loaded or unloaded on the Guest OS by the root user. Any attempt to load the driver as a regular user will fail. 2.1. Loading the driver To load the driver, run the following command: [root@host]# modprobe cxgb4vf Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter III. Virtual Function Network (vNIC) 3. Software/Driver Unloading 3.1. Unloading the driver To unload the driver, execute the following command: [root@host]# rmmod cxgb4vf Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter III. Virtual Function Network (vNIC) 4. Software/Driver Configuration and Fine-tuning 4.1. Instantiate Virtual Functions To instantiate Chelsio Virtual Functions, please refer to the Network (NIC/TOE) section (click here) Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IV. iWARP (RDMA) IV. iWARP (RDMA) Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IV. iWARP (RDMA) 1. Introduction Chelsio‟s T4 engine implements a feature rich RDMA implementation which adheres to the IETF standards with optional markers and MPA CRC-32C. The iWARP RDMA operation benefits from the virtualization, traffic management and QoS mechanisms provided by T4. It is possible to ACL process iWARP RDMA packets.
Chapter IV. iWARP (RDMA) Suse Linux Enterprise Server 11 kernel Suse Linux Enterprise Server 11 SP1 kernel Suse Linux Enterprise Server 11 SP2 kernel Fedora release 13 Fedora release 14 Ubuntu 12.04, 3.2.0-23 Kernel.org linux-2.6.34 Kernel.org linux-2.6.35 Kernel.org linux-2.6.36 Kernel.org linux-2.6.37 Kernel.org linux-2.6.39 Kernel.org linux-3.1 Kernel.org linux-3.5 Kernel.org linux-3.6* (SLES11) , 2.6.27.19-5 (SLES11SP1),2.6.32.12-0.7 (SLES11SP2), 3.0.13-0.27 * (FC 13), 2.6.33.3-85.fc13 * (FC 14), 2.6.
Chapter IV. iWARP (RDMA) 2. Software/Driver Loading Before loading the iWARP driver, please install OFED software as mentioned below: 2.1. Installing OFED software 2.1.1. Installing OFED-1.5.4 software i. Download OFED-1.5.4 package from the following location http://www.openfabrics.org/downloads/OFED/ofed-1.5.4/OFED-1.5.4.tgz ii. Untar the OFED package using following command: [root@host]# tar -xf iii.
Chapter IV. iWARP (RDMA) 2.1.2. Installing OFED-1.5.2 software i. Download OFED-1.5.2 package from the following location http://www.openfabrics.org/downloads/OFED/ofed-1.5.2/OFED-1.5.2.tgz ii. Follow steps (ii) – (vii) mentioned in section 2.1.1. above before proceeding. iii. Navigate to the share directory of OPENMPI as: [root@host]# cd /usr/mpi/gcc/openmpi-1.4.3/share iv. Navigate to OPENMPI in share directory: [root@host]# cd openmpi v.
Chapter IV. iWARP (RDMA) 2.2. Compiling and Loading iWARP driver The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. i. Change your current working directory to driver package directory & run the following commands: [root@host]# make [root@host]# make install ii. To load the iWARP driver we need to load the NIC driver & core RDMA drivers first.
Chapter IV. iWARP (RDMA) 3. Software/Driver Unloading To unload the iWARP driver, run the following command: [root@host]# rmmod iw_cxgb4 Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IV. iWARP (RDMA) 4. Software/Driver Configuration and Fine-tuning 4.1. Testing connectivity with ping and rping Load the NIC, iWARP & core RDMA modules as mentioned in Software/Driver Loading section. After which, you will see one or two ethernet interfaces for the T4 device. Configure them with an appropriate ip address, netmask, etc. You can use the Linux ping command to test basic connectivity via the T4 interface.
Chapter IV. iWARP (RDMA) 4.2. Enabling various MPIs 4.2.1. DAPL Library configuration for Intel MPI and Platform MPI You must set the iWARP (iw_cxgb4) module option peer2peer=1 on all systems. This can be done by writing to the /sys/module/ file system during boot. E.g.: For RHEL 5 and SLES platforms use following commands: [root@host]# echo 1 > /sys/module/iw_cxgb4/parameters/peer2peer OR You can add the following line to /etc/modprobe.
Chapter IV. iWARP (RDMA) iii. Press enter upon prompting to accept default setup and empty password phrase iv. Create authorization file [root@host]# cd .ssh [root@host]# cat *.pub > authorized_keys [root@host]# chmod 600 authorized_keys v. Copy directory .ssh to all systems in the cluster [root@host]# cd [root@host]# scp -r /root/.ssh remotehostname-or-ipaddress: 4.2.3. Configuration of various MPIs (Installation and Setup) 4.2.3.1. Intel-MPI i. ii. iii. iv. v.
Chapter IV. iWARP (RDMA) ix. Populate mpd.hosts with node names. Note The hosts in this file should be Chelsio interface IP addresses. I_MPI_DEVICE=rdssm:chelsio assumes you have an entry /etc/dat.conf named chelsio. MPIEXEC_TIMEOUT value might be required to increase if heavy traffic is going across the systems. in x. Contact Intel for obtaining their MPI with DAPL support. xi. To run Intel MPI applications: mpdboot -n -r ssh --ncpus= mpiexec -ppn -n 2 /opt/intel/impi/3.1/tests/IMB-3.
Chapter IV. iWARP (RDMA) v. Start the License Manager and run the test as: mpirun -prot -e DAPL_MAX_INLINE=64 -UDAPL -hostlist -np 4 /opt/platform_mpi/tests/IMB-3.2/IMB-MPI1 4.2.3.3. Open MPI (Installation and Setup) The latest release of Open-MPI-1.4.3 comes with OFED package only. Select Open MPI package while installing OFED package. Open MPI iWARP support is only available in Open MPI version 1.3 or greater.
Chapter IV. iWARP (RDMA) i. Set MVAPICH2 with mpi-selector (do this on all nodes). [root@host]# mpi-selector --set mvapich2-1.7 ii. Edit .bashrc and add these lines: export MVAPICH2_HOME=/usr/mpi/gcc/mvapich2-1.7/ export MV2_USE_IWARP_MODE=1 export MV2_USE_RDMA_CM=1 iii. Logout & log back in. iv. Populate mpd.hosts with node names. v. On each node, create /etc/mv2.conf with a single line containing the IP address of the local T4 interface.
Chapter IV. iWARP (RDMA) iii. Navigate to OFED-1.5.3.x directory [root@host]# cd OFED-1.5.3.x iv. Generate a config file: [root@host]# ./install.pl –p A config file ofed-all.conf will be generated in the same directory. v. Open the file and append the following entries at the end nfsrdma=y rnfs-utils=y vi. Save and Exit. vii. Now, start OFED installation with NFS-RDMA support: [root@host]# ./install –c ofed-all.conf Note If you are using OFED-1.5.
Chapter IV. iWARP (RDMA) iv. Start OFED installation [root@host]#./install.py –all 4.3.2. Starting NFS-RDMA Server-side settings Follow the steps mentioned below to set up an NFS-RDMA server. i. Make entry in /etc/exports file for the directories you need to export using NFS-RDMA on server as: /share/rdma /share/rdma1 *(fsid=0,async,insecure,no_root_squash) *(fsid=1,async,insecure,no_root_squash) Note that for each directory you export, you should have DIFFERENT fsid‟s. ii.
Chapter IV. iWARP (RDMA) vii. Run exportfs to make local directories available for Network File System (NFS) clients to mount. [root@host]# exportfs Now the NFS-RDMA server is ready. Client-side settings Follow the steps mentioned below at the client side. i. Load the iwarp modules and make sure peer2peer is set to 1. Make sure you are able to ping and ssh to the server Chelsio interface through which directories will be exported. ii. Load the xprtrdma module. [root@host]# modprobe xprtrdma iii.
Chapter V. WD-UDP V. WD-UDP Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter V. WD-UDP 1. Introduction Chelsio WD-UDP (Wire Direct-User Datagram Protocol) with Multicast is a user-space UDP stack with Multicast address reception and socket acceleration that enables users to run their existing UDP socket applications unmodified.
Chapter V. WD-UDP Suse Linux Enterprise Server 11 SP2 kernel (SLES11SP2), 3.0.13-0.27 Fedora release 13 (FC 13), 2.6.33.3-85.fc13 * Fedora release 14 (FC 14), 2.6.35.6-45.fc14 * Ubuntu 12.04 , 3.2.0-23 Kernel.org linux-2.6.34 Kernel.org linux-2.6.35 Kernel.org linux-2.6.36 Kernel.org linux-2.6.37 Kernel.org linux-2.6.39 Kernel.org linux-3.1 Kernel.org linux-3.5 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Copyright ©2013.
Chapter V. WD-UDP 2. Software/Driver Compiling and Loading The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. i. Change your current working directory to driver package directory & run the following commands : [root@host]# make [root@host]# make install ii. RDMA core modules from the OFED package should be loaded before proceeding.
Chapter V. WD-UDP 3. Software/Driver Unloading To unload the WD-UDP driver, run the following command: [root@host]# rmmod iw_cxgb4 Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter V. WD-UDP 4. Software/Driver Configuration and Fine-tuning 4.1. Accelerating UDP Socket communications The libcxgb4_sock library is a LD_PRELOAD-able library that accelerates UDP Socket communications transparently and without recompilation of the user application. This section describes how to use libcxgb4_sock. By preloading libcxgb4_sock, all sockets created by the application are intercepted and possibly accelerated based on the user‟s configuration.
Chapter V. WD-UDP egress, if the destination IP address will not route out via the T4 device, then it too will not be accelerated. 4.1.2. Using libcxgb4_sock The libcxgb4_sock library utilizes the Linux RDMA Verbs subsystem, and thus requires the RDMA modules be loaded.
Chapter V. WD-UDP Now, preload libcxgb4_sock_debug using the command mentioned below when starting your application: root@host]# LD_PRELOAD=libcxgb4_sock_debug.so CXGB4_SOCK_DEBUG=-1 /your_application In addition to preloading libcxgb4_sock.so, you must create a configuration file that defines which UDP endpoints should be accelerated, their vlan and priority if any, as well as which T4 interface/port should be used. The file /etc/libcxgb4_sock.conf contains these endpoint entries.
Chapter V. WD-UDP If you have eth2 and eth3 configured for example, you can define certain endpoints to eth2 and others to eth3: endpoint {interface=eth2 port=9999} endpoint {interface=eth3 port=8888} For VLAN support, create your VLANs using the normal OS service (like vconfig, for example), then add entries to define the VLAN and priority for each endpoint to be accelerated: endpoint {interface = eth2.5 port=10000} endpoint {interface = eth2.
Chapter V. WD-UDP To preload libcxb4_sock and load the config file, use the LD_PRELOAD environment variable along with the config file when starting your application: [root@host]# CXGB4_SOCK_CFG= LD_PRELOAD=libcxgb4_sock.so /your_application 4.1.4. Example with hpcbench/udp The udp benchmark from the hpcbench suite can be used to show the benefits of libcxgb4_sock. The hpcbench suite can be found at: Source: http://hpcbench.sourceforge.net/index.html Sample: http://hpcbench.
Chapter V. WD-UDP [root@r10 ~]# /usr/local/src/hpcbench/udp/udpserver -p 9000 & [1] 11453 [root@r10 ~]# TCP socket listening on port [9000] [root@r10 ~]# LD_PRELOAD=libcxgb4_sock.so /usr/local/src/hpcbench/udp/udpserver -p 9001 & [2] 11454 [root@r10 ~]# TCP socket listening on port [9001] [root@r10 ~]# Then on r9, we run udptest to port 9000 to see the host stack UDP latency: [root@r9 ~]# /usr/local/src/hpcbench/udp/udptest -r 5 -a -h 192.168.1.
Chapter VI. WD-TOE VI. WD-TOE Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VI. WD-TOE 1. Introduction Chelsio WD-TOE (Wire Direct-Transmission Control Protocol) with a user-space TCP stack enables users to run their existing TCP socket applications unmodified. It features software modules that enable direct wire access from user space to the Chelsio T4 network adapter with complete bypass of the kernel, which results in a low latency 10Gb Ethernet solution for high frequency trading and other delay-sensitive applications. 1.1. Hardware Requirements 1.1.1.
Chapter VI. WD-TOE Suse Linux Enterprise Server 11 SP2 kernel (SLES11SP2), 3.0.13-0.27 Kernel.org linux-3.1 Kernel.org linux-3.5 Kernel.org linux-3.6 Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VI. WD-TOE 2. Software/Driver Loading The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. Run the following commands to load the drivers: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VI. WD-TOE 3. Software/Driver Unloading Reboot the system to unload the driver. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VI. WD-TOE 4. Software/Driver Configuration and Fine-tuning 4.1. Running the application To run an application with WD-TOE library, use the following command: [root@host]# PROT=TCP wdload Example: To run Netperf application with WD-TOE library. i. Start netserver at the PEER, using the following command: [root@host]# netserver ii. On the Test Machine, run the following command to run netperf application.
Chapter VII. iSCSI PDU Offload Target VII. iSCSI PDU Offload Target Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VII. iSCSI PDU Offload Target 1. Introduction This section describes how to install and configure iSCSI PDU Offload Target software for use as a key element in your iSCSI SAN. The software runs on Linux-based systems that use Chelsio or non-Chelsio based Ethernet adapters. However to guarantee highest performance, Chelsio recommends using Chelsio adapters.
Chapter VII.
Chapter VII. iSCSI PDU Offload Target 1.2.2. Adapter Requirements The Chelsio iSCSI PDU Offload Target software can be used with or without hardware protocol offload technology. When used with protocol offload, a Chelsio TOE adapter must be used. There are four modes of operation using the iSCSI PDU Offload Target software on Ethernetbased adapters: Regular NIC (non-Chelsio NIC Adapter) – The software can be used in non-offloaded (regular NIC) mode.
Chapter VII. iSCSI PDU Offload Target 1.3.1.
Chapter VII. iSCSI PDU Offload Target License Key Requirement A license key file is required for each copy of the iSCSI software installed. To obtain the key file a binary program called “chinfotool” must be run on the host system where the iSCSI software is to be installed. That program generates an information file that contains data about the system. The information file must be sent back to Chelsio where a license key file will be generated.
Chapter VII. iSCSI PDU Offload Target 2. Software/Driver Loading There are two main steps to installing the Chelsio iSCSI PDU Offload Target software. They are: 1. Installing the iSCSI software – The majority of this section deals with how to install the iSCSI software. 2. Configuring the iSCSI software – Information on configuring the software can be found in a section further into this user‟s guide. 2.1.
Chapter VII. iSCSI PDU Offload Target Note i. While using rpm-tar-ball for installation a. Uninstallation will result into chiscsi.conf file renamed into chiscsi.conf.rpmsave, but if again uninstallation is done then it will lead to overwriting of the old chiscsi.rpmsave file. b. Its advised to take a backup of chiscsi.conf file before you do an uninstallation and installation of new/same unified wire package. As re-installing/upgrading unified-wire package may lead to loss of chiscsi.conf file. ii.
Chapter VII. iSCSI PDU Offload Target 2.2. Generating single RPM for T3 and T4 adapters Follow the below procedure to generate a single iSCSI PDU Offload Target RPM for both T3 and T4 adapters: 1. If you haven‟t done already, download Unified Wire for Linux from Chelsio Download Center, http://service.chelsio.com. 2. Untar the tarball using the following command: [root@host]# tar -zxvfm ChelsioUwire-x.x.x.x.tar.gz 3. Browse to the ChelsioUwire-x.x.x.
Chapter VII. iSCSI PDU Offload Target 2.3. Obtaining the iSCSI Software License A license file is required for each copy of the Chelsio iSCSI PDU Offload Target software installed. The license is tied to the selected NIC present in the system. The license file will be generated depending on your requirement for a Chelsio iSCSI Target. 2.3.1. Linux Requirements To obtain an iSCSI license key file, which could be either a production or an evaluation version, please follow the steps below. 1.
Chapter VII. iSCSI PDU Offload Target [root@host]# chinfotool Scanning System for network devices.... License key will be tied to any of the following interfaces. Please select the interface 1. Interface eth1 with INTEL Adapter Linkspeed is 1000 Mbps/s MAC is: 00:30:48:00:00:10. 2. Interface eth2 with CHELSIO Adapter Linkspeed is 10000 Mbps/s MAC is: 00:07:43:00:00:10.
Chapter VII. iSCSI PDU Offload Target 3. Software/Driver Unloading Use the following command to unload the module: [root@host]# rmmod chiscsi_t4 Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VII. iSCSI PDU Offload Target 4. Software/Driver Configuration and Fine-tuning The Chelsio iSCSI software needs configuration before it can become useful. The following sections describe how this is done. There are two main components used in configuring the Chelsio iSCSI software: the configuration file and the iSCSI control tool. This section describes in some detail what they are and their relationship they have with one another. 4.1.
Chapter VII. iSCSI PDU Offload Target There are many specific parameters that can be configured, some of which are iSCSI specific and the rest being Chelsio specific. An example of an iSCSI specific item is “HeaderDigest” which is defaulted to “None” but can be overridden to “CRC32C”. An example of a Chelsio specific configurable item is “ACL” (for Access Control List). “ACL” is one of the few items that have no default. Before starting any iSCSI target, an iSCSI configuration file must be created.
Chapter VII. iSCSI PDU Offload Target A target can serve multiple devices, each device will be assigned a Logical Unit Number (LUN) according to the order it is specified (i.e., the first device specified is assigned LUN 0, the second one LUN 1, …, and so on and so forth). Multiple TargetDevice key=value pairs are needed to indicate multiple devices. Here is a sample of a minimum iSCSI target configuration located at /etc/chelsioiscsi/chiscsi.conf: target: TargetName=iqn.2006-02.com.chelsio.diskarray.
Chapter VII. iSCSI PDU Offload Target To stop a specific target execute iscsictl with “-s” followed by the target name. [root@host]# iscsictl –s target=iqn.2006-02.com.chelsio.diskarray.san1 View Configuration: To see the configuration of all the active iSCSI targets, execute iscsictl with “-c” option. [root@host]# iscsictl –c To see the more detailed configuration settings of a specific target, execute iscsictl with “-c” option followed by the target name. [root@host]# iscsictl –c target=iqn.2006-02.
Chapter VII. iSCSI PDU Offload Target 4.4. The iSCSI Configuration File The iSCSI configuration file consists of a series of blocks consisting of the following types of iSCSI entity blocks: 1. global 2. target There can be only one global entity block whereas multiple target entity blocks are allowed. The global entity block is optional but there must be at least one target entity block. An entity block begins with a block type (global or target).
Chapter VII. iSCSI PDU Offload Target Table of Chelsio Global Entity Settings Key iscsi_offload_mode Valid Values “AUTO” “TOE” “ULP” Default Value “AUTO” Multiple Values No Description Defines the offload mode AUTO: iSCSI software will make the decision. If the connection goes through Chelsio‟sHBA which has the iSCSI acceleration enabled, then ULP. TOE: Use Chelsio HBA TCP Offloading Engine (TOE) capabilities.
Chapter VII.
Chapter VII. iSCSI PDU Offload Target IFMarker “Yes” “No” “No” No OFMarkInt 1 to 65535 2048 No IFMarkInt 1 to 65535 2048 No To turn on or off the target to initiator markers on the connection. Chelsio only supports “No”. To set the interval for the initiator to target markers on a connection. To set the interval for the target to initiator markers on a connection. 4.4.3. Chelsio Entity Settings Description Chelsio Entity Parameters pass control information to the Chelsio iSCSI module.
Chapter VII. iSCSI PDU Offload Target allowed. must be between 6 and 255 characters. Commas “,” are not allowed. The initiator user id and secret are used by the target to authenticate the initiator. Auth_CHAP_Challenge Length Auth_CHAP_Policy 16 to 1024 16 No NOTE: The double quotes are required as part of the format.
Chapter VII. iSCSI PDU Offload Target TargetSessionMaxCmd 1 to 2048 TargetDevice* [,FILE|MEM|BL K] [,NULLRW] [,SYNC] [,RO] [,size=xMB] [,ScsiID=xxxx xx] [,WWN=xxxxxxx xx] 64 No No The maximum number of outstanding iSCSI commands per session. A device served up by the associated target. The device mode can be a: Block Device (e.g. /dev/sda) Virtual Block Device (e.g.
Chapter VII. iSCSI PDU Offload Target WWN=xxxxxx is a 16 character unique value set for multipath aware iSCSI initiator host. When multipath aware initiator host is accessing the storage Logical Unit Number( LUN) via multiple iSCSI session, ScsiID and WWN values must be set for the TargetDevice. These values will be returned in Inquiry response (VPD 0x83). Multiple TargetDevice key=value pairs are needed to indicate multiple devices. There can be multiple devices for any particular target.
Chapter VII. iSCSI PDU Offload Target sip, and/or dip. lun=: controls how the initiators access the luns. The supported value for is ALL. can be: R: Read Only RW or WR: Read and Write If permissions are specified then the associated LUN list is required. If no lun=:[R|RW] is specified then it defaults to ALL:RW. NOTE: For the Chelsio Target Software release with lun-masking included, is in the format of <0..N | 0~N | ALL> Where: 0..
Chapter VII. iSCSI PDU Offload Target # lun 0: a ramdisk with default size of 16MB TargetDevice=ramdisk,MEM PortalGroup=5@192.0.2.178:3260 # # an iSCSI Target “iqn.2005-8.com.chelsio:diskarrays.san.328” # being served by the portal group "1" and "2" # target: # # iSCSI configuration # TargetName=iqn.2005-8.com.chelsio:diskarrays.san.
Chapter VII. iSCSI PDU Offload Target # Auth_CHAP_Policy=Mutual Auth_CHAP_target=“iTarget1ID”:“iTarget1Secret” Auth_CHAP_Initiator=“iInitiator1”:“InitSecret1” Auth_CHAP_Initiator=“iInitiator2”:“InitSecret2” Auth_CHAP_ChallengeLength=16 # # ACL configuration # # initiator “iqn.2006-02.com.chelsio.san1” is allowed full access # to this target ACL=iname=iqn.2006-02.com.chelsio.san1 # any initiator from IP address 102.50.50.101 is allowed full access # of this target ACL=sip=102.50.50.
Chapter VII. iSCSI PDU Offload Target 4.5.1. Oneway CHAP authentication With Oneway CHAP (also called unidirectional CHAP) the target uses CHAP to authenticate the initiator. The initiator does not authenticate the target. This method is the default method. For Oneway CHAP, the initiator CHAP id and secret are configured and stored on a per-initiator with Chelsio Entity parameter “Auth_CHAP_Initiator”. 4.5.2.
Chapter VII. iSCSI PDU Offload Target a) no CHAP b) CHAP Oneway or c) CHAP Mutual Auth_CHAP_Policy=Mutual, the Chelsio iSCSI target will accept a relevant initiator if it does a) no CHAP or b) CHAP Mutual With AuthMethod=None, regardless the setting of the key Auth_CHAP_Policy, the Chelsio iSCSI target will only accept a relevant initiator if it does no CHAP. With AuthMethod=CHAP, CHAP is enforced on the target: i.
Chapter VII. iSCSI PDU Offload Target # full read-write access to this target ACL=iname=iqn.2006-02.com.chelsio.san1 # any initiator from IP address 102.50.50.101 is allowed full # read-write access of this target ACL=sip=102.50.50.101 # any initiator connected via the target portal 102.60.60.25 # is allowed full read-write access to this target ACL=dip=102.60.60.25 # initiator “iqn.2005-09.com.chelsio.san2” from 102.50.50.22 # and connected via the target portal 102.50.50.
Chapter VII. iSCSI PDU Offload Target When option NULLRW is specified, on writes the data is dropped without being copied to the storage device, and on reads the data is not actually read from the storage device but instead random data is used. This option is usefuly for measuring network performance. The details of the parameters for the key TargetDevice are found in the table of Chelsio Entity Settings section earlier in this document. 4.7.1.
Chapter VII. iSCSI PDU Offload Target TargetDevice=[,FILE][,SYNC] Where: Is the path to the actual storage device, such as /dev/sdb for a block device or /dev/md0 for a software RAID. The path must exist in the system. SYNC When specified, the Target will flush all the data in the system cache to the storage driver before sending response back to the Initiator. 4.7.3.
Chapter VII. iSCSI PDU Offload Target 4.8. Target Redirection Support An iSCSI Target can redirect an initiator to use a different IP address and port (often called a portal) instead of the current one to connect to the target. The redirected target portal can either be on the same machine, or a different one. 4.8.1. ShadowMode for Local vs. Remote Redirection The ShadowMode setting specifies whether the Redirected portal groups should be present on the same machine or not.
Chapter VII. iSCSI PDU Offload Target 4.8.2. Redirecting to Multiple Portal Groups The Chelsio iSCSI Target Redirection allows redirecting all login requests received on a particular portal group to multiple portal groups in a round robin manner. Below is an example Redirection to Multiple Portal Groups: target: # # any login requests received on 10.193.184.81:3260 will be # redirected to 10.193.184.85:3261 and 10.193.184.85:3262 in a # Round Robin Manner. PortalGroup=1@10.193.184.
Chapter VII. iSCSI PDU Offload Target 4.9.3. iscsictl options Options Mandatory Parameters Optional Parameters -h Display the help messages. -v -f Description Display the version. <[path/] filename> Specifies a pre-written iSCSI configuration text file, used to start, update, save, or reload the iSCSI node(s). This option must be specified with one of the following other options: “-S”, “-U”, or “-W”. For the “-S” option “-f” must be specified first. All other options will ignore this “-f” option.
Chapter VII. iSCSI PDU Offload Target If the target= option is specified, the -k option can optionally be specified along with this option to display only the selected entity parameter setting. -F target= -k lun= Example: iscsictl -c target=iqn.com.cc.target1 -k HeaderDigest Flush the cached data to the target disk(s). target= parameter: Where name is the name of the target to be flushed.
Chapter VII. iSCSI PDU Offload Target -s target= If any of the specified var=const parameter is invalid, the command will reject only the invalid parameters, but will continue on and complete all other valid parameters if any others are specified. Stop the specified active iSCSI targets. target= parameter: See the description of option -c for the target= parameter definition. The target= parameter is mandatory.
Chapter VII. iSCSI PDU Offload Target currently active, they will be started. For Rules 2 & 3, please note the differences – they are not the same! -r target= -k initiator= The global settings are also reloaded from the configuration file with this option. Retrieve active iSCSI sessions under a target. target= parameter: Where name must be a single target name.
Chapter VII. iSCSI PDU Offload Target In the first example the minimum command set is given where the IP address of the iSNS server is specified. In the second example a fully qualified command is specified by also setting three optional parameters. Here, the mandatory IP address and the corresponding optional port number are specified. Also set is the iSNS entity ID to “isnscln2” as well as the query interval to 30 seconds. 4.10. Rules of Target Reload (i.e.
Chapter VII.
Chapter VII. iSCSI PDU Offload Target Note iscsi_offload_mode has no meaning when the iSCSI software is used on a nonTOE based NIC.keyfile from Chelsio support. 4.11.2. iscsi_auth_order Options: “ACL” or “CHAP”, defaults to “CHAP” On an iSCSI target when ACL_Enable is set to “Yes”, iscsi_auth_order decides whether to perform CHAP first then ACL or perform ACL then CHAP.
Chapter VIII. iSCSI PDU Offload Initiator VIII. iSCSI PDU Offload Initiator Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VIII. iSCSI PDU Offload Initiator 1. Introduction The Chelsio T4 series Adapters support iSCSI acceleration and iSCSI Direct Data Placement (DDP) where the hardware handles the expensive byte touching operations, such as CRC computation and verification, and direct DMA to the final host memory destination: iSCSI PDU digest generation and verification On transmitting, Chelsio h/w computes and inserts the Header and Data digest into the PDUs.
Chapter VIII. iSCSI PDU Offload Initiator T420-BT T404-BT 1.2. Software Requirements 1.2.1.
Chapter VIII. iSCSI PDU Offload Initiator 2. Software/Driver Loading The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail.
Chapter VIII. iSCSI PDU Offload Initiator ii. Now, unload them using the following command: [root@host]# rmmod Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VIII. iSCSI PDU Offload Initiator 3. Software/Driver Unloading To unload the driver, execute the following commands: [root@host]# rmmod cxgb4i [root@host]# rmmod libcxgbi Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter VIII. iSCSI PDU Offload Initiator 4. Software/Driver Configuration and Fine-tuning 4.1. Accelerating open-iSCSI Initiator The following steps need to be taken to accelerate the open-iSCSI initiator: 4.1.1. Configuring iscsid.conf file Edit the iscsi/iscsid.conf file and change the setting for MaxRecvDataSegmentLength: node.conn[0].iscsi.MaxRecvDataSegmentLength = 8192 The login would fail for a normal session if MaxRecvDataSegmentLength is too big.
Chapter VIII. iSCSI PDU Offload Initiator E.g.:iface.iscsi_ifacename = cxgb4i.00:07:43:04:5b:da iface.hwaddress = 00:07:43:04:5b:da iface.transport_name = cxgb4i iface.net_ifacename = eth3 iface.ipaddress = 102.2.2.137 Alternatively, you can create the file automatically by executing the following command: [root@host]# iscsiadm -m iface Here, iface.iscsi_ifacename denotes the name of interface file in /etc/iscsi/ifaces/. iface.
Chapter VIII. iSCSI PDU Offload Initiator iscsiadm -m discovery -t st -p : -I E.g.:[root@host]# iscsiadm -m discovery -t st -p 102.2.2.155:3260 -I cxgb4i.00:07:43:04:5b:da iii. Logging into an iSCSI Target Log into an iSCSI target using the following format: iscsiadm -m node -T -p : -I -l E.g.:[root@host]# iscsiadm -m node -T iqn.2004-05.com.chelsio.target1 -p 102.
Chapter VIII. iSCSI PDU Offload Initiator 4.2. Auto login from cxgb4i initiator at OS bootup For iSCSI auto login (via cxgb4i) to work on OS startup, please add the following line to start() in /etc/rc.d/init.d/iscsid file on RHEL: modprobe -q cxgb4i E.g.
Chapter IX. FCoE Full Offload Initiator IX. FCoE Full Offload Initiator Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator 1. Introduction Fibre Channel over Ethernet (FCoE) is a mapping of Fibre Channel over selected full duplex IEEE 802.3 networks. The goal is to provide I/O consolidation over Ethernet, reducing network complexity in the Datacenter. Chelsio FCoE initiator maps Fibre Channel directly over Ethernet while being independent of the Ethernet forwarding scheme. The FCoE protocol specification replaces the FC0 and FC1 layers of the Fibre Channel stack with Ethernet.
Chapter IX. FCoE Full Offload Initiator 2. Software/Driver Loading The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver, execute the following: [root@host]# modprobe csiostor Note To load the driver on SLES11sp1 and SLES11sp2, execute the following command: [root@host]# modprobe csiostor --allow Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator 3. Software/Driver Unloading To unload the driver: [root@host]# modprobe -r Note csiostor If multipath services are running, unload of FCoE driver is not possible. Stop the multipath service and then unload the driver. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator 4. Software/Driver Configuration and Fine-tuning 4.1. Configuring Cisco Nexus 5010 switch 4.1.1. Configuring the DCBX parameters Note By default the Cisco Nexus switch enables DCBX functionality and configures PFC for FCoE traffic making it no drop with bandwidth of 50% assigned to FCoE class of traffic and another 50% for the rest(like NIC). If you wish to configure custom bandwidth, then follow the procedure below.
Chapter IX. FCoE Full Offload Initiator v. Configure qos policy-maps. switch(config)#policy-map type qos policy-test switch(config-pmap-qos)#class type qos class-nic switch(config-pmap-c-qos)#set qos-group 2 vi. Configure queuing policy-maps and assign network bandwidth. Divide the network bandwidth between FcoE and NIC traffic.
Chapter IX. FCoE Full Offload Initiator i. Following steps will enable FCoE services on a particular VLAN and does a VSAN-VLAN mapping. Need not do these steps every time, unless a new mapping has to be created. switch(config)# vlan 2 switch(config-vlan)# fcoe vsan 2 switch(config-vlan)#exit ii. Following steps help in creating a virtual fibre channel (VFC) and binds that VFC to a Ethernet interface so that the Ethernet port begins functioning as a FCoE port.
Chapter IX. FCoE Full Offload Initiator iv. Enabling DCBX: switch(config)# interface ethernet 1/13 switch(config-if)# priority-flow-control mode auto switch(config-if)# flowcontrol send off switch(config-if)# flowcontrol receive off switch(config-if)# lldp transmit switch(config-if)# lldp receive switch(config-if)# no shutdown v.
Chapter IX. FCoE Full Offload Initiator ii. Create a CEE Map to carry LAN and SAN traffic if it does not exist. Example of creating a CEE map. switch(config)# cee-map default switch(conf-cee-map)#priority-group-table 1 weight 40 pfc switch(conf-cee-map)#priority-group-table 2 weight 60 switch(conf-cee-map)#priority-table 2 2 2 1 2 2 2 2 iii. Configure the CEE interface as a Layer 2 switch port. Example of configuring the switch port as a 10-Gigabit Ethernet interface.
Chapter IX. FCoE Full Offload Initiator v. Save the Configuration switch#copy running-config startup-config 4.2. FCoE fabric discovery verification The verification is done using “cxgbtool”. 4.2.1. Verifying the DCBX parameters To verify the current DCBX information being exchanged, execute the below commands using cxgbtool. i. First step is to find the adapter number. Find it using the following command [root@host]# cxgbtool stor –s ii. Now execute the following command to check the DCBX information.
Chapter IX. FCoE Full Offload Initiator 4.2.2. Verifying Local Ports Once connected to the switch, use the following command to see if the FIP has gone through and a VN_Port MAC address has been assigned. Verify if all the FCoE ports are online/ready and a successful FIP has taken place using the following command. The wwpn and state of the initiator local port can be found under sysfs. [root@host]# cat /sys/class/fc_host/hostX/port_name Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator Note The hosts under fc_host depends on the number of ports on the adapter used. Inorder to identify chelsio fc_host from other vendor fc_host, the WWPN always begins with 0x5000743 Alternatively, the local port information can also be found using: [root@host]# cxgbtool stor –a --show-lnode Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator 4.2.3. Verifying the target discovery To check the targets being discovered use cxgbtool. To check the list of targets that are being discovered from a particular FCoE port, use the following commands from cxgbtool. i. Check for the adapter number using the following command. [root@host]# cxgbtool stor –s Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator ii. To check the list of targets discovered from a particular FCoE port, first find out the wwpn of the initiator local port under sysfs. The hosts under fc_host depends on the number of ports on the adapter used. [root@host]# cat /sys/class/fc_host/hostX/port_name iii. After finding the localport, go to the corresponding Remote port under sysfs # cat /sys/class/fc_remote_ports/rport-X:B:R where X is the Host ID, B is the bus ID and R is the remote port.
Chapter IX. FCoE Full Offload Initiator Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator 4.3. Formatting the LUNs and Mounting the Filesystem Use lsscsi –g to list the LUNs discovered by the initiator [root@host]# lsscsi –g Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator Alternatively, the LUNs discovered by the Chelsio FCoE initiators can be accessed via easilyidentifiable „udev‟ path device files like: [root@host]# ls /dev/disk/by-path/pci-0000:04:00.0-csio-fcoe :: Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator 4.4. Creating Filesystem Create an ext3 filesystem using the following command: [root@host]# mkfs.ext3 /dev/sdx 4.5. Mounting the formatted LUN The formatted LUN can be mounted on the specified mountpoint using the following command: [root@host]# mount /dev/sdx /mnt Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter IX. FCoE Full Offload Initiator Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter X. Offload Bonding driver X. Offload Bonding driver Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter X. Offload Bonding driver 1. Introduction The Chelsio Offload bonding driver provides a method to aggregate multiple network interfaces into a single logical bonded interface effectively combining the bandwidth into a single connection. It also provides redundancy in case one of link fails. The traffic running over the bonded interface can be fully offloaded to the T4 Adapter, thus freeing the CPU from TCP/IP overhead. 1.1. Hardware Requirements 1.1.1.
Chapter X. Offload Bonding driver Suse Linux Enterprise Server 11 SP2 kernel Ubuntu 12.04, 3.2.0-23 Kernel.org linux-3.1 Kernel.org linux-3.5 Kernel.org linux-3.6* (SLES11SP2), 3.0.13-0.27 Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter X. Offload Bonding driver 2. Software/Driver Loading To use the offload bonding driver, IPv6 should be enabled on the machine and the drivers should be built and installed with IPv6 support. The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host]# modprobe bonding Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter X. Offload Bonding driver 3. Software/Driver Unloading To unload the driver, run the following command: [root@host]# rmmod bonding Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter X. Offload Bonding driver 4. Software/Driver Configuration and Fine-tuning 4.1. Creating a Bond Interface The Chelsio Offload Bonding driver supports active-backup (mode=1), 802.3ad (mode=4) and balance-xor (mode=2) modes. To create a bond, use the following command: [root@host]# modprobe bonding mode=1 miimon=100 This would create a bond interface (bondX) in active-backup mode.
Chapter XI. UDP Segmentation Offload and Pacing XI. UDP Segmentation Offload and Pacing Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XI. UDP Segmentation Offload and Pacing 1. Introduction Chelsio‟s T4 series of adapters provide UDP segmentation offload and per-stream rate shaping to drastically lower server CPU utilization, increase content delivery capacity, and improve service quality. Tailored for UDP content, UDP Segmentation Offload (USO) technology moves the processing required to packetize UDP data and rate control its transmission from software running on the host to the network adapter.
Chapter XI. UDP Segmentation Offload and Pacing 1.1. Hardware Requirements 1.1.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with the UDP Segmentation Offload and Pacing driver. T420-CR T420-CX T440-CR T404-BT T422-CR T440-LP-CR T420-LL-CR T420-BCH T420-BT 1.2. Software Requirements 1.2.1.
Chapter XI. UDP Segmentation Offload and Pacing Kernel.org linux-3.1 Kernel.org linux-3.5 Kernel.org linux-3.6 Other kernel versions have not been tested and are not guaranteed to work. *Only IPv4 supported Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XI. UDP Segmentation Offload and Pacing 2. Software/Driver Loading The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. Run the following commands to load the drivers: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Though normally associated with the Chelsio TCP Offload engine, the t4_tom module is required in order to allow for the proper redirection of UDP socket calls. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XI. UDP Segmentation Offload and Pacing 3. Software/Driver Unloading Reboot the system to unload the driver. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XI. UDP Segmentation Offload and Pacing 4. Software/Driver Configuration and Fine-tuning 4.1. Modifying the application To use the UDP offload functionality, the application needs to be modified. Follow the steps mentioned below: i. Determine the UDP socket file descriptor in the application through which data is sent ii. Declare and initialize two variables in the application: int fs=1316; int cl=1; Here, fs is the UDP packet payload size in bytes that is transmitted on the wire.
Chapter XI. UDP Segmentation Offload and Pacing Here: sockfd : The file descriptor of the UDP socket &fs / &cl : Pointer to the framesize and class variables sizeof(fs) / sizeof(cl) : The size of the variables v. Now, compile the application. 4.1.1. UDP offload functionality for RTP data In case of RTP data, the video server application sends the initial sequence number and the RTP payload. The USO engine segments the payload data, increments the sequence number and sends out the data.
Chapter XI. UDP Segmentation Offload and Pacing 4.2. Configuring UDP Pacing Now that the application has been modified to associate the application‟s UDP socket to a particular UDP traffic class, the pacing of that socket‟s traffic can be set using the cxgbtool utility. The command and its parameters are explained below: [root@host]# cxgbtool sched-class params type packet level cl-rl mode flow rate-unit bits rate-mode absolute channel
Chapter XII. Offload IPv6 driver XII. Offload IPv6 driver Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XII. Offload IPv6 driver 1. Introduction The growth of the Internet has created a need for more addresses than are possible with IPv4. Internet Protocol version 6 (IPv6) is a version of the Internet Protocol (IP) designed to succeed the Internet Protocol version 4 (IPv4). Chelsio‟s Offload IPv6 driver provides support to fully offload IPv6 and UDP Segmentation traffic to the T4 adapter. 1.1. Hardware Requirements 1.1.1.
Chapter XII. Offload IPv6 driver Suse Linux Enterprise Server 11 SP2 kernel (SLES11SP2), 3.0.13-0.27 Ubuntu 12.04, 3.2.0-23 Kernel.org linux-2.6.39 Kernel.org linux-2.6.37 Kernel.org linux-2.6.36 Kernel.org linux-2.6.35 Kernel.org linux-2.6.34 Kernel.org linux-3.1 Kernel.org linux-3.5 Kernel.org linux-3.6* Other kernel versions have not been tested and are not guaranteed to work. *Limited QA peformed. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XII. Offload IPv6 driver 2. Software/Driver Loading The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. After installing the driver package and rebooting the host, load the offload drivers as: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom All the IPv6 traffic will be offloaded now. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XII. Offload IPv6 driver 3. Software/Driver Unloading Please reboot the system to unload the TOE driver. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XII. Offload IPv6 driver 4. Software/Driver Configuration and Fine-tuning 4.1. Offloading IPv6 traffic Once the IPv6 Offload driver is loaded successfully, load the t4_tom driver: [root@host]# modprobe t4_tom All the IPv6 TCP traffic will be offloaded now. 4.2. Network Device Configuration Please refer to the operating system documentation for administration and configuration of network devices.
Chapter XIII. Bypass Driver XIII. Bypass Driver Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XIII. Bypass Driver 1. Introduction Chelsio‟s B420 and B404 Bypass Adapters are Ethernet cards that provide bypass functionality and an integrated L2, L3, and L4 Ethernet switch. The integrated switch allows for selective bypass on a per-packet basis at line rate. To use the bypass adapters, you must have both the Chelsio NIC driver and the bypass CLI user space application loaded. 1.1.
Chapter XIII. Bypass Driver Disconnect Mode The Bypass cards can also be programmed to drop all the packets. Selective Bypass In Normal mode, the Bypass Adapters can be programmed to perform redirection of packets depending on the certain portion of the packet. The specification of the match criteria is called a rule. When a rule is matched an action is applied to the ingress packet. The actions that are supported are drop, forward and input. The drop action causes the packet to be discarded.
Chapter XIII. Bypass Driver 1.3. Software Requirements 1.3.1.
Chapter XIII. Bypass Driver 2. Software/Driver Loading The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. Run the following command to load the Bypass driver: [root@host]# modprobe cxgb4 Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XIII. Bypass Driver 3. Software/Driver Unloading Run the following command to unload the Bypass driver: [root@host]# rmmod cxgb4 Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XIII. Bypass Driver 4. Software/Driver Configuration and Fine-tuning 4.1. Starting ba server 4.1.1. For IPv4 only Execute the following command to start the ba server only for IPv4: [root@host]# ba_server –i ethX 4.1.2. For IPv4 and IPv6 Execute the following command to start the ba server for IPv4 and IPv6: [root@host]# ba_server -6 –i ethX 4.2. Bypass API (CLI) A CLI will be created that implements the Bypass API as specified below. This CLI will then communicate the requests to the SDK server.
Chapter XIII. Bypass Driver E.g.
Chapter XIII. Bypass Driver Getting the default state: [root@host]# bypass ethX get --default_state Setting the default state: [root@host]# bypass ethX set --default_state [bypass |disconnect] 4.2.4. Using the bypass watchdog timer The watchdog timer is used to ensure that if there is a software failure, the switch will enter the default state.
Chapter XIII. Bypass Driver Redirect Command List Command redirect list Key Value ethX redirect ethX add update match list of all configured tables and rules table proto srcaddr dstaddr vlan action port srcmask dstmask srcport dstport srcportmask dstportmask rule index. If not specified the rule is appended. Otherwise it is inserted at the specified index.
Chapter XIII. Bypass Driver old_id new_id redirect ethX count redirect ethX create_table redirect ethX activate_table redirect ethx deactivate_table redirect ethX delete_table redirect ethX dump Existing rule index. New rule index. table table id index rule index table table id table table id table table id table table id Display a count of the number of packets and bytes has matched a rule. Create a new table that is used to hold a set of rules.
Chapter XIII. Bypass Driver To make the current configured rules & tables persistent, redirect the output to /etc/ba.cfg file only: [root@host]# redirect ethX dump > /etc/ba.cfg where the /etc/ba.cfg is read by the bad service at boot time. To apply the saved configuration after machine reboots, start the bad service. This service is available only in the IPv4 mode. For IPv6, the ba_server needs to be started manually.
Chapter XIV. WD Sniffing and Tracing XIV. WD Sniffing and Tracing Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XIV. WD Sniffing and Tracing 1. Theory of Operation The objective of these utilities (wd_sniffer and wd_tcpdump_trace) is to provide sniffing and tracing capabilities by making use of T4's hardware features. Sniffer- Involves targeting specific multicast traffic and sending it directly to user space. a) Get a Queue (raw QP) idx. b) Program a filter to redirect specific traffic to the raw QP queue.
Chapter XIV. WD Sniffing and Tracing Schematic diagram of T4 sniffer and tracer 1.1. Hardware Requirements 1.1.1. Supported Adapters The following are the currently shipping Chelsio Adapters that are compatible with the tools: T420-CR T420-LL-CR T440-CR T440-LP-CR T420-BCH Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XIV. WD Sniffing and Tracing T422-CR T420-CX T420-BT T404-BT 1.2. Software Requirements 1.2.1.
Chapter XIV. WD Sniffing and Tracing 2. Installation and Usage 2.1. Installing basic support iw_cxgb4 (Chelsio iWARP driver) and cxgb4 (Chelsio NIC driver) drivers have to be compiled and loaded before running the utilities. Refer to the Software/Driver Loading section for each driver and follow the instructions mentioned before proceeding. 2.2. Using Sniffer (wd_sniffer) 1. Setup: Wire filter sniffing requires 2 systems with one machine having a T4 card.
Chapter XIV. WD Sniffing and Tracing DUT: Machine B PEER: Machine A <-----> (port 0) (port 1) 192.168.1.100 IP-dont-care IP-dont-care <-----> PEER: Machine C 192.168.1.200 2. Procedure: Run wd_tcpdump_trace -i iface on the command prompt where iface is one of the interfaces whose traffic you want to trace. In the above diagram its port 0 or port 1. [root@host]# wd_tcpdump_trace -i Try ping or ssh between machines A and B.
Chapter XV. Classification and Filtering XV. Classification and Filtering Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XV. Classification and Filtering 1. Introduction Classification and Filtering feature enhances network security by controlling incoming traffic as they pass through network interface based on source and destination addresses, protocol, source and receiving ports, or the value of some status bits in the packet. This feature can be used in the ingress path to: Steer ingress packets that meet ACL (Access Control List) accept criteria to a particular receive queue.
Chapter XV. Classification and Filtering Redhat Enterprise Linux 6 base kernel Redhat Enterprise Linux 6 update 1 kernel Redhat Enterprise Linux 6 update 2 kernel Redhat Enterprise Linux 6 update 3 kernel Suse Linux Enterprise Server 10 SP2 kernel Suse Linux Enterprise Server 10 SP3 kernel Suse Linux Enterprise Server 11 kernel Suse Linux Enterprise Server 11 SP1 kernel Suse Linux Enterprise Server 11 SP2 kernel Fedora release 13 Fedora release 14 Ubuntu 12.04, 3.2.0-23 Kernel.org linux-2.6.34 Kernel.
Chapter XV. Classification and Filtering 2. Usage 2.1. Configuration The Classification and Filtering feature is configured by specifying the filter selection combination set in the T4 firmware configuration file t4-config.
Chapter XV. Classification and Filtering Where, ethX index action pass switch drop : Chelsio interface : positive integer set as filter id : Ingress packet disposition : Ingress packets will be passed through set ingress queues : Ingress packets will be routed to an output port with optional header rewrite. : Ingress packets will be dropped. Note In case of multiple filter rules, the rule with the lowest filter index takes higher priority. 2.2.1.
Chapter XV. Classification and Filtering For offloaded ingress packets, use the prio argument with the above command: [root@host]# cxgbtool ethx filter action prio 1 Note For more information on additional parameters, refer to cxgbtool manual by running the man cxgbtool command 2.3. Listing Filter Rules To list the filters set, run the following command: [root@host]# cxgbtool ethX filter show 2.4.
XVI. Traffic Management XVI. Traffic Management Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVI. Traffic Management 1. Introduction Traffic Management capabilities built-in to Chelsio T4 CNAs can shape transmit data traffic through the use of sophisticated queuing and scheduling algorithms built-in to the T4 ASIC hardware which provides fine-grained software control over latency and bandwidth parameters such as packet rate and byte rate. These features can be used in a variety of data center application environments to solve traffic management problems.
XVI.
XVI. Traffic Management 2. Software/Driver Loading Traffic Management can be performed on non-offloaded connections as well as on offloaded connections. The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. Run the following commands to load the drivers: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVI. Traffic Management 3. Software/Driver Unloading Reboot the system to unload the driver. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVI. Traffic Management 4. Software/Driver Configuration and Fine-tuning 4.1. Traffic Management Rules Traffic Management supports the following types of scheduler hierarchy levels which can be configured using the cxgbtool utility: i. Class Rate Limiting ii. Class Weighted Round Robin iii. Channel Rate Limiting 4.1.1. Class Rate Limiting This scheduler hierarchy level can be used to rate limit individual traffic classes or individual connections (flow) in a traffic class.
XVI. Traffic Management 4.1.2. Class Weighted Round Robin Incoming traffic flows from various applications can be prioritized and provisioned using a weighted round-robin scheduling algorithm. Class weighted round robin can be configured using the following command: [root@host]# cxgbtool sched-class params type packet level cl-wrr mode class rate-unit bits rate-mode absolute channel class weight Here, ethX is the Chelsio interface Channel No.
XVI. Traffic Management 4.2. Configuring Traffic Management 4.2.1. For Non-offloaded connections Traffic Management of non-offloaded connections can be configured by binding the indicated NIC TX queue to the specified TX Scheduler class using the cxgbtool utility. 1. Bring up the interface: [root@host]# ifconfig ethX up 2.
XVI. Traffic Management 2. Create a new policy file (say new_policy_file) and add the following line to associate connections with the given scheduling class. E.g.: src host 102.1.1.1 => offload class 0 The above example will associate all connections originating from IP address 102.1.1.1 with scheduling class 0 Note COP supports association of connections with scheduling classes 0 and 1. 3. Compile the policy file using COP [root@host]# cop -d –o 4.
XVI. Traffic Management 2. Declare and initialize a variable in the application: int cl=1; Here, cl is the TCP traffic class(scheduler-class-index) that the user wishes to assign the data stream to. This value needs to be in the range of 0 to 7. The application will function according to the parameters set for that traffic class. 3.
XVI. Traffic Management 5. Usage 5.1. Non-Offloaded Connections The following example demonstrates the method to rate limit all TCP connections on class 0 at a rate of 300 Mbps for Non-offload connections: 1. Load the Network driver in NIC mode. [root@host]# modprobe cxgb4 2. Bind the NIC TX queue to class 0 [root@host]# cxgbtool ethX sched-queue 0 0 3.
XVI. Traffic Management 3. Compile the policy file using COP [root@host]# cop -d –o 4. Apply the COP policy: [root@host]# cxgbtool ethX policy 5. Set the appropriate rule for class 0 [root@host]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 5.3.
XVII. Unified Wire Manager (UM) XVII. Unified Wire Manager (UM) Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 1. Introduction Chelsio's Unified Wire Manager is a powerful management software tool, allowing you to view and configure different aspects of the system, including Chelsio hardware installed in the system. The software includes a command line interface (CLI) tool and a web management interface (Web GUI) to help you manage all Chelsio network adapter on the network across multiple operating systems.
XVII. Unified Wire Manager (UM) 1.2. Reference Architecture Chelsio’s Web GUI is a web-based management interface that lets you remotely manage several Chelsio CNAs from anywhere, at anytime on the network using a web browser. The Web GUI provides a great amount of flexibility, efficiency and accessibility to system administrators in managing Network and SAN resources.
XVII. Unified Wire Manager (UM) properties. You can use either the CLI or Web GUI client to manage agents based on your preference. It makes service requests based on the command issued by the user and returns the appropriate information. CLI Client The CLI Client (chelsio_uwcli) is an executable binary which allows you to manage and configure agents using the command-line interface.
XVII. Unified Wire Manager (UM) 2. Hardware and Software 2.1.
XVII. Unified Wire Manager (UM) Intel Myricom Broadcom S2io nVIDIA REALTEK D-Link Alteon DEC 2.2. Platform/Component Matrix The table below lists the Linux distributions and the supported UM components. Distribution Supported UM Components RHEL5.8, 2.6.18-308.el5 (64-bit) Management Agent, Management Client RHEL6.3, 2.6.32-279.el6 (64-bit) Management Agent, Management Client, Web Management Interface SLES11SP2, 3.0.13-0.27 (64-bit) Management Agent, Management Client, Web Management Interface 2.3.
XVII. Unified Wire Manager (UM) 3. Installing Unified Wire Manager Chelsio Unified Wire has been designed to install Unified Wire Manager (UM) by default. All the three UM components, i.e. Management Agent, Client and Station, will be installed on selecting any of the Terminator 4 configuration tuning options during installation. Hence, no separate installation is required. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 4. Licensing Unified Wire Manager The Unified Wire Manager software has a licensing component, which unlocks the functionality of the software. On installing the software, and using the Web GUI or CLI, the software is restricted to work only on the system summary module or system service. The Chelsio cards on the system are listed, but no management functionality is enabled for them.
XVII. Unified Wire Manager (UM) 7. Start/Restart the Management Station [root@host]# /etc/init.d/chelsio-mgmtstd 8. Start/restart Apache Server. 4.2.2. Licensing using Web GUI 1. 2. 3. 4. Follow steps 1-5 mentioned above to use CLI to receive the license key. Access the system‟s Web Management Interface using a web browser by entering its IP. Enter user credentials and click Login. A prompt “Management Station is not licensed. License It” will be displayed. Click on License It. 5.
XVII. Unified Wire Manager (UM) 4.3. Licensing Management Agent 4.3.1. Licensing using CLI 1. Please make sure that at least one Chelsio card is present in the system where Management agent is installed. Also, the required drivers should be installed before proceeding. 2. Now, run the following command: [root@host]# chelsio_uwcli system createinfofile The above command will generate an information file, chinfofile, in the current working directory. 3.
XVII. Unified Wire Manager (UM) 4. The license key file that is received from Chelsio support needs to be installed on the system running the management agent. In the Licensing module, locate the license key using the Select button. 5. Now, click the Install License Key button. The license key will now be installed and details like license type, device serial Id, Chelsio card used for licensing, etc will be displayed. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 5. Verifying UM components status The following section explains how to verify status of various UM components. 5.1. Verifying Management Agent 1. Execute the following query command : [root@chelsio]# ps –eaf | grep UW The above query should confirm that Management Agent is running by displaying a similar result: root root root root 30531 1 30534 1 30537 1 30581 28384 0 0 0 0 09:27 09:27 09:27 09:45 ? ? ? pts/1 00:00:00 00:00:00 00:00:00 00:00:00 ./UWMgrServer .
XVII. Unified Wire Manager (UM) 5.2. Verifying Management Client Execute the following query command to determine if Management Client is installed: [root@host]# chelsio_uwcli -V The above query should confirm that Management Client is installed by displaying a similar result: Unified Manager client CLI version : 2.x.yy 5.3. Verifying Management Station Execute the following query command to determine the status of Management Station: [root@host]# /etc/init.
XVII. Unified Wire Manager (UM) 6. Management Agent 6.1. Communication The agent uses a TCP connection over IP to communicate with the client. After the connection is established, SSL (Secure Sockets Layer) encryption is enabled using the Open SSL libraries. The agent listens on a TCP port for new incoming connections from clients. This port is set to 35001 by default. It may be changed in the configuration file for the agent. The agent needs to be restarted after the change. 6.2.
XVII. Unified Wire Manager (UM) 6.3.2. Service start/stop/restart You can start, stop or restart the service by using the following command: [root@host]#/etc/init.d/chelsio-uwire_mgmtd [start|stop|restart] 6.4. Firewall If the system has a firewall configured, such as iptables, it should be configured to allow traffic to the management agent TCP port configured above in the configuration section, or the default port that the management agent uses, 35001.
XVII. Unified Wire Manager (UM) 7. CLI client 7.1. CLI Help system A detailed help and usage documentation is built into the CLI, and is accessible through its help system. The help can be invoked by the usual argument of /? or --help. 7.1.1. Viewing help Use the chelsio_uwcli command to view the help file as shown below: [root@host]# chelsio_uwcli /? 7.2. Client conflict resolution The CLI and Web GUI cannot manage the same system at the same time by default.
XVII. Unified Wire Manager (UM) 8. Web GUI client 8.1. Management Station In order to access the Web Management Interface, Apache HTTP server should be installed and running on a machine. Also, Cookies and Javascript must be enabled in the browser. 8.1.1. Running Management Station on RHEL 6.3 1. Start/Restart Apache httpd daemon: [root@host]# service httpd [start|restart] 2. Start/Restart the Management Station: [root@host]# /etc/init.d/chelsio-mgmtstd [start|restart] 8.1.2.
XVII. Unified Wire Manager (UM) 3. Start/Restart the Management Station: [root@host]# /etc/init.d/chelsio-mgmtstd [start/restart] 8.2. Accessing Web Management Interface 1. To access the Web GUI, type in the URL https:// in a web browser. 2. The security certificate used by the web server is a generic one. It may cause the following types of prompts in different browsers. You will need to select the correct option to continue. Figure 8.
XVII. Unified Wire Manager (UM) Figure 8.2 (b) - Security Certificate prompt in Mozilla Firefox Figure 8.2 (c) - Security Certificate prompt in Apple Safari Figure 8.2 (d) - Security Certificate prompt in Google Chrome 3. The web interface requires password authorization to be accessed. Enter the username and corresponding password that was set up on the management station system and click on the Login button. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Figure 8.2 (e) - Web GUI Login page Note Not performing any operation/action for 5 minutes will result in session timeout. You will have to re-login and connect to the Agents again. 8.3. Layout and Navigation The Web Management Interface consists of the following: Title bar displaying the username on the left, Unified Wire Manager logo and name in the centre; and a Logout button on the right.
XVII. Unified Wire Manager (UM) Figure 8.3 – Web Management Interface 8.4. Home page The home page is displayed by default on launching the Web GUI. It displays Bookmarks and History, Service Discovery and Bulk Driver Installation modules. Options to go back to home page, add/remove system, refresh and configure email alerts are also available. 8.4.1. Home This option will display the home page. 8.4.2. Add System Use this option to connect to new Agents using their IP or Hostname.
XVII. Unified Wire Manager (UM) Figure 8.4(a) – Adding a UM Agent 8.4.3. Remove System Use this option to disconnect an Agent. To remove an agent, click on the name of the system in the tree menu in the left and click Remove System. Then click Yes to confirm. Figure 8.4(b) – Removing a UM Agent 8.4.4. Refresh This option can be used to reload the Web GUI or UM Agent. To reload the Web GUI, navigate to the Home page (by clicking on the “Home” button and click Refresh.
XVII. Unified Wire Manager (UM) 8.4.5. Subscribe (Email Alerts) This feature is available only on the Web Management Interface. Using this option, you can receive email alerts regarding the link status of a Chelsio Network Interface Card. This feature sends email notifications regarding the port and the card, on which the link up/down event is occurring. Not only can you configure multiple email addresses to receive notifications, but also customize the email id of the sender for troubleshooting purposes.
XVII. Unified Wire Manager (UM) 8.4.6. Bookmarks and History A history of the last 128 systems that were managed from this system, by the current user, will be shown here in a list. Each system's management IP address, TCP port, and Login details are also stored. This may be edited and saved. Any systems that are not required in the list may be deleted. Important Storing login passwords for the managed systems is inherently insecure.
XVII. Unified Wire Manager (UM) Removing a system Select the system from the Bookmark list and click Delete system to remove it. Note Once removed, the system will no longer appear in the Bookmarks and History module. If you wish to manage that system again, you will have to use the Add system option. 8.4.7. Service Discovery Using this module, all the Unified Wire Manager agents connected in the same or different subnet can be discovered.
XVII. Unified Wire Manager (UM) selected and the operating system (Linux, Windows, ESX, Macintosh, Solaris, FreeBSD) selected. Installing Driver 1. In the Choose the card fields, select T3 or T4 depending on the chip revision of the network card. 2. Select the operating system for which drivers are to be installed in the Choose the OS Type field. All the systems with the selected operating system will be displayed in the list below. 3.
XVII. Unified Wire Manager (UM) Note Agents that report errors or whose passwords are not stored, will be automatically skipped during the driver installation. 8.5. System page The system page is displayed, when the system hostname / IP address is selected in the tree menu on the left. On adding a system, this item is automatically selected, and this page is displayed. The system page contains generic system and support modules, which work even without a license installed. The modules available are: 8.
XVII. Unified Wire Manager (UM) 8.5.2. Drivers Installation Using this module, one can install various Chelsio drivers for different operating systems. You can choose the configuration file type (Linux Agents only). You can also update Unified Wire Manager software by using the Unified Manager Package section. Figure 8.5 (b) - Drivers Installation module connected to Linux Agent Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Figure 8.5 (c) - Drivers Installation module connected to FreeBSD Agent 8.5.3. Licensing The licensing module allows generating and saving the information file required for obtaining a license key from Chelsio support. It also allows installing the license key file to the managed system. If a license key is installed, the details of the license are displayed. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Figure 8.5 (d) - Licensing using a T3 HBA Figure 8.5 (e) - Unified Wire Manager licensed with a T4 CNA Note Refer to Licensing Unified Wire Manager section for a detailed explanation on licensing various UM components on different operating systems. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.5.4. Driver Details A list of the Chelsio device drivers installed on the system is shown in this module, with version information and the current status, whether the driver is loaded and active or not. Figure 8.5 (f) - Driver Details module for a Linux Agent 8.5.5. System Diagnostics Using this module, you can run various diagnostic tests on Chelsio (T3 and T4) HBAs to troubleshoot adapter related issues. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Figure 8.5 (g) – System Diagnostics module for a T4 CNA 8.5.6. Unified Wire Manager Component Versions A list of the Unified Wire Manager agent components installed on the managed system is shown in this module. The versions of the components are useful in case of reporting an issue to support. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Figure 8.5 (h) - Unified Wire Manager Component Versions module for a Windows Agent 8.5.7. Managed system application logs The management agent logs its activities and any errors that occur, in /var/log/chelsio in Linux and FreeBSD and in the Event log, in Windows. This log can be obtained in this module. Only 20 entries can be obtained and viewed at a time. Logs can be viewed by either choosing from a list of fixed range or by specifying a custom starting point. Figure 8.
XVII. Unified Wire Manager (UM) Figure 8.6 (a) – Network Summary module 8.6.2. Chelsio card page When a Chelsio card is selected in the tree menu on the left, this page is displayed. It provides details of the card and associated settings. It also displays any card specific statistics that the hardware provides. The modules available on this page are as below: 8.6.3. Card summary This module provides PCI, firmware and other details of the card.
XVII. Unified Wire Manager (UM) Figure 8.6.1 (a) - Card Summary module for a Windows Agent 8.6.4. TCP Offload settings (Linux & FreeBSD) The TCP offload settings applicable to the card are shown here. These settings are only available when using the TOE capable drivers (t3_tom and toecore for T3 cards; t4_tom and toecore for T4 cards). On changing the settings, the changed settings may not reflect immediately on refreshing the data.
XVII. Unified Wire Manager (UM) Figure 8.6.1 (b) – TCP Offload Settings module for a Linux Agent Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Figure 8.6.1 (c – TCP Offload Settings module for a FreeBSD Agent 8.6.5. Device Driver settings (Windows) The device driver settings applicable to the card are shown here. For Chelsio T4 cards, only the MaxVMQueues field will be displayed. On changing the settings, the changed settings may not reflect immediately on refreshing the data.
XVII. Unified Wire Manager (UM) 8.6.6. Card statistics Certain statistics are maintained on a per card basis (instead of a per port basis), since the card has a TCP/IP offload capability. The statistics are for TCP and IP protocol processing done in the card's hardware. These statistics may only be applicable if the card is TOE enabled. Figure 8.6.
XVII. Unified Wire Manager (UM) Figure 8.6.1.1 (a) - Port Summary of T4 CNA on Linux Agent Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.6.8. Port settings Port settings such as MTU, Link speed and others can be set in this module. The settings depend on the device driver installed. Figure 8.6.1.1 (b) - Port Settings of T4 CNA on Linux Agent Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.6.9. Port statistics Ethernet statistics and additional hardware statistics for the port are displayed in this module. Figure 8.6.1.1 (c) - Port Statistics of T4 CNA on Linux Agent 8.6.10. Networking Management page The system networking and teaming / bonding configurations are shown on this page. IP addresses, MTU, VLAN Ids, DNS and default gateway settings can be viewed and modified here. Network adapters can also be enabled or disabled as required.
XVII. Unified Wire Manager (UM) addresses or aliases for the specified adapter. Use the option to add additional IP addresses with caution, since multiple IP addresses configured on the same adapter, for the same network, may result in unpredictable behavior of the system's networking stack. Maximum Transfer Unit (MTU) can be set between 1500-9000 bytes. VLAN id can also be set for an adapter within the range 0-4094 (enter 0 to disable it).
XVII. Unified Wire Manager (UM) 8.6.12. System network statistics Using this module, one can generate reports based on Throughput pkts/sec and Throughput Mbs (Receive, Transmit, Bi-direction) in Table and Graph format for a network adapter. A report for hardware statistics can be generated based on different parameters, only in the Table view in the Advanced NIC characteristics. The polling time field sets the average time (in seconds) based on which the table/graph updates the report. Figure 8.6.
XVII. Unified Wire Manager (UM) Figure 8.6.2 (d) - Network Throughput Vs Time instant Graph Figure 8.6.2 (e) - Network Throughput Vs Time instant Table Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.6.13. Default Gateway and DNS configuration The DNS servers list can be set here. The default gateway for remote networks and the Internet can also be set here. On Linux and FreeBSD, only one default gateway is allowed. On Windows, you may set multiple default gateways. Use the option to set multiple default gateways with caution, since it may cause the system to stop communicating with external networks. Figure 8.6.
XVII. Unified Wire Manager (UM) when creating the team, although both DHCP and Static IP addressing is supported for the team adapter, after it is created successfully. Please check with the driver documentation for the supported modes for creating a team / bond, with offload enabled Chelsio cards. All modes may not be available with all configurations / combinations. Also, the team members can only be 2 ports of a single offload-enabled card, and not across Chelsio cards.
XVII. Unified Wire Manager (UM) 8.6.15. Network troubleshooting This module allows detecting and troubleshooting various network connectivity issues. The Ping utility helps to contact a system by specifying IP address, Number of ICMP packets to send and packet timeout. The result of the ping can be viewed by clicking on the Ping Result button. Using TraceRoute one can determine the route taken by packets across an IP network. Figure 8.6.2 (h) - Ping Utility Figure 8.6.
XVII. Unified Wire Manager (UM) Figure 8.6.2 (j) - GetConnections Utility Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.6.16. iWARP On Linux Agents, iWARP parameter settings for Chelsio's RDMA capable NICs (T3 and T4) can be set using this module. These settings can be set only when iWARP driver (iw_cxgb3 for T3; iw_cxgb4 for T4) is loaded. If you set any parameter for a T3 card, it applies for all the T3 cards present. Same applies for T4 cards. On Windows Agents, only T3 HBAs are supported currently. Parameters can be set per port. On FreeBSD Agents, only T4 CNAs are supported.
XVII. Unified Wire Manager (UM) Figure 8.6.3 (a) – iWARP settings for T4 CNA for FreeBSD Agent Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Figure 8.6.3 (b) – iWARP settings for T4 CNA for Linux Agent 8.6.17. WD-UDP The WD-UDP module lists the process ids (pid) of UDP traffic running on the agent and displays the corresponding statistics and attributes. Note Please ensure that WD-UDP traffic is running on the agent before accessing this module. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7. Storage (Linux and Windows) Storage Summary The Storage module lists the status of configuration modules under Storage section, running on the agent. Figure 8.7 – Storage Summary Module Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.1. FCoE Initiator All supported Chelsio FCoE initiators available on the operating system can be managed from this page. FCoE support is extended on Linux and Windows platforms. Please refer to Platform/Driver matrix section on the list of operating systems that are supported. FCoE Initiator Summary This module provides details about the driver installed; such as driver name and its version.
XVII. Unified Wire Manager (UM) Figure 8.7.1.1 (a) – FCoE Card Summary module FCoE Attributes Information such as Interrupt modes (MSI/MSI-X/INTx), SCSI mode and the card state are provided in this module. Figure 8.7.1.1 (b) – FCoE Attributes module 8.7.1.2. FCoE Port This is an actual N_Port which communicates with the fabric and performs FIP and FCoE device discovery. This page lets the user to retrieve all the FCoE specific port information and also extend NPIV management support.
XVII. Unified Wire Manager (UM) The SCSI adapter name and the underlying ENODE MAC address of the physical port can be found here. Figure 8.7.1.2 (a) – FCoE Port Summary module for Linux Agent Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) FCoE Port Attributes This module provides details about link status and port identifiers such as WWPN, WWNN, FC ID and NPort MAC Address. The module also contains fabric information such as fabric name, VLAN on which the FCoE service is currently running and the number of SCSI targets that are being discovered by this port. Port speed being mentioned in this section varies on the card type (10G/1G) being used.
XVII. Unified Wire Manager (UM) FCoE NPIV management NPIV is a fibre channel facility allowing multiple N_Port IDs to share a single physical N_Port. This module allows the user to manage virtual ports on the corresponding FCoE Port. To create a virtual port, select the option Create and the GUI allows two ways of creating a virtual port. i. Manual: Where the user can manually create a virtual port by providing a value to the WWPN and WWNN fields. ii.
XVII. Unified Wire Manager (UM) 8.7.1.3. FCoE Virtual Port A virtual port allows multiple Fibre Channel initiators to occupy a single physical port, easing hardware requirements in SAN design, especially where virtual SANs are called for. The virtual ports appear under their respective N_Ports after creation and the GUI conveys it via a tree structure so that the end user knows the N_port-VN_Port mapping.
XVII. Unified Wire Manager (UM) FCoE Virtual Port Attributes The module provides details about link status and port identifiers such as WWPN, WWNN, FC ID and Virtual NPort MAC Address. The module also contains fabric information such as fabric name, VLAN on which the FCoE service is currently running and the number of SCSI targets that are being discovered by this virtual port. Port speed being mentioned in this section varies on the card type (10G/1G) being used.
XVII. Unified Wire Manager (UM) 8.7.1.4. FCoE Remote Port Remote ports are the SCSI targets that are discovered by their respective N_port/virtual ports. The GUI conveys the same via a tree structure so that the end user knows the initiator-target mapping. FCoE Remote Port Summary This module displays the name of the FCoE target. Figure 8.7.1.
XVII. Unified Wire Manager (UM) 8.7.1.5. LUN This page provides the LUN information such as size of the LUN, SCSI address, OS device name and the 64 bit LUN address. For Linux, the SCSI address is displayed in H:C:T:L (Host:Channel:Target:Lun) format and for Windows, results are displayed in P:B:T:L(SCSI Port:Bus:Target:Lun) format. Figure 8.7.1.5 – LUN Summary module 8.7.1.6. iSCSI initiator All supported iSCSI initiators available on the OS can be managed from this page.
XVII. Unified Wire Manager (UM) Figure 8.7.1.6 (a) - Open iSCSI initiator Figure 8.7.1.6 (b) - Microsoft iSCSI initiator Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) Figure 8.7.1.6 (c) - Chelsio iSCSI initiator Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.3. Discover targets iSCSI targets can be discovered by providing the IP address and TCP port (usually 3260) of the target. In Windows, you can specify the initiator HBA to use and its IP address. The discovery operation fetches the targets found at that Portal (combination of IP address and TCP port). The discovery operation also fetches all the other Portals that the target(s) are listening on. The discovered target can be deleted if required.
XVII. Unified Wire Manager (UM) Figure 8.7.1.6 (e) - Discover targets module Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.4. Targets The iSCSI targets that have been discovered, or are currently connected, are listed here. You may login, logout and delete the target from the initiator's configuration. In Windows, for the Microsoft iSCSI initiator, connections to an already established iSCSI session can be added or deleted .For the Microsoft iSCSI initiator or the Open iSCSI initiator, you may specify the authentication details and digest settings while logging in.
XVII. Unified Wire Manager (UM) 8.7.5. iSCSI Target page (Linux) This page allows to create new Targets and manage them (add/delete portals, add/delete LUNs, add/delete ACLs).It also provides information on Session details. Viewing and modifying Target properties is also available. The modules available on this page are as below: 8.7.6. Target Stack Globals This module displays various global properties of a currently connected iSCSI target. Authentication priority between CHAP and ACL can be set here.
XVII. Unified Wire Manager (UM) 8.7.7. Target properties Properties such as Target name and Alias, Max Data Receive Length, Authentication mode related to a specific iSCSI target can be viewed and modified here. iSCSI targets can be started/stopped or deleted. Figure 8.7.2 (b) - Target properties module Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.8. Session details Details including Session ID, Initiator IQN and Connections List of all discovered and currently connected iSCSI targets are listed here. Figure 8.7.2 (c) – Session Details module Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.9. New Target Creation New iSCSI target can be created here by specifying the Target IQN and Target Alias name. Figure 8.7.2 (d) - New Target Creation module Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.9.1. LUNs Various Logical Units created in an iSCSI Target can be managed here. The modules available on this page are as below: 8.7.10. View/Edit iSCSI Target LUNs This module displays various Logical Units created in an iSCSI Target. Selected LUNs can be deleted. Figure 8.7.2.1 (a) - View/Edit iSCSI Target LUNs module Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.11. Add LUN New LUNs can be added here by providing various parameters like Target Name, Target Device and RAM Disk Size etc. RW (Read-Write) and RO (Read Only) are the two kinds of permissions that can be set. If Ram Disk is selected, then a minimum of 16 MB should be provided. Figure 8.7.2.1 (b) - Adding a new LUN Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.11.1. Portal Groups Portal details for currently connected iSCSI Targets can be viewed and added here. The modules available on this page are as below: 8.7.12. View/Edit iSCSI Target Portals Portal List on the left displays details of the portal group on which an iSCSI target is listening and the related info is displayed on the right under Portal Details. Selected portals can be deleted. Figure 8.7.2.2 (a) - View/Edit iSCSI Target Portals module Copyright ©2013.
XVII. Unified Wire Manager (UM) 8.7.13. Add Portal New Portals can be added here by choosing the specific target and Portal IP address. The Port number should be 3260. Figure 8.7.2.2 (b) – Adding a new Portal Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.13.1. ACLs ACLs configured for currently connected iSCSI Targets can be managed here. The modules available on this page are as below: 8.7.14. View/Edit iSCSI Target ACLs This module displays details for all the ACLs configured for an iSCSI Target. Selected ACLs can be deleted. Figure 8.7.2.3 (a) - Target ACL operations module Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.7.15. Add ACL New ACLs can be configured by specifying Target name, initiator IQN name, IP address and permission type. Figure 8.7.2.3 (b) - Adding new ACL Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.8. Hardware Features The Hardware module lists the status of configuration modules under Hardware Features section, running on the agent. Figure 8.9 – Hardware module for a Linux Agent 8.8.1. Filtering (Linux) * applicable only to T3 Offload capable HBAs Using this page, one can control the traffic from a specific IP. The module available on this page is as below: 8.8.2.
XVII. Unified Wire Manager (UM) Figure 8.8.1 - Filtering Configuration module Note Results for actions like adding a new filter or setting maximum filters make some time to reflect. Highlight the system item in the tree menu on the left, and click "Refresh system", to refresh data from the system, in case the updated settings are not being shown. Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 8.8.3. Traffic Management (Linux) Using this page, one can add/delete/modify offload policies only in the presence of offload driver (t3_tom for T3 adapters; t4_tom for T4 adapters). 8.8.4. Traffic Management configuration The Chelsio Card section on the left displays all the cards available in the server and their corresponding policies on the right. Policies can be added and deleted. Policy Details displays the primitives (maximum 8) and actions which can be modified.
XVII. Unified Wire Manager (UM) 8.8.5. Boot T4 Save Config File (Linux) This module displays the current T4 configuration tuning option selected. You can also change the tuning option by selecting the config file for each option located in /ChelsioUwirex.x.x.x/src/network/firmware. For instance, to select Low latency Networking, locate the file, t4config.txt, in /ChelsioUwire-x.x.x.x/src/network/firmware/low_latency_config directory. Figure 8.8.3 (a) – T4 Save Config File module Copyright ©2013.
XVII. Unified Wire Manager (UM) T3 iSCSI / T3 PXE Boot Option Rom Management (T3 HBAs) The Chelsio T3 card may be used for PXE or iSCSI boot. This module allows managing the boot capability. The Option ROM (PXE/iSCSI) may be installed to the card, or erased from the card. iSCSI boot Option ROM settings can be configured for the card here. Enable the iSCSI Option ROM only if you are planning to boot the system via iSCSI. Refer the iSCSI boot documentation for more details about its configuration. Figure 8.
XVII. Unified Wire Manager (UM) T4 Boot Option ROM management (T4 CNAs) This module allows managing the PXE and FCoE boot capability for Chelsio T4 cards. The Option ROM (PXE and FCoE) may be installed to or erased from the card. The version of Option ROM flashed can be viewed here. Figure 8.8.3 (c) – T4 Option ROM Management module Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) T4 Boot Configuration (T4 CNAs) This module can be used to view and configure T4 PXE and FCoE Option ROM settings for Chelsio T4 cards. PXE physical functions and order of ports for PXE boot can be selected here. The same module can be used to configure FCoE Option ROM. Here port order for target discovery and discovery timeout can be set. Under the Boot section, LUN can be configured as boot device from the discovered FCoE targets.
XVII. Unified Wire Manager (UM) Figure 8.8.3 (e) - FCoE Boot configuration for T4 CNAs Copyright ©2013.Chelsio Communications. All Rights Reserved.
XVII. Unified Wire Manager (UM) 9. Uninstalling Unified Wire Manager This section describes the method to uninstall components of Chelsio Unified Manger. 9.1. Uninstalling Management Agent 1. Use the following query command to determine the name of the agent RPM: [root@host]# rpm –qa | grep chelsio-uwire_mgmt-agent 2. Now, execute the following command with the result from the above query to uninstall Management Agent: E.g. for RHEL 6.3: [root@host]# rpm –e chelsio-uwire_mgmt-agent-rhel6u3-2.2-xyz.
XVII. Unified Wire Manager (UM) 9.3. Uninstalling Management Station 1. Use the following query command to determine the name of the Management Station RPM: [root@host]# rpm –qa | grep chelsio-uwire_mgmt-station 2. Now, execute the following command with the result from the above query to uninstall Management Station: E.g. for RHEL 6.3: [root@host]# rpm –e chelsio-uwire_mgmt-station-rhel6u3-2.2-xyz.x86_64 Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot XVIII. Unified Boot Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot 1. Introduction PXE is short for Preboot eXecution Environment and is used for booting computers over an ethernet network using a Network Interface Card (NIC). FCoE SAN boot process involves installation of an operating system to an FC/FCoE disk and then booting from it. This section of the guide explains how to configure and use Chelsio T4 Unified Boot Option ROM which flashes PXE and FCoE Option ROM onto Chelsio‟s converged network adapters (CNAs).
Chapter XVIII. Unified Boot Linux Distribution Driver/Software (DUDs) RHEL6.3,2.6.32-279.el6 PXE,FCoE SLES11SP2,3.0.13-0.27 PXE,FCoE Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot 2. Flashing configuration file, firmware & option ROM Chelsio legacy T4 Flash utility (cfut4.exe) is used to program the PXE Option ROM image onto the Chelsio HBAs. Example # 1: This example assumes you are using a USB flash drive as a storage media for the necessary files. Follow the steps below: i. Create a DOS bootable USB flash drive. ii. Create CHELSIO folder on the USB flash drive. iii. If you haven‟t done already, download ChelsioUwire-x.x.x.x.tar.
Chapter XVIII. Unified Boot xi. Delete any previous version of Option ROM flashed on the HBA: C:\CHELSIO>cfut4 –d -xb –xc Here, idx is the HBA index found in step ix (0 in this case) xii. Delete any previous firmware and configuration file using the following command: C:\CHELSIO>cfut4 -d -xh –xf Here, idx is the HBA index found in step ix (0 in this case) xiii. Run the following command to flash the configuration file (t4-config.txt).
Chapter XVIII. Unified Boot xiv. Run the following command to flash the firmware (t4fw-x.x.xx.x.bin). C:\CHELSIO>cfut4 -d -uf Here, idx is the HBA index found in step ix (0 in this case) and t4_firmware is the T4 firmware present in the CHELSIO folder. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xv. Flash the unified option ROM onto the Chelsio HBA, using the following command: C:\CHELSIO>cfut4 -d -ub cubt4.bin Here, idx is the HBA index found in step ix (0 in this case) and cubt4.bin is the name of the unified option ROM image file. xvi. Reboot the system for changes to take effect. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot 3. Configuring PXE Server The following components are required to configure a server as PXE Server: DHCP Server TFTP Server PXE server configuration steps for Linux can be found on following links: http://linux-sxs.org/internet_serving/pxeboot.html http://www.howtoforge.com/ubuntu_pxe_install_server PXE server configuration steps for Windows can be found on following links: http://technet.microsoft.com/en-us/library/cc771670%28WS.10%29.aspx http://tftpd32.
Chapter XVIII. Unified Boot 4. PXE boot process Before proceeding, please ensure that the Chelsio HBA has been flashed with the provided configuration file, firmware and option ROM by following the procedure mentioned in section 3 (Click here). 4.1. For Legacy PXE boot i. After configuring the PXE server, make sure the PXE server works. Then reboot the client machine. ii.
Chapter XVIII. Unified Boot v. Enable the Adapter BIOS using arrow keys if not already enabled. Hit [ENTER]. Note Use the default values for Boot Mode, EDD and EBDA Relocation parameters, unless instructed otherwise. vi. Choose PXE from the list to configure. Hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot vii. Use the arrow keys to highlight the appropriate function among the 4 supported NIC functions and hit [Enter] to select. NIC and FCoE is the only function type that is currently supported in Legacy environment. viii. Enable NIC function bios if not already enabled. ix. Choose the boot port to try the PXE boot. Each function must be configured to boot from same port number as the function number i.e.
Chapter XVIII. Unified Boot PXE configuration for 4 port Chelsio T4 Adapter NIC Functions Ports to be configured NIC Func00 00 NONE NONE NONE NIC Func01 01 NONE NONE NONE NIC Func02 02 NONE NONE NONE NIC Func03 03 NONE NONE NONE PXE configuration for 2 port Chelsio T4 Adapter Note NIC Functions Ports to be configured NIC Func00 00 NONE NONE NONE NIC Func01 01 NONE NONE NONE NIC Func02 NONE NONE NONE NONE NIC Func03 NONE NONE NONE NONE VLAN id is not currently supported.
Chapter XVIII. Unified Boot x. Hit [Esc] and then [Y] to save configuration changes. xi. Reboot the system. xii. Hit [F2] or [DEL] or any other key as mentioned during system startup to enter the system setup. xiii. Allow the Chelsio T4 option ROM to initialize and setup PXE devices. DO NOT PRESS ALT-S to skip Chelsio T4 option ROM. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xiv. In the system setup, choose any of the Chelsio T4 PXE devices as the first boot device. xv. Reboot. DO NOT PRESS ALT-S to skip Chelsio T4 option ROM, during POST. xvi. Hit [F12] key when prompted to start PXE boot. 4.2. For uEFI PXE Boot This section describes the method to configure and use Chelsio T4 uEFI PXE interfaces. i. After configuring the PXE server, make sure the PXE server works. Then reboot the client machine. Copyright ©2013.Chelsio Communications.
Chapter XVIII. Unified Boot ii. Configure the system having T4 CNA to boot in UEFI mode. iii. Boot to EFI Shell. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot iv. Issue command drivers to note down the handle# of Chelsio driver. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot v. Then issue command drvcfg -s to invoke the configuration utility. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot vi. In the configuration utility, choose the HBA on which the option ROM image was flashed and Hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot vii. Choose config utility if you want to configure T4 adapter for PXE boot, else choose flash utility if you want to flash/erase firmware, option ROM image or configuration parameters. viii. If config utility is chosen, go to step (ix), else goto step (xviii). Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot ix. Enable the Adapter BIOS if not already enabled. Hit [Enter]. x. Choose PXE as the function to use and configure. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xi. Choose the appropriate function among the 4 NIC functions available. xii. Enable function Bios if not already enabled. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot Note VLAN is currently not supported. xiii. Choose the boot port to try the PXE boot. It is recommended to only enable functions and ports which are going to be used. Please note that enabling NIC Func 00 will enable port 0 for PXE, enabling NIC Func 01 will enable port 1 and so on for NIC function. Please refer the table below: NIC Function enabled Ports enabled NIC Func00 00 NIC Func01 01 NIC Func02 02 NIC Func03 03 xiv.
Chapter XVIII. Unified Boot xvii. To confirm if uEFI Chelsio PXE has loaded successfully, please boot to uEFI shell and issue the command drivers. Now, note that the type field should be B as shown below. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xviii.If flash utility is chosen, choose any of available options and Hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xix. If you have chosen to update option ROM image, then enter the full path to the file. E.g.:- If the pen drive is device fs0 and the option ROM image file is in fs0:\CHELSIO\cubt4.bin, then enter CHELSIO\cubt4.bin. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xx. Restart the system for the changes to take place. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot 5. FCoE boot process Before proceeding, please ensure that the Chelsio HBA has been flashed with the provided configuration file, firmware and option ROM by following the procedure mentioned in section 3 (Click here). Note NIC and FCoE is the only function type that is currently supported in Legacy environment. 5.1. For Legacy FCoE boot i. Reboot the system. ii.
Chapter XVIII. Unified Boot v. Enable the Adapter BIOS if not already enabled. Hit [ENTER]. Note Use the default values for Boot Mode, EDD and EBDA Relocation parameters, unless instructed otherwise. vi. Choose FCoE from the list to configure and hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot vii. Choose the first option (function parameters) from the list of parameter type and hit [Enter]. viii. Enable FCoE BIOS if not already enabled. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot ix. Choose the order of the ports to discover FCoE targets. x. Set discovery timeout to a suitable value. Recommended value is >= 30. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xi. Hit [Esc] and then [Y] to save the configuration. xii. Go back and choose boot parameters to configure. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xiii. Select the first boot device and hit [Enter] to discover FC/FCoE targets connected to the switch. Wait till all targets reachable are discovered. xiv. List of discovered targets will be displayed. Highlight a target using the arrow keys and hit [Enter] to select. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xv. From the list of LUNs displayed for the selected target, choose one on which operating system has to be installed. Hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xvi. Hit [Esc] and then [Y] to save the configuration. xvii. Reboot the machine. xviii. During POST, allow the Chelsio T4 option ROM to discover FCoE targets. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xix. Enter BIOS setup and choose FCoE disk discovered via Chelsio T4 adapter as the first boot device. xx. Reboot and install the required OS to the FCoE disk. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot 5.2. For uEFI FCoE Boot This section describes the method to perform FCoE boot on uEFI platforms. i. Configure the system having T4 CNA to boot in uEFI mode. ii. Boot to EFI Shell. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot iii. Issue command drivers to note down the handle# of Chelsio driver. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot iv. Then issue command drvcfg -s to invoke the configuration utility. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot v. In the configuration utility, choose the HBA on which the option ROM image was flashed and Hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot vi. Choose config utility if you want to configure T4 adapter for PXE boot, else choose flash utility if you want to flash/erase firmware, option ROM image or configuration parameters. vii. If config utility is chosen, go to step (viii), else goto step (xxii). Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot viii. Enable the Adapter BIOS if not already enabled. Hit [Enter]. ix. Choose FCoE from the list to configure and hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot x. Choose the first option (function parameters) from the list of parameter type and hit [Enter]. xi. Enable FCoE BIOS if not already enabled. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xii. Choose the order of the ports to discover FCoE targets. xiii. Set discovery timeout to a suitable value. Recommended value is >= 30. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xiv. Hit [Esc] and then [Y] to save the configuration. xv. Go back and choose boot parameters to configure. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xvi. Select the first boot device and hit [Enter] to discover FC/FCoE targets connected to the switch. Wait till all targets reachable are discovered. xvii. List of discovered targets will be displayed. Highlight a target using the arrow keys and hit [Enter] to select. xviii.From the list of LUNs displayed for the selected target, choose one on which operating system has to be installed. Hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xix. Hit [Esc] and then [Y] to save the configuration. xx. Reboot the machine. xxi. Enter uEFI BIOS Manager, set DVD and discovered LUN as first and second boot device respectively. Save changes and start the OS installation. xxii. If flash utility is chosen, choose any of available options and Hit [Enter]. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xxiii.If you have chosen to update option ROM image, then enter the full path to the file. E.g.:- If the pen drive is device fs0 and the option ROM image file is in fs0:\CHELSIO\cubt4.bin, then enter CHELSIO\cubt4.bin. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot xxiv. Restart the system for the changes to take place. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot 6. Creating Driver Update Disk (DUD) 6.1. Creating DUD for RedHat Enterprise Linux i. If you haven‟t done already, download ChelsioUwire-x.x.x.x.tar.gz from Chelsio Download Center, service.chelsio.com ii. Untar the package, [root@host]# tar -zxvfm ChelsioUwire-x.x.x.x.tar.gz iii. Navigate to the LinuxDUD directory, [root@host]# cd ChelsioUwire-x.x.x.x/Uboot/LinuxDUD iv. Insert a blank, formatted USB flash drive. v.
Chapter XVIII. Unified Boot iv. Format the USB drive [root@host]# mkfs.vfat /dev/sda v. Depending on the distribution to be installed, copy the corresponding image file to the USB stick. For example, execute the following command for SLES11sp2. [root@host]# dd if=/root/ ChelsioUwire-x.x.x.x/Uboot/LinuxDUD/ChelsioDriverUpdateDisk-SLES11sp2-x86_64-x.x.x.x-y.img of=/dev/sda Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot 7. OS Installation 7.1. Installation using Chelsio NIC DUD (PXE only) This is the recommended method for installing Linux OS using Chelsio T4 PXE boot. The Chelsio Driver Update Disk (DUD) has support for all the new adapters. Use Network Boot (PXE Boot) media to install the OS, and provide the Driver Update Disk as per the detailed instructions for each OS. The DUD supports installation of RHEL and SLES distributions using Chelsio adapters over Network.
Chapter XVIII. Unified Boot 7.1.1. RHEL 6.X installation using Chelsio DUD i. PXE boot prompt Please make sure that the USB drive with DUD image is inserted. Type dd at the boot prompt for the installation media. The dd option specifies that you will be providing a Driver Update Disk during the installation. ii. Driver disk prompt: The installer will load and prompt you for the driver update disk. Now select “Yes” and hit [Enter] to proceed. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot iii. Driver disk source prompt: You will be asked to select the Driver Update Disk device from a list. USB drives usually show up as SCSI disks in Linux. So if there are no other SCSI disks connected to the system, the USB drive would assume the first drive letter “a”. Hence the drive name would be “sda”. You can view the messages from the Linux kernel and drivers to determine the name of the USB drive, by pressing [Alt] + [F3/F4] and [Alt] + [F1] to get back to the list. iv.
Chapter XVIII. Unified Boot vi. Choose language and Keyboard type: Select the required language from the list. vii. Select Keyboard type Select the type of keyboard you have from the list. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot viii. Select Installation method: In this step, you can choose the source which contains the OS installation ISO image. In this case, select “NFS directory”. ix. Select Displayed Network Devices: The Chelsio Network Devices will be displayed. Select the appropriate Chelsio NIC interface to proceed with installation. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot x. Configure TCP/IP settings: Here you can specify if you want to configure your network interfaces using DHCP or manually using IPv4. IPv6 is currently not supported. Hence disable IPv6 before proceeding. xi. Provide NFS/FTP/HTTP Server Name/IP and Path: Proceeding with the installation will get NFS/FTP/HTTP setup page. Here, provide NFS server details to proceed with the installation. Then the graphical Installation screens for RHEL will appear.
Chapter XVIII. Unified Boot 7.1.2. SLES installation using Chelsio DUD i. PXE boot prompt: Please make sure that the USB drive with DUD image is inserted. Type dd at the boot prompt for the installation media. The dd option specifies that you will be providing a Driver Update Disk during the installation. The DUD gets detected and the driver is automatically loaded by installer. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot ii. Select network as method of install: Select “Network” as the source of medium to install the SLES Operating System. iii. Select the Network protocol: Select the desired Network protocol from the list presented. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot iv. Select appropriate Chelsio network Interface: Select the appropriate Chelsio interface from the list to proceed with installation. You can view the messages from the Linux kernel and drivers to determine the name of NIC interface by pressing [Alt] + [F3/F4] and [Alt] + [F1] to get back to the list. Then follow the instructions provided by the installer. When the graphical Installation screen for SLES appears, proceed with the installation as usual. v.
Chapter XVIII. Unified Boot vi. Provide operating system Directory Path: Provide a valid directory path to the operating system to be installed. 7.2. Installation on FCoE LUN 7.2.1. Using CD/DVD ROM Please make sure that the USB drive with DUD image is inserted. Also, change the boot priority to boot from CD/DVD in the BIOS setup. i. Insert the OS installation disc into your CD/DVD ROM. ii. On the Grub menu, choose Install or upgrade an existing system option if not already selected. iii.
Chapter XVIII. Unified Boot iv. Driver disk prompt: The installer will load and prompt you for the driver update disk. Now select “Yes” and hit [Enter] to proceed. v. Driver disk source prompt: You will be asked to select the Driver Update Disk device from a list. USB drives usually show up as SCSI disks in Linux. So if there are no other SCSI disks connected to the system, the USB drive would assume the first drive letter “a”. Hence the drive name would be “sda”. Copyright ©2013.Chelsio Communications.
Chapter XVIII. Unified Boot You can view the messages from the Linux kernel and drivers to determine the name of the USB drive, by pressing [Alt] + [F3/F4] and [Alt] + [F1] to get back to the list. vi. Select the Driver Update File: Select the Appropriate image file and Choose “OK”. Now the installer will search for the appropriate drivers from the driver disk and load them. This step may take some time. Check on the [Alt] + [F3] or [Alt] + [F4] screens for log messages. vii.
Chapter XVIII. Unified Boot viii. Testing Installation media If you want to test the installation media, choose OK or else Skip. Hit [Enter]. ix. Graphical Installer Red Hat graphical installer will now start. Click Next. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XVIII. Unified Boot x. Select storage device Select the FC/FCoE LUN which was saved as boot device in system BIOS and click Next. Then proceed with the installation as usual. Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XIX. Appendix XIX. Appendix A Copyright ©2013.Chelsio Communications. All Rights Reserved.
Chapter XIX. Appendix 1. Troubleshooting Cannot bring up T4 interface Make sure you have created the corresponding network-script configuration file as stated in Section 5.2 of Cheslsio Unified Wire chapter (Click here). If the file does exist, make sure the structure and contents are correct. A sample is given in Section 5.1 of Chelsio Unified Wire chapter (Click here). Another reason may be that the IP address mentioned in the configuration file is already in use on the network.
Chapter XIX. Appendix priority-flow-control mode on the switch On the switch, make sure priority-flow-control mode is always set to auto and flow control is disabled. Configuring Ethernet interfaces on Cisco switch Always configure Ethernet interfaces on Cisco switch in trunk mode. Binding VFC to MAC If you are binding the VFC to MAC address in case of Cisco Nexus switch, then make sure you make the Ethernet interface part of both Ethernet VLAN and FCoE VLAN.
Chapter XIX. Appendix 2. Chelsio End-User License Agreement (EULA) Installation and use of the driver/software implies acceptance of the terms in the Chelsio EndUser License Agreement (EULA). IMPORTANT: PLEASE READ THIS SOFTWARE LICENSE CAREFULLY BEFORE DOWNLOADING OR OTHERWISE USING THE SOFTWARE OR ANY ASSOCIATED DOCUMENTATION OR OTHER MATERIALS (COLLECTIVELY, THE "SOFTWARE"). BY CLICKING ON THE "OK" OR "ACCEPT" BUTTON YOU AGREE TO BE BOUND BY THE TERMS OF THIS AGREEMENT.
Chapter XIX. Appendix (including the related documentation), together with all copies or modifications in any form. 6. Limited Warranty. Chelsio warrants only that the media upon which the Software is furnished will be free from defects in material or workmanship under normal use and service for a period of thirty (30) days from the date of delivery to you. CHELSIO DOES NOT AND CANNOT WARRANT THE PERFORMANCE OR RESULTS YOU MAY OBTAIN BY USING THE SOFTWARE OR ANY PART THEREOF.
Chapter XIX. Appendix Federal Acquisition Regulations and its successors and 49 C.F.R. 227.7202-1 of the DoD FAR Supplement and its successors. 12. General. You acknowledge that you have read this Agreement, understand it, and that by using the Software you agree to be bound by its terms and conditions.
