Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1

ManualsBrandsHP ManualsServerHP ProLiant DL580 G7 Server

Technical white paper

Implementing SR-IOV on

HP ProLiant Servers with

VMware vSphere 5.1

Table of contents

Introduction ............................................................................................................................................................................ 2

SR-IOV Overview ................................................................................................................................................................... 2

vSphere 5.1 recommended system configurations ......................................................................................................... 4

SR-IOV-compliant ProLiant servers .................................................................................................................................... 5

SR-IOV-compliant network adapters ................................................................................................................................. 6

Supported Guest Operating System Versions ................................................................................................................... 6

SR-IOV limitations ................................................................................................................................................................. 6

Enabling the SR-IOV feature ................................................................................................................................................ 8

Prepare ............................................................................................................................................................................... 9

Install and enable .............................................................................................................................................................. 9

Configure ......................................................................................................................................................................... 10

Deploying across multiple systems ............................................................................................................................. 14

Troubleshooting ................................................................................................................................................................. 15

Known issues and workarounds ...................................................................................................................................... 17

Resources, contacts, or additional links .......................................................................................................................... 19

Click here to verify the latest version of this document

Summary of content (19 pages)

PAGE 1
Technical white paper Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 Table of contents Introduction............................................................................................................................................................................2 SR-IOV Overview ...................................................................................................................................................................2 vSphere 5.
PAGE 2
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 Introduction HP and VMware enable Single Root I/O Virtualization (SR-IOV) for VMware vSphere 5.1 on HP ProLiant Servers. SR-IOV enables a single PCI Express (PCIe) network adapter to appear to the VMware vSphere (ESXi) hypervisor as multiple special-purpose network adapters. These special-purpose network adapters, termed Virtual Functions (VF), are only available for direct presentation to virtual machines (VMs).
PAGE 3
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere Figure 1 shows the PF and VF concepts on a 2-port, SR-IOV-capable network adapter. Figure 1. SR-IOV PF and VF Conceptual Overview When planning an SR-IOV implementation in vSphere 5.1, take into account the following key requirements, which are addressed in more detail throughout the remainder of this paper. • Each VF may utilize the entire potential bandwidth of a PF.
PAGE 4
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 – Available Interrupt Vectors on host VMware provides 256 interrupt vectors to each ESXi host. Of those 256 interrupt vectors, 128 are available for use for SR-IOV. As VMs power on and the Guest OS VF driver starts, interrupt vectors are consumed. Depending on the network adapter and VF driver, the quantity consumed per VF differs.
PAGE 5
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere SR-IOV-compliant ProLiant servers Table 2 lists the HP ProLiant servers and minimum System ROM version that are required for SR-IOV functionality. Important SR-IOV requires both hardware and System ROM compliance to work. Your server must have the minimum System ROM version listed in the table to enable SR-IOV. If necessary, update the System ROM before beginning to enable SR-IOV.
PAGE 6
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 SL270s Gen8 P75 12/11/2012 or later SR-IOV-compliant network adapters At this time, you must use one of the following network adapters to implement SR-IOV: • HP Ethernet 10Gb 2-port 560FLB Adapter: A dual port 10Gb Ethernet FlexibleLOM mezzanine adapter based on the Intel 82599 controller. The 560FLB is designed for use with HP ProLiant BladeSystem c-Class Gen8 servers.
PAGE 7
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere • When a network adapter port is selected for SR-IOV use by enabling VFs on the PF, then that port instance becomes hidden. In this hidden state, the virtual machine NIC (vmnic) typically associated with each network adapter port cannot use vSwitches. • There is no method to limit the bandwidth allocation to individual VFs. Each VF has access to the entire potential bandwidth of the physical link.
PAGE 8
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 Enabling the SR-IOV feature Figure 2 provides an overview of the process for enabling the SR-IOV feature. These steps are for a single server and a single VM deployment, additional recommendations are provided for deploying at scale. These steps do not cover an Auto Deploy environment or customer created ESXi ISO images.
PAGE 9
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere Prepare To prepare for enabling the SR-IOV feature, make sure that your server and components are supported with vSphere 5.1 and SR-IOV: • Ensure that the server and network adapter support SR-IOV; refer to the sections “SR-IOV-compliant ProLiant servers” and “SR-IOV-compliant network adapters.” • Ensure that the server has a DVD drive (either installed or attached to the server receiving the installation).
PAGE 10
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 Figure 3. Use the RBSU to enable SR-IOV 5. Install ESXi 5.1 using the HP Customized ESXi Image for vSphere 5.1 No specific settings are requried during installation to provide SR-IOV support. Configure After installing ESXi on the host, enumerate the VFs and configure the VM: 1. 2. Determine the network adapters to use for SR-IOV.
PAGE 11
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere Note The example esxcli commands are applicable for SSH or VMware ESXi Shell. vSphere CLI execution requires additional connection parameters such as the target VMware ESXi host. Note There is currently an issue with setting the VF enumeration value to zero for the first PF. In this scenario, none of the PFs will enumerate their VFs. See the section “Known issues and workarounds” for additional details.
PAGE 12
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 5. Determine the PF to VF mappings. If more than one PF has been configured to enumerate VFs, we recommend that you determine the vmnic and VF associations. This is especially true when a Guest OS uses teaming or bonding to present multiple VFs to a VM for redundancy . To resolve the PF to VF mappings, execute the following esxcli commands from the vSphere CLI, SSH, or ESXi Shell. See Figure 5 for example output. A.
PAGE 13
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 6. Create a new VM, and then add a PCI device to it. Each PCI Device for the VM will be associated with a unique VF. If additional VFs are desired on a VM, then add PCI Devices as required, see Figure 6. Figure 6: Create PCI Device for VM After selecting add a PCI device, the available VFs will be displayed for selection, see Figure 7. Figure 7: Select VF to Attach 7. 8.
PAGE 14
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 9. If unique virtual LAN (VLAN) settings are required on a per-VF basis, then the VMs configuration (.vmx) file must be modified manually. Make sure the VM is shutdown prior to editing the file. To edit the vmx file, connect to the host using SSH or directly access the host using the ESXi shell. If you are unsure of the vmx file location, check the VM properties using vSphere Client (see Figure 8).
PAGE 15
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere • Utilize HP Scripting Toolkit – BIOS settings, storage controller configuration, and iLO configuration can be automated. Although the HP Scripting toolkit is listed as providing support for Linux and Windows, the toolkit can be used for applying configuration settings for a server prior to installation of ESXi. For additional details refer to hp.com/go/ProLiant/STK.
PAGE 16
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 Figure 11: VF configured and device driver loaded B. If the VF is listed under “Other devices” as shown in Figure 12, then the VF device driver has not been installed. Attach the SPP ISO image to the VM and install the device driver using HP SUM.
PAGE 17
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere Known issues and workarounds This section outlines potential issues that you may encounter with vSphere 5.1 SR-IOV enablement and ProLiant servers (Table 3). We are actively working to resolve all issues. Table 3. Known issues with vSphere 5.1 SR-IOV and ProLiant servers.
PAGE 18
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere 5.1 VM powers off automatically after starting up Description A VM configured with a VF shuts down shortly after powering on. This behavior is observed when all available interrupt vectors on the host have been consumed. No additional VMs with VFs attached will be able to startup successfully. The following entry in the vmkernel.log will be logged when this occurs.
PAGE 19
Technical white paper | Implementing SR-IOV on HP ProLiant Servers with VMware vSphere Resources, contacts, or additional links HP VMware Home Page hp.com/go/vmware HP Custom ESXi Images hp.com/go/esxidownload HP ProLiant Server and Option Support Recipe for VMware vibsdepot.hp.com HP Service Pack for ProLiant hp.com/go/spp HP Ethernet 10Gb 2-port 560FLB Adapter hp.com/products/servers/networking/560FLB/index.html HP Ethernet 10Gb 2-port 560SFP+ Adapter hp.com/products/servers/networking/560SFP/index.