HP IO Accelerator version 3.1.1 VMware ESX and ESXi User Guide Abstract This document describes software requirements for all relevant HP IO Accelerators using VMware ESX or ESXi operating systems. This document is intended for system administrators who plan to install and use HP IO Accelerators with a VMware ESX or ESXi operating system. It is helpful to have previous experience with HP IO Accelerators and a VMware ESX or ESXi operating system.
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Confidential computer software.
Contents About this guide ........................................................................................................................... 5 Contents summary ..................................................................................................................................... 5 Introduction .................................................................................................................................. 6 Overview ...................................................
Introduction to performance and tuning ...................................................................................................... 28 Disabling DVFS ....................................................................................................................................... 28 Limiting APCI C-states .............................................................................................................................. 28 Utilities .................................................
About this guide Contents summary • Instructions on downloading and installing the approved driver and utilities • Instructions on maintaining the IO Accelerator • Description of the following IO Accelerator models: o HP IO Accelerator for BladeSystem c-Class o HP PCIe IO Accelerator o HP PCIe IO Accelerator Duo CAUTION: Before upgrading to 3.x.x software and firmware, back up all data on the IO Accelerator. The 3.1.
Introduction Overview Designed around ioMemory, a revolutionary storage architecture, HP IO Accelerator is an advanced NAND flash storage device. With performance comparable to DRAM and storage capacity on par with hard disks, the IO Accelerator increases performance so that every server can contain internal storage that exceeds the I/O performance of an enterprise SAN. HP IO Accelerator is the first data accelerator designed specifically to improve the bandwidth for I/O-bound applications.
Performance attributes IO Accelerator capacity Models AJ878B and BK836A 320GB 640GB NAND type MLC (Multi Level Cell) MLC (Multi Level Cell) Read Bandwidth (64kB) 735 MB/s 750 MB/s Write Bandwidth (64kB) 510 MB/s 550 MB/s Read IOPS (512 Byte) 100,000 93,000 Write IOPS (512 Byte) 141,000 145,000 Mixed IOPS* (75/25 r/w) 67,000 74,000 Access Latency (512 Byte) 30 µs 30 µs Bus Interface PCI-Express x4 PCI-Express Gen1 x4 IO Accelerator capacity Models QK762A and QK763A 785GB 1.
CAUTION: Version 3.1.1 of the driver software is not backward-compatible. When you install version 3.1.1, you cannot revert to any previous version. IMPORTANT: All operating systems must be 64-bit architecture. NOTE: IO Accelerators cannot be used as hibernation devices. The following scenarios are supported: • Using the IO Accelerator as a VMFS datastore within the hypervisor, and then sharing that storage with guest operating systems.
BladeSystem c-Class IO Accelerators have two distinct designs for the respective server product lines. The G1 through G7 IO Accelerator adapter is provided in a c-Class Type 1 Mezzanine card form factor. It can be installed in both Type 1 and Type 2 mezzanine slots within the c-Class blade G1 through G7 servers, enabling a total of two cards in a half-height server blade, and three cards in a full-height server blade and up to 6 in a double-high, double-wide server (BL680c).
• Sufficient RAM to operate—The amount of RAM that the driver requires to manage the NAND flash varies according to the block size you select when formatting the device (filesystem format, not low-level format). For a virtual machine using an IO Accelerator directly (using PCI pass-through), consult the user guide for the installed operating system. The following table lists the amount of RAM required per 100GB of storage space, using various block sizes. The amount of RAM used in driver version 3.
IO Accelerators cannot be installed as part of an ESX or ESXi installation.
Software installation VMDirectPathIO The VSL software is only required for ESX or ESXi if you plan to use the IO Accelerator device as a VMFS datastore. If you are passing the devices through using VMDirectPathIO, you do not need to install the VSL on your the ESX or ESXi system. Instead, install the VSL on the guest system. For example, pass the device through to a Windows VM, and then install the Windows VSL on that VM.
Installation overview Before installing the IO Accelerator driver, make sure you have properly installed the IO Accelerator devices. For more information, see the HP IO Accelerator Hardware Installation Guide. 1. If necessary, uninstall previous versions of the VSL and utilities. For instructions, see "Common maintenance tasks (on page 25)." 2. Install the latest version of the VSL and command-line utilities. 3. To load the driver and attach the IO Accelerator devices, reboot the ESX or ESXi system.
Transferring the VSL files to the ESX or ESXi server You must transfer the firmware file to the ESX or ESXi host. Depending on your ESX or ESXi version and your preferred installation method, you might also have to transfer the two bundle installation files to the host. HP recommends transferring all the files at this point and then choosing the installation method later. With any method that you select for transferring the files, HP recommends saving the files to a datastore on the host.
Installing the VSL on ESXi 5.0 These instructions describe how to install the VSL on a single hypervisor. However, if you are familiar with the VUM plugin for the Virtual Center Server (vCenter Server), you can use that method to install the VSL on multiple hosts. For more details on VUM, see the vCenter Server documentation. IMPORTANT: HP does not recommend performing an upgrade installation. Instead, uninstall the previous version of the VSL software before you install this version.
Installing the VSL on ESX or ESXi 4.x using vCLI IMPORTANT: Do not save the installation bundles to the ESX or ESXi 4.x host when using the vCLI. Store them on the remote system. 1. On your remote system, navigate to the directory that contains the downloaded files. 2. To install the bundle, run the following command against your ESXi or ESXi 4.x system using the vCLI: vihostupdate --server --install --bundle ./*offline-bundle.
NOTE: When using PCI pass-through, if you upgrade the firmware on an IO Accelerator, you must cycle the power for the change to take place. Restarting the virtual machine does not apply the change. Upgrading device firmware from VSL 1.x.x or 2.x.x to 3.x.x CAUTION: You cannot downgrade an HP IO Accelerator device firmware to an earlier version after you have upgraded the device. CAUTION: Upgrading IO Accelerator devices that were previously configured for VSL 1.x.x or 2.x.x to work with VSL 3.x.
CAUTION: Upgrading IO Accelerator devices that were previously configured for VSL 1.x.x or 2.x.x to work with VSL 3.x.x requires a low-level media format of the device. No user data is maintained during the media format process. Be sure to backup all data on your IO Accelerator device as instructed before upgrading the firmware. Do not back up the data onto another IO Accelerator device on the same system. The backup must be to a local disk or to an externally attached volume. b.
c. 4. Reboot the system. Update the firmware on each device to the latest version using the fio-update-iodrive TSM or COS command-line utility. CAUTION: Do not turn off the power during a firmware upgrade, because this might cause device failure. If a UPS is not in place, consider adding one to the system before performing a firmware upgrade. Sample syntax: fio-update-iodrive Where is the path to the firmware archive.
CAUTION: ESX and ESXi require 512B sector sizes. New IO Accelerator devices come pre-formatted with 512B sector sizes from the factory. If yours is a new device, there is no need to format it. However, if your IO Accelerator device was previously used in a system that allowed for larger sector sizes (such as Linux and 4KB sectors), then you must perform a format using the fio-format utility. To prevent data loss, follow formatting instructions carefully, including disabling and re-enabling auto attach.
An ESXi host has the following: • Memory capacity of 36,852 MB • Total IO Accelerator device datastore capacity of 320 GB (or approximately 320,000 MB). 320,000 MB device capacity * 0.5% of device capacity approximately 1600 MB of RAM equivalent. 36,852 MB total memory capacity - 1600 MB free = 35,252 MB of memory limited to the host. The new value under Limit in Memory Resources is 35,252 MB.
Maintenance Maintenance tools The IO Accelerator includes both software utilities to maintain the device and external LED indicators to display the status. You can also install SNMP as a monitoring option. The following are the most common tasks for maintaining your IO Accelerator. You can also use the IO Accelerator Management Tool application to perform firmware upgrades. For more information, see the HP IO Accelerator Management Tool User Guide.
Command line utility Purpose fio-format (on page Used to perform a low-level format of an IO Accelerator 32) fio-pci-check (ESX Checks for errors on the PCI bus tree, specifically for PCIe IO Accelerator devices. This utility is only supported on only) ESX. fio-status (on page 34) Displays information about the device fio-update-iodrive (on page 36) Updates the IO Accelerator firmware For more information on command-line utilities, see "Utilities (on page 29).
Important considerations • If you are installing more than one dual IO Accelerator device and enabling the override parameter for each device, be sure the motherboard is rated to provide 55W power to each slot used. For example, some motherboards safely provide up to 75W to any one slot, but run into power constraints when multiple slots are used to provide that much power. Installing multiple devices in this situation might result in server hardware damage.
1149D0969,1159E0972,24589 NOTE: The remote option (--server) is only required for the vCLI. 3. Reboot the system to enable the parameter. Common maintenance tasks In ESX, these tasks require the COS. In ESXi, some of these maintenance tasks are only accessible through VMware TSM, also known as SSH. IMPORTANT: VMware suggests that TSM only be used for the purposes of troubleshooting and remediation.
Disabling auto attach When the IO Accelerator driver is installed, it is configured to automatically attach any devices when the driver is loaded. When necessary, disable the auto attach feature. To disable auto attach using ESX: 1. To set the auto_attach parameter to 0, run the following command: esxcfg-module -s 'auto_attach=0' iomemory-vsl 2. Restart the system. To disable auto attach using ESXi: 1.
vihostupdate --server --query Sample output: ---------Bulletin ID--------iomemory-vsl-2.2.0.7601742 -----Installed----2011-02-08T10:37:05 ----------------Summary----------------- iomemory-vsl: block driver for ESXi 4.1.X 2. To remove the bulletin containing the driver and utilities, run the following command: vihostupdate --server --remove --bulletin iomemory-vsl-2.2.0.7601742 Uninstalling the software in ESXi 5.
Performance and tuning Introduction to performance and tuning HP IO Accelerator devices provide high bandwidth and high IOPS and are specifically designed to achieve low latency. As IO Accelerator devices improve in IOPS and low latency, the device performance may be limited by operating system settings and BIOS configuration. To take advantage of the revolutionary performance of IO Accelerator devices, you might have to tune these settings.
Utilities Utilities reference These command-line utilities are only accessible through VMware TSM, also known as SSH, on ESXi and the COS on ESX. IMPORTANT: VMware suggests that TSM only be used for the purposes of troubleshooting and remediation. VMware recommends using the vSphere Client or any other VMware Administration Automation Product to perform routine ESXi host configuration tasks that do not involve a troubleshooting scenario.
NOTE: If multiple IO Accelerator devices are installed in a computer, they attach in parallel. Syntax fio-attach [options] Where is the name of the device node (/dev/fctx), where x indicates the card number: 0, 1, 2, and so on. For example, /dev/fct0 indicates the first IO Accelerator installed on the system. Options Option Description -c Attach only if clean. -q Quiet: disables the display of the progress bar and percentage.
NOTE: If the utility recommends that you contact Fusion-io support, disregard that message and contact HP support (http://www.hp.com/support) instead. Syntax fio-bugreport Sample output ~ # fio-bugreport VMkernel-4.1.0 Report output: /var/tmp/fio-bugreport-20111006.223733-sc07HE.tar.gz OS: VMware-ESX-4.1.0 Collecting esxupdate query... done Collecting dc-status -v... done Collecting dc-status -fk... done Collecting fio-status -a... done Collecting fio-status -fj...
For example, the filename for a bug report file named fio-bugreport-20111006.173256-sc07HE.tar.gz indicates the following: • Date (20111006) • Time (173256, or 17:32:56) • Misc. information (sc07HE.tar.gz) fio-detach CAUTION: Before using this utility, be sure that the IO Accelerator to be detached is not currently mounted or otherwise in use (for example, as a datastore). The driver cannot determine whether the drive is in use at the time of detaching.
Syntax fio-format [options] where is the name of the device node (/dev/fctx), where x indicates the card number: 0, 1, 2, and so on. For example, /dev/fct0 indicates the first IO Accelerator installed on the system. Options NOTE: ESX only supports 512 byte sector sizes for use in VMDKs. Do not format your IO Accelerator device with any other sector size if you plan to use VMDKs.
fio-pci-check [options] Options Description -d 1 = Disable the link; 0 = bring the link up (HP does not recommend this parameter) -f Scan every device in the system. -i Print the device serial number. This option is invalid when the VSL is loaded. -r Force the link to retrain. -v Verbose: Print extra data about the hardware. fio-status Description The fio-status command provides detailed information about the specified or all installed IO Accelerators.
IMPORTANT: Starting with version 3.0.0 and later, the standard formatting of the fio-status output changed. The changing of the format affects any custom management tools that used the output of this utility.
• o Date of manufacture o Power loss protection status o PCIe bus voltage (average, minimum, maximum) o PCIe bus current (average, maximum) o PCIe bus power (average, maximum o PCIe power limit threshold (watts) o PCIe slot available power (watts) o PCIe negotiated link information (lanes and throughput) Block device information o Manufacturer code o Manufacturing date o Vendor and sub-vendor information o Format status and sector information (if device is attached) o FPGA ID and L
To ensure a clean firmware update, you must detach the device from the system using the fio-detach utility. For more information, see "fio-detach (on page 32)". To update one or more specific drives (rather than all drives): • Make sure the IO Accelerator driver is loaded. • Use the -d option with the device number. CAUTION: Do not turn off the power during a firmware upgrade, because this might cause device failure.
CAUTION: Use the -f option with care because it might damage your board. CAUTION: Use the {{-d}} option with care, because updating the wrong device might damage the device.
Monitoring IO Accelerator health NAND flash and component failure The IO Accelerator is a highly fault-tolerant storage subsystem that provides many levels of protection against component failure and the loss nature of solid state storage. However, as in all storage subsystems, component failures might occur. When a large enough number of data blocks is retired due to error, the NAND flash media is considered worn out.
Lifetime data volumes: Physical bytes written: 6,423,563,326,064 Physical bytes read : 5,509,006,756,312 Flashback substitution events The IO Accelerator is equipped with Flashback substitution capability. This capability allows for predictive NAND flash failures to occur and keep the IO Accelerator functioning by substituting out bad locations or even a whole NAND chip. The output of the fio-status –a command will show if any flashback substitution events have occurred.
Module parameters Using module parameters The following table describes the module parameters you can set using the esxcfg-module command. The remote option (--server) is only required for the vCLI. Sample command: esxcfg-module --server iomemory-vsl -s '=' You must reboot the ESX or ESXi system to enforce any parameter changes. Module parameter Default (min/max) Description auto_attach 1 1 = Always attach the device on startup. 0 = Do not attach the device on startup.
VMDirectPathIO Working with IO Accelerators and VMDirectPathIO Each IO Accelerator device can either be used as a VMFS datastore in ESX or ESXi, or it can be passed through directly to a VM. In VMware documentation, passing through to a VM this is referred to as VMDirectPathIO. If you are passing the devices through, you do not have to install the VSL on the ESX or ESXi system. Instead, install the software on the guest system. Install the driver only if you plan to create a VMFS on the devices.
Resources Subscription service HP recommends that you register your product at the Subscriber’s Choice for Business website (http://www.hp.com/support). After registering, you will receive e-mail notification of product enhancements, new driver versions, firmware updates, and other product resources. For more information For additional information, see the following HP websites: • HP BladeSystem technical resources (http://www.hp.
Support and other resources Before you contact HP Be sure to have the following information available before you call HP: • Active Health System log Download and have available an Active Health System log for 3 days before the failure was detected. For more information, see the HP iLO 4 User Guide or HP Intelligent Provisioning User Guide on the HP website (http://www.hp.com/go/ilo/docs).
providers or service partners) identifies that the repair can be accomplished by the use of a CSR part, HP will ship that part directly to you for replacement. There are two categories of CSR parts: • Mandatory—Parts for which customer self repair is mandatory. If you request HP to replace these parts, you will be charged for the travel and labor costs of this service. • Optional—Parts for which customer self repair is optional. These parts are also designed for customer self repair.
Pour plus d'informations sur le programme CSR de HP, contactez votre Mainteneur Agrée local. Pour plus d'informations sur ce programme en Amérique du Nord, consultez le site Web HP (http://www.hp.com/go/selfrepair). Riparazione da parte del cliente Per abbreviare i tempi di riparazione e garantire una maggiore flessibilità nella sostituzione di parti difettose, i prodotti HP sono realizzati con numerosi componenti che possono essere riparati direttamente dal cliente (CSR, Customer Self Repair).
HINWEIS: Einige Teile sind nicht für Customer Self Repair ausgelegt. Um den Garantieanspruch des Kunden zu erfüllen, muss das Teil von einem HP Servicepartner ersetzt werden. Im illustrierten Teilekatalog sind diese Teile mit „No“ bzw. „Nein“ gekennzeichnet. CSR-Teile werden abhängig von der Verfügbarkeit und vom Lieferziel am folgenden Geschäftstag geliefert. Für bestimmte Standorte ist eine Lieferung am selben Tag oder innerhalb von vier Stunden gegen einen Aufpreis verfügbar.
sustituciones que lleve a cabo el cliente, HP se hará cargo de todos los gastos de envío y devolución de componentes y escogerá la empresa de transporte que se utilice para dicho servicio. Para obtener más información acerca del programa de Reparaciones del propio cliente de HP, póngase en contacto con su proveedor de servicios local. Si está interesado en el programa para Norteamérica, visite la página web de HP siguiente (http://www.hp.com/go/selfrepair).
Opcional – Peças cujo reparo feito pelo cliente é opcional. Essas peças também são projetadas para o reparo feito pelo cliente. No entanto, se desejar que a HP as substitua, pode haver ou não a cobrança de taxa adicional, dependendo do tipo de serviço de garantia destinado ao produto. OBSERVAÇÃO: Algumas peças da HP não são projetadas para o reparo feito pelo cliente. A fim de cumprir a garantia do cliente, a HP exige que um técnico autorizado substitua a peça.
Support and other resources 50
Support and other resources 51
Regulatory compliance notices Regulatory compliance identification numbers For the purpose of regulatory compliance certifications and identification, this product has been assigned a unique regulatory model number. The regulatory model number can be found on the product nameplate label, along with all required approval markings and information. When requesting compliance information for this product, always refer to this regulatory model number.
Disposal of waste equipment by users in private households in the European Union This symbol on the product or on its packaging indicates that this product must not be disposed of with your other household waste. Instead, it is your responsibility to dispose of your waste equipment by handing it over to a designated collection point for the recycling of waste electrical and electronic equipment.
Acronyms and abbreviations CIM common information model COS console operating system DA direct attached DVFS dynamic voltage and frequency scaling IOPS input/output operations per second LFM linear feet per minute NAND Not AND PCIe peripheral component interconnect express SSH Secure Shell TSM technical support mode vCLI vSphere Command Line Interface VIB vSphere Installation Bundle Acronyms and abbreviations 54
VM Virtual Machine VMDK virtual machine disk format VMFS virtual machine file system VSL virtual storage layer VUM VMware Update Manager WBEM Web-Based Enterprise Management Acronyms and abbreviations 55
Documentation feedback HP is committed to providing documentation that meets your needs. To help us improve the documentation, send any errors, suggestions, or comments to Documentation Feedback (mailto:docsfeedback@hp.com). Include the document title and part number, version number, or the URL when submitting your feedback.
Index A About this guide 5 authorized reseller 44 B before you begin 10 C command-line installation 12 command-line utilities 22 configuring the device to support VM disks 19 contents summary 5 CSR (customer self repair) 44 customer self repair (CSR) 44 D disabling auto attach 26 disabling DVFS 28 disabling the driver 25 documentation feedback 56 downloading the driver 13 fio-update-iodrive utility 36 flashback substitution events 40 For more information 43 H health metrics 39 health monitoring techniq
P PCIe power override, enabling 23 PCIe power, enabling 19 performance and tuning 28 performance and tuning, introduction 28 performance attributes 7 phone numbers 44 product naming 6 R regulatory compliance identification numbers 52 regulatory compliance notices 52, 53 required information 44 resources 43, 44 S series number 52 support 44 supported firmware revisions 8 supported hardware 8 T technical support 44 telephone numbers 44 transferring VSL files 14 U uninstalling the driver 26 unmanaged shutd
