vSphere Troubleshooting Update 1 ESXi 6.0 vCenter Server 6.0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see http://www.vmware.com/support/pubs.
vSphere Troubleshooting You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com Copyright © 2010–2015 VMware, Inc. All rights reserved. Copyright and trademark information. VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com 2 VMware, Inc.
Contents About vSphere Troubleshooting 5 Updated Information 7 1 Troubleshooting Overview 9 Guidelines for Troubleshooting 9 Troubleshooting with Logs 11 2 Troubleshooting Virtual Machines 13 Troubleshooting Fault Tolerant Virtual Machines 13 Troubleshooting USB Passthrough Devices 18 Recover Orphaned Virtual Machines 19 Virtual Machine Does Not Power On After Cloning or Deploying from Template 20 3 Troubleshooting Hosts 23 Troubleshooting vSphere HA Host States 23 Troubleshooting Auto Deploy 27 A
vSphere Troubleshooting Virtual Machines with RDMs Need to Ignore SCSI INQUIRY Cache 66 Software iSCSI Adapter Is Enabled When Not Needed 66 Failure to Mount NFS Datastores 67 VMkernel Log Files Contain SCSI Sense Codes 67 Troubleshooting Storage Adapters 68 Checking Metadata Consistency with VOMA 68 Troubleshooting Flash Devices 70 Troubleshooting Virtual Volumes 73 Troubleshooting VAIO Filters 75 8 Troubleshooting Networking 77 Troubleshooting MAC Address Allocation 78 The Conversion to the Enhanced LA
About vSphere Troubleshooting vSphere Troubleshooting describes troubleshooting issues and procedures for vCenter Server implementations and related components. Intended Audience This information is for anyone who wants to troubleshoot virtual machines, ESXi hosts, clusters, and related storage solutions. The information in this book is for experienced Windows or Linux system administrators who are familiar with virtual machine technology and data center operations. VMware, Inc.
vSphere Troubleshooting 6 VMware, Inc.
Updated Information This vSphere Troubleshooting is updated with each release of the product or when necessary. This table provides the update history of the vSphere Troubleshooting. Revision Description EN-001811-02 Added embedded video titled "Troubleshooting Basics". See “Guidelines for Troubleshooting,” on page 9. EN-001811-01 Added a new topic about troubleshooting VIB downloads while using a custom vCenter Server reverse proxy port.
vSphere Troubleshooting 8 VMware, Inc.
Troubleshooting Overview 1 vSphere Troubleshooting contains common troubleshooting scenarios and provides solutions for each of these problems. You can also find guidance here for resolving problems that have similar origins. For unique problems, consider developing and adopting a troubleshooting methodology. The following approach for effective troubleshooting elaborates on how to gather troubleshooting information, such as identifying symptoms and defining the problem space.
vSphere Troubleshooting n Can the affected task be divided into subtasks that you can evaluate separately? n Is the task ending in an error? Is an error message associated with it? n Is the task completing but in an unacceptably long time? n Is the failure consistent or sporadic? n What has changed recently in the software or hardware that might be related to the failure? Defining the Problem Space After you identify the symptoms of the problem, determine which components in your setup are affect
Chapter 1 Troubleshooting Overview Troubleshooting with Logs You can often obtain valuable troubleshooting information by looking at the logs provided by the various services and agents that your implementation is using. Most logs are located in C:\ProgramData\VMware\CIS\logs. Common logs are available in all implementations. Other logs are unique to certain deployment options (Management Node or Platform Services Controller). Common Logs The following logs are common to all deployments on Windows.
vSphere Troubleshooting Table 1‑2. Management Node Logs (Continued) Log Description Vws VMware System and Hardware Health Manager Workflow VMware vCenter workflow manager Platform Services Controller Logs You can examine the following logs if a Platform Services Controller node deployment is chosen. Table 1‑3.
Troubleshooting Virtual Machines 2 The virtual machine troubleshooting topics provide solutions to potential problems that you might encounter when using your virtual machines.
vSphere Troubleshooting Solution If the ESXi server hardware supports HV, but HV is not currently enabled, enable HV in the BIOS on that server. The process for enabling HV varies among BIOSes. See the documentation for your hosts' BIOSes for details on how to enable HV. If the ESXi server hardware does not support HV, switch to hardware that uses processors that support Fault Tolerance.
Chapter 2 Troubleshooting Virtual Machines n To resolve a CPU resources problem, set an explicit CPU reservation for the Primary VM at an MHz value sufficient to run its workload at the desired performance level. This reservation is applied to both the Primary and Secondary VMs, ensuring that both VMs can execute at a specified rate.
vSphere Troubleshooting n If both the Primary VM and Secondary VM cannot access the metadata datastore, the VMs might fail unexpectedly. Typically, an unrelated failure that terminates FT must also occur when access to the FT metadata datastore is lost by both VMs. vSphere HA then tries to restart the Primary VM on a host with access to the metadata datastore. n The VM might stop being recognized as an FT VM by vCenter Server.
Chapter 2 Troubleshooting Virtual Machines Cause When EVC is disabled on a DRS cluster, a VM override that disables DRS on an FT VM might be added. Even if EVC is later reenabled, this override is not canceled. Solution If DRS does not place or evacuate FT VMs in the cluster, check the VMs for a VM override that is disabling DRS. If you find one, remove the override that is disabling DRS. NOTE For more information on how to edit or delete VM overrides, see vSphere Resource Management.
vSphere Troubleshooting Troubleshooting USB Passthrough Devices Information about feature behavior can help you troubleshoot or avoid potential problems when USB devices are connected to a virtual machine. Error Message When You Try to Migrate Virtual Machine with USB Devices Attached Migration with vMotion cannot proceed and issues a confusing error message when you connect multiple USB devices from an ESXi host to a virtual machine and one or more devices are not enabled for vMotion.
Chapter 2 Troubleshooting Virtual Machines Cannot Copy Data From an ESXi Host to a USB Device That Is Connected to the Host You can connect a USB device to an ESXi host and copy data to the device from the host. For example, you might want to gather the vm-support bundle from the host after the host loses network connectivity. To perform this task, you must stop the USB arbitrator.
vSphere Troubleshooting 2 Return to the virtual machine in the vSphere Web Client, right-click it, and select All Virtual Infrastructure Actions > Remove from Inventory. 3 Click Yes to confirm the removal of the virtual machine. 4 Reregister the virtual machine with vCenter Server. a Right-click the datastore where the virtual machine file is located and select Register VM. b Browse to the .vmx file and click OK. c Select the location for the virtual machine and click Next.
Chapter 2 Troubleshooting Virtual Machines d Click Edit. NOTE If the host is part of a cluster that specifies that the virtual machine swap files are stored in the same directory as the virtual machine, you cannot click Edit. You must use the Cluster Settings dialog box to change the swap file location policy for the cluster. VMware, Inc. e Select Use a specific datastore and select a datastore from the list. f Click OK.
vSphere Troubleshooting 22 VMware, Inc.
Troubleshooting Hosts 3 The host troubleshooting topics provide solutions to potential problems that you might encounter when using your vCenter Servers and ESXi hosts.
vSphere Troubleshooting Solution Determine if vCenter Server is reporting the host as not responding. If so, there is a networking problem, an ESXi host agent failure, or a total cluster failure. After the condition is resolved, vSphere HA should work correctly. If not, reconfigure vSphere HA on the host. Similarly, if vCenter Server reports the hosts are responding but a host's state is Agent Unreachable, reconfigure vSphere HA on that host.
Chapter 3 Troubleshooting Hosts Solution When a Configure HA task fails, a reason for the failure is reported. Reason for Failure Action Host communication errors Resolve any communication problems with the host and retry the configuration operation. Timeout errors Possible causes include that the host crashed during the configuration task, the agent failed to start after being installed, or the agent was unable to initialize itself after starting up.
vSphere Troubleshooting Solution Check for the noted failure conditions and resolve any that are found. vSphere HA Agent is in the Network Partitioned State The vSphere HA agent on a host is in the Network Partitioned state. User intervention might be required to resolve this situation. Problem While the virtual machines running on the host continue to be monitored by the master hosts that are responsible for them, vSphere HA's ability to restart the virtual machines after a failure is affected.
Chapter 3 Troubleshooting Hosts n The vSphere HA agent on the host is unable to access any of the agents running on the other cluster hosts. NOTE If your vSphere HA cluster has Virtual SAN enabled, a host is determined to be isolated if it cannot communicate with the other vSphere HA agents in the cluster and cannot reach the configured isolation addresses.
vSphere Troubleshooting Auto Deploy Host Boots with Wrong Configuration A host is booting with a different ESXi image, host profile, or folder location than the one specified in the rules. Problem A host is booting with a different ESXi image profile or configuration than the image profile or configuration that the rules specify. For example, you change the rules to assign a different image profile, but the host still uses the old image profile.
Chapter 3 Troubleshooting Hosts Cause Each VIB in an image profile has a stateless-ready flag that indicates that the VIB is meant for use with Auto Deploy. You get the error if you attempt to write an Auto Deploy rule that uses an image profile in which one or more VIBs have that flag set to FALSE. NOTE You can use hosts provisioned with Auto Deploy that include VIBs that are not stateless ready without problems.
vSphere Troubleshooting The host might also display the following details: Details: This host has been added to VC, but no Image Profile is associated with it. You can use Apply-ESXImageProfile in the PowerCLI to associate an Image Profile with this host. Alternatively, you can reevaluate the rules for this host with the Test-DeployRuleSetCompliance and Repair-DeployRuleSetCompliance cmdlets. The console then displays the host's machine attributes including vendor, serial number, IP address, and so on.
Chapter 3 Troubleshooting Hosts Solution 1 Log in to the system on which you installed the Auto Deploy server. 2 Check that the Auto Deploy server is running. 3 a Click Start > Settings > Control Panel > Administrative Tools. b Double-click Services to open the Services Management panel. c In the Services field, look for the VMware vSphere Auto Deploy Waiter service and restart the service if it is not running.
vSphere Troubleshooting 3 As a temporary workaround, turn off the firewall to see whether that resolves the problem. a Open the command prompt by clicking Start > Program > Accessories > Command prompt. b Type the following command to temporarily turn off the firewall. Do not turn off the firewall in a production environment. netsh firewall set opmode disable c Attempt to provision the host with Auto Deploy. d Type the following command to turn the firewall back on.
Chapter 3 Troubleshooting Hosts n Passwords containing characters from three character classes must be at least seven characters long. n Passwords containing characters from all four character classes must be at least six characters long. NOTE An uppercase character that begins a password does not count toward the number of character classes used. A number that ends a password does not count toward the number of character classes used.
vSphere Troubleshooting Cause If vCenter Server is using a custom port for the reverse proxy, the custom port is not automatically enabled in the ESXi firewall and the VIB downloads fail. Solution 1 Open an SSH connection to the host and log in as root. 2 (Optional) List the existing firewall rules. esxcli network firewall ruleset list 3 (Optional) Back up the /etc/vmware/firewall/service.xml file. cp /etc/vmware/firewall/service.xml /etc/vmware/firewall/service.xml.
Chapter 3 Troubleshooting Hosts 10 (Optional) If you want the firewall configuration to persist after a reboot of the ESXi host, copy the service.xml onto persistent storage and modify the local.sh file. a Copy the modified service.xml file onto persistent storage, for example /store/, or onto a VMFS volume, for example /vmfs/volumes/volume/. cp /etc/vmware/firewall/service.xml location_of_xml_file You can store a VMFS volume in a single location and copy it to multiple hosts. b Add the service.
vSphere Troubleshooting 36 VMware, Inc.
Troubleshooting vCenter Server and the vSphere Web Client 4 The vCenter Server and vSphere Web Client troubleshooting topics provide solutions to problems you might encounter when you set up and configure vCenter Server and the vSphere Web Client, including vCenter Single Sign-On.
vSphere Troubleshooting Microsoft SQL Database Set to Unsupported Compatibility Mode Causes vCenter Server Installation or Upgrade to Fail vCenter Server installation with a Microsoft SQL database fails when the database is set to compatibility mode with an unsupported version. Problem The following error message appears: The DB User entered does not have the required permissions needed to install and configure vCenter Server with the selected DB.
Chapter 4 Troubleshooting vCenter Server and the vSphere Web Client Unable to Start the Virtual Machine Console When you attempt to open a virtual machine console from the vSphere Web Client, the console does not open. Problem When you attempt to open a virtual machine console from the vSphere Web Client, the console does not open. The following error message appears: HTTP ERROR 404 Problem accessin /. Reason: Not Found Errors similar to the following appear in the virgo-server.
vSphere Troubleshooting Troubleshooting vCenter Server and ESXi Host Certificates Certificates are automatically generated when you install vCenter Server. These default certificates are not signed by a commercial certificate authority (CA) and might not provide strong security. You can replace default vCenter Server certificates with certificates signed by a commercial CA. When you replace vCenter Server and ESXi certificates, you might encounter errors.
Chapter 4 Troubleshooting vCenter Server and the vSphere Web Client Cannot Configure vSphere HA When Using Custom SSL Certificates After you install custom SSL certificates, attempts to enable vSphere High Availability (HA) fail. Problem When you attempt to enable vSphere HA on a host with custom SSL certificates installed, the following error message appears: vSphere HA cannot be configured on this host because its SSL thumbprint has not been verified.
vSphere Troubleshooting 42 VMware, Inc.
Troubleshooting Availability 5 The availability troubleshooting topics provide solutions to potential problems that you might encounter when using your hosts and datastores in vSphere HA clusters. You might get an error message when you try to use vSphere HA or vSphere FT. For information about these error messages, see the VMware knowledge base article at http://kb.vmware.com/kb/1033634.
vSphere Troubleshooting Another possible cause of this problem is if your cluster contains any virtual machines that have much larger memory or CPU reservations than the others. The Host Failures Cluster Tolerates admission control policy is based on the calculation on a slot size consisting of two components, the CPU and memory reservations of a virtual machine.
Chapter 5 Troubleshooting Availability Fewer Available Slots Shown Than Expected The Advanced Runtime Info box might display a smaller number of available slots in the cluster than you expect. Problem When you select the Host Failures Cluster Tolerates admission control policy, view the Advanced Runtime Info pane that appears in the vSphere HA section of the cluster's Monitor tab in the vSphere Web Client.
vSphere Troubleshooting n A specified datastore is not optimal for host accessibility and storage backing redundancy. More specifically, the datastore might not be chosen if it is accessible to only a small set of hosts in the cluster. A datastore also might not be chosen if it is on the same LUN or the same NFS server as datastores that vCenter Server has already chosen.
Chapter 5 Troubleshooting Availability Incorrect Virtual Machine Protection State A virtual machine in a vSphere HA cluster is reported as vSphere HA unprotected although it has been powered on for several minutes. Problem When a virtual machine is powered on for several minutes, yet its vSphere HA protection state remains as unprotected, if a failure occurs, vSphere HA might not attempt to restart the virtual machine.
vSphere Troubleshooting Cause vSphere HA might not restart a virtual machine after a failure or might delay its restart for several reasons. n Virtual machine is not protected by vSphere HA at the time the failure occurred n Insufficient spare capacity on hosts with which the virtual machine is compatible n vSphere HA attempted to restart the virtual machine but encountered a fatal error each time it tried.
Chapter 5 Troubleshooting Availability Solution To fix this condition, check the vSphere HA host states reported by vCenter Server. If hosts are identified as partitioned, isolated, or unreachable, resolve the problem before proceeding. In some situations, you can resolve a restart problem by reconfiguring vSphere HA on the host that vCenter Server is reporting as the master host. However, in most situations, this step is insufficient, and you must resolve all host state problems.
vSphere Troubleshooting However, if a VM's swapfile is on a host-local datastore, such a datastore might not be configured on other hosts in the cluster. This situation prevents VMCP from finding a compatible host to fail over the VM, and the VM continues running on the host which experienced an APD failure. Solution u Keep the VM swap file in the default directory, or ensure that the host-local datastore that the VM's swapfile resides on is shared among a set of hosts.
Troubleshooting Resource Management 6 The resource management troubleshooting topics provide solutions to potential problems that you might encounter when using your hosts and datastores in vSphere DRS or vSphere Storage DRS cluster.
vSphere Troubleshooting n If the virtual machine has system files on a separate datastore from the home datastore (legacy), Storage DRS is disabled on the home disk. If you use Storage vMotion to manually migrate the home disk, the system files on different datastores will be all be located on the target datastore and Storage DRS will be enabled on the home disk.
Chapter 6 Troubleshooting Resource Management n Alternatively, if Storage DRS rules are preventing Storage DRS from making migration recommendations, you can set the Storage DRS advanced option IgnoreAffinityRulesForMaintenance to 1. a Browse to the datastore cluster in the vSphere Web Client object navigator. b Click the Manage tab and click Settings. c Select SDRS and click Edit. d In Advanced Options > Configuration Parameters, click Add.
vSphere Troubleshooting Cause Until each placement recommendation is applied, the space resources appear to be available to Storage DRS. Therefore, Storage DRS might reallocate space resources to subsequent requests for space.
Chapter 6 Troubleshooting Resource Management Storage DRS Rule Violation Fault Is Displayed Multiple Times When you attempt to put a datastore into maintenance mode, the same affinity or anti-affinity rule violation fault might appear to be listed more than once in the Faults dialog box. Problem The Faults dialog box appears to display multiple instances of identical faults, but in fact, each fault refers to a different datastore.
vSphere Troubleshooting Cause If the destination host explicitly specifies the virtual machine's swap file location as a datastore in the target datastore cluster, the disks to be placed in that cluster do not form a single affinity group. Storage DRS generates alternative placement recommendations only for a single item or a single affinity group. Solution Accept the single recommendation.
Chapter 6 Troubleshooting Resource Management Unmanaged Workload Detected on Datastore In the vSphere Web Client, an alarm is triggered when vCenter Server detects that a workload from a host might be affecting performance. Problem The alarm Unmanaged workload is detected on the datastore is triggered. Cause The array is shared with non-vSphere workloads, or the array is performing system tasks such as replication. Solution There is no solution.
vSphere Troubleshooting 58 VMware, Inc.
Troubleshooting Storage 7 The storage troubleshooting topics provide solutions to potential problems that you might encounter when using vSphere in different storage environments that include SAN, Virtual SAN, or Virtual Volumes.
vSphere Troubleshooting Resolving SAN Storage Display Problems When you use the vSphere Web Client to display Fibre Channel SAN or iSCSI storage devices, you might not be able to see all devices available to your host. A number of troubleshooting tasks exist that you can perform to resolve storage display problems. Resolving Fibre Channel Storage Display Problems If Fibre Channel storage devices do not display correctly in the vSphere Web Client, perform troubleshooting tasks. Table 7‑1.
Chapter 7 Troubleshooting Storage Resolving iSCSI Storage Display Problems Perform troubleshooting tasks if iSCSI storage devices do not display correctly in the vSphere Web Client. Table 7‑2. Troubleshooting iSCSI LUN Display Troubleshooting Task Description Check cable connectivity. If you do not see a port, the problem could be cable connectivity or routing. Check the cables first. Ensure that cables are connected to the ports and a link light indicates that the connection is good.
vSphere Troubleshooting Excessive SCSI Reservations Cause Slow Host Performance Operations that require getting a file lock or a metadata lock in VMFS result in short-lived SCSI reservations. SCSI reservations lock an entire LUN. Excessive SCSI reservations by a host can cause performance degradation on other servers accessing the same VMFS. Problem Excessive SCSI reservations cause performance degradation and SCSI reservation conflicts. Cause Several operations require VMFS to use SCSI reservations.
Chapter 7 Troubleshooting Storage Path Thrashing Causes Slow LUN Access If your ESXi host is unable to access a LUN, or access is very slow, you might have a problem with path thrashing, also called LUN thrashing. Problem Your host is unable to access a LUN, or access is very slow. The host's log files might indicate frequent path state changes. For example: Frequent path state changes are occurring for path vmhba2:C0:T0:L3. This may indicate a storage problem. Affected device: naa.600600000000000000edd1.
vSphere Troubleshooting Adjust Queue Depth for QLogic, Emulex, and Brocade HBAs If you are not satisfied with the performance of your hardware bus adapters (HBAs), change the maximum queue depth on your ESXi host. The maximum value refers to the queue depths reported for various paths to the LUN. When you lower this value, it throttles the host's throughput and alleviates SAN contention concerns if multiple hosts are overutilizing the storage and are filling its command queue.
Chapter 7 Troubleshooting Storage Adjust Maximum Queue Depth for Software iSCSI If you notice unsatisfactory performance for your software iSCSI LUNs, change their maximum queue depth by running the esxcli commands. Prerequisites n Install vCLI or deploy the vSphere Management Assistant (vMA) virtual machine. See Getting Started with vSphere Command-Line Interfaces. For troubleshooting, you can run esxcli commands in the ESXi Shell.
vSphere Troubleshooting 3 Verify your changes by entering the following command: esxcli storage core device list -d device_ID Virtual Machines with RDMs Need to Ignore SCSI INQUIRY Cache Storage vendors might require that virtual machines with RDMs ignore SCSI INQUIRY data cached by ESXi. Problem Certain guest operating systems or applications run in virtual machines with RDMs display unpredictable behavior.
Chapter 7 Troubleshooting Storage Failure to Mount NFS Datastores Attempts to mount NFS datastores with names in international languages result in failures. Problem The use of non-ASCII characters for directory and file names on NFS storage might cause unpredictable behavior. For example, you might fail to mount an NFS datastore or not be able to power on a virtual machine.
vSphere Troubleshooting For example, the 0x5 0x25 0x0 field from the above error message can be represented as sense=5 asc=25 ascq=0. To interpret Sense Keys, see http://www.t10.org/lists/2sensekey.htm. To determine the meaning of the Additional Sense Code (asc) and Additional Sense Code Qualifier (ascq), use the two codes together. See http://www.t10.org/lists/2asc.htm for details.
Chapter 7 Troubleshooting Storage n You see metadata errors in the vmkernel.log file. n You are unable to access files on a VMFS. n You see corruption being reported for a datastore in events tabs of vCenter Server. Solution To check metadata consistency, run VOMA from the CLI of an ESXi host. VOMA can be used to check and fix metadata inconsistency issues for a VMFS datastore or a virtual flash resource. To resolve errors reported by VOMA, consult VMware Support.
vSphere Troubleshooting Table 7‑4. VOMA Command Options Command Option -m | --module -f | --func Description The module to run: n vmfs. This is a default option. You can check VMFS3 and VMFS 5 datastores. If you specify this module, minimal checks are performed for LVM as well. n vmfsl. Check file systems that back virtual flash volumes. n lvm. Check logical volumes that back VMFS datastores. Functions to be performed: n query. List functions supported by module. n check. Check for errors.
Chapter 7 Troubleshooting Storage n To use the flash disk as a virtual flash resource, do not claim this disk for Virtual SAN. If the disk is claimed by Virtual SAN, remove the disk from Virtual SAN. The flash disk is released from Virtual SAN and becomes available on the list of disks to use with virtual flash. For information about removing disks from Virtual SAN, see the Administering VMware Virtual SAN documentation.
vSphere Troubleshooting Cause ESXi does not recognize certain devices as flash disks when their vendors do not support automatic flash disk detection. In other cases, some non-SATA SAS flash disks might not be detected as local. When disks are not recognized as local flash disks, they are excluded from the list of flash disks available for such features that require only local flash disks. Solution You might need to manually tag disks as flash disks or as local.
Chapter 7 Troubleshooting Storage 3 Click Storage Devices. 4 From the list of storage devices, select one or several remote devices that need to be marked as local and click the Mark as Local for the Host icon. 5 Click Yes to save your changes. Troubleshooting Virtual Volumes Virtual volumes are encapsulations of virtual machine files, virtual disks, and their derivatives. Virtual volumes are stored natively inside a storage system that is connected through Ethernet or SAN.
vSphere Troubleshooting Failures When Migrating VMs or Deploying VM OVFs to Virtual Volumes Datastores Your attempts to migrate a virtual machine or to deploy a VM OVF to virtual datastores fail. Problem An OVF template or a VM being migrated from a nonvirtual datastore might include additional large files, such as ISO disk images, DVD images, and image files. If these additional files cause the configuration virtual volume to exceed its 4-GB limit, migration or deployment to a virtual datastore fails.
Chapter 7 Troubleshooting Storage Non-VVols virtual machines of hardware version 11 or later use separate files to store their memory snapshots. This usage is consistent with VMs on vSphere Virtual Volumes storage, where memory snapshots are created as separate VVols instead of being stored as part of a .vmsn file in the VM home directory. In contrast, non-VVols VMs with hardware version 10 continue to store their memory snapshots as part of the .vmsn file in the VM home directory.
vSphere Troubleshooting Install I/O Filters on a Single ESXi Host For troubleshooting purposes, you can download an ESXi component of the I/O filter, packaged as a VIB file, and install it on the ESXi host. Use the esxcli command to install the VIB file. When you specify a target server by using --server=server_name, the server prompts you for a user name and password. Other connection options, such as a configuration file or session file, are supported.
Troubleshooting Networking 8 The troubleshooting topics about networking in vSphere provide solutions to potential problems that you might encounter with the connectivity of ESXi hosts, vCenter Server and virtual machines.
vSphere Troubleshooting Troubleshooting MAC Address Allocation In vSphere, certain restrictions on the range of MAC addresses that can be assigned to virtual machines might cause lost of connectivity or inability to power on workloads. Duplicate MAC Addresses of Virtual Machines on the Same Network You encounter loss of packets and connectivity because virtual machines have duplicate MAC addresses generated by vCenter Server.
Chapter 8 Troubleshooting Networking n If the vCenter Server instance generates the MAC addresses of virtual machines according to the default allocation, VMware OUI, change the vCenter Server instance ID or use another allocation method to resolve conflicts. NOTE Changing the vCenter Server instance ID or switching to a different allocation scheme does not resolve MAC address conflicts in existing virtual machines.
vSphere Troubleshooting n Enforce MAC address regeneration when transferring a virtual machine between vCenter Server instances by using the virtual machine files from a datastore. a Power off a virtual machine, remove it from the inventory, and in its configuration file (.vmx), set the ethernetX.addressType parameter to generated. X next to ethernet stands for the sequence number of the virtual NIC in the virtual machine.
Chapter 8 Troubleshooting Networking The Conversion to the Enhanced LACP Support Fails Under certain conditions, the conversion from an existing LACP configuration to the enhanced LACP support on a vSphere Distributed Switch 5.5 and later might fail. Problem After you upgrade a vSphere distributed switch to version 5.5 and later, when you initiate the conversion to the enhanced LACP support from an existing LACP configuration, the conversion fails at a certain stage of the process.
vSphere Troubleshooting For example, suppose you verify that a new LAG has been created on the distributed switch and that an intermediate teaming and failover configuration has been created for the distributed port groups. You continue with checking whether there are physical NICs assigned to the LAG ports. You find out that not all hosts have physical NICs assigned to the LAG ports, and you assign the NICs manually.
Chapter 8 Troubleshooting Networking Hosts on a vSphere Distributed Switch 5.1 and Later Lose Connectivity to vCenter Server Hosts on a vSphere Distributed Switch 5.1 and later cannot connect to vCenter Server after a port group configuration. Problem After you change the networking configuration of a port group on a vSphere Distributed Switch 5.1 and later that contains the VMkernel adapters for the management network, the hosts on the switch lose connectivity to vCenter Server.
vSphere Troubleshooting 4 Apply the configuration of the distributed port group and VMkernel adapter from vCenter Server to the host. n Push the correct configuration of the distributed port group and VMkernel adapter from vCenter Server to the host. a In the vSphere Web Client, navigate to the host. b Under the Manage tab, click Networking c From the Virtual switches list, select the distributed switch and click Rectify. n Wait until vCenter Server applies the settings within the next 24 hours.
Chapter 8 Troubleshooting Networking 4 In the vSphere Distributed Switch view, migrate the VMkernel adapter for the network to a standard switch. a Select the vSphere Distributed Switch view, and for the distributed switch, click Manage Virtual Adapters. b In the Manage Virtual Adapters wizard, select the VMkernel adapter from the list and click Migrate. c Select the newly created or another standard switch to migrate the adapter to, and click Next.
vSphere Troubleshooting Virtual Machines Lose Connectivity After Changing the Uplink Failover Order of a Distributed Port Group Changes in the failover NIC order on a distributed port group cause the virtual machines associated with the group to disconnect from the external network.
Chapter 8 Troubleshooting Networking Unable to Add a Physical Adapter to a vSphere Distributed Switch That Has Network I/O Control Enabled You might be unable to add a physical adapter with low speed, for example, 1 Gbps, to a vSphere Distributed Switch that has vSphere Network I/O Control version 3 configured. Problem You try to add a physical adapter with low speed, for example, 1 Gbps, to a vSphere Distributed Switch that is connected to physical adapters with high speed, for example, 10 Gbps.
vSphere Troubleshooting The virtual machine log file vmware.log contains the following message about the VF: PCIPassthruChangeIntrSettings: vf_name failed to register interrupt (error code 195887110) The VMkernel log file vmkernel.
Chapter 8 Troubleshooting Networking Solution u In the guest operating system, reset the interface to cause the passthrough network adapter to regain its valid MAC address. If the interface is configured to use DHCP for address assignment, the interface acquires an IP address automatically. For example, on a Linux virtual machine run the ifconfig console command.
vSphere Troubleshooting Solution n If the VPN software must continue its work on the virtual machine, allow the traffic out of the virtual machine and configure the physical switch port individually to pass the BPDU frames. Network Device Configuration Distributed or standard switch Set the Forged Transmit security property on the port group to Accept to allow BPDU frames to leave the host and reach the physical switch port.
Chapter 8 Troubleshooting Networking Low Throughput for UDP Workloads on Windows Virtual Machines When a Windows virtual machine in vSphere 5.1 and later transmits large UDP packets, the throughput is lower than expected or is oscillating even when other traffic is negligible. Problem When a Windows virtual machine transmits UDP packets larger than 1024 bytes, you experience lower than expected or oscillating throughput even when other traffic is negligible.
vSphere Troubleshooting Parameter in the vSphere Web Client Parameter for the esxcli system settings sdvanced set Command Value Set a default interrupt rate higher than the expected packet rate. For example, set the interrupt rate to 16000 if 15000 interrupts are expected per second. Net.CoalesceScheme Net.CoalesceParams /Net/CoalesceScheme /Net/CoalesceParams rbc 16000 Disable coalescing for low throughput or latency-sensitive workloads.
Chapter 8 Troubleshooting Networking Attempt to Power On a Migrated vApp Fails Because the Associated Protocol Profile Is Missing You cannot power on a vApp or virtual machine that you transferred to a data center or a vCenter Server system because a network protocol profile is missing. Problem After you cold migrate a vApp or a virtual machine to another data center or vCenter Server system, an attempt to power it on fails.
vSphere Troubleshooting Networking Configuration Operation Is Rolled Back and a Host Is Disconnected from vCenter Server When you attempt to add or configure networking on a vSphere Distributed Switch on a host, the operation is rolled back and the host is disconnected from vCenter Server. Problem In vSphere 5.
Chapter 8 Troubleshooting Networking c Under the section, increase the timeout, in the element. 60 VMware, Inc. d Save and close the file. e Restart the vCenter Server system to apply the changes.
vSphere Troubleshooting 96 VMware, Inc.
Troubleshooting Licensing 9 The troubleshooting licensing topics provide solutions to problems that you might encounter as a result of an incorrect or incompatible license setup in vSphere.
vSphere Troubleshooting n Upgrade the license edition to match the resources and features on the host, or disable the features and resources that do not match the license edition. n Assign a vSphere license whose edition is compatible with the license edition of vCenter Server. ESXi Host Disconnects from vCenter Server An ESXi host might disconnect from vCenter Server or all ESXi hosts might disconnect from vCenter Server at the same time.
Chapter 9 Troubleshooting Licensing Cause The host or the vCenter Server system is assigned a license that does not support the features that you want to configure. Solution Check the licensed features on the host and on the vCenter Server system. Upgrade the edition of the license assigned to the host or vCenter Server if they do not include the features that you try to configure or use. VMware, Inc.
vSphere Troubleshooting 100 VMware, Inc.
Index A access to the FT metadata datastore, losing 15 Active Directory 33 address already in use, Jetty server 39 advanced settings, Disk.
vSphere Troubleshooting I I/O filters installation failures 75 installing 76 troubleshooting 75 iBFT 66 identifying symptoms, troubleshooting 9 imbalanced placement of FT VMs 15 IPv4 26 IPv6 26 J Jetty server, address already in use 39 L license capacity 98 License capacity 97 License edition 97, 98 license key 98 local flash disks, undetectable 71 local flash disks are unavailable 70 logging 11 losing access to the FT metadata datastore 15 loss of uplink redundancy 85 lost virtual machine connectivity,
Index storage devices, marking as local 72 Storage DRS affinity rules 55 cannot apply recommendations 56 deleting affinity rules 55 disabled 51 disabling 54 faults 55 OVF templates 54 placement 55 recommendations 55 rule violation 55 troubleshooting 51 Storage I/O Control monitoring 56, 57 troubleshooting 56, 57 swapfile 49 T testing solutions 10 TFTP server, Auto Deploy 30 timeout error, Auto Deploy 27 Tomcat service, vCenter Server upgrade failure 37 troubleshooting certificates 40 extensions 41 plug-in
vSphere Troubleshooting vSphere distributed switch hosts not responding 83, 84 lost virtual machine connectivity 86 vSphere license 98 vSphere DRS 47 vSphere Fault Tolerance 48, 49 vSphere HA Denial of Service 89 troubleshooting SSL certificates 41 vSphere HA admission control 43 vSphere HA Admission Control 43 vSphere HA agent 23–26 vSphere HA cluster 48 vSphere HA failovers 46 vSphere HA host state Agent Uninitialized 24 Agent Unreachable 23 Host Failed 25 Initialization Error 24 Network Isolated 26 Netw
