vSphere Monitoring and Performance Update 1 vSphere 6.0 vCenter Server 6.0 ESXi 6.0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see http://www.vmware.com/support/pubs.
vSphere Monitoring and Performance You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com Copyright © 2010–2016 VMware, Inc. All rights reserved. Copyright and trademark information. VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com 2 VMware, Inc.
Contents About vSphere Monitoring and Performance 5 Updated Information 7 1 Monitoring Inventory Objects with Performance Charts 9 Performance Chart Types 10 Data Counters 10 Metric Groups in vSphere 11 Data Collection Intervals 12 Data Collection Levels 13 View Performance Charts 14 Performance Charts Options Available Under the View Menu Overview Performance Charts 15 Working with Advanced and Custom Charts 92 Troubleshoot and Enhance Performance 94 14 2 Monitoring Guest Operating System Performance 1
vSphere Monitoring and Performance 6 Monitoring the Health of Services and Nodes 125 View the Health Status of Services and Nodes 125 7 Performance Monitoring Utilities: resxtop and esxtop 127 Using the esxtop Utility 127 Using the resxtop Utility 128 Using esxtop or resxtop in Interactive Mode 128 Using Batch Mode 142 Using Replay Mode 143 8 Using the vimtop Plug-In to Monitor the Resource Usage of Services 145 Monitor Services by Using vimtop in Interactive Mode 145 Interactive Mode Command-Line Optio
About vSphere Monitoring and Performance VMware provides several tools to help you monitor your virtual environment and to locate the source of potential issues and current problems. Performance charts Allow you to see performance data on a variety of system resources including CPU, Memory, Storage, and so on. Performance monitoring command-line utilities Allow you to access detailed information on system performance through the command line.
vSphere Monitoring and Performance 6 VMware, Inc.
Updated Information This vSphere Monitoring and Performance documentation is updated with each release of the product or when necessary. This table provides the update history of the vSphere Monitoring and Performance documentation. Revision Description EN-001901-01 Adding information to “Memory Panel,” on page 134 about CNSM statistic that is part of the memory utilization statistics that esxtop utility provides. EN-001901-00 Initial release. VMware, Inc.
vSphere Monitoring and Performance 8 VMware, Inc.
Monitoring Inventory Objects with Performance Charts 1 The vSphere statistics subsystem collects data on the resource usage of inventory objects. Data on a wide range of metrics is collected at frequent intervals, processed, and archived in the vCenter Server database. You can access statistical information through command-line monitoring utilities or by viewing performance charts in the vSphere Web Client.
vSphere Monitoring and Performance n “View Performance Charts,” on page 14 n “Performance Charts Options Available Under the View Menu,” on page 14 n “Overview Performance Charts,” on page 15 n “Working with Advanced and Custom Charts,” on page 92 n “Troubleshoot and Enhance Performance,” on page 94 Performance Chart Types Performance metrics are displayed in different types of charts, depending on the metric type and object. Table 1‑1.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑2. Data Counter Attributes (Continued) Attribute Description Statistics Type Measurement used during the statistics interval. Related to the unit of measurement. n Rate – Value over the current statistics interval n Delta – Change from previous statistics interval. n Absolute – Absolute value (independent of the statistics interval). Rollup Type Calculation method used during the statistics interval to roll up data.
vSphere Monitoring and Performance Table 1‑3. Metric Groups (Continued) Metric group Description Network Network utilization for both physical and virtual network interface controllers (NICs) and other network devices, such as the virtual switches that support connectivity among all components, such as hosts, virtual machines, VMkernel, and so on. Power Energy usage statistics per host. Storage Adapter Data traffic statistics per host bus adapter (HBA).
Chapter 1 Monitoring Inventory Objects with Performance Charts Data Collection Levels Each collection interval has a default collection level that determines the amount of data gathered and which counters are available for display in charts. Collection levels are also referred to as statistics levels. Table 1‑5.
vSphere Monitoring and Performance View Performance Charts The vCenter Server statistics settings, the type of object selected, and the features that are enabled on the selected object determine the amount of information displayed in charts. Charts are organized into views. You can select a view to see related data together on one screen. You can also specify the time range, or data collection interval. The duration extends from the selected time range to the present time.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑6. Performance Chart Views by Inventory Object (Continued) Object View List Items Cluster n n n Host n n Resource Pool and vApps n Virtual Machine n n n n Home - CPU and memory charts for the cluster. Resource Pools & Virtual Machines - thumbnail charts for resource pools and virtual machines, and stacked charts for total CPU and memory usage in the cluster.
vSphere Monitoring and Performance Table 1‑7. Data Counters Chart Label Description Usage Sum of the average CPU usage values, in Megahertz, of all virtual machines in the cluster. n Counter: usagemhz n Stats Type: Rate n Unit: Megahertz (MHz) n Rollup Type: Average (Minimum/Maximum) n Total Collection Level: 1 (4) Total amount of CPU resources available in the cluster. The maximum value is equal to the number of cores multiplied by the frequency of the processors.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑9. Data Counters Chart Label Description , , or Amount of CPU actively used by the host, resource pool, or virtual machine in the cluster. n Counter: usagemhz n Stats Type: Rate n Unit: MegaHertz (MHz) n n Rollup Type: Average (Minimum/Maximum) Collection Level: 1 (4) Chart Analysis A short spike in CPU usage indicates that you are making the best use of cluster resources.
vSphere Monitoring and Performance Chart Analysis Use the disk charts to monitor average disk loads and to determine trends in disk usage. For example, you might notice a performance degradation with applications that frequently read from and write to the hard disk. If you see a spike in the number of disk read or write requests, check whether any such applications were running at that time.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑12. Disk I/O Performance Enhancement Advice (Continued) # Resolution 11 For ESXi hosts, create virtual disks as preallocated. When you create a virtual disk for a guest operating system, select Allocate all disk space now. The performance degradation associated with reassigning additional disk space does not occur, and the disk is less likely to become fragmented. 12 Use the most current host hardware.
vSphere Monitoring and Performance Table 1‑14. Memory Performance Enhancement Advice (Continued) # Resolution 3 If the balloon value is high, check the resource shares, reservations, and limits for the virtual machines and resource pools on the hosts. Verify that the host's settings are adequate and not lower than those set for the virtual machine.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑15. Data Counters (Continued) Chart Label Description Granted Sum of the guest physical memory granted for all powered on virtual machines. Granted memory is mapped to the host's machine memory. Granted memory for a host includes the shared memory of each virtual machine on the host.
vSphere Monitoring and Performance Table 1‑16. Memory Performance Enhancement Advice (Continued) # Resolution 3 Reduce the memory space on the virtual machine, and correct the cache size if it is too large. This frees up memory for other virtual machines. 4 If the memory reservation of the virtual machine is set to a value much higher than its active memory, decrease the reservation setting so that the VMkernel can reclaim the idle memory for other virtual machines on the host.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑18. Memory Performance Enhancement Advice (Continued) # Resolution 5 If the cluster is a DRS cluster: n Increase the number of hosts, and migrate one or more virtual machines to the new host. n Check the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots in the cluster. 6 Add more physical memory to one or more hosts.
vSphere Monitoring and Performance Table 1‑20. Networking Performance Enhancement Advice (Continued) # Resolution 4 Assign each physical NIC to a port group and a vSwitch. 5 Use separate physical NICs to handle the different traffic streams, such as network packets generated by virtual machines, iSCSI protocols, vMotion tasks. 6 Ensure that the physical NIC capacity is large enough to handle the network traffic on that vSwitch.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑22. CPU Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. 2 If the cluster is not a DRS cluster, enable DRS. Select the cluster, and click Manage > Settings > vSphere DRS > Edit > Turn ON vSphere DRS. 3 If the cluster is a DRS cluster: n Increase the number of hosts, and migrate one or more virtual machines to the new host. n Check the aggressiveness threshold.
vSphere Monitoring and Performance Table 1‑24. Data Counters Chart Label Description Amount of used storage space on the 10 datastores with the most used space. Counter: used n Stats Type: Absolute n Unit: GigaBytes (GB) n Rollup Type: Latest n n Collection Level: 1 Chart Analysis The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑25. Data Counters (Continued) File Type Description Snapshots Amount of disk space used by virtual machine snapshot files. Snapshot files store information about virtual machine snapshots. They include snapshot state files and delta disk files. A snapshot state file stores the running state of the virtual machine at the time of the snapshot. It has the extension .vmsn.
vSphere Monitoring and Performance Table 1‑26. Data Counters Chart Label Description Allocated Amount of physical space provisioned by an administrator for the datastore. It is the storage size up to which files on the datastore can grow. Allocated space is not always in use. n Counter: provisioned n Stats Type: Absolute n Unit: Gigabytes (GB) n n Rollup Type: Latest Collection Level: 1 Used Amount of physical datastore space in use.
Chapter 1 Monitoring Inventory Objects with Performance Charts Datastore Counters Table 1‑27. Data Counters File Type Description Virtual Disks Amount of disk space used by virtual disk files. Virtual disk files store the contents of the virtual machine's hard disk drive, including information that you write to a virtual machine's hard disk, such as the operating system, program files, and data files. The files have the extension .vmdk and appear as a physical disk drive to a guest operating system.
vSphere Monitoring and Performance If snapshot files are consuming a lot of datastore space, consider consolidating them to the virtual disk when they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the snapshots from the vSphere Web Client user interface. For information about consolidating the data center, see the vSphere documentation.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑29. Data Counters Chart Label Description Storage I/O Control Normalized Latency This is the latency that is monitored by Storage I/O Control to detect congestion on the datastore.
vSphere Monitoring and Performance Table 1‑32. Data Counters Chart Label Description Average Device Latency per Host Measures the amount of time, in milliseconds, to complete a SCSI command issued from the physical device. n Counter: deviceLatency n Stats Type: Absolute n Unit: Milliseconds (ms) n n Rollup Type: Average Collection Level: 3 Maximum Queue Depth per Host This chart displays maximum queue depth that hosts are currently maintaining for the datastore.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑35. Data Counters Chart Label Description Write IOPs per Host Number of disk write commands completed on each disk on the host, per second.
vSphere Monitoring and Performance Table 1‑38. Data Counters Chart Label Description Read IOPs per Virtual Machine Disk Number of disk read commands completed on each virtual machine disk, per second. Read rate = blocks read per second × block size n Counter: numberReadAveraged n Stats Type: Rate n Unit: Number n n Rollup Type: Average Collection Level: 3 Write IOPs per Virtual Machine Disk This chart displays the ten virtual machines with the highest number of write operations.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑41. Data Counters Chart Label Description Usage Actively used CPU, as a percentage of the total available CPU, for each physical CPU on the host. Active CPU is approximately equal to the ratio of the used CPU to the available CPU. Available CPU = # of physical CPUs × clock rate. 100% represents all CPUs on the host.
vSphere Monitoring and Performance Table 1‑43. Data Counters Chart Label Description Usage The sum, in megahertz, of the actively used CPU of all powered on virtual machines on a host. The maximum possible value is the frequency of the processors multiplied by the number of processors. For example, if you have a host with four 2GHz CPUs running a virtual machine that is using 4000MHz, the host is using two CPUs completely. 4000 ÷ (4 × 2000) = 0.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑45. Counters Name Description virtual_machine Amount of CPU actively being used by each virtual machine on the host. 100% represents all CPUs. For example, if a virtual machine has one virtual CPU that is running on a host with four CPUs and the CPU usage is 100%, the virtual machine is using one CPU resource.
vSphere Monitoring and Performance Table 1‑47. Data Counters Chart Label Usage Description Average data I/O rate across all LUNs on the host. Counter: usage n Stats Type: Rate n Unit: Kilobytes per second (KBps) n Rollup Type: Average (Minimum/Maximum) n n Collection Level: 1 (4) Chart Analysis Use the disk charts to monitor average disk loads and to determine trends in disk usage.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑48. Disk I/O Performance Enhancement Advice (Continued) # Resolution 8 For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with the system page file. This alleviates disk spindle contention during periods of high use. 9 On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtual machine's VMX file.
vSphere Monitoring and Performance n The deviceLatency data counter measures the average amount of time, in milliseconds, to complete a SCSI command from the physical device. Depending on your hardware, a number greater than 15ms indicates probable problems with the storage array. Move the active VMDK to a volume with more spindles or add disks to the LUN. n The queueLatency data counter measures the average amount of time taken per SCSI command in the VMkernel queue. This value must always be zero.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑51. Data Counters Chart Label Description Read Requests Number of disk read commands completed on each LUN on the host. The aggregate number of all disk read commands is also displayed in the chart. n Counter: numberRead n Stats Type: Absolute n Unit: Number n n Write Requests Rollup Type: Summation Collection Level: 3 Number of disk write commands completed on each LUN on the host.
vSphere Monitoring and Performance Table 1‑52. Disk I/O Performance Enhancement Advice (Continued) # Resolution 4 Use the vendor's array tools to determine the array performance statistics. When too many servers simultaneously access common elements on an array, the disks might have trouble keeping up. Consider array-side improvements to increase throughput. 5 Use Storage vMotion to migrate I/O-intensive virtual machines across multiple hosts.
Chapter 1 Monitoring Inventory Objects with Performance Charts n The deviceLatency data counter measures the average amount of time, in milliseconds, to complete a SCSI command from the physical device. Depending on your hardware, a number greater than 15ms indicates probable problems with the storage array. Move the active VMDK to a volume with more spindles or add disks to the LUN. n The queueLatency data counter measures the average amount of time taken per SCSI command in the VMkernel queue.
vSphere Monitoring and Performance Table 1‑55. Data Counters Chart Label Description Highest Disk Latency Highest latency value of all disks used by the host. Latency measures the time used to process a SCSI command issued by the guest OS to the virtual machine. The kernel latency is the time VMkernel takes to process an I/O request. The device latency is the time it takes the hardware to handle the request. Total latency = kernelLatency + deviceLatency.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑56. Disk I/O Performance Enhancement Advice (Continued) # Resolution 6 Balance the disk load across all physical resources available. Spread heavily used storage across LUNs that are accessed by different adapters. Use separate queues for each adapter to improve disk efficiency. 7 Configure the HBAs and RAID controllers for optimal use. Verify that the queue depths and cache settings on the RAID controllers are adequate.
vSphere Monitoring and Performance n The queueLatency data counter measures the average amount of time taken per SCSI command in the VMkernel queue. This value must always be zero. If not, the workload is too high and the array cannot process the data fast enough. If the disk latency values are high, or if you notice other problems with disk I/O performance, consider taking the actions listed below. Table 1‑58. Disk I/O Performance Enhancement Advice # Resolution 1 Increase the virtual machine memory.
Chapter 1 Monitoring Inventory Objects with Performance Charts If the memory usage value is high, and the host has high ballooning or swapping, check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host cannot handle the demand for memory. This leads to memory reclamation, which might degrade performance. If the host has enough free memory, check the resource shares, reservation, and limit settings of the virtual machines and resource pools on the host.
vSphere Monitoring and Performance If the memory usage value is high, and the host has high ballooning or swapping, check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host cannot handle the demand for memory. This leads to memory reclamation, which might degrade performance. If the host has enough free memory, check the resource shares, reservation, and limit settings of the virtual machines and resource pools on the host.
Chapter 1 Monitoring Inventory Objects with Performance Charts However, a consistently high memory usage value (94% or greater) indicates the host does not have the memory resources required to meet the demand. If the memory balloon and swap values are not high, performance is probably not affected. If the memory usage value is high, and the host has high ballooning or swapping, check the amount of free physical memory on the host.
vSphere Monitoring and Performance Table 1‑64. Data Counters (Continued) Chart Label Description Balloon Target Sum of the balloon target memory of all powered on virtual machines on the host. If the balloon target value is greater than the balloon value, the VMkernel inflates the balloon, causing more virtual machine memory to be reclaimed.
Chapter 1 Monitoring Inventory Objects with Performance Charts Chart Analysis To ensure best performance, the host memory must be large enough to accommodate the active memory of the virtual machines. The active memory can be smaller than the virtual machine memory size. This allows you to over-provision memory, but still ensures that the virtual machine active memory is smaller than the host memory. Transient high-usage values usually do not cause performance degradation.
vSphere Monitoring and Performance Table 1‑66. Data Counters Chart Label Description Usage Amount of guest physical memory currently in use on the virtual machine. Counter: usage n Stats Type: Absolute n Unit: Percentage (%) n Rollup Type: Average (Minimum/Maximum) n n Collection Level: 1 (4) Chart Analysis A virtual machine's memory size must be slightly larger than the average guest memory usage. This enables the host to accommodate workload spikes without swapping memory among guests.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑68. Host Counters Chart Label Description Usage Average rate at which data is transmitted and received across all NIC instances connected to the host. n Counter: usage n Stats Type: Rate n Unit: Megabits per second (Mbps) n n Rollup Type: Average (Minimum/Maximum) Collection Level: 1 (4) Chart Analysis Network performance depends on application workload and network configuration.
vSphere Monitoring and Performance Table 1‑69. Networking Performance Enhancement Advice (Continued) # Resolution 9 Verify that all NICs are running in full duplex mode. Hardware connectivity problems might result in a NIC resetting itself to a lower speed or half duplex mode. 10 Use vNICs that are TCP Segmentation Offload (TSO)-capable, and verify that TSO-Jumbo Frames are enabled where possible. Network Rate (Mbps) The Network Rate chart displays network bandwidth on a host.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑71. Networking Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. 2 If possible, use vmxnet3 NIC drivers, which are available with VMware Tools. They are optimized for high performance. 3 If virtual machines running on the same host communicate with each other, connect them to the same vSwitch to avoid transferring packets over the physical network.
vSphere Monitoring and Performance If packets are being dropped, adjust the virtual machine shares. If packets are not being dropped, check the size of the network packets and the data receive and transfer rates. In general, the larger the network packets, the faster the network speed. When the packet size is large, fewer packets are transferred, which reduces the amount of CPU required to process the data.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑74. Data Counters Chart Label Description Sum of the data transmitted and received across all virtual NIC instances connected to the virtual machine. n Counter: usage n Stats Type: Rate n Unit: Megabits per second (Mbps) n n Rollup Type: Average (Minimum/Maximum) Collection Level: 1 (4) Chart Analysis Network performance depends on application workload and network configuration.
vSphere Monitoring and Performance Table 1‑75. Networking Performance Enhancement Advice (Continued) # Resolution 9 Verify that all NICs are running in full duplex mode. Hardware connectivity problems might result in a NIC resetting itself to a lower speed or half duplex mode. 10 Use vNICs that are TCP Segmentation Offload (TSO)-capable, and verify that TSO-Jumbo Frames are enabled where possible.
Chapter 1 Monitoring Inventory Objects with Performance Charts CPU Usage The CPU Usage chart displays CPU usage of virtual machines in the resource pool or vApp. The chart displays the top ten virtual machines with the highest CPU usage. This chart is located in the Resource Pools & Virtual Machines view of the Resource Pool or vApp Performance tab. Table 1‑78. Data Counters Chart Label Description virtual_machine Amount of CPU actively used by virtual machines.
vSphere Monitoring and Performance Memory (MB) The Memory (MB) chart displays memory usage in the resource pool or vApp. This chart is located in the Home view of the resource pool or vApp Performance tab. Table 1‑80. Data Counters Chart Label Description resource_pool or vApp Sum of the active memory used by all virtual machines in the resource pool or vApp. Active memory is determined by the VMkernel and includes overhead memory.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑82. Data Counters Chart Label Description virtual_machine Amount of host memory used by the virtual machine for its guest operating system's physical memory. Memory overhead is not included in consumed memory.
vSphere Monitoring and Performance Memory (MB) The Memory (MB) chart displays memory data counters for resource pools or vApps. Description This chart is located in the Home view of the resource pool or vApp Performance tab. NOTE These data counter definitions are for virtual machines. At the resource pool level, the values are collected and totaled. The counter values in the chart represent the aggregate amounts of the virtual machine data.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑84. Data Counters (Continued) Chart Label Description Shared Amount of guest physical memory shared with other virtual machines in the resource pool. Swapped Sum of the memory swapped by all powered on virtual machines in the resource pool.
vSphere Monitoring and Performance vApps The vApp charts contain information about CPU and memory usage for vApps. The help topic for each chart contains information about the data counters displayed in that chart. The counters available are determined by the collection level set for vCenter Server. CPU (MHz) The CPU (MHz) chart displays CPU usage in the vApp or resource pool. This chart is located in the Home view of the vApp or resource pool Performance tab. Counters Table 1‑86.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑88. Data Counters Chart Label Description virtual_machine Amount of CPU actively used by virtual machines. Counter: usagemhz n Stats Type: Rate n Unit: Megahertz (MHz) n Rollup Type: Average (Minimum/Maximum) n n Collection Level: 1 (4) Chart Analysis A short spike in CPU usage or CPU ready indicates that you are making the best use of the virtual machine resources.
vSphere Monitoring and Performance Table 1‑90. Data Counters Chart Label Description resource_pool or vApp Sum of the active memory used by all virtual machines in the resource pool or vApp. Active memory is determined by the VMkernel and includes overhead memory.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑92. Data Counters Chart Label Description virtual_machine Amount of host memory used by the virtual machine for its guest operating system's physical memory. Memory overhead is not included in consumed memory.
vSphere Monitoring and Performance Virtual Machines The virtual machine charts contain information about CPU, disk, memory, network, storage, and fault tolerance for virtual machines. The help topic for each chart contains information about the data counters displayed in that chart. The counters available are determined by the collection level set for vCenter Server. CPU (%) The CPU (%) chart displays virtual machine CPU usage and ready values.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑95. CPU Performance Enhancement Advice (Continued) # Resolution 5 Increase the CPU shares to give the virtual machine more opportunities to run. The total ready time on the host might remain at the same level if the host system is constrained by CPU. If the host ready time doesn't decrease, set the CPU reservations for high-priority virtual machines to guarantee that they receive the required CPU cycles.
vSphere Monitoring and Performance Table 1‑97. CPU Performance Enhancement Advice (Continued) # Resolution 5 Increase the CPU shares to give the virtual machine more opportunities to run. The total ready time on the host might remain at the same level if the host system is constrained by CPU. If the host ready time doesn't decrease, set the CPU reservations for high-priority virtual machines to guarantee that they receive the required CPU cycles.
Chapter 1 Monitoring Inventory Objects with Performance Charts If the disk latency values are high, or if you notice other problems with disk I/O performance, consider taking the actions listed below. Table 1‑99. Disk I/O Performance Enhancement Advice # Resolution 1 Increase the virtual machine memory. This should allow for more operating system caching, which can reduce I/O activity. Note that this might require you to also increase the host memory.
vSphere Monitoring and Performance Table 1‑100. Data Counters Chart Label Description Read Number of disk read commands completed on each virtual disk on the virtual machine, per second. The aggregate number of all disk read commands per second is also displayed in the chart.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑101. Disk I/O Performance Enhancement Advice # Resolution 1 Increase the virtual machine memory. This should allow for more operating system caching, which can reduce I/O activity. Note that this might require you to also increase the host memory. Increasing memory might reduce the need to store data because databases can utilize system memory to cache data and avoid disk access.
vSphere Monitoring and Performance Table 1‑102. Data Counters Chart Label Description Read Requests Number of disk read commands completed on each virtual disk on the virtual machine. The aggregate number of all disk read commands is also displayed in the chart. n Counter: numberRead n Stats Type: Absolute n n n Write Requests Unit: Number Rollup Type: Summation Collection Level: 3 Number of disk write commands completed on each virtual disk on the virtual machine.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑103. Disk I/O Performance Enhancement Advice (Continued) # Resolution 4 Use the vendor's array tools to determine the array performance statistics. When too many servers simultaneously access common elements on an array, the disks might have trouble keeping up. Consider array-side improvements to increase throughput. 5 Use Storage vMotion to migrate I/O-intensive virtual machines across multiple hosts.
vSphere Monitoring and Performance Table 1‑105. Data Counters Chart Label Description Read Requests Number of virtual disk read commands completed on each virtual disk on the virtual machine. The aggregate number of all virtual disk read commands per second is also displayed in the chart.
Chapter 1 Monitoring Inventory Objects with Performance Charts If the host has enough free memory, check the resource shares, reservation, and limit of the virtual machines and resource pools on the host. Verify that the host settings are adequate and not lower than those set for the virtual machine. If little free memory is available, or if you notice degradation in performance, consider taking the following actions. Table 1‑107.
vSphere Monitoring and Performance If little free memory is available, or if you notice degradation in performance, consider taking the following actions. Table 1‑109. Memory Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical to performance. 2 Verify that the balloon driver is enabled. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑111. Memory Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical to performance. 2 Verify that the balloon driver is enabled. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, this does not impact virtual machine performance.
vSphere Monitoring and Performance Table 1‑112. Data Counters (Continued) Chart Label Description Consumed Amount of guest physical memory consumed by the virtual machine for guest memory. Consumed memory does not include overhead memory. It includes shared memory and memory that might be reserved, but not actually used.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑113. Memory Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical to performance. 2 Verify that the balloon driver is enabled. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, this does not impact virtual machine performance.
vSphere Monitoring and Performance Table 1‑115. Networking Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. 2 If possible, use vmxnet3 NIC drivers, which are available with VMware Tools. They are optimized for high performance. 3 If virtual machines running on the same host communicate with each other, connect them to the same vSwitch to avoid transferring packets over the physical network.
Chapter 1 Monitoring Inventory Objects with Performance Charts If packets are being dropped, adjust the virtual machine shares. If packets are not being dropped, check the size of the network packets and the data receive and transfer rates. In general, the larger the network packets, the faster the network speed. When the packet size is large, fewer packets are transferred, which reduces the amount of CPU required to process the data.
vSphere Monitoring and Performance Table 1‑118. Data Counters Chart Label Description Packets Transmitted Number of network packets transmitted across the top ten virtual NIC instances on the virtual machine. The chart also displays the aggregated value for each NIC. n Counter: packetTx n Stats Type: Absolute n Unit: Number n n Packets Received Rollup Type: Summation Collection Level: 3 Number of network packets received across the top ten virtual NIC instances on the virtual machine.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑119. Networking Performance Enhancement Advice (Continued) # Resolution 8 Verify that the reported speed and duplex settings for the physical NIC match the hardware expectations and that the hardware is configured to run at its maximum capability. For example, verify that NICs with 1Gbps are not reset to 100Mbps because they are connected to an older switch. 9 Verify that all NICs are running in full duplex mode.
vSphere Monitoring and Performance Space Utilization by Datastores The Space Utilization by Datastores chart displays the amount of space used by a virtual machine on different datastores in the data center. NOTE This chart does not show historical statistics. It only shows the most recently available data, which may be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statistics are not collected across all datastores at one time. They are collected asynchronously.
Chapter 1 Monitoring Inventory Objects with Performance Charts Datastore counters Table 1‑122. Data Counters File Type Description Virtual Disks Amount of disk space used by virtual disk files. Virtual disk files store the contents of the virtual machine's hard disk drive, including information that you write to a virtual machine's hard disk - the operating system, program files, and data files. The files have the extension .vmdk and appear as a physical disk drive to a guest operating system.
vSphere Monitoring and Performance Fault Tolerance Performance Counters The fault tolerance charts contain information about CPU and memory for fault tolerant virtual machines. NOTE The performance charts and help topics for fault tolerance are available only when you have vSphere Fault Tolerance enabled.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑125. Data Counters Chart Label Description System Amount of time spent on system processes on each virtual CPU in the virtual machine. NOTE This is the host view of the CPU usage, not the guest operating system view.
vSphere Monitoring and Performance Chart Analysis A large discrepancy in CPU usage between the primary and secondary virtual machines might indicate performance problems. The CPU ready, system, and wait times of each virtual machine should be synchronized. A large discrepancy in these values might indicate performance problems. Consider taking the following actions. Table 1‑128.
Chapter 1 Monitoring Inventory Objects with Performance Charts If a virtual machine has high ballooning or swapping, check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host cannot meet the memory requirements. This leads to memory reclamation, which might degrade performance. If the active memory size is the same as the granted memory size, demand for memory is greater than the memory resources available.
vSphere Monitoring and Performance If a virtual machine has high ballooning or swapping, check the amount of free physical memory on the host. The host might require more memory resources. If it does not, check the resource shares, reservation, and limit of the virtual machines and resource pools on the host. Verify that the host settings are adequate and not lower than those set for the virtual machine.
Chapter 1 Monitoring Inventory Objects with Performance Charts Change Advanced Chart Settings You can customize a performance chart by specifying the objects to monitor, the counters to include, the time range, and chart type. You can customize preconfigured chart views and create new chart views. Procedure 1 Select an inventory object in the vSphere Web Client. 2 Click the Monitor tab, and click Performance. 3 Click Advanced. 4 Click Chart Options.
vSphere Monitoring and Performance 6 Click OK. The chart settings are saved and an entry for your chart is added to the View menu. Delete a Custom Advanced Chart View You can delete custom chart views from the vSphere Web Client. Procedure 1 Select an inventory object in the vSphere Web Client. 2 Click the Monitor tab, and click Performance. 3 Click Advanced. 4 Click Chart Options. 5 Select a chart and click Delete Options. 6 Click OK to confirm deletion.
Chapter 1 Monitoring Inventory Objects with Performance Charts Solutions for Consistently High CPU Usage Temporary spikes in CPU usage indicate that you are making the best use of CPU resources. Consistently high CPU usage might indicate a problem. You can use the CPU performance charts to monitor CPU usage for hosts, clusters, resource pools, virtual machines, and vApps. Problem n Host CPU usage constantly is high.
vSphere Monitoring and Performance Solutions for Memory Performance Problems Host machine memory is the hardware backing for guest virtual memory and guest physical memory. Host machine memory must be at least slightly larger than the combined active memory of the virtual machines on the host. A virtual machine's memory size must be slightly larger than the average guest memory usage. Increasing the virtual machine memory size results in more overhead memory usage.
Chapter 1 Monitoring Inventory Objects with Performance Charts n You can provision more space to the datastore if possible, or you can add disks to the datastore or use shared datastores. Solutions for Disk Performance Problems Use the disk charts to monitor average disk loads and to determine trends in disk usage. For example, you might notice a performance degradation with applications that frequently read from and write to the hard disk.
vSphere Monitoring and Performance n For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with the system page file. This alleviates disk spindle contention during periods of high use. n On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtual machine's .VMX file. n If the combined disk I/O is higher than a single HBA capacity, use multipathing or multiple links.
Chapter 1 Monitoring Inventory Objects with Performance Charts n Ensure that the physical NIC capacity is large enough to handle the network traffic on that virtual switch. If the capacity is not enough, consider using a high-bandwidth physical NIC (10Gbps) or moving some virtual machines to a virtual switch with a lighter load or to a new virtual switch. n If packets are being dropped at the virtual switch port, increase the virtual network driver ring buffers where applicable.
vSphere Monitoring and Performance 100 VMware, Inc.
Monitoring Guest Operating System Performance 2 This section describes how to install and view VMware-specific performance data for virtual machines that run Microsoft Windows operating systems. VMware provides performance counters that enable you to view data on many aspects of guest operating system performance for the Microsoft Windows Perfmon utility.
vSphere Monitoring and Performance 5 In the Add Counters dialog box, select Use local computer counters. 6 Select a virtual machine performance object. Virtual machine performance object names begin with VM. 7 Select the counters that you want to display for that object. 8 If the performance object has multiple instances, select the instances you want to display. 9 Click Add. The Performance dialog box displays data for the selected performance object.
Monitoring Host Health Status 3 You can use the vSphere Web Client to monitor the state of host hardware components, such as CPU processors, memory, fans, and other components.
vSphere Monitoring and Performance This chapter includes the following topics: n “Monitor Hardware Health Status,” on page 104 n “Reset Health Status Sensors,” on page 104 Monitor Hardware Health Status You can monitor the health status of host hardware in the vSphere Web Client. Procedure 1 Select a host in the vSphere Web Client navigator. 2 Click the Monitor tab, and click Hardware Status. 3 Select the type of information to view.
Monitoring Events, Alarms, and Automated Actions 4 vSphere includes a user-configurable events and alarms subsystem. This subsystem tracks events happening throughout vSphere and stores the data in log files and the vCenter Server database. This subsystem also enables you to specify the conditions under which alarms are triggered. Alarms can change state from mild warnings to more serious alerts as system conditions change, and can trigger automated alarm actions.
vSphere Monitoring and Performance n Actions - Defines operations that occur in response to triggered alarms. VMware provides sets of predefined actions that are specific to inventory object types. Alarms have the following severity levels: n Normal – green n Warning – yellow n Alert – red Alarm definitions are associated with the object selected in the inventory. An alarm monitors the type of inventory objects specified in its definition.
Chapter 4 Monitoring Events, Alarms, and Automated Actions 2 Click the Monitor tab, and click Events. 3 Select an event to see details. 4 (Optional) Use the filter controls above the list to filter the list. 5 (Optional) Click a column heading to sort the list. View System Logs vSphere records events in the vCenter Server database. System log entries include such information as who generated the event, when the event was created, and the type of event. Prerequisites n Required privilege: Global.
vSphere Monitoring and Performance n To view alarms triggered on a selected inventory object, click the Monitor tab, click Issues, and click Triggered Alarms. n To view a list of available alarm definitions for a selected inventory object, click the Manage tab, and click Alarm Definitions. Live Refresh of Recent Tasks and Alarms You can configure the vSphere Web Client to live refresh the recent tasks and the alarms that result from operations that other users perform in your environment.
Chapter 4 Monitoring Events, Alarms, and Automated Actions n You log in to vSphere Web Client A from https://hostnameA:9443/vsphere-client/. You can see live refresh of recent tasks and alarms for all the users currently performing operations on both vCenter Server systems in the vCenter Server Single-Sign On domain. n You log in to vSphere Web Client B from https://hostnameB/vsphere-client/.
vSphere Monitoring and Performance Prerequisites Required Privilege: Alarms.Create alarm or Alarms.Modify alarm Procedure n n Create or edit alarms in the Manage tab. a Select an inventory object and click the Manage tab. b Click Alarm Definitions. c Right-click the list of alarms and select one of the options to add or edit an alarm. Add an alarm to an object in the object navigator. a Right-click an inventory object and select Alarms > New Alarm Definition.
Chapter 4 Monitoring Events, Alarms, and Automated Actions 3 Click in the Status column, and select an option from the drop-down menu. 4 (Optional) Configure additional conditions to be met before the alarm triggers. a Click the Add icon to add an argument. b Click in the Argument column, and select an option from the drop-down menu. c Click in the Operator column, and select an option from the drop-down menu. d Click in the Value column, and enter a value into the text field.
vSphere Monitoring and Performance Specify Alarm Actions You can define actions that the system performs when the alarm is triggered or changes status. You can enable and disable alarms and alarm actions independently of each other. vCenter Server can perform alarm actions in response to triggered alarms. Prerequisites Verify that you have navigated to the Actions page of the alarm definition wizard. See “View and Edit Alarm Settings,” on page 109. Required Privilege: Alarms.Create alarm or Alarms.
Chapter 4 Monitoring Events, Alarms, and Automated Actions 6 Click Finish. You configured the alarm general settings, triggers, and actions. The alarm monitors the object on which it is defined, as well as child objects. Send Email as an Alarm Action You can use the SMTP agent included with vCenter Server to send email notifications when alarms are triggered. Prerequisites Ensure that the vCenter Server SMTP agent is properly configured to send email notifications. Required Privilege: Alarms.
vSphere Monitoring and Performance Send SNMP Traps as an Alarm The SNMP agent included with vCenter Server can be used to send traps when alarms are triggered on a vCenter Server instance. The default hardware health alarms send SNMP traps by default. Prerequisites Ensure that vCenter Server SNMP agents and ESXi SNMP agents are properly configured. Ensure that SNMP trap receiver agents are properly configured. Required Privilege: Alarms.Create alarm or Alarms.
Chapter 4 Monitoring Events, Alarms, and Automated Actions n VMWARE_ALARM_ALARMVALUE You can attach the script to any alarm on any object without changing the script. The script runs on the vCenter Server machine, and it runs even if you close the vSphere Web Client. Prerequisites Required Privilege: Alarms.Create alarm or Alarms.Modify alarm Procedure 1 On the Actions tab of the alarm definitions wizard, click Add to add an action.
vSphere Monitoring and Performance Table 4‑4. Alarm Environment Variables (Continued) Variable Name Variable Description Supported Alarm Type VMWARE_ALARM_TRIGGERINGSUMMARY A multiline summary of the alarm. Condition, State, Event VMWARE_ALARM_DECLARINGSUMMARY A single-line declaration of the alarm expression. Condition, State, Event VMWARE_ALARM_ALARMVALUE The value that triggered the alarm. Condition, State VMWARE_ALARM_EVENTDESCRIPTION A description of the alarm status change event.
Chapter 4 Monitoring Events, Alarms, and Automated Actions Table 4‑5. Command-Line Parameters for Alarm Action Scripts (Continued) Variable Description {oldStatus} The alarm status before the alarm is triggered. {newStatus} The alarm status after the alarm is triggered. {target} The inventory object on which the alarm is set. Acknowledge Triggered Alarms After you acknowledge an alarm in the vSphere Web Client, its alarm actions are discontinued.
vSphere Monitoring and Performance d Select the alarms you want to reset. Use Shift+left-click or Ctrl+left-click to select multiple alarms. e Right-click an alarm and select Reset to Green. Preconfigured vSphere Alarms vCenter Server provides a list of default alarms, which monitor the operations of vSphere inventory objects. You only need to set up actions for these alarms. Some alarms are stateless. vCenter Server does not keep data on stateless alarms and neither computes nor displays their status.
Chapter 4 Monitoring Events, Alarms, and Automated Actions Table 4‑6. Default vSphere Alarms (Continued) Alarm Name Description Virtual Machine Fault Tolerance vLockStep interval Status Changed Monitors changes in the Fault Tolerance Secondary vLockStep interval. Host processor status Monitors the host processors. Host memory status Monitors host memory usage. Host hardware fan status Monitors host fans. Host hardware voltage Monitors host hardware voltage.
vSphere Monitoring and Performance Table 4‑6. Default vSphere Alarms (Continued) 120 Alarm Name Description vSphere HA virtual machine monitoring action Monitors whether vSphere High Availability has restarted a virtual machine. vSphere HA virtual machine monitoring error Monitors whether vSphere High Availability failed to reset a virtual machine.
Chapter 4 Monitoring Events, Alarms, and Automated Actions Table 4‑6. Default vSphere Alarms (Continued) Alarm Name Description Appliance Management Health Alarm Monitors the health status of the Appliance Management Service. Inventory Health Alarm Monitors the health status of the Inventory Service. vCenter Server Health Alarm Monitors the health status of vCenter Server. Data Service Health Alarm Monitors the health status of the Data Service.
vSphere Monitoring and Performance Table 4‑6. Default vSphere Alarms (Continued) Alarm Name Description Migration error * Monitors whether a virtual machine cannot be migrated or relocated, or is orphaned. Exit standby error Monitors whether a host cannot exit standby mode. Table 4‑7. Deprecated vSphere Alarms 122 Alarm name Description Cannot connect to network Monitors the network connectivity on a virtual switch.
Monitoring Solutions with the vCenter Solutions Manager 5 In the vSphere Web Client, you can view an inventory of installed solutions, view detailed information about the solutions, and monitor the solution health status. A solution is an extension of vCenter Server that adds new functions to a vCenter Server instance. VMware products that integrate with vCenter Server are also considered solutions.
vSphere Monitoring and Performance 4 Select a solution. The Summary tab displays more information about the solution. 5 To view vService provider information, click Monitor, and click vServices. Monitoring Agents The vCenter Solutions Manager displays the vSphere ESX Agent Manager agents that you use to deploy and manage related agents on ESX/ESXi hosts. You can use the Solutions Manager to keep track of whether the agents of a solution are working as expected.
Monitoring the Health of Services and Nodes 6 You can monitor the health status of services and nodes to determine whether problems exist in your environment. The vSphere Web Client provides an overview of all services and nodes across the management stack of the vCenter Server system. A list of default services is available for each vCenter Server instance. View the Health Status of Services and Nodes In the vSphere Web Client, you can view the health status of vCenter Server services and nodes.
vSphere Monitoring and Performance 3 (Optional) In the Services Health and Nodes Health panes, click the hyperlink next to the health badge to view all services and nodes in this health state. For example, in the Services Health pane, click the hyperlink of the Warning health status, and in the dialog box that pops up, select a service to view more information about the service and attempt to resolve the health issues of the service. 126 VMware, Inc.
7 Performance Monitoring Utilities: resxtop and esxtop The resxtop and esxtop command-line utilities provide a detailed look at how ESXi uses resources in real time. You can start either utility in one of three modes: interactive (default), batch, or replay. The fundamental difference between resxtop and esxtop is that you can use resxtop remotely, whereas you can start esxtop only through the ESXi Shell of a local ESXi host.
vSphere Monitoring and Performance Using the resxtop Utility The resxtop utility is a vSphere CLI command. Before you can use any vSphere CLI commands, you must either download and install a vSphere CLI package or deploy the vSphere Management Assistant (vMA) to your ESXi host or vCenter Server system. After it is set up, start resxtop from the command line. For remote connections, you can connect to a host either directly or through vCenter Server.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑2. Interactive Mode Command-Line Options (Continued) Option Description server The name of the remote server host to connect to (required for resxtop only). vihost If you connect indirectly (through vCenter Server), this option should contain the name of the ESXi host you connect to. If you connect directly to the ESXi host, this option is not used.
vSphere Monitoring and Performance Table 7‑3. Interactive Mode Single-Key Commands Key Description h or ? Displays a help menu for the current panel, giving a brief summary of commands, and the status of secure mode. space Immediately updates the current panel. ^L Erases and redraws the current panel. f or F Displays a panel for adding or removing statistics columns (fields) to or from the current panel. o or O Displays a panel for changing the order of statistics columns on the current panel.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑4. CPU Panel Statistics Line Description PCPU USED(%) A PCPU refers to a physical hardware execution context. This can be a physical CPU core if hyperthreading is unavailable or disabled, or a logical CPU (LCPU or SMT thread) if hyperthreading is enabled.
vSphere Monitoring and Performance Table 7‑4. CPU Panel Statistics (Continued) 132 Line Description %RDY Percentage of time the resource pool, virtual machine, or world was ready to run, but was not provided CPU resources on which to execute. 100% = %RUN + %RDY + %CSTP + %WAIT %MLMTD (max limited) Percentage of time the ESXi VMkernel deliberately did not run the resource pool, virtual machine, or world because doing so would violate the resource pool, virtual machine, or world's limit setting.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑4. CPU Panel Statistics (Continued) Line Description %DMD CPU demand in percentage. It represents the average active CPU load in the past minute. CORE UTIL(%) Percentage of CPU cycles per core when at least one of the PCPUs in this core is unhalted, and its average over all cores. This statistic only appears when hyperthreading is enabled. In batch mode, the correspondening CORE UTIL(%) statistic is displayed for each PCPU.
vSphere Monitoring and Performance Table 7‑6. CPU Power Panel Statistics (Continued) Line Description %Px Percentage of time the PCPU spent in P-State 'x'. On systems with Processor Clocking Control, P-states are not directly visible to ESXi, so esxtop instead shows the percentage of time spent at full speed under the heading 'P0' and the percentage of time spent at any lower speed under 'P1'. %Tx Percentage of time the PCPU spent in T-State 'x'.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑7. Memory Panel Statistics (Continued) Field Description PSHARE (MB) Displays the ESXi page-sharing statistics. All numbers are in megabytes. shared Amount of physical memory that is being shared. common Amount of machine memory that is common across worlds. saving Amount of machine memory that is saved because of page sharing. shared = common + saving SWAP (MB) ZIP (MB) MEMCTL (MB) Displays the ESXi swap usage statistics.
vSphere Monitoring and Performance Table 7‑7. Memory Panel Statistics (Continued) 136 Field Description CNSM Amount of the memory currently consumed by the virtual machine. The memory currently consumed by the virtual machine is equal to the amount of memory that the VM guest operating system currently uses, excluding the amount of memory saved for sharing if memory sharing is enabled on the VM, excluding the amount of memory saved if some of the VM memory is compressed.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑7. Memory Panel Statistics (Continued) Field Description OVD_NDx (MB) VMM overhead memory allocated for a resource pool on NUMA node x. This statistic is applicable on NUMA systems only. TCHD_W (MB) Write working set estimate for resource pool. CACHESZ (MB) Compression memory cache size. CACHEUSD (MB) Used compression memory cache. ZIP/s (MB/s) Compressed memory per second. UNZIP/s (MB/s) Decompressed memory per second.
vSphere Monitoring and Performance Table 7‑9. Storage Adapter Panel Statistics (Continued) Column Description GAVG/rd Average guest operating system read latency per read operation, in milliseconds. QAVG/rd Average queue latency per read operation, in milliseconds. DAVG/wr Average device write latency per write operation, in milliseconds. KAVG/wr Average ESXi VMkernel write latency per write operation, in milliseconds.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑11. Storage Device Panel Statistics (Continued) Column Description WORLD World ID. This ID is visible only if the corresponding device is expanded to worlds. See the interactive command e in Table 7-12. The world statistics are per world per device. PARTITION Partition ID. This ID is visible only if the corresponding device is expanded to partitions. See interactive command t in Table 7-12. NPH Number of paths.
vSphere Monitoring and Performance Table 7‑11. Storage Device Panel Statistics (Continued) Column Description PAECP/s Number of PAE copies per second. This statistic applies to only paths. SPLTCMD/s Number of split commands per second. This statistic applies to only paths. SPLTCP/s Number of split copies per second. This statistic applies to only paths. The following table displays the interactive commands you can use with the storage device panel. Table 7‑12.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑13. Virtual Machine Storage Panel Statistics (Continued) Column Description LAT/rd Average latency (in milliseconds) per read. LAT/wr Average latency (in milliseconds) per write. The following table lists the interactive commands you can use with the virtual machine storage panel. Table 7‑14. Virtual Machine Storage Panel Interactive Commands Command Description e Expand or roll up storage VSCSI statistics.
vSphere Monitoring and Performance Table 7‑15. Network Panel Statistics (Continued) Column Description PKTTXBRD/s Number of broadcast packets transmitted per second. PKTRXBRD/s Number of broadcast packets received per second. The following table displays the interactive commands you can use with the network panel. Table 7‑16. Network Panel Interactive Commands Command Description T Sorts by Mb Tx column. R Sorts by Mb Rx column. t Sorts by Packets Tx column. r Sorts by Packets Rx column.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Use esxtop or resxtop in Batch Mode After you have prepared for batch mode, you can use esxtop or resxtop in this mode. Procedure 1 Start resxtop (or esxtop) to redirect the output to a file. For example: esxtop -b > my_file.csv The filename must have a .csv extension. The utility does not enforce this, but the post-processing tools require it. 2 Process statistics collected in batch mode using tools such as Microsoft Excel and Perfmon.
vSphere Monitoring and Performance Prepare for Replay Mode To run in replay mode, you must prepare for replay mode. Procedure 1 Run vm-support in snapshot mode in the ESXi Shell. Use the following command. vm-support -S -d duration -I interval 2 Unzip and untar the resulting tar file so that esxtop can use it in replay mode. You can now use esxtop in replay mode. Use esxtop in Replay Mode You can use esxtop in replay mode.
Using the vimtop Plug-In to Monitor the Resource Usage of Services 8 You can use the vimtop utility plug-in to monitor vSphere services that run in the vCenter Server Appliance. vimtop is a tool similar to esxtop, which runs in the environment of the vCenter Server Appliance. By using the text-based interface of vimtop in the appliance shell, you can view overall information about the vCenter Server Appliance, and a list of vSphere services and their resource usage.
vSphere Monitoring and Performance Table 8‑1. Interactive Mode Command-Line Options (Continued) Option Description -n number Sets the number of performed iterations before the vimtop exits interactive mode. vimtop updates the display number number of times and exits. The default value is 10000. -p / -d seconds Sets the update period in seconds. Interactive Mode Single-Key Commands for vimtop When running in interactive mode, vimtop recognizes several single-key commands.
Chapter 8 Using the vimtop Plug-In to Monitor the Resource Usage of Services Table 8‑2. Interactive Mode Single-Key Commands (Continued) Key Names Description x Return the column widths to their default values. + Expand selected item. - Collapse selected item. w Write the current setup to a vimtop configuration file. The default file name is the one specified by -c option, or /root/vimtop/vimtop.xml if the -c option is not used.
vSphere Monitoring and Performance 148 VMware, Inc.
Monitoring Networked Devices with SNMP and vSphere 9 Simple Network Management Protocol (SNMP) is commonly used by management programs to monitor a variety of networked devices. vSphere systems run SNMP agents, which can provide information to a management program in at least one of the following ways: n In response to a GET, GETBULK, or GETNEXT operation, which is a specific request for information from the management system.
vSphere Monitoring and Performance Configure SNMP Settings for vCenter Server If you plan to use SNMP with vCenter Server, you must use the vSphere Web Client to configure the SNMP settings. Prerequisites n Verify that the vSphere Web Client is connected to a vCenter Server instance. n Verify that you have the domain name or IP address of the SNMP receiver, the port number of the receiver, and the community string. Procedure 1 In the vSphere Web Client, navigate to a vCenter Server instance.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere n Configure the SNMP Agent for Polling on page 151 If you configure the ESXi SNMP agent for polling, it can listen for and respond to requests from SNMP management client systems, such as GET, GETNEXT and GETBULK requests. n Configure ESXi for SNMP v1 and v2c on page 152 When you configure the ESXi SNMP agent for SNMP v1 and v2c, the agent supports sending notifications and receiving GET requests.
vSphere Monitoring and Performance Configure ESXi for SNMP v1 and v2c When you configure the ESXi SNMP agent for SNMP v1 and v2c, the agent supports sending notifications and receiving GET requests. In SNMP v1 and v2c authentication is performed by using community strings. Community strings are namespaces which contain one or more managed objects. This form of authentication does not secure the communication between the SNMP agent and the management system.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere To configure SNMP v3 traps, see “Configure SNMP v3 Targets,” on page 156. If you run ESXCLI commands through vCLI, you must supply connection options that specify the target host and login credentials. If you use ESXCLI commands directly on a host using the ESXi Shell, you can use the commands as given without specifying connection options. For more information on connection options see vSphere Command-Line Interface Concepts and Examples.
vSphere Monitoring and Performance Configure the SNMP Engine ID Every SNMP v3 agent has an engine ID which serves as a unique identifier for the agent. The engine ID is used with a hashing function to generate keys for authentication and encryption of SNMP v3 messages. If you do not specify an engine ID, when you enable the SNMP agent, an engine ID is automatically generated. If you run ESXCLI commands through vCLI, you must supply connection options that specify the target host and login credentials.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere 2 (Optional) Run the esxcli system snmp set command with the --privacy option to configure privacy. For example, run the following command: esxcli system snmp set --privacy protocol Here, protocol must be either none (for no privacy) or AES128. Configure SNMP Users You can configure up to 5 users who can access SNMP v3 information. User names must be no more than 32 characters long.
vSphere Monitoring and Performance Parameter Description userid The user name. authhash The authentication hash value. privhash The privacy hash value. security The level of security enabled for that user, which can be auth (for authentication only), priv (for authentication and privacy), or none (for no authentication or privacy).
Chapter 9 Monitoring Networked Devices with SNMP and vSphere Procedure 1 (Optional) If you are configuring informs, configure the remote users by running the esxcli system snmp set command with the --remote-users option. For example, run the following command: esxcli system snmp set --remote-users userid/auth-protocol/auth-hash/priv-protocol/privhash/engine-id The command accepts the following parameters: 2 Parameter Description userid The user name.
vSphere Monitoring and Performance If you run ESXCLI commands through vCLI, you must supply connection options that specify the target host and login credentials. If you use ESXCLI commands directly on a host using the ESXi Shell, you can use the commands as given without specifying connection options. For more information on connection options see vSphere Command-Line Interface Concepts and Examples. Prerequisites Configure the ESXi SNMP agent by using the ESXCLI commands.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere Configure SNMP Management Client Software After you have configured a vCenter Server instance or an ESXi host to send traps, you must configure your management client software to receive and interpret those traps. To configure your management client software, specify the communities for the managed device, configure the port settings, and load the VMware MIB files.
vSphere Monitoring and Performance Procedure u Install the SNMP agents you normally would use for that purpose in the guest operating systems. VMware MIB Files VMware MIB files define the information provided by ESXi hosts and vCenter Server to SNMP management software. You can download these MIB files from http://communities.vmware.com/community/developer/forums/managementapi#SNMP-MIB.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere Table 9‑2. Other MIB Files (Continued) MIB File Description IEEE8021-BRIDGE-MIB Defines objects for managing devices that support IEEE 802.1D. IEEE8023-LAG-MIB Defines objects for managing devices that support IEEE 802.3ad link aggregation. IEEE8021-Q-BRIDGE-MIB Defines objects for managing Virtual Bridged Local Area Networks. IF-MIB.mib Defines attributes related to physical NICs on the host system. IP-MIB.
vSphere Monitoring and Performance Table 9‑3. Diagnostic Counters from SNMPv2-MIB (Continued) 162 Variable ID Mapping Description snmpEnableAuthenTraps snmp 30 Indicates whether the SNMP entity is permitted to generate authenticationFailure traps. The value of this object overrides any configuration information. It therefore provides a means of disabling all authenticationFailure traps.
System Log Files 10 In addition to lists of events and alarms, vSphere components generate assorted logs. These logs contain additional information about activities in your vSphere environment.
vSphere Monitoring and Performance 3 Press Enter or the spacebar to scroll through the messages. 4 (Optional) Perform a regular expression search. a Press the slash key (/). b Type the text to find. c Press Enter The found text is highlighted on the screen. 5 Press q to return to the direct console. External System Logs VMware technical support might request several files to help resolve any issues you have with the product.
Chapter 10 System Log Files n To manage diagnostic data: Global.Licenses Procedure 1 In the inventory, navigate to a vCenter Server instance and click the Monitor tab. 2 Click System Logs. 3 Click Export System Logs. 4 If you are connected to vCenter Server, select the object for which you want to export data. Selecting an object selects all of its child objects.
vSphere Monitoring and Performance n Query the syslog configuration to make sure that a valid syslog server has been configured, including the correct port. Configure Syslog on ESXi Hosts All ESXi hosts run a syslog service (vmsyslogd), which logs messages from the VMkernel and other system components to log files. You can use the vSphere Web Client or the esxcli system syslog vCLI command to configure the syslog service.
Chapter 10 System Log Files Configuring Logging Levels for the Guest Operating System Virtual machines can write support and troubleshooting information into a virtual machine log file stored on a VMFS volume. The default settings for virtual machines are appropriate for most situations. If your environment relies heavily on using VMotion, or if the defaults do not seem suitable for other reasons, you can modify the logging settings for virtual machine guest operating systems.
vSphere Monitoring and Performance Control When to Switch to New Virtual Machine Log Files The vmx.log.rotateSize parameter specifies the log file size at which the switch to new log files happens for the logs for individual virtual machines. Use this parameter together with the vmx.log.keepOld parameter to ensure acceptable log file sizes without losing critical logging information. The vmx.log.
Chapter 10 System Log Files 3 Select Verbose from the pop-up menu. 4 Click OK. Collect vSphere Log Files You can collect vSphere log files into a single location. Procedure u View the log file using one of the following methods. Task Action View the viclient-*.log file Change to the directory, %temp%. Download the log bundle from vSphere Web Client connected to a vCenter Server system To download the log bundle, do the following: a Select Administration > System Configuration.
vSphere Monitoring and Performance Component Location Purpose Shell log /var/log/shell.log Contains a record of all commands typed into the ESXi Shell as well as shell events (for example, when the shell was enabled). Authentication /var/log/auth.log Contains all events related to authentication for the local system. System messages /var/log/syslog.log Contains all general log messages and can be used for troubleshooting. This information was formerly located in the messages log file.
Chapter 10 System Log Files n logRegexp specifies a case-sensitive phrase with Python regular expression syntax to filter the log messages by their content. For example, if you want to set a limit of maximum two log entries from the hostd component for messages that resemble the SOCKET connect failed, error 2: No such file or directory phrase with any error number, add the following entry: 2 | hostd | SOCKET connect failed, error .
vSphere Monitoring and Performance 2 Click the Monitor tab. 3 Click Log Browser. 4 (Optional) If no logs for the host or vCenter Server are available, click Retrieve now to retrieve the logs for that object. The retrieved logs are based on a current snapshot of the system. Retrieving logs can take a few minutes. You can perform other tasks while the logs are being retrieved. 5 (Optional) Click Refresh to retrieve newer logs. 6 Select the type of log you want to browse.
Chapter 10 System Log Files 3 Enter the conditions you want included in the filter. 4 Type a filter name. 5 Click Save to save the filter. The filter is saved on the vSphere Web Client server and is available the next time you start the vSphere Web Client. 6 Click Filter to view the results in the Log Browser. What to do next You can load saved filters from your local system by clicking Save to local system. The filters are saved in XML format.
vSphere Monitoring and Performance 2 Select Actions > New Browser Window to open a window in the Log Browser. 3 In the Log Browser window, retrieve another log file to view. You can perform the same actions with the log file opened in the new window as you can with the original Log Browser window. Manage Logs Using the Log Browser From the Log Browser, you can update, remove, and see a list of available log file bundles.
Index A C agent manager 124 alarm actions described 105 email notifications 113 notification traps 112 running a script 112 setting up 112 alarm environment variables 116 alarms acknowledging triggered alarms 117 automatic refresh 108 commands 114 condition or state-based 111 create 109 definitions 107 described 105 general settings 110 live refresh 108 network 114 notification emails 112 resetting triggered event alarms 117 scripts 114, 115 send email as alarm action 113 set on inventory object 109 SMTP
vSphere Monitoring and Performance fault tolerance virtual machines 88 hosts 16, 35 resource pools 58 resource pools and virtual machines 16 vApps 64 virtual machines 64, 69 CPU used time, fault tolerance virtual machines 89 CPU,troubleshooting 95 create alarms 109 D data centers CPU usage (MHz), clusters 24 memory usage, clusters 25 data collection interval 12 data collection levels 13 data counter 10 datacenter charts 24 datacenters disk space by datastores 25 disk space by file type 26 datastore, troub
Index G guest operating systems logging levels 167 SNMP 159 guest operating statistics, enabling 101 H hardware health, reset sensors 104 health status, monitoring 104 host counters 75 host security, logging 165 host charts 34 host health, reset sensors 104 hosts CPU usage 34 CPU usage (MHz) 35 disk latency 31, 43 disk rate 30–34, 39, 42 disk size 40 disk usage 37 hardware monitoring 103 health status 104 memory balloon 47 memory data counters 49 memory swap rates 48 memory usage 46 memory usage, virtual
vSphere Monitoring and Performance monitoring, health status 125 N network SNMP traps 114 SNMP 149 troubleshooting 98 network rate hosts 54 virtual machines 82 network size hosts 55 virtual machines 83 network usage clusters 23 hosts 52 virtual machines 56, 81 networks, health monitoring 103 notification emails, alarms 112 notification traps, alarms 112 notifications, See traps O object identifiers (OIDs) 160 overview charts cluster charts 15 datacenter charts 24 datastore charts 27 fault tolerance chart
Index SNMP authentication 154 CIM indications 157 communities 152 configuring 149, 150 configuring for ESXi 150 configuring traps 152 diagnostics 159, 161 encryption 154 engine ID 154 filtering traps 158 GET 151 guest operating systems 159 IPMI sensors 157 management software 159 polling 151 ports 151 privacy 154 targets 156 traps 149 users 155 v3 traps 156 SNMP traps 114 SNMP v1 152 SNMP v2c 152 SNMP v3 153 solutions monitor 123 view 123 stacked charts, description 10 statistics about vCenter Server data
vSphere Monitoring and Performance vMA 128 VMkernel, logs 171 vmx.log.keepOld 168 vmx.log.rotateSize 168 vpxd, log files 171 vServices 123 vServices,monitoring 124 vSphere Client, logs 164 vSphere CLI 128 vSphere events 106 vSphere logs 107 vSphere Management Assistant 128 W watchdog, health monitoring 103 Windows, performance statistics 101 180 VMware, Inc.
