vSphere Monitoring and Performance Update 1 VMware vSphere 6.5 VMware ESXi 6.5 vCenter Server 6.5 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see http://www.vmware.com/support/pubs.
vSphere Monitoring and Performance You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com Copyright © 2010–2017 VMware, Inc. All rights reserved. Copyright and trademark information. VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com 2 VMware, Inc.
Contents About vSphere Monitoring and Performance 5 1 Monitoring Inventory Objects with Performance Charts 7 Performance Chart Types 8 Data Counters 8 Metric Groups in vSphere 9 Data Collection Intervals 10 Data Collection Levels 11 View Performance Charts 12 Performance Charts Options Available Under the View Menu Overview Performance Charts 13 Working with Advanced and Custom Charts 91 Troubleshoot and Enhance Performance 93 12 2 Monitoring Guest Operating System Performance 99 Enable Statistics Coll
vSphere Monitoring and Performance 6 Monitoring the Health of Services and Nodes 127 View the Health Status of Services and Nodes 127 7 Performance Monitoring Utilities: resxtop and esxtop 129 Using the esxtop Utility 129 Using the resxtop Utility 130 Using esxtop or resxtop in Interactive Mode Using Batch Mode 144 Using Replay Mode 145 130 8 Using the vimtop Plug-In to Monitor the Resource Use of Services 147 Monitor Services By Using vimtop in Interactive Mode 147 Interactive Mode Command-Line Options
About vSphere Monitoring and Performance VMware provides several tools to help you monitor your virtual environment and to locate the source of potential issues and current problems. Performance charts Allow you to see performance data on a variety of system resources including CPU, Memory, Storage, and so on. Performance monitoring command-line utilities Allow you to access detailed information on system performance through the command line.
vSphere Monitoring and Performance vSphere Web Client and vSphere Client Task instructions in this guide are based on the vSphere Web Client. You can also perform most of the tasks in this guide by using the new vSphere Client. The new vSphere Client user interface terminology, topology, and workflow are closely aligned with the same aspects and elements of the vSphere Web Client user interface. You can apply the vSphere Web Client instructions to the new vSphere Client unless otherwise instructed.
Monitoring Inventory Objects with Performance Charts 1 The vSphere statistics subsystem collects data on the resource usage of inventory objects. Data on a wide range of metrics is collected at frequent intervals, processed, and archived in the vCenter Server database. You can access statistical information through command-line monitoring utilities or by viewing performance charts in the vSphere Web Client.
vSphere Monitoring and Performance n “View Performance Charts,” on page 12 n “Performance Charts Options Available Under the View Menu,” on page 12 n “Overview Performance Charts,” on page 13 n “Working with Advanced and Custom Charts,” on page 91 n “Troubleshoot and Enhance Performance,” on page 93 Performance Chart Types Performance metrics are displayed in different types of charts, depending on the metric type and object. Table 1‑1.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑2. Data Counter Attributes (Continued) Attribute Statistics Type Description Measurement used during the statistics interval. Related to the unit of measurement. Rate – Value over the current statistics interval n Delta – Change from previous statistics interval. n Absolute – Absolute value (independent of the statistics interval). n Rollup Type Calculation method used during the statistics interval to roll up data.
vSphere Monitoring and Performance Table 1‑3. Metric Groups (Continued) Metric group Description Network Network utilization for both physical and virtual network interface controllers (NICs) and other network devices. The virtual switches that support connectivity among all components, such as hosts, virtual machines, VMkernel. Power Energy usage statistics per host. Storage Adapter Data traffic statistics per host bus adapter (HBA). Storage Path Data traffic statistics per path.
Chapter 1 Monitoring Inventory Objects with Performance Charts Data Collection Levels Each collection interval has a default collection level that determines the amount of data gathered and which counters are available for display in the charts. Collection levels are also referred to as statistics levels. Table 1‑5.
vSphere Monitoring and Performance View Performance Charts The vCenter Server statistics settings, the type of object selected, and the features that are enabled on the selected object determine the amount of information displayed in charts. Charts are organized into views. You can select a view to see related data together on one screen. You can also specify the time range, or data collection interval. The duration extends from the selected time range to the present time.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑6. Performance Chart Views by Inventory Object (Continued) Object View List Items Cluster n n n Host n n Resource Pool and vApps n Virtual Machine n n n n Home - CPU and memory charts for the cluster. Resource Pools & Virtual Machines - thumbnail charts for resource pools and virtual machines, and stacked charts for total CPU and memory usage in the cluster.
vSphere Monitoring and Performance Table 1‑7. Data Counters Chart Label Description Usage Sum of the average CPU usage values, in Megahertz, of all virtual machines in the cluster. n Counter: usagemhz n Stats Type: Rate n Unit: Megahertz (MHz) n Rollup Type: Average (Minimum/Maximum) n Total Collection Level: 1 (4) Total amount of CPU resources available in the cluster. The maximum value is equal to the number of cores multiplied by the frequency of the processors.
Chapter 1 Monitoring Inventory Objects with Performance Charts CPU Usage The cluster CPU Usage charts monitor the CPU utilization of the hosts, resource pools, and virtual machines in the cluster. This chart displays the 10 child objects in the cluster with the most CPU usage. This chart is located in the Resource Pools and Virtual Machines view of the Cluster Performance tab. Table 1‑9.
vSphere Monitoring and Performance Table 1‑11. Data Counters Chart Label Description host_name Average data I/O rate across all hosts in the cluster. Counter: usage n Stats Type: Rate n Unit: Kilobytes per second (KBps) n Rollup Type: Average (Minimum/Maximum) n n Collection Level: 1 (4) Chart Analysis Use the disk charts to monitor average disk loads and to determine trends in disk usage.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑12. Disk I/O Performance Enhancement Advice (Continued) # Resolution 8 For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with the system page file. It alleviates disk spindle contention during periods of high use. 9 On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtual machine's VMX file.
vSphere Monitoring and Performance Table 1‑14. Memory Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. 2 Verify that the balloon driver is enabled. The balloon driver is installed with VMware Tools and is critical to performance. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, it does not impact virtual machine performance.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑15. Data Counters (Continued) Chart Label Description Consumed Amount of machine memory used on the host. Consumed memory includes virtual machine memory, service console memory, and VMkernel memory.
vSphere Monitoring and Performance If the host has little free memory available, or if you notice a degradation in performance, consider taking the following actions. Table 1‑16. Memory Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical to performance. 2 Verify that the balloon driver is enabled.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑18. Memory Performance Enhancement Advice (Continued) # Resolution 3 If the balloon value is high, check the resource shares, reservations, and limits for the virtual machines and resource pools on the hosts. Verify that the host's settings are adequate and not lower than those set for the virtual machine.
vSphere Monitoring and Performance If packets are not being dropped and the data receive rate is slow, the host is probably lacking the CPU resources required to handle the load. Check the number of virtual machines assigned to each physical NIC. If necessary, perform load balancing by moving virtual machines to different vSwitches or by adding more NICs to the host. You can also move virtual machines to another host or increase the host CPU or virtual machine CPU.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑21. Data Counters Chart Label Description Amount of CPU currently in use by the cluster. The active CPU usage is approximately equal to the ratio of the used CPU cycles to the available CPU cycles. The maximum possible value is the frequency of the processors multiplied by the number of cores.
vSphere Monitoring and Performance Table 1‑23. Data Counters Chart Label Description Amount of host machine memory used by all powered on virtual machines in the cluster. n Counter: consumed n Stats Type: Absolute n Unit: MegaBytes (MB) n n Rollup Type: Average (Minimum/Maximum) Collection Level: 1 (4) Chart Analysis A cluster's consumed memory consists of virtual machine consumed memory and overhead memory.
Chapter 1 Monitoring Inventory Objects with Performance Charts Datastore Counters Table 1‑25. Data Counters File Type Description Virtual Disks Amount of disk space used by virtual disk files. Virtual disk files store the contents of the virtual machine's hard disk drive. It includes information that you write to a virtual machine's hard disk, such as the operating system, program files, and data files. The files have the extension .vmdk and appear as a physical disk drive to a guest operating system.
vSphere Monitoring and Performance If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the snapshots from the vSphere Web Client user interface. For information about consolidating the data center, see the vSphere documentation.
Chapter 1 Monitoring Inventory Objects with Performance Charts Space Utilization By File Type The Space Utilization by File Type chart displays space used by virtual disks, swap files, snapshot files, and other virtual machine files on the datastore or the datastore cluster. Note This chart does not show historical statistics. It only shows the most recently available data, which may be up to 30 minutes late, depending on when the last statistics rollup occurred.
vSphere Monitoring and Performance Table 1‑27. Data Counters (Continued) File Type Description Free Space Amount of disk space not currently in use. Total Space Amount of disk space available to the datastore. It defines the datastore capacity. The chart displays the information for datastores but not for data centers.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑29. Data Counters Chart Label Description virtual_machine Amount of datastore space used by the five virtual machines with the most used datastore space. n Counter: used n Stats Type: Absolute n Unit: Gigabytes (GB) n n Rollup Type: Latest Collection Level: 1 Chart Analysis The datastore is at full capacity when the used space is equal to the capacity.
vSphere Monitoring and Performance Table 1‑31. Data Counters Chart Label Description Storage I/O Control Aggregate IOPs Number of I/O operations per second on the datastore, aggregated across all hosts, and virtual machines accessing the datastore.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑34. Data Counters Chart Label Description Max Queue Depth per Host Maximum queue depth. Queue depth is the number of commands the SCSI driver queues to the HBA. n Counter: maxQueueDepth n Stats Type: Absolute n Unit: Number n n Rollup Type: Average Collection Level: 3 Read IOPs per Host This chart displays the per-host disk read rates for a datastore. The chart displays information about the ten hosts with the highest values.
vSphere Monitoring and Performance Table 1‑37. Data Counters Chart Label Description Average Read Latency per Virtual Machine Disk Latency measures the time used to process a SCSI command issued by the guest OS to the virtual machine. The kernel latency is the time VMkernel takes to process an I/O request. The device latency is the time it takes the hardware to handle the request. Total latency = kernelLatency + deviceLatency.
Chapter 1 Monitoring Inventory Objects with Performance Charts Write IOPs Per Virtual Machine Disk This chart displays the 10 virtual machines with the highest number of write operations. Data is not displayed when the virtual machine is powered off. This chart is located in the Performance view of the datastore Performance tab. The counter can also be displayed for datastore cluster charts. Table 1‑40.
vSphere Monitoring and Performance Table 1‑42. Data Counters Chart Label Description Usage Actively used CPU, as a percentage of the total available CPU, for each physical CPU on the host. Active CPU is approximately equal to the ratio of the used CPU to the available CPU. Available CPU = # of physical CPUs × clock rate. 100% represents all CPUs on the host. For example, if a four-CPU host is running a virtual machine with two CPUs, and the usage is 50%, the host is using two CPUs completely.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑44. Data Counters Chart Label Description Usage The sum, in megahertz, of the actively used CPU of all powered on virtual machines on a host. The maximum possible value is the frequency of the processors multiplied by the number of processors. For example, if you have a host with four 2GHz CPUs running a virtual machine that is using 4000MHz, the host is using two CPUs completely. 4000 ÷ (4 × 2000) = 0.
vSphere Monitoring and Performance Table 1‑46. Counters Name Description virtual_machine Amount of CPU actively being used by each virtual machine on the host. 100% represents all CPUs. For example, if a virtual machine has one virtual CPU that is running on a host with four CPUs and the CPU usage is 100%, the virtual machine is using one CPU resource.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑48. Data Counters Chart Label Description Usage Average data I/O rate across all LUNs on the host. Counter: usage n Stats Type: Rate n Unit: Kilobytes per second (KBps) n Rollup Type: Average (Minimum/Maximum) n n Collection Level: 1 (4) Chart Analysis Use the disk charts to monitor average disk loads and to determine trends in disk usage.
vSphere Monitoring and Performance Table 1‑49. Disk I/O Performance Enhancement Advice (Continued) # Resolution 8 For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with the system page file. It alleviates disk spindle contention during periods of high use. 9 On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtual machine's VMX file.
Chapter 1 Monitoring Inventory Objects with Performance Charts n The deviceLatency data counter measures the average amount of time, in milliseconds, to complete a SCSI command from the physical device. Depending on your hardware, a number greater than 15 ms indicates probable problems with the storage array. Move the active VMDK to a volume with more spindles or add disks to the LUN. n The queueLatency data counter measures the average amount of time taken per SCSI command in the VMkernel queue.
vSphere Monitoring and Performance Table 1‑52. Data Counters Chart Label Description Read Requests Number of disk read commands completed on each LUN on the host. The aggregate number of all disk read commands is also displayed in the chart. n Counter: numberRead n Stats Type: Absolute n Unit: Number n n Write Requests Rollup Type: Summation Collection Level: 3 Number of disk write commands completed on each LUN on the host.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑53. Disk I/O Performance Enhancement Advice (Continued) # Resolution 4 Use the vendor's array tools to determine the array performance statistics. When too many servers simultaneously access common elements on an array, the disks might have trouble keeping up. To increase throughput, consider array-side improvements. 5 Use Storage vMotion to migrate I/O-intensive virtual machines across multiple hosts.
vSphere Monitoring and Performance n The deviceLatency data counter measures the average amount of time, in milliseconds, to complete a SCSI command from the physical device. Depending on your hardware, a number greater than 15 ms indicates probable problems with the storage array. Move the active VMDK to a volume with more spindles or add disks to the LUN. n The queueLatency data counter measures the average amount of time taken per SCSI command in the VMkernel queue. This value must always be zero.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑56. Data Counters Chart Label Description Highest Disk Latency Highest latency value of all disks used by the host. Latency measures the time used to process a SCSI command issued by the guest OS to the virtual machine. The kernel latency is the time VMkernel takes to process an I/O request. The device latency is the time it takes the hardware to handle the request. Total latency = kernelLatency + deviceLatency.
vSphere Monitoring and Performance Table 1‑57. Disk I/O Performance Enhancement Advice (Continued) # Resolution 6 Balance the disk load across all physical resources available. Spread heavily used storage across LUNs that are accessed by different adapters. Use separate queues for each adapter to improve disk efficiency. 7 Configure the HBAs and RAID controllers for optimal use. Verify that the queue depths and cache settings on the RAID controllers are adequate.
Chapter 1 Monitoring Inventory Objects with Performance Charts n The queueLatency data counter measures the average amount of time taken per SCSI command in the VMkernel queue. This value must always be zero. If not, the workload is too high and the array cannot process the data fast enough. If the disk latency values are high, or if you notice other problems with disk I/O performance, consider taking the following actions. Table 1‑59.
vSphere Monitoring and Performance If the memory usage value is high, and the host has high ballooning or swapping, check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host cannot handle the demand for memory. It leads to memory reclamation, which might degrade performance. If the host has enough free memory, check the resource shares, reservation, and limit settings of the virtual machines and resource pools on the host.
Chapter 1 Monitoring Inventory Objects with Performance Charts If the memory usage value is high, and the host has high ballooning or swapping, check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host cannot handle the demand for memory. It leads to memory reclamation, which might degrade performance. If the host has enough free memory, check the resource shares, reservation, and limit settings of the virtual machines and resource pools on the host.
vSphere Monitoring and Performance However, a consistently high memory usage value (94% or greater) indicates the host does not have the memory resources required to meet the demand. If the memory balloon and swap values are not high, performance is probably not affected. If the memory usage value is high, and the host has high ballooning or swapping, check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host requires more memory resources.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑65. Data Counters (Continued) Chart Label Description Balloon Target Sum of the balloon target memory of all powered on virtual machines on the host. If the balloon target value is greater than the balloon value, the VMkernel inflates the balloon, causing more virtual machine memory to be reclaimed.
vSphere Monitoring and Performance Chart Analysis To ensure best performance, the host memory must be large enough to accommodate the active memory of the virtual machines. The active memory can be smaller than the virtual machine memory size. It allows you to over-provision memory, but still ensures that the virtual machine active memory is smaller than the host memory. Transient high-usage values usually do not cause performance degradation.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑67. Data Counters Chart Label Description Usage Amount of guest physical memory currently in use on the virtual machine. Counter: usage n Stats Type: Absolute n Unit: Percentage (%) n Rollup Type: Average (Minimum/Maximum) n n Collection Level: 1 (4) Chart Analysis A virtual machine's memory size must be slightly larger than the average guest memory usage.
vSphere Monitoring and Performance Table 1‑69. Host Counters Chart Label Description Usage Average rate at which data is transmitted and received across all NIC instances connected to the host. n Counter: usage n Stats Type: Rate n Unit: Megabits per second (Mbps) n n Rollup Type: Average (Minimum/Maximum) Collection Level: 1 (4) Chart Analysis Network performance depends on the application workload and network configuration. Dropped network packets indicate a bottleneck in the network.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑70. Networking Performance Enhancement Advice (Continued) # Resolution 9 Verify that all NICs are running in full duplex mode. Hardware connectivity problems might result in a NIC resetting itself to a lower speed or half duplex mode. 10 Use vNICs that are TCP Segmentation Offload (TSO)-capable, and verify that TSO-Jumbo Frames are enabled where possible.
vSphere Monitoring and Performance Table 1‑72. Networking Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. 2 If possible, use vmxnet3 NIC drivers, which are available with VMware Tools. They are optimized for high performance. 3 If virtual machines running on the same host communicate with each other, connect them to the same vSwitch to avoid transferring packets over the physical network.
Chapter 1 Monitoring Inventory Objects with Performance Charts If packets are being dropped, adjust the virtual machine shares. If packets are not being dropped, check the size of the network packets and the data receive and transfer rates. In general, the larger the network packets, the faster the network speed. When the packet size is large, fewer packets are transferred, which reduces the amount of CPU required to process the data.
vSphere Monitoring and Performance Table 1‑75. Data Counters Chart Label Description Sum of the data transmitted and received across all virtual NIC instances connected to the virtual machine. n Counter: usage n Stats Type: Rate n Unit: Megabits per second (Mbps) n n Rollup Type: Average (Minimum/Maximum) Collection Level: 1 (4) Chart Analysis Network performance depends on the application workload and network configuration.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑76. Networking Performance Enhancement Advice (Continued) # Resolution 9 Verify that all NICs are running in full duplex mode. Hardware connectivity problems might result in a NIC resetting itself to a lower speed or half duplex mode. 10 Use vNICs that are TCP Segmentation Offload (TSO)-capable, and verify that TSO-Jumbo Frames are enabled where possible.
vSphere Monitoring and Performance CPU Usage The CPU Usage chart displays CPU usage of virtual machines in the resource pool or vApp. The chart displays the top 10 virtual machines with the highest CPU usage. This chart is located in the Resource Pools & Virtual Machines view of the Resource Pool or vApp Performance tab. Table 1‑79. Data Counters Chart Label Description virtual_machine Amount of CPU actively used by virtual machines.
Chapter 1 Monitoring Inventory Objects with Performance Charts Memory (MB) The Memory (MB) chart displays memory usage in the resource pool or vApp. This chart is located in the Home view of the resource pool or vApp Performance tab. Table 1‑81. Data Counters Chart Label Description resource_pool or vApp Sum of the active memory used by all virtual machines in the resource pool or vApp. Active memory is determined by the VMkernel and includes overhead memory.
vSphere Monitoring and Performance Memory Consumed The Memory Consumed chart displays the memory performance of all virtual machines in the resource pool or vApp. This chart is located in the Resource Pools & Virtual Machines view of the resource pool or vApp Performance tab. For resource pools and virtual machines in a resource pool or vApp, this chart is located in the Resource Pools & Virtual Machines view of the resource pool or vApp Performance tab. Table 1‑83.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑84. Memory Performance Enhancement Advice (Continued) # Resolution 4 If the memory reservation of the virtual machine is set to a value much higher than its active memory, decrease the reservation setting so that the VMkernel can reclaim the idle memory for other virtual machines on the host. 5 Migrate one or more virtual machines to a host in a DRS cluster. 6 Add physical memory to the host.
vSphere Monitoring and Performance Table 1‑85. Data Counters (Continued) Chart Label Description Shared Amount of the guest physical memory shared with other virtual machines in the resource pool. Swapped Sum of the memory swapped by all powered on virtual machines in the resource pool.
Chapter 1 Monitoring Inventory Objects with Performance Charts vApps The vApp charts contain information about CPU and memory usage for vApps. The help topic for each chart contains information about the data counters displayed in that chart. The counters available are determined by the collection level set for vCenter Server. CPU (MHz) The CPU (MHz) chart displays CPU usage in the vApp or resource pool. This chart is located in the Home view of the vApp or resource pool Performance tab.
vSphere Monitoring and Performance Table 1‑89. Data Counters Chart Label Description virtual_machine Amount of CPU actively used by virtual machines. n Counter: usagemhz n Stats Type: Rate n Unit: Megahertz (MHz) n Rollup Type: Average (Minimum/Maximum) n Collection Level: 1 (4) Chart Analysis A short spike in CPU usage or CPU ready indicates that you are making the best use of the virtual machine resources.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑91. Data Counters Chart Label Description resource_pool or vApp Sum of the active memory used by all virtual machines in the resource pool or vApp. Active memory is determined by the VMkernel and includes overhead memory.
vSphere Monitoring and Performance Table 1‑93. Data Counters Chart Label Description virtual_machine Amount of host memory used by the virtual machine for its guest operating system's physical memory. Memory overhead is not included in consumed memory. consumed memory = memory granted - memory saved from page sharing For example, if a virtual machine has 100 MB of memory that is shared equally with three other virtual machines, its portion of the shared memory is 25 MB (100 MB ÷ 4 VMs).
Chapter 1 Monitoring Inventory Objects with Performance Charts Virtual Machines The virtual machine charts contain information about CPU, disk, memory, network, storage, and fault tolerance for virtual machines. The help topic for each chart contains information about the data counters displayed in that chart. The counters available are determined by the collection level set for vCenter Server. CPU (%) The CPU (%) chart displays virtual machine CPU usage and ready values.
vSphere Monitoring and Performance Table 1‑96. CPU Performance Enhancement Advice (Continued) # Resolution 5 Increase the CPU shares to give the virtual machine more opportunities to run. The total ready time on the host might remain at the same level if the host system is constrained by CPU. If the host ready time doesn't decrease, set the CPU reservations for high-priority virtual machines to guarantee that they receive the required CPU cycles.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑98. CPU Performance Enhancement Advice (Continued) # Resolution 5 Increase the CPU shares to give the virtual machine more opportunities to run. The total ready time on the host might remain at the same level if the host system is constrained by CPU. If the host ready time doesn't decrease, set the CPU reservations for high-priority virtual machines to guarantee that they receive the required CPU cycles.
vSphere Monitoring and Performance If the disk latency values are high, or if you notice other problems with disk I/O performance, consider taking the following actions. Table 1‑100. Disk I/O Performance Enhancement Advice # Resolution 1 Increase the virtual machine memory. It allows more operating system caching, which reduces I/O activity. Note: It might require you to increase the host memory.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑101. Data Counters Chart Label Description Read Number of disk read commands completed on each virtual disk on the virtual machine, per second. The aggregate number of all disk read commands per second is also displayed in the chart.
vSphere Monitoring and Performance Table 1‑102. Disk I/O Performance Enhancement Advice # Resolution 1 Increase the virtual machine memory. It allows more operating system caching, which reduces I/O activity. Note: It might require you to increase the host memory. Increasing memory might reduce the need to store data because databases can utilize the system memory to cache data and avoid disk access.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑103. Data Counters Chart Label Description Read Requests Number of disk read commands completed on each virtual disk on the virtual machine. The aggregate number of all disk read commands is also displayed in the chart. n Counter: numberRead n Stats Type: Absolute n n n Write Requests Unit: Number Rollup Type: Summation Collection Level: 3 Number of disk write commands completed on each virtual disk on the virtual machine.
vSphere Monitoring and Performance Table 1‑104. Disk I/O Performance Enhancement Advice (Continued) # Resolution 4 Use the vendor's array tools to determine the array performance statistics. When too many servers simultaneously access common elements on an array, the disks might have trouble keeping up. To increase throughput, consider array-side improvements. 5 Use Storage vMotion to migrate I/O-intensive virtual machines across multiple hosts.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑106. Data Counters Chart Label Description Read Requests Number of virtual disk read commands completed on each virtual disk on the virtual machine. The aggregate number of all virtual disk read commands per second is also displayed in the chart.
vSphere Monitoring and Performance If the host has enough free memory, check the resource shares, reservation, and limit of the virtual machines and resource pools on the host. Verify that the host settings are adequate and not lower than those set for the virtual machine. If little free memory is available, or if you notice degradation in performance, consider taking the following actions. Table 1‑108.
Chapter 1 Monitoring Inventory Objects with Performance Charts If little free memory is available, or if you notice degradation in performance, consider taking the following actions. Table 1‑110. Memory Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical to performance. 2 Verify that the balloon driver is enabled.
vSphere Monitoring and Performance Table 1‑112. Memory Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical to performance. 2 Verify that the balloon driver is enabled. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, this does not impact virtual machine performance.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑113. Data Counters (Continued) Chart Label Description Consumed Amount of guest physical memory consumed by the virtual machine for guest memory. Consumed memory does not include overhead memory. It includes shared memory and memory that might be reserved, but not actually used.
vSphere Monitoring and Performance Table 1‑114. Memory Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical to performance. 2 Verify that the balloon driver is enabled. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, this does not impact virtual machine performance.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑116. Networking Performance Enhancement Advice # Resolution 1 Verify that VMware Tools is installed on each virtual machine. 2 If possible, use vmxnet3 NIC drivers, which are available with VMware Tools. They are optimized for high performance. 3 If virtual machines running on the same host communicate with each other, connect them to the same vSwitch to avoid transferring packets over the physical network.
vSphere Monitoring and Performance If packets are being dropped, adjust the virtual machine shares. If packets are not being dropped, check the size of the network packets and the data receive and transfer rates. In general, the larger the network packets, the faster the network speed. When the packet size is large, fewer packets are transferred, which reduces the amount of CPU required to process the data.
Chapter 1 Monitoring Inventory Objects with Performance Charts Table 1‑119. Data Counters Chart Label Description Packets Transmitted Number of network packets transmitted across the top ten virtual NIC instances on the virtual machine. The chart also displays the aggregated value for each NIC.
vSphere Monitoring and Performance Table 1‑120. Networking Performance Enhancement Advice (Continued) # Resolution 8 Verify that the reported speed and duplex settings for the physical NIC match the hardware expectations and that the hardware is configured to run at its maximum capability. For example, verify that NICs with 1 Gbps are not reset to 100 Mbps because they are connected to an older switch. 9 Verify that all NICs are running in full duplex mode.
Chapter 1 Monitoring Inventory Objects with Performance Charts Space Utilization by Datastores The Space Utilization by Datastores chart displays the amount of space used by a virtual machine on different datastores in the data center. Note This chart does not show historical statistics. It only shows the most recently available data, which may be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statistics are not collected across all datastores at one time.
vSphere Monitoring and Performance Datastore counters Table 1‑123. Data Counters File Type Description Virtual Disks Amount of disk space used by virtual disk files. Virtual disk files store the contents of the virtual machine's hard disk drive, including information that you write to a virtual machine's hard disk - the operating system, program files, and data files. The files have the extension .vmdk and appear as a physical disk drive to a guest operating system.
Chapter 1 Monitoring Inventory Objects with Performance Charts Fault Tolerance Performance Counters The fault tolerance charts contain information about CPU and memory for fault tolerant virtual machines. Note The performance charts and help topics for fault tolerance are available only when you have vSphere Fault Tolerance enabled.
vSphere Monitoring and Performance Table 1‑126. Data Counters Chart Label Description System Amount of time spent on system processes on each virtual CPU in the virtual machine. Note This is the host view of the CPU usage, not the guest operating system view. n Counter: system n Stats Type: Delta n Unit: Percentage (%) n n Rollup Type: Summation Collection Level: 3 Chart Analysis A large discrepancy in CPU usage between the primary and secondary virtual machines might indicate performance problems.
Chapter 1 Monitoring Inventory Objects with Performance Charts Chart Analysis A large discrepancy in CPU usage between the primary and secondary virtual machines might indicate performance problems. The CPU ready, system, and wait times of each virtual machine should be synchronized. A large discrepancy in these values might indicate performance problems. Consider taking the following actions. Table 1‑129.
vSphere Monitoring and Performance If a virtual machine has high ballooning or swapping, check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host cannot meet the memory requirements. This leads to memory reclamation, which might degrade performance. If the active memory size is the same as the granted memory size, demand for memory is greater than the memory resources available.
Chapter 1 Monitoring Inventory Objects with Performance Charts If a virtual machine has high ballooning or swapping, check the amount of free physical memory on the host. The host might require more memory resources. If it does not, check the resource shares, reservation, and limit of the virtual machines and resource pools on the host. Verify that the host settings are adequate and not lower than those set for the virtual machine.
vSphere Monitoring and Performance Change Advanced Chart Settings You can customize a performance chart by specifying the objects to monitor, the counters to include, the time range, and chart type. You can customize preconfigured chart views and create chart views. Procedure 1 Select an inventory object in the vSphere Web Client. 2 Click the Monitor tab, and click Performance. 3 Click Advanced. 4 Click Chart Options. 5 In Chart Metrics, select a metric group for the chart.
Chapter 1 Monitoring Inventory Objects with Performance Charts 6 Click OK. The chart settings are saved and an entry for your chart is added to the View menu. Delete a Custom Advanced Chart View You can delete custom chart views from the vSphere Web Client. Procedure 1 Select an inventory object in the vSphere Web Client. 2 Click the Monitor tab, and click Performance. 3 Click Advanced. 4 Click Chart Options. 5 Select a chart and click Delete Options. 6 Click OK to confirm deletion.
vSphere Monitoring and Performance Solutions for Consistently High CPU Usage Temporary spikes in CPU usage indicate that you are making the best use of CPU resources. Consistently high CPU usage might indicate a problem. You can use the CPU performance charts to monitor CPU usage for hosts, clusters, resource pools, virtual machines, and vApps. Problem n Host CPU usage constantly is high. A high CPU usage value can lead to increased ready time and processor queuing of the virtual machines on the host.
Chapter 1 Monitoring Inventory Objects with Performance Charts Solutions for Memory Performance Problems Host machine memory is the hardware backing for guest virtual memory and guest physical memory. Host machine memory must be at least slightly larger than the combined active memory of the virtual machines on the host. A virtual machine's memory size must be slightly larger than the average guest memory usage. Increasing the virtual machine memory size results in more overhead memory usage.
vSphere Monitoring and Performance n You can provision more space to the datastore if possible, or you can add disks to the datastore or use shared datastores. Solutions for Disk Performance Problems Use the disk charts to monitor average disk loads and to determine trends in disk usage. For example, you might notice a performance degradation with applications that frequently read from and write to the hard disk.
Chapter 1 Monitoring Inventory Objects with Performance Charts n For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with the system page file. This alleviates disk spindle contention during periods of high use. n On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtual machine's VMX file. n If the combined disk I/O is higher than a single HBA capacity, use multipathing or multiple links.
vSphere Monitoring and Performance n Ensure that the physical NIC capacity is large enough to handle the network traffic on that virtual switch. If the capacity is not enough, consider using a high-bandwidth physical NIC (10Gbps) or moving some virtual machines to a virtual switch with a lighter load or to a new virtual switch. n If packets are being dropped at the virtual switch port, increase the virtual network driver ring buffers where applicable.
Monitoring Guest Operating System Performance 2 This section describes how to install and view VMware-specific performance data for virtual machines that run Microsoft Windows operating systems. VMware provides performance counters that enable you to view data on many aspects of guest operating system performance for the Microsoft Windows Perfmon utility.
vSphere Monitoring and Performance 5 In the Add Counters dialog box, select Use local computer counters. 6 Select a virtual machine performance object. Virtual machine performance object names begin with VM. 7 Select the counters that you want to display for that object. 8 If the performance object has multiple instances, select the instances you want to display. 9 Click Add. The Performance dialog box displays data for the selected performance object.
Monitoring Host Health Status 3 You can use the vSphere Web Client to monitor the state of host hardware components, such as CPU processors, memory, fans, and other components.
vSphere Monitoring and Performance Monitor Hardware Health Status You can monitor the health status of host hardware in the vSphere Web Client. Procedure 1 Select a host in the vSphere Web Client navigator. 2 Click the Monitor tab, and click Hardware Status. 3 Select the type of information to view. Option Description Sensors Displays all sensors arranged in a tree view. If the status is blank, the health monitoring service cannot determine the status of the component.
Monitoring Events, Alarms, and Automated Actions 4 vSphere includes a user-configurable events and alarms subsystem. This subsystem tracks events happening throughout vSphere and stores the data in log files and the vCenter Server database. This subsystem also enables you to specify the conditions under which alarms are triggered. Alarms can change state from mild warnings to more serious alerts as system conditions change, and can trigger automated alarm actions.
vSphere Monitoring and Performance n Actions - Defines operations that occur in response to triggered alarms. VMware provides sets of predefined actions that are specific to inventory object types. Alarms have the following severity levels: n Normal – green n Warning – yellow n Alert – red Alarm definitions are associated with the object selected in the inventory. An alarm monitors the type of inventory objects specified in its definition.
Chapter 4 Monitoring Events, Alarms, and Automated Actions View Events You can view events associated with a single object or view all vSphere events. The events list for a selected inventory object includes events associated with child objects. vSphere keeps information about tasks and events for 30 days. Procedure 1 Select an inventory object in the vSphere Web Client. 2 Click the Monitor tab, and click Events. 3 To see details, select an event.
vSphere Monitoring and Performance Streaming Events to a Remote Syslog Server You can stream to a remote syslog server the information about the events that your vSphere system generates. Streaming events is supported only for the vCenter Server Appliance. The streaming of events to a remote syslog server is disabled by default. You can enable and configure the streaming of vCenter Server events to a remote syslog server from the vCenter Server Appliance Management Interface.
Chapter 4 Monitoring Events, Alarms, and Automated Actions Procedure 1 In the vCenter Server Appliance Management Interface, select Syslog Configuration. 2 Click Edit. 3 From the Common Log Level drop-down menu select the log files to redirect. Option Description * All log files are redirected to the remote machine. info Only informational log files are redirected to the remote machine. notice Only notices are redirected to the remote machine. Notice indicates normal but significant condition.
vSphere Monitoring and Performance 6 Enable or disable the vpxd.event.syslog option. The default value for the setting is enabled. Retention of Events in the vCenter Server Database You can configure vCenter Server to retain events in the database for a limited period. Discarding events periodically ensures optimal performance of the database. In new installations of vCenter Server 6.
Chapter 4 Monitoring Events, Alarms, and Automated Actions View Triggered Alarms and Alarm Definitions Triggered alarms are visible in several locations throughout the vSphere Web Client. Procedure n To view all triggered alarms, click All in the Alarms sidebar panel. Note The list of alarms in the sidebar refreshes every 120 seconds. For information about changing the default refresh period, see the VMware knowledge base article at http://kb.vmware.com/kb/2020290.
vSphere Monitoring and Performance In an environment with multiple vCenter Server systems that are connected to the same vCenter Server Single-Sign On domain, the vSphere Web Client that you configured for live refresh displays recent tasks and alarms for all the vCenter Server instances in the domain. However, if you log in to a different vSphere Web Client, you will not see live refresh for recent tasks or alarms for any of the vCenter Server systems in the vCenter Server Single-Sign On domain.
Chapter 4 Monitoring Events, Alarms, and Automated Actions n Specify How a Condition-Based or State-Based Alarm is Triggered on page 112 You can select and configure the events, states, or conditions that trigger the alarm from the Triggers page of the alarm definition wizard. n Specify How an Event-Based Alarm is Triggered on page 112 You can specify the events, states, or conditions that trigger the alarm on the Triggers page of the alarm definition wizard.
vSphere Monitoring and Performance 4 Click the Enable this alarm check-box if you want to enable the alarm immediately after creation. 5 Click Next. What to do next Set alarm triggers. Specify How a Condition-Based or State-Based Alarm is Triggered You can select and configure the events, states, or conditions that trigger the alarm from the Triggers page of the alarm definition wizard.
Chapter 4 Monitoring Events, Alarms, and Automated Actions Procedure 1 Select the trigger that you want to change, or click the Add icon to add a trigger. 2 Click in the Event column, and select an option from the drop-down menu. 3 Click in the Status column, and select an option from the drop-down menu. 4 (Optional) Configure additional conditions to be met before the alarm triggers. a Click the Add icon to add an argument.
vSphere Monitoring and Performance 3 Click in the Configuration column, and enter configuration information for actions that require additional information: Option Action Send a notification email Type email addresses, separated by a comma. Migrate VM Complete the virtual machine migration wizard. Run a command Take one of the following actions and press Enter: n If the command is a .exe file, enter the full path name of the command and include any parameters. For example, to run the cmd.
Chapter 4 Monitoring Events, Alarms, and Automated Actions vCenter Server Email Agent Notifications The following tables describe the information that is included in Alarm-based and Event-based email notifications. The first table described the information included in all email notifications. The second table describes additional information that is included in Event-based notifications. Table 4‑1.
vSphere Monitoring and Performance SNMP Trap Notifications The following table describes the information that is included in vCenter Server and ESXi trap notifications. Table 4‑3. SNMP Trap Notification Details Trap Entry Description Type The state vCenter Server is monitoring for the alarm. Options include Host Processor (or CPU) usage, Host Memory usage, Host State, Virtual Machine Processor (or CPU) usage, Virtual Machine Memory usage, Virtual Machine State, Virtual Machine Heartbeat.
Chapter 4 Monitoring Events, Alarms, and Automated Actions 3 In the Configuration column, type script or command information: For this type of command... Enter this... EXE executable files Full pathname of the command. For example, to run the cmd.exe command in the C:\tools directory, type: c:\tools\cmd.exe. BAT batch file Full pathname of the command as an argument to the c:\windows\system32\cmd.exe command. For example, to run the cmd.bat command in the C:\tools directory, type: c:\windows\system32
vSphere Monitoring and Performance Table 4‑4. Alarm Environment Variables (Continued) Variable Name Variable Description Supported Alarm Type VMWARE_ALARM_EVENT_USERNAME The user name associated with the event. Event VMWARE_ALARM_EVENT_DATACENTER The name of the data center in which the event occurred. Event VMWARE_ALARM_EVENT_COMPUTERESOURCE The name of the cluster or resource pool in which the event occurred. Event VMWARE_ALARM_EVENT_HOST The name of the host on which the event occurred.
Chapter 4 Monitoring Events, Alarms, and Automated Actions Acknowledge Triggered Alarms After you acknowledge an alarm in the vSphere Web Client, its alarm actions are discontinued. Alarms are not cleared, or reset when acknowledged. Acknowledging an alarm lets other users know that you are taking ownership of the issue. For example, a host has an alarm set to monitor CPU usage. It sends an email to an administrator when the alarm is triggered.
vSphere Monitoring and Performance Preconfigured vSphere Alarms vCenter Server provides a list of default alarms, which monitor the operations of vSphere inventory objects. You must only set up actions for these alarms. Some alarms are stateless. vCenter Server does not keep data on stateless alarms, does not compute, or display their status. Stateless alarms cannot be acknowledged or reset. Stateless alarms are indicated by an asterisk next to their name. Table 4‑6.
Chapter 4 Monitoring Events, Alarms, and Automated Actions Table 4‑6. Default vSphere Alarms (Continued) Alarm Name Description Host hardware fan status Monitors host fans. Host hardware voltage Monitors host hardware voltage. Host hardware temperature status Monitors the temperature status of the host hardware. Host hardware power status Monitors the host power status. Host hardware system board status Monitors the status of host system boards.
vSphere Monitoring and Performance Table 4‑6. Default vSphere Alarms (Continued) 122 Alarm Name Description vSphere HA VM Component Protection could not power off a virtual machine Monitors whether vSphere High Availability VM Component Protection cannot power off a virtual machine with an inaccessible datastore. License error * Monitors license errors. Health status changed * Monitors changes to service and extension health status.
Chapter 4 Monitoring Events, Alarms, and Automated Actions Table 4‑6. Default vSphere Alarms (Continued) Alarm Name Description vCenter Server Health Alarm Monitors the health status of vCenter Server. Database Health Alarm Monitors the database health status. When database space reaches 80%, vCenter Server displays a warning event. When database space reaches 95%, vCenter Server displays an error event and shuts down.
vSphere Monitoring and Performance Table 4‑6. Default vSphere Alarms (Continued) Alarm Name Description Network uplink redundancy degraded * Monitors network uplink redundancy degradation on a virtual switch. VMKernel NIC not configured correctly * Monitors incorrectly configured VMkernel NICs. Cannot connect to storage * Monitors host connectivity to a storage device. Migration error * Monitors whether a virtual machine cannot be migrated or relocated, or is orphaned.
Monitoring Solutions with the vCenter Solutions Manager 5 In the vSphere Web Client, you can view an inventory of installed solutions, view detailed information about the solutions, and monitor the solution health status. A solution is an extension of vCenter Server that adds new functions to a vCenter Server instance. VMware products that integrate with vCenter Server are also considered solutions.
vSphere Monitoring and Performance 4 Select a solution. The Summary tab displays more information about the solution. 5 To view vService provider information, click Monitor, and click vServices. Monitoring Agents The vCenter Solutions Manager displays the vSphere ESX Agent Manager agents that you use to deploy and manage related agents on ESX/ESXi hosts. You can use the Solutions Manager to keep track of whether the agents of a solution are working as expected.
Monitoring the Health of Services and Nodes 6 You can monitor the health status of services and nodes to determine whether problems exist in your environment. The vSphere Web Client provides an overview of all services and nodes across the management stack of the vCenter Server system. A list of default services is available for each vCenter Server instance. View the Health Status of Services and Nodes In the vSphere Web Client, you can view the health status of vCenter Server services and nodes.
vSphere Monitoring and Performance 3 (Optional) In the Services Health and Nodes Health panes, click the hyperlink next to the health badge to view all services and nodes in this health state. For example, in the Services Health pane, click the hyperlink of the Warning health status. In the dialog box that pops up, select a service to view more information about the service and attempt to resolve the health issues of the service. 128 VMware, Inc.
7 Performance Monitoring Utilities: resxtop and esxtop The resxtop and esxtop command-line utilities provide a detailed look at how ESXi uses resources in real time. You can start either utility in one of three modes: interactive (default), batch, or replay. The fundamental difference between resxtop and esxtop is that you can use resxtop remotely, whereas you can start esxtop only through the ESXi Shell of a local ESXi host.
vSphere Monitoring and Performance Using the resxtop Utility The resxtop utility is a vSphere CLI command. Before you can use any vSphere CLI commands, you must either download and install a vSphere CLI package or deploy the vSphere Management Assistant (vMA) to your ESXi host or vCenter Server system. After it is set up, start resxtop from the command line. For remote connections, you can connect to a host either directly or through vCenter Server.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑2. Interactive Mode Command-Line Options (Continued) Option Description server The name of the remote server host to connect to (required for resxtop only). vihost If you connect indirectly (through vCenter Server), this option should contain the name of the ESXi host you connect to. If you connect directly to the ESXi host, this option is not used.
vSphere Monitoring and Performance Table 7‑3. Interactive Mode Single-Key Commands Key Description h or? Displays a help menu for the current panel, giving a brief summary of commands, and the status of secure mode. space Immediately updates the current panel. ^L Erases and redraws the current panel. f or F Displays a panel for adding or removing statistics columns (text boxes) to or from the current panel.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑4. CPU Panel Statistics Line Description PCPU USED(%) A PCPU refers to a physical hardware execution context. It can be a physical CPU core if the hyperthreading is unavailable or disabled, or a logical CPU (LCPU or SMT thread) if the hyperthreading is enabled.
vSphere Monitoring and Performance Table 7‑4. CPU Panel Statistics (Continued) 134 Line Description %RDY Percentage of time the resource pool, virtual machine, or world was ready to run, but was not provided CPU resources on which to execute. 100% = %RUN + %RDY + %CSTP + %WAIT %MLMTD (max limited) Percentage of time the ESXi VMkernel deliberately did not run the resource pool, virtual machine, or world because doing so would violate the resource pool, virtual machine, or world's limit setting.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑4. CPU Panel Statistics (Continued) Line Description %DMD CPU demand in percentage. It represents the average active CPU load in the past minute. CORE UTIL(%) Percentage of CPU cycles per core when at least one of the PCPUs in this core is unhalted, and its average over all cores. This statistic only appears when hyperthreading is enabled. In batch mode, the corresponding CORE UTIL(%) statistic is displayed for each PCPU.
vSphere Monitoring and Performance Table 7‑6. CPU Power Panel Statistics (Continued) Line Description %Px Percentage of time the physical CPU spent in P-State 'x'. On systems with Processor Clocking Control, P-states are not directly visible to ESXi. Theesxtop shows the percentage of time spent at full speed under the heading 'P0' and the percentage of time spent at any lower speed under 'P1'. %Tx Percentage of time the physical CPU spent in T-State 'x'.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑7. Memory Panel Statistics (Continued) Field Description PSHARE (MB) Displays the ESXi page-sharing statistics. All numbers are in megabytes. shared Amount of the physical memory that is being shared. common Amount of the machine memory that is common across worlds. saving Amount of the machine memory that is saved because of page sharing.
vSphere Monitoring and Performance Table 7‑7. Memory Panel Statistics (Continued) 138 Field Description CNSM Amount of the memory currently consumed by the virtual machine. The memory currently consumed by the virtual machine is equal to the amount of memory that the VM guest operating system currently uses, excluding the amount of memory saved for sharing if memory sharing is enabled on the VM, excluding the amount of memory saved if some of the VM memory is compressed.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑7. Memory Panel Statistics (Continued) Field Description OVD_NDx (MB) VMM overhead memory allocated for a resource pool on NUMA node x. This statistic is applicable on NUMA systems only. TCHD_W (MB) Write working set estimate for resource pool. CACHESZ (MB) Compression memory cache size. CACHEUSD (MB) Used compression memory cache. ZIP/s (MB/s) Compressed memory per second. UNZIP/s (MB/s) Decompressed memory per second.
vSphere Monitoring and Performance Table 7‑9. Storage Adapter Panel Statistics (Continued) Column Description GAVG/rd Average guest operating system read latency per read operation, in milliseconds. QAVG/rd Average queue latency per read operation, in milliseconds. DAVG/wr Average device write latency per write operation, in milliseconds. KAVG/wr Average ESXi VMkernel write latency per write operation, in milliseconds.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑11. Storage Device Panel Statistics (Continued) Column Description WORLD World ID. This ID is visible only if the corresponding device is expanded to worlds. See the interactive command e in Table 7-12. The world statistics are per world per device. PARTITION Partition ID. This ID is visible only if the corresponding device is expanded to partitions. See interactive command t in Table 7-12. NPH Number of paths.
vSphere Monitoring and Performance Table 7‑11. Storage Device Panel Statistics (Continued) Column Description PAECP/s Number of PAE copies per second. This statistic applies to only paths. SPLTCMD/s Number of split commands per second. This statistic applies to only paths. SPLTCP/s Number of split copies per second. This statistic applies to only paths. The following table displays the interactive commands you can use with the storage device panel. Table 7‑12.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Table 7‑13. Virtual Machine Storage Panel Statistics (Continued) Column Description LAT/rd Average latency (in milliseconds) per read. LAT/wr Average latency (in milliseconds) per write. The following table lists the interactive commands you can use with the virtual machine storage panel. Table 7‑14. Virtual Machine Storage Panel Interactive Commands Command Description e Expand or roll up storage VSCSI statistics.
vSphere Monitoring and Performance Table 7‑15. Network Panel Statistics (Continued) Column Description PKTTXBRD/s Number of broadcast packets transmitted per second. PKTRXBRD/s Number of broadcast packets received per second. The following table displays the interactive commands you can use with the network panel. Table 7‑16. Network Panel Interactive Commands Command Description T Sorts by Mb Tx column. R Sorts by Mb Rx column. t Sorts by Packets Tx column. r Sorts by Packets Rx column.
Chapter 7 Performance Monitoring Utilities: resxtop and esxtop Use esxtop or resxtop in Batch Mode After you have prepared for batch mode, you can use esxtop or resxtop in this mode. Procedure 1 Start resxtop (or esxtop) to redirect the output to a file. For example: esxtop -b > my_file.csv The filename must have a .csv extension. The utility does not enforce this, but the post-processing tools require it. 2 Process statistics collected in batch mode using tools such as Microsoft Excel and Perfmon.
vSphere Monitoring and Performance Prepare for Replay Mode To run in replay mode, you must prepare for replay mode. Procedure 1 Run vm-support in snapshot mode in the ESXi Shell. Use the following command. vm-support -S -d duration -I interval 2 Unzip and untar the resulting tar file so that esxtop can use it in replay mode. You can now use esxtop in replay mode. Use esxtop in Replay Mode You can use esxtop in replay mode.
Using the vimtop Plug-In to Monitor the Resource Use of Services 8 You can use the vimtop utility plug-in to monitor vSphere services that run in the vCenter Server Appliance. vimtop is a tool similar to esxtop, which runs in the environment of the vCenter Server Appliance. By using the text-based interface of vimtop in the appliance shell, you can view overall information about the vCenter Server Appliance, and a list of vSphere services and their resource use.
vSphere Monitoring and Performance Table 8‑1. Interactive Mode Command-Line Options (Continued) Option Description -n number Sets the number of performed iterations before the vimtop exits interactive mode. vimtop updates the display number number of times and exits. The default value is 10000. -p / -d seconds Sets the update period in seconds. Interactive Mode Single-Key Commands for vimtop When running in interactive mode, vimtop recognizes several single-key commands.
Chapter 8 Using the vimtop Plug-In to Monitor the Resource Use of Services Table 8‑2. Interactive Mode Single-Key Commands (Continued) Key Names Description x Return the column widths to their default values. + Expand selected item. - Collapse selected item. w Write the current setup to a vimtop configuration file. The default file name is the one specified by -c option, or /root/vimtop/vimtop.xml if the -c option is not used.
vSphere Monitoring and Performance 150 VMware, Inc.
Monitoring Networked Devices with SNMP and vSphere 9 Simple Network Management Protocol (SNMP) is commonly used by management programs to monitor a variety of networked devices. vSphere systems run SNMP agents, which can provide information to a management program in at least one of the following ways: n In response to a GET, GETBULK, or GETNEXT operation, which is a specific request for information from the management system.
vSphere Monitoring and Performance Configure SNMP Settings for vCenter Server If you plan to use SNMP with vCenter Server, you must use the vSphere Web Client to configure the SNMP settings. Prerequisites n Verify that the vSphere Web Client is connected to a vCenter Server instance. n Verify that you have the domain name or IP address of the SNMP receiver, the port number of the receiver, and the community string. Procedure 1 In the vSphere Web Client, navigate to a vCenter Server instance.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere As an alternative to configuring SNMP manually using esxcli commands, you can use host profiles to configure SNMP for an ESXi host. See the vSphere Host Profiles documentation for more information. Note For information on configuring SNMP for ESXi 5.0 or earlier or ESX 4.1 or earlier, see the documentation for the appropriate product version.
vSphere Monitoring and Performance Here, port is the port the SNMP agent uses to listen for polling requests. Note The port you specify must not be already in use by other services. Use IP addresses from the dynamic range, port 49152 and up.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere Configure the SNMP Agent to Send SNMP v1 or v2c Notifications You can use the ESXi SNMP agent to send virtual machine and environmental notifications to management systems. To send SNMP v1/v2c notifications with the SNMP agent, you must configure the target (receiver) unicast address, community, and an optional port. If you do not specify a port, the SNMP agent sends traps to UDP port 162 on the target management system by default.
vSphere Monitoring and Performance 2 Configure SNMP Authentication and Privacy Protocols on page 156 SNMPv3 optionally supports authentication and privacy protocols. 3 Configure SNMP Users on page 157 You can configure up to 5 users who can access SNMP v3 information. User names must be no more than 32 characters long. 4 Configure SNMP v3 Targets on page 158 Configure SNMP v3 targets to allow the ESXi SNMP agent to send SNMP v3 traps and informs.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere Procedure 1 (Optional) Run the esxcli system snmp set command with the --authentication option to configure authentication. For example, run the following command: esxcli system snmp set --authentication protocol Here, protocol must be either none (for no authentication), SHA1, or MD5. 2 (Optional) Run the esxcli system snmp set command with the --privacy option to configure privacy.
vSphere Monitoring and Performance The produced output might be the following: Authhash: 08248c6eb8b333e75a29ca0af06b224faa7d22d6 Privhash: 232ba5cbe8c55b8f979455d3c9ca8b48812adb97 The authentication and privacy hash values are displayed. 2 Configure the user by running the esxcli system snmp set command with the --users flag.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere n If you are configuring informs, you need the engine ID for the SNMP agent on the remote system that receives the inform message. n Configure the ESXi SNMP agent by using the ESXCLI commands. See Getting Started with vSphere Command-Line Interfaces for more information on how to use ESXCLI.
vSphere Monitoring and Performance Configure the Source of Hardware Events Received by the SNMP Agent You can configure the ESXi SNMP agent to receive hardware events either from IPMI sensors or CIM indications. IPMI sensors are used for hardware monitoring in ESX/ESXi 4.x and earlier. The conversion of CIM indications to SNMP notifications is available in ESXi 5.0 and later. If you run ESXCLI commands through vCLI, you must supply connection options that specify the target host and login credentials.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere Configure SNMP Management Client Software After you have configured a vCenter Server instance or an ESXi host to send traps, configure your management client software to receive and interpret those traps. To configure your management client software, specify the communities for the managed device, configure the port settings, and load the VMware MIB files. See the documentation for your management system for specific instructions for these steps.
vSphere Monitoring and Performance Monitor Guest Operating Systems with SNMP You can use SNMP to monitor guest operating systems or applications running in virtual machines. The virtual machine uses its own virtual hardware devices. Do not install agents in the virtual machines that are intended to monitor physical hardware. Procedure u Install the SNMP agents you normally would use for that purpose in the guest operating systems.
Chapter 9 Monitoring Networked Devices with SNMP and vSphere Table 9‑2. Other MIB Files MIB File Description ENTITY-MIB.mib Allows the description of relationships between physical entities and logical entities managed by the same SNMP agent. See RFC 4133 for more information. HOST-RESOURCES-MIB.mib Defines objects that are useful for managing host computers. HOST-RESOURCES-TYPES.mib Defines storage, device, and filesystem types for use with HOST-RESOURCES-MIB.mib.
vSphere Monitoring and Performance Table 9‑3. Diagnostic Counters from SNMPv2-MIB (Continued) 164 Variable ID Mapping Description snmpInBadCommunityUses snmp 5 The total number of community-based SNMP messages delivered to the SNMP entity that represented an SNMP operation that was not allowed for the community named in the message. snmpInASNParseErrs snmp 6 The total number of ASN.1 or BER errors encountered by the SNMP entity when decoding received SNMP messages.
System Log Files 10 In addition to lists of events and alarms, vSphere components generate assorted logs. These logs contain additional information about activities in your vSphere environment.
vSphere Monitoring and Performance 2 Press a corresponding number key to view a log. vCenter Server agent (vpxa) logs appear if the host is managed by vCenter Server. 3 Press Enter or the spacebar to scroll through the messages. 4 (Optional) Perform a regular expression search. a Press the slash key (/). b Type the text to find. c Press Enter The found text is highlighted on the screen. 5 Press q to return to the direct console.
Chapter 10 System Log Files Table 10‑2. vSphere Web Client Logs and Location on a vCenter Server That Runs on Windows (Continued) File Path Log File Description C:\ProgramData\VMware\vCenterServer\logs\vspher e-client\logs\virgo-server\ log.log ??? Contains information about the plug-in bundle communication. C:\ProgramData\VMware\vCenterServer\logs\vspher e-ui\logs eventlog.log Contains time stamps of startup and shutdown of the vSphere Web Client service. C:\ProgramData\VMware\vCenterServer\logs\
vSphere Monitoring and Performance 8 (Optional) Select to apply a password for encrypted core dumps to the support package. You can make that password available to your support representative in a secure channel. If only some of the host in your environment use encryption, some of the files in the package are encrypted. 9 Click Finish. 10 Specify the location to which to save the log files. The host or vCenter Server generates a .zip file containing the log files. 11 Click Save.
Chapter 10 System Log Files Procedure 1 In the vSphere Web Client, navigate to Administration. 2 Click Support, and click Upload File to Service Request. An Upload File to Service Request dialog box opens. 3 Enter your Service Request ID with VMware. 4 Click Choose File, and select the log bundle you want to attach to your service request with VMware, and click OK. 5 If you protected your support package with a password, provide the password to VMware Technical Support by using a secure channel.
vSphere Monitoring and Performance 6 7 (Optional) To overwrite the default log size and log rotation for any of the logs. a Click the name of the log that you want to customize. b Click Edit and enter the number of rotations and the log size you want. Click OK. Changes to the syslog options take effect immediately. Configuring Logging Levels for the Guest Operating System Virtual machines can write support and troubleshooting information into a virtual machine log file stored on a VMFS volume.
Chapter 10 System Log Files 3 Select VM Options. 4 Click Advanced and click Edit Configuration. 5 Add or edit the vmx.log.keepOld parameter to the number of files to keep for this virtual machine. For example, to keep 20 log files and begin deleting the oldest files as new ones are created, enter 20. 6 Click OK. Control When to Switch to New Virtual Machine Log Files The vmx.log.
vSphere Monitoring and Performance Collecting Log Files VMware technical support might request several files to help resolve technical issues. The following sections describe script processes for generating and collecting some of these files. Set Verbose Logging You can set the verbose log file specification You can only set verbose logging for vpxd logs. Procedure 1 Select Administration > vCenter Server Settings. 2 Select Logging Options. 3 Select Verbose from the pop-up menu. 4 Click OK.
Chapter 10 System Log Files ESXi Log File Locations ESXi records host activity in log files, using a syslog facility. Component Location Purpose VMkernel /var/log/vmkernel.log Records activities related to virtual machines and ESXi. VMkernel warnings /var/log/vmkwarning.log Records activities related to virtual machines. VMkernel summary /var/log/vmksummary.log Used to determine uptime and availability statistics for ESXi (comma separated). ESXi host agent log /var/log/hostd.
vSphere Monitoring and Performance Procedure 1 Log in to the ESXi Shell as root. 2 In the /etc/vmware/logfilters file, add the following entry to create a log filter. numLogs | ident | logRegexp where: n numLogs sets the maximum number of log entries for the specified log messages. After reaching this number, the specified log messages are filtered and ignored. Use 0 to filter and ignore all the specified log messages.
Chapter 10 System Log Files Viewing Log Files with the Log Browser Log browser is a plug-in to the vSphere Web Client, and is part of the vCenter Server installation package. To use the log browser, you must deploy the log browser plug-in. If you deploy the log browser plug-in, you can view, search, and export one or more vCenter Server and ESXi log files at a time using the log browser. You can also export, manage, and view different log types.
vSphere Monitoring and Performance 4 From the vSphere Web Client, restart the VMware Service Lifecycle Manager API service. For more information how to restart services in the vSphere Web Client, see vCenter Server and Host Management documentation. The vCenter Server instance restarts. The Log Browser tab appears under the Monitor tab in the vSphere Web Client.
Chapter 10 System Log Files 3 Type the text that you want filtered in the search box. When displaying adjacent lines, groups of consecutive entries are highlighted with a different background color. The log browser displays the lines in the log that contain the word you typed, with the number of adjacent (before and after) lines. Create Advanced Log Filters You can create and save log file filters to your local system. Procedure 1 Navigate to the Log Browser and retrieve a log file object.
vSphere Monitoring and Performance 3 Select the type of file that you want to export. Selecting the log file bundle downloads a potentially large file. 4 Click Export. After a few seconds a new web browser displays. 5 Enter the location where you want to save the file. The log file is downloaded to your local machine and you can close the new browser window. Compare Log Files You can open multiple windows in the Log Browser to compare log files.
Index A C agent manager 126 alarm actions described 103 email notifications 115 notification traps 113 running a script 113 setting up 113 alarm environment variables 118 alarms acknowledging triggered alarms 119 automatic refresh 109 commands 116 condition or state-based 112 create 110 definitions 109 described 103 general settings 111 live refresh 109 network 115, 116 notification emails 113 resetting triggered event alarms 119 scripts 116, 117 send email as alarm action 114 set on inventory object 110
vSphere Monitoring and Performance resource pools 57 resource pools and virtual machines 15 vApps 63 virtual machines 63, 68 CPU usage (%) hosts 33 resource pools, virtual machines 58 virtual machines 35, 67 CPU used time, fault tolerance virtual machines 88 CPU,troubleshooting 94 create alarms 110 D data centers CPU usage (MHz), clusters 22 memory usage, clusters 23 data collection interval 10 data collection levels 11 data counter 8 database limiting size 108 retention policy 108 datacenter charts 22 da
Index F fans, monitoring 101 fault tolerance charts 87 fault tolerance virtual machines CPU system time 87 CPU usage (MHz) 87 CPU used time 88 memory active 89 memory swapout 90 filtering traps, SNMP 160 G guest operating systems logging levels 170 SNMP 162 guest operating statistics, enabling 99 H hardware health, reset sensors 102 health status, monitoring 102 host counters 74 host security, logging 168 host charts 33 host health, reset sensors 102 hosts CPU usage 33 CPU usage (MHz) 34 disk latency 30,
vSphere Monitoring and Performance resource pools 61 virtual machines 78 memory swap rates hosts 47 virtual machines 77 memory swapout, fault tolerance virtual machines 90 memory, troubleshooting 95 metric groups 7 metric groups,description 9 MIB files 162 monitor, guest operating system performance 99 monitoring health status, services, nodes 127 performance charts 7 virtual environment 5 monitoring, health status 127 N network SNMP traps 115, 116 SNMP 151 troubleshooting 97 network size hosts 54 virtual
Index interactive mode command-line options 130 interactive mode single-key commands 131 interrupt panel 144 memory panel 136 network panel 143 options 130 order pages 131 performance monitoring 129 statistics column 131 storage adapter panel 139 storage device panel 140 virtual machine storage panel 142 S scripts, alarms 116, 117 services monitoring in interactive mode 147 syslogd 169 SIOC activity report 30 SMASH 101 SMTP, configuring 114 SNMP authentication 156 CIM indications 160 communities 154 confi
vSphere Monitoring and Performance vimtop command-line options 147 interactive mode single-key commands 148 overview 147 using 147 virtual machine counters 74 virtual machine charts 67 virtual machines CPU usage (%) 35, 67 CPU usage (MHz) 68 disk rate 70 disk size 72 disk space by datastores 85 disk space by file type 85 disk space data counters 84 disk usage 69 memory balloon 76 memory data counters 78 memory swap rates 77 memory usage 75 network usage 80 network rate 81 network size 82 performance 99 vMA
