vSphere Resource Management Guide ESX 4.0 ESXi 4.0 vCenter Server 4.
vSphere Resource Management Guide You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com © 2006–2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws.
Contents About This Book 5 1 Getting Started with Resource Management 7 What Is Resource Management? 7 Configuring Resource Allocation Settings 8 Viewing Resource Allocation Information 11 Admission Control 14 2 Managing CPU Resources 15 CPU Virtualization Basics 15 Administering CPU Resources 16 3 Managing Memory Resources 23 Memory Virtualization Basics 23 Administering Memory Resources 26 4 Managing Resource Pools 35 Why Use Resource Pools? 36 Create Resource Pools 37 Add Virtual Machines to a Re
vSphere Resource Management Guide 7 Viewing DRS Cluster Information 65 Viewing the Cluster Summary Tab Using the DRS Tab 67 65 8 Using NUMA Systems with ESX/ESXi 71 What is NUMA? 71 How ESX/ESXi NUMA Scheduling Works 72 VMware NUMA Optimization Algorithms and Settings 73 Resource Management in NUMA Architectures 74 Specifying NUMA Controls 75 A Performance Monitoring Utilities: resxtop and esxtop 79 Using the esxtop Utility 79 Using the resxtop Utility 79 Using esxtop or resxtop in Interactive Mode 80
About This Book ® The vSphere Resource Management Guide describes resource management for vSphere environments.
vSphere Resource Management Guide Customers with appropriate support contracts should use telephone support for the fastest response on priority 1 issues. Go to http://www.vmware.com/support/phone_support.html. 6 Support Offerings To find out how VMware support offerings can help meet your business needs, go to http://www.vmware.com/support/services.
Getting Started with Resource Management 1 To understand resource management, you must be aware of its components, its goals, and how best to implement it in a cluster setting. Resource allocation settings for a virtual machine (shares, reservation, and limit) are discussed, including how to set them and how to view them. Also, admission control, the process whereby resource allocation settings are validated against existing resources is explained.
vSphere Resource Management Guide ® A cluster is a group of hosts. You can create a cluster using VMware vCenter Server, and add multiple hosts to the cluster. vCenter Server manages these hosts’ resources jointly: the cluster owns all of the CPU and memory of all hosts. You can enable the cluster for joint load balancing or failover. See Chapter 5, “Creating a DRS Cluster,” on page 43 for more information. Resource Consumers Virtual machines are resource consumers.
Chapter 1 Getting Started with Resource Management n Guarantee that a particular virtual machine is always allocated a higher percentage of the physical resources than other virtual machines. n Set an upper bound on the resources that can be allocated to a virtual machine. Resource Allocation Shares Shares specify the relative priority or importance of a virtual machine (or resource pool).
vSphere Resource Management Guide Resource Allocation Reservation A reservation specifies the guaranteed minimum allocation for a virtual machine. vCenter Server or ESX/ESXi allows you to power on a virtual machine only if there are enough unreserved resources to satisfy the reservation of the virtual machine. The server guarantees that amount even when the physical server is heavily loaded. The reservation is expressed in concrete units (megahertz or megabytes).
Chapter 1 Getting Started with Resource Management Changing Resource Allocation Settings—Example The following example illustrates how you can change resource allocation settings to improve virtual machine performance. Assume that on an ESX/ESXi host, you have created two new virtual machines—one each for your QA (VMQA) and Marketing (VM-Marketing) departments. Figure 1-1.
vSphere Resource Management Guide CPU Section The following information about CPU resource allocation is shown: Table 1-2. CPU Resource Allocation Field Description Total Capacity Guaranteed CPU allocation, in megahertz (MHz), reserved for this object. Reserved Capacity Number of megahertz (MHz) of the reserved allocation that this object is using. Available Capacity Number of megahertz (MHz) not reserved.
Chapter 1 Getting Started with Resource Management Virtual Machine Resource Allocation Tab A Resource Allocation tab is available when you select a virtual machine from the inventory panel. This Resource Allocation tab displays information about the CPU and memory resources for the selected virtual machine. CPU Section These bars display the following information about host CPU usage: Table 1-5. Host CPU Field Description Consumed Actual consumption of CPU resources by the virtual machine.
vSphere Resource Management Guide Table 1-9. Resource Settings Field Description Reservation Guaranteed memory allocation for this virtual machine. Limit Upper limit for this virtual machine’s memory allocation. Shares Memory shares for this virtual machine. Configured User-specified guest physical memory size.
Managing CPU Resources 2 ESX/ESXi hosts support CPU virtualization. When you utilize CPU virtualization, you should understand how it works, its different types, and processorspecific behavior. Also, you need to be aware of the performance implications of CPU virtualization.
vSphere Resource Management Guide Hardware-Assisted CPU Virtualization Certain processors (such as Intel VT and AMD SVM) provide hardware assistance for CPU virtualization. When using this assistance, the guest can use a separate mode of execution called guest mode. The guest code, whether application code or privileged code, runs in the guest mode. On certain events, the processor exits out of guest mode and enters root mode.
Chapter 2 Managing CPU Resources View Processor Information You can access information about current CPU configuration through the vSphere Client or using the vSphere SDK. Procedure 1 In the vSphere Client, select the host and click the Configuration tab. 2 Select Processors. You can view the information about the number and type of physical processors and the number of logical processors. NOTE In hyperthreaded systems, each hardware thread is a logical processor.
vSphere Resource Management Guide The ESX CPU scheduler can interpret processor topology, including the relationship between sockets, cores, and logical processors. The scheduler uses topology information to optimize the placement of virtual CPUs onto different sockets to maximize overall cache utilization, and to improve cache affinity by minimizing virtual CPU migrations. In undercommitted systems, the ESX CPU scheduler spreads load across all sockets by default.
Chapter 2 Managing CPU Resources If there is no work for a logical processor, it is put into a halted state, which frees its execution resources and allows the virtual machine running on the other logical processor on the same core to use the full execution resources of the core. The VMware scheduler properly accounts for this halt time, and charges a virtual machine running with the full resources of a core more than a virtual machine running on a half core.
vSphere Resource Management Guide Table 2-1. Hyperthreaded Core Sharing Modes Option Description Any The default for all virtual machines on a hyperthreaded system. The virtual CPUs of a virtual machine with this setting can freely share cores with other virtual CPUs from this or any other virtual machine at any time. None Virtual CPUs of a virtual machine should not share cores with each other or with virtual CPUs from other virtual machines.
Chapter 2 Managing CPU Resources In some cases, such as display-intensive workloads, significant communication might occur between the virtual CPUs and these other virtual machine threads. Performance might degrade if the virtual machine's affinity setting prevents these additional threads from being scheduled concurrently with the virtual machine's virtual CPUs (for example, a uniprocessor virtual machine with affinity to a single CPU, or a two-way SMP virtual machine with affinity to only two CPUs).
vSphere Resource Management Guide CPU Power Management To improve CPU power efficiency, you can configure your ESX/ESXi hosts to dynamically switch CPU frequencies based on workload demands. This type of power management is called Dynamic Voltage and Frequency Scaling (DVFS). It uses processor performance states (P-states) made available to the VMkernel through an ACPI interface. ESX/ESXi supports the Enhanced Intel SpeedStep and Enhanced AMD PowerNow! CPU power management technologies.
Managing Memory Resources 3 All modern operating systems provide support for virtual memory, allowing software to use more memory than the machine physically has. Similarly, the ESX/ESXi hypervisor provides support for overcommitting virtual machine memory, where the amount of guest memory configured for all virtual machines might be larger than the amount of physical host memory. If you intend to use memory virtualization, you should understand how ESX/ESXi hosts allocate, tax, and reclaim memory.
vSphere Resource Management Guide For example, consider a virtual machine with a configured size of 1GB. When the guest operating system boots, it detects that it is running on a dedicated machine with 1GB of physical memory. The actual amount of physical host memory allocated to the virtual machine depends on its memory resource settings and memory contention on the ESX/ESXi host. In some cases, the virtual machine might be allocated the full 1GB. In other cases, it might receive a smaller allocation.
Chapter 3 Managing Memory Resources The amount of memory saved by memory sharing depends on workload characteristics. A workload of many nearly identical virtual machines might free up more than thirty percent of memory, while a more diverse workload might result in savings of less than five percent of memory. Software-Based Memory Virtualization ESX/ESXi virtualizes guest physical memory by adding an extra level of address translation.
vSphere Resource Management Guide Figure 3-1. ESX/ESXi Memory Mapping virtual machine 1 virtual machine 2 a b c b guest virtual memory a b b c guest physical memory c machine memory a b b n The boxes represent pages, and the arrows show the different memory mappings. n The arrows from guest virtual memory to guest physical memory show the mapping maintained by the page tables in the guest operating system.
Chapter 3 Managing Memory Resources View Memory Allocation Information You can use the vSphere Client to view information about current memory allocations. You can view the information about the total memory and memory available to virtual machines. In ESX, you can also view memory assigned to the service console. Procedure 1 In the vSphere Client, select a host and click the Configuration tab. 2 Click Memory. You can view the information shown in “Host Memory Information,” on page 27.
vSphere Resource Management Guide ESX/ESXi memory virtualization adds little time overhead to memory accesses. Because the processor's paging hardware uses page tables (shadow page tables for software-based approach or nested page tables for hardware-assisted approach) directly, most memory accesses in the virtual machine can execute without address translation overhead. The memory space overhead has two components. n A fixed, system-wide overhead for the VMkernel and (for ESX only) the service console.
Chapter 3 Managing Memory Resources An ESX/ESXi host determines allocations for each virtual machine based on the number of shares allocated to it and an estimate of its recent working set size. n Shares — ESX/ESXi hosts use a modified proportional-share memory allocation policy. Memory shares entitle a virtual machine to a fraction of available physical memory.
vSphere Resource Management Guide Figure 3-2. Memory Ballooning in the Guest Operating System 1 memory 2 swap space memory 3 memory swap space NOTE You must configure the guest operating system with sufficient swap space. Some guest operating systems have additional limitations. If necessary, you can limit the amount of memory vmmemctl reclaims by setting the sched.mem.maxmemctl parameter for a specific virtual machine.
Chapter 3 Managing Memory Resources Swap File Location By default, the swap file is created in the same location as the virtual machine's configuration file. A swap file is created by the ESX/ESXi host when a virtual machine is powered on. If this file cannot be created, the virtual machine cannot power on. Instead of accepting the default, you can also: n Use per-virtual machine configuration options to change the datastore to another shared storage location.
vSphere Resource Management Guide If you are overcommitting memory with ESX/ESXi, to support the intra-guest swapping induced by ballooning, ensure that your guest operating systems also have sufficient swap space. This guest-level swap space must be greater than or equal to the difference between the virtual machine’s configured memory size and its Reservation.
Chapter 3 Managing Memory Resources To determine the effectiveness of memory sharing for a given workload, try running the workload, and use resxtop or esxtop to observe the actual savings. Find the information in the PSHARE field of the interactive mode in the Memory page. Measuring and Differentiating Types of Memory Usage The Performance tab of the vSphere Client displays a number of metrics that can be used to analyze memory usage.
vSphere Resource Management Guide A similar result is obtained when determining Memory Shared and Memory Shared Common for the ESX/ ESXi host. n Memory Shared for the host is the sum of each virtual machine's Memory Shared. Calculate this by looking at each virtual machine's guest physical memory and counting the number of blocks that have arrows to machine memory blocks that themselves have more than one arrow pointing at them.
Managing Resource Pools 4 A resource pool is a logical abstraction for flexible management of resources. Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources. Each standalone host and each DRS cluster has an (invisible) root resource pool that groups the resources of that host or cluster. The root resource pool is not displayed because the resources of the host (or cluster) and the root resource pool are always the same.
vSphere Resource Management Guide Why Use Resource Pools? Resource pools allow you to delegate control over resources of a host (or a cluster), but the benefits are evident when you use resource pools to compartmentalize all resources in a cluster. Create multiple resource pools as direct children of the host or cluster and configure them. You can then delegate control over the resource pools to other individuals or organizations. Using resource pools can result in the following benefits.
Chapter 4 Managing Resource Pools Create Resource Pools You can create a child resource pool of any ESX/ESXi host, resource pool, or DRS cluster. NOTE If a host has been added to a cluster, you cannot create child resource pools of that host. You can create child resource pools of the cluster if the cluster is enabled for DRS. When you create a child resource pool, you are prompted for resource pool attribute information.
vSphere Resource Management Guide Resource Pool Creation Example This procedure example demonstrates how you can create a resource pool with the ESX/ESXi host as the parent resource. Assume that you have an ESX/ESXi host that provides 6GHz of CPU and 3GB of memory that must be shared between your marketing and QA departments. You also want to share the resources unevenly, giving one department (QA) a higher priority.
Chapter 4 Managing Resource Pools n If the virtual machine has custom shares assigned, the share value is maintained. NOTE Because share allocations are relative to a resource pool, you might have to manually change a virtual machine’s shares when you move it into a resource pool so that the virtual machine’s shares are consistent with the relative values in the new resource pool. A warning appears if a virtual machine would receive a very large (or very small) percentage of total shares.
vSphere Resource Management Guide How Unreserved CPU and memory are computed and whether actions are performed depends on the Reservation Type. Table 4-2. Reservation Types Reservation Type Description Fixed The system checks whether the selected resource pool has sufficient unreserved resources. If it does, the action can be performed. If it does not, a message appears and the action cannot be performed.
Chapter 4 Managing Resource Pools Figure 4-3. Admission Control with Expandable Resource Pools: Successful Power-On 6GHz RP-MOM VM-M1, 1GHz 2GHz VM-K1, 2GHz RP-KID VM-K2, 2GHz Now, consider another scenario with VM-M1 and VM-M2 (shown in Figure 4-4): n Power on two virtual machines in RP-MOM with a total reservation of 3GHz. n You can still power on VM-K1 in RP-KID because 2GHz are available locally. n When you try to power on VM-K2, RP-KID has no unreserved CPU capacity so it checks its parent.
vSphere Resource Management Guide 42 VMware, Inc.
Creating a DRS Cluster 5 A DRS cluster is a collection of ESX/ESXi hosts and associated virtual machines with shared resources and a shared management interface. Before you can obtain the benefits of cluster-level resource management you must create a DRS cluster. When you add a host to a DRS cluster, the host’s resources become part of the cluster’s resources.
vSphere Resource Management Guide Admission Control and Initial Placement When you attempt to power on a single virtual machine or a group of virtual machines in a DRS-enabled cluster, vCenter Server performs admission control. It checks that there are enough resources in the cluster to support the virtual machine(s). If the cluster does not have sufficient resources to power on a single virtual machine, or any of the virtual machines in a group power-on attempt, a message appears.
Chapter 5 Creating a DRS Cluster Group Power-On Example The user selects three virtual machines in the same datacenter for a group power-on attempt. The first two virtual machines (VM1 and VM2) are in the same DRS cluster (Cluster1), while the third virtual machine (VM3) is on a standalone host. VM1 is in automatic mode and VM2 is in manual mode.
vSphere Resource Management Guide When a cluster becomes unbalanced, DRS makes recommendations or migrates virtual machines, depending on the default automation level: n If the cluster or any of the virtual machines involved are manual or partially automated, vCenter Server does not take automatic actions to balance resources.
Chapter 5 Creating a DRS Cluster n Satisfy DRS (affinity or anti-affinity) rule. n Host is entering maintenance mode or standby mode. NOTE If you are using the VMware Distributed Power Management feature, in addition to migration recommendations, DRS provides host power state recommendations. DRS Cluster Prerequisites Any host that is added to a DRS cluster must meet certain prerequisites to use cluster features successfully. Shared Storage Ensure that the managed hosts use shared storage.
vSphere Resource Management Guide vCenter Server provides features that help ensure that virtual machines migrated with VMotion meet processor compatibility requirements. These features include: n Enhanced VMotion Compatibility (EVC) – You can use EVC to help ensure VMotion compatibility for the hosts in a cluster. EVC ensures that all hosts in a cluster present the same CPU feature set to virtual machines, even if the actual CPUs on the hosts differ.
Chapter 5 Creating a DRS Cluster 5 Select a default automation level for DRS. Initial Placement Migration Manual Recommended host(s) is displayed. Migration recommendation is displayed. Partially Automated Automatic placement. Migration recommendation is displayed. Fully Automated Automatic placement. Migration recommendation is executed automatically. 6 Set the migration threshold for DRS. 7 Click Next. 8 Specify the default power management setting for the cluster.
vSphere Resource Management Guide Disable DRS You can turn off DRS for a cluster. When DRS is disabled, the cluster’s resource pool hierarchy and DRS rules (see “Using DRS Rules,” on page 51) are not reestablished when DRS is turned back on. So if you disable DRS, the resource pools are removed from the cluster. To avoid losing the resource pools, instead of disabling DRS, you should suspend it by changing the DRS automation level to manual (and disabling any virtual machine overrides).
Using DRS Clusters to Manage Resources 6 After you create a DRS cluster, you can customize it and use it to manage resources. To customize your DRS cluster and the resources it contains you can configure DRS rules and you can add and remove hosts and virtual machines. When a cluster’s settings and resources have been defined, you should ensure that it is and remains a valid cluster. You can also use a valid DRS cluster to manage power resources and interoperate with VMware HA.
vSphere Resource Management Guide Create DRS Rules You can create DRS rules to specify virtual machine affinity or anti-affinity. Procedure 1 Select the cluster in the vSphere Client inventory. 2 Right-Click and select Edit Settings. 3 In the left panel under VMware DRS select Rules. 4 Click Add. 5 In the Virtual Machine Rule dialog box, name the rule.
Chapter 6 Using DRS Clusters to Manage Resources Delete DRS Rules You can delete DRS rules. Procedure 1 Select the cluster in the vSphere Client inventory. 2 Select Edit Settings from the right-click menu. 3 In the left panel, select Rules under VMware DRS. 4 Select the rule you want to remove and click Remove. The rule is deleted.
vSphere Resource Management Guide Add an Unmanaged Host to a Cluster You can add an unmanaged host to a cluster. Such a host is not currently managed by the same vCenter Server system as the cluster and it is not visible in the vSphere Client. Procedure 1 Select the cluster to which to add the host and select Add Host from the right-click menu. 2 Enter the host name, user name, and password, and click Next. 3 View the summary information and click Next.
Chapter 6 Using DRS Clusters to Manage Resources Remove Hosts from a Cluster You can remove hosts from a cluster. Prerequisites Before you remove a host from a DRS cluster, consider the issues involved. n Resource Pool Hierarchies – When you remove a host from a cluster, the host retains only the root resource pool, even if you used a DRS cluster and decided to graft the host resource pool when you added the host to the cluster. In that case, the hierarchy remains with the cluster.
vSphere Resource Management Guide When no more running virtual machines are on the host, the host’s icon changes to include under maintenance and the host’s Summary panel indicates the new state. While in maintenance mode, the host does not allow you to deploy or power on a virtual machine.
Chapter 6 Using DRS Clusters to Manage Resources Nonexpandable resource pools Reservation minus reservation used. Expandable resource pools (Reservation minus reservation used) plus any unreserved resources that can be borrowed from its ancestor resource pools. Valid DRS Clusters A valid cluster has enough resources to meet all reservations and to support all running virtual machines.
vSphere Resource Management Guide Figure 6-2.
Chapter 6 Using DRS Clusters to Manage Resources Figure 6-3.
vSphere Resource Management Guide A cluster can also turn red if you reconfigure a resource pool while a virtual machine is failing over. A virtual machine that is failing over is disconnected and does not count toward the reservation used by the parent resource pool. You might reduce the reservation of the parent resource pool before the failover completes. After the failover is complete, the virtual machine resources are again charged to the parent resource pool.
Chapter 6 Using DRS Clusters to Manage Resources VMware DPM can use one of three power management protocols to bring a host out of standby mode: Intelligent Platform Management Interface (IPMI), Hewlett-Packard Integrated Lights-Out (iLO), or Wake-OnLAN (WOL). Each protocol requires its own hardware support and configuration. If a host does not support any of these protocols it cannot be put into standby mode by VMware DPM.
vSphere Resource Management Guide Test Wake-on-LAN for VMware DPM The use of Wake-on-LAN (WOL) for the VMware DPM feature is fully supported, if you configure and successfully test it according to the VMware guidelines. You must perform these steps before enabling VMware DPM for a cluster for the first time or on any host that is being added to a cluster that is using VMware DPM. Prerequisites Before testing WOL, ensure that your cluster meets the prerequisites.
Chapter 6 Using DRS Clusters to Manage Resources Automation Level Whether the host power state and migration recommendations generated by VMware DPM are executed automatically or not depends upon the power management automation level selected for the feature. The automation level is configured under Power Management in the cluster’s Settings dialog box. The options available are: n Off – The feature is disabled and no recommendations will be made.
vSphere Resource Management Guide After enabling and running VMware DPM, you can verify that it is functioning properly by viewing each host’s Last Time Exited Standby information displayed on the Host Options page in the cluster Settings dialog box and on the Hosts tab for each cluster. This field shows a timestamp and whether vCenter Server Succeeded or Failed the last time it attempted to bring the host out of standby mode. If no such attempt has been made, the field displays Never.
Viewing DRS Cluster Information 7 You can view information about a DRS cluster using the cluster Summary and DRS tabs in the vSphere Client. You can also apply the DRS recommendations that appear in the DRS tab. This chapter includes the following topics: n “Viewing the Cluster Summary Tab,” on page 65 n “Using the DRS Tab,” on page 67 Viewing the Cluster Summary Tab You can access a cluster’s Summary tab from the inventory panel of the vSphere Client.
vSphere Resource Management Guide Cluster Summary Tab VMware DRS Section The VMware DRS section appears in the cluster's Summary tab only if VMware DRS is enabled. Table 7-2. VMware DRS Section Field Description Migration Automation Level Manual, Partially Automated, Fully Automated. Power Management Automation Level Off, Manual, Automatic. DRS Recommendations Number of DRS migration recommendations awaiting user confirmation.
Chapter 7 Viewing DRS Cluster Information Using the DRS Tab The DRS tab is available when you select a DRS cluster object from the inventory panel in the vSphere Client. This tab displays information about the DRS recommendations made for the cluster, faults that have occurred in applying such recommendations, and the history of DRS actions. You can access three pages from this tab. These pages are named Recommendations, Faults, and History.
vSphere Resource Management Guide Table 7-4. DRS Recommendations Information Column Description Priority Priority level (1-5) for the recommendation. Priority one, the highest, indicates a mandatory move because of a host entering maintenance or standby mode or DRS rule violations. Other priority ratings denote how much the recommendation would improve the cluster’s performance; from priority two (significant improvement) to priority five (slight). Prior to ESX/ESXi 4.
Chapter 7 Viewing DRS Cluster Information DRS History Page The History page of the DRS tab displays recent actions taken as a result of DRS recommendations. You can reach this page by clicking the History button on the DRS tab. For each action, DRS provides the information shown in Table 7-6. Table 7-6. DRS History Page Field Description DRS Actions Details of the action taken. Time Timestamp of when the action occurred.
vSphere Resource Management Guide 70 VMware, Inc.
Using NUMA Systems with ESX/ESXi 8 ESX/ESXi supports memory access optimization for Intel and AMD Opteron processors in server architectures that support NUMA (non-uniform memory access). After you understand how ESX/ESXi NUMA scheduling is performed and how the VMware NUMA algorithms work, you can specify NUMA controls to optimize the performance of your virtual machines.
vSphere Resource Management Guide There are many disadvantages to using such an operating system on a NUMA platform. The high latency of remote memory accesses can leave the processors under-utilized, constantly waiting for data to be transferred to the local node, and the NUMA connection can become a bottleneck for applications with high-memory bandwidth demands. Furthermore, performance on such a system can be highly variable.
Chapter 8 Using NUMA Systems with ESX/ESXi VMware NUMA Optimization Algorithms and Settings This section describes the algorithms and settings used by ESX/ESXi to maximize application performance while still maintaining resource guarantees. Home Nodes and Initial Placement When a virtual machine is powered on, ESX/ESXi assigns it a home node. A virtual machine runs only on processors within its home node, and its newly allocated memory comes from the home node as well.
vSphere Resource Management Guide When a virtual machine moves to a new node, the ESX/ESXi host immediately begins to migrate its memory in this fashion. It manages the rate to avoid overtaxing the system, particularly when the virtual machine has little remote memory remaining or when the destination node has little free memory available. The memory migration algorithm also ensures that the ESX/ESXi host does not move memory needlessly if a virtual machine is moved to a new node for only a short period.
Chapter 8 Using NUMA Systems with ESX/ESXi The IBM Enterprise X-Architecture supports servers with up to four nodes (also called CECs or SMP Expansion Complexes in IBM terminology). Each node can contain up to four Intel Xeon MP processors for a total of 16 CPUs. The next generation IBM eServer x445 uses an enhanced version of the Enterprise X-Architecture, and scales to eight nodes with up to four Xeon MP processors for a total of 32 CPUs.
vSphere Resource Management Guide An administrator can also manually move a virtual machine to another node after the virtual machine has started running. In this case, the page migration rate of the virtual machine must be set manually, so that memory from the virtual machine’s previous node can be moved to its new node. Manual NUMA placement might interfere with the ESX/ESXi resource management algorithms, which try to give each virtual machine a fair share of the system’s processor resources.
Chapter 8 Using NUMA Systems with ESX/ESXi Example 8-1. Binding a Virtual Machine to a Single NUMA Node The following example illustrates manually binding the last four physical CPUs to a single NUMA node for a two-way virtual machine on an eight-way server. The CPUs—for example, 4, 5, 6, and 7—are the physical CPU numbers. 1 In the vSphere Client inventory panel, select the virtual machine and select Edit Settings. 2 Select Options and click Advanced. 3 Click the Configuration Parameters button.
vSphere Resource Management Guide 78 VMware, Inc.
A Performance Monitoring Utilities: resxtop and esxtop The resxtop and esxtop command-line utilities provide a detailed look at how ESX/ESXi uses resources in real time. You can start either utility in one of three modes: interactive (default), batch, or replay. The fundamental difference between resxtop and esxtop is that you can use resxtop remotely (or locally), whereas esxtop can be started only through the service console of a local ESX host.
vSphere Resource Management Guide After it is set up, start resxtop from the command line. For remote connections, you can either connect to an ESX/ESXi host either directly or through vCenter Server. The command-line options are the same as for esxtop (except for the R option) with additional connection options. NOTE resxtop does not use all the options shared by other vSphere CLI commands. Table A-1.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-2. Interactive Mode Command-Line Options (Continued) Option Description a Show all statistics. This option overrides configuration file setups and shows all statistics. The configuration file can be the default ~/.esxtop4rc configuration file or a user-defined configuration file. c Load a user-defined configuration file. If the -c option is not used, the default configuration filename is ~/.esxtop4rc.
vSphere Resource Management Guide Table A-3. Interactive Mode Single-Key Commands (Continued) Key Description # Prompts you for the number of statistics rows to display. Any value greater than 0 overrides automatic determination of the number of rows to show, which is based on window size measurement. If you change this number in one resxtop (or esxtop) panel, the change affects all four panels. s Prompts you for the delay between updates, in seconds.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-4. CPU Panel Statistics (Continued) Line Description CCPU(%) Percentages of total CPU time as reported by the ESX service console. This field is not displayed if you are using ESXi. n us — Percentage user time. n sy — Percentage system time. n id — Percentage idle time. n wa — Percentage wait time. n cs/sec — Context switches per second recorded by the service console.
vSphere Resource Management Guide Table A-4. CPU Panel Statistics (Continued) Line Description AFFINITY BIT MASK Bit mask showing the current scheduling affinity for the world. HTSHARING Current hyperthreading configuration. CPU The physical or logical processor on which the world was running when resxtop (or esxtop) obtained this information. HTQ Indicates whether the world is currently quarantined or not. N means no and Y means yes. TIMER/s Timer rate for this world.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-6. Memory Panel Statistics Field Description PMEM (MB) Displays the machine memory statistics for the server. All numbers are in megabytes. VMKMEM (MB) COSMEM (MB) total Total amount of machine memory in the server. cos Amount of machine memory allocated to the ESX service console. vmk Amount of machine memory being used by the ESX/ESXi VMkernel.
vSphere Resource Management Guide Table A-6. Memory Panel Statistics (Continued) Field Description SWAP (MB) Displays the ESX/ESXi swap usage statistics. All numbers are in megabytes. MEMCTL (MB) 86 curr Current swap usage. target Where the ESX/ESXi system expects the swap usage to be. r/s Rate at which memory is swapped in by the ESX/ESXi system from disk. w/s Rate at which memory is swapped to disk by the ESX/ESXi system. Displays the memory balloon statistics. All numbers are in megabytes.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-6. Memory Panel Statistics (Continued) Field Description SWTGT (MB) Target where the ESX/ESXi host expects the swap usage by the resource pool or virtual machine to be. SWR/s (MB) Rate at which the ESX/ESXi host swaps in memory from disk for the resource pool or virtual machine. SWW/s (MB) Rate at which the ESX/ESXi host swaps resource pool or virtual machine memory to disk. CPTRD (MB) Amount of data read from checkpoint file.
vSphere Resource Management Guide Table A-8. Storage Adapter Panel Statistics (Continued) Column Description NTGTS Number of targets. NLUNS Number of LUNs. NWDS Number of worlds. BLKSZ Block size in bytes. This statistic is applicable only to LUNs. AQLEN Storage adapter queue depth. Maximum number of ESX/ESXi VMkernel active commands that the adapter driver is configured to support. LQLEN LUN queue depth. Maximum number of ESX/ESXi VMkernel active commands that the LUN is allowed to have.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-9. Storage Adapter Panel Interactive Commands Command Description e Toggles whether storage adapter statistics are displayed expanded or unexpanded. Allows viewing storage resource utilization statistics broken down by individual channels belonging to an expanded storage adapter. You are prompted for the adapter name. P Toggles whether storage adapter statistics are displayed expanded or unexpanded.
vSphere Resource Management Guide Table A-10. Storage Device Panel Statistics (Continued) Column Description WQLEN World queue depth. This is the maximum number of ESX/ESXi VMkernel active commands that the world is allowed to have. This is a per device maximum for the world. It is valid only if the corresponding device is expanded to worlds. ACTV Number of commands in the ESX/ESXi VMkernel that are currently active. This statistic is applicable only to worlds and devices.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-11. Storage Device Panel Interactive Commands Command Description e Expand or roll up storage world statistics. This command allows you to view storage resource utilization statistics separated by individual worlds belonging to an expanded storage device. You are prompted for the device name. The statistics are per world per device. p Expand or roll up storage path statistics.
vSphere Resource Management Guide Table A-12. Virtual Machine Storage Panel Statistics (Continued) Column Description QUED Number of commands in the ESX/ESXi VMkernel that are currently queued. This number is applicable only to worlds and devices. %USD Percentage of queue depth used by ESX/ESXi VMkernel active commands. This number is applicable only to worlds and devices. LOAD Ratio of ESX/ESXi VMkernel active commands plus ESX/ESXi VMkernel queued commands to queue depth.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-13. Virtual Machine Storage Panel Interactive Commands (Continued) Command Description N Sort first by virtual machine column, and then by WORLD column. This is the default sort order. L Changes the displayed length of the NAME column. Network Panel The Network panel displays server-wide network utilization statistics. Statistics are arranged by port for each virtual network device configured.
vSphere Resource Management Guide Interrupt Panel The interrupt panel displays information about the use of interrupt vectors. Table A-16. Interrupt Panel Statistics Column Description VECTOR Interrupt vector ID. COUNT/s Total number of interrupts per second. This value is cumulative of the count for every CPU. COUNT_x Interrupts per second on CPU x. TIME/int Average processing time per interrupt (in microseconds). TIME_x Average processing time per interrupt on CPU x (in microseconds).
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-17. Command-Line Options in Batch Mode Option Description a Show all statistics. This option overrides configuration file setups and shows all statistics. The configuration file can be the default ~/.esxtop4rc configuration file or a user-defined configuration file. b Runs resxtop (or esxtop) in batch mode. c Load a user-defined configuration file.
vSphere Resource Management Guide Replay Mode Command-Line Options You can use replay mode with command-line options. Table A-18 lists the command-line options available for esxtop replay mode. Table A-18. Command-Line Options in Replay Mode 96 Option Description R Path to the vm-support collected snapshot’s directory. a Show all statistics. This option overrides configuration file setups and shows all statistics. The configuration file can be the default ~/.
Advanced Attributes B You can set advanced attributes for hosts or individual virtual machines to help you customize resource management. In most cases, adjusting the basic resource allocation settings (reservation, limit, shares) or accepting default settings results in appropriate resource allocation. However, you can use advanced attributes to customize resource management for a host or a specific virtual machine.
vSphere Resource Management Guide Advanced Memory Attributes You can use the advanced memory attributes to customize memory resource usage. Table B-2. Advanced Memory Attributes 98 Attribute Description Default Mem.CtlMaxPercent Limits the maximum amount of memory reclaimed from any virtual machine using vmmemctl, based on a percentage of its configured memory size. Specify 0 to disable reclamation using vmmemctl for all virtual machines. 65 Mem.
Appendix B Advanced Attributes Advanced NUMA Attributes You can use the advanced NUMA attributes to customize NUMA usage. Table B-3. Advanced NUMA Attributes Attribute Description Default Numa.RebalanceEnable Set this option to 0 to disable all NUMA rebalancing and initial placement of virtual machines, effectively disabling the NUMA scheduling system. 1 Numa.PageMigEnable If you set this option to 0, the system does not automatically migrate pages between nodes to improve memory locality.
vSphere Resource Management Guide 3 Click the Configuration Parameters button. 4 In the dialog box that appears, click Add Row to enter a new parameter and its value. Advanced Virtual Machine Attributes You can use the advanced virtual machine attributes to customize virtual machine configuration. Table B-4. Advanced Virtual Machine Attributes 100 Attribute Description sched.mem.maxmemctl Maximum amount of memory reclaimed from the selected virtual machine by ballooning, in megabytes (MB).
Index A D admission control CPU 21 resource pools 39 with expandable resource pools 40 advanced attributes CPU 97 hosts 97 memory 98 NUMA 99 virtual machines 99 alarms 64 AMD Opteron-based systems 47, 71, 74, 99 applications CPU-bound 16 single-threaded 16 automation modes, virtual machines 49 DPM and admission control 14 automation level 63 enabling 62 Last Time Exited Standby 63 monitoring 64 overrides 63 threshold 63 DRS disabling 50 fully automated 48 group power on 44 information 66 initial placeme
vSphere Resource Management Guide DRS rules creating 52 deleting 53 disabling 52 editing 52 DRS tab Faults page 68 History page 69 Recommendations page 67 using 67 DRS Troubleshooting Guide 68 dual-processor virtual machine 15 dynamic load balancing, NUMA 73 Dynamic Voltage and Frequency Scaling (DVFS) 22 E Enhanced VMotion Compatibility (EVC) 16, 47, 48, 65 entering maintenance mode 55 ESX/ESXi memory allocation 28 memory reclamation 29 esxtop batch mode 94 common statistics description 81 CPU panel 82 i
Index memory advanced attributes 98 balloon driver 29 managing allocation 23, 26 overcommitment 24, 31 overhead 23 overhead, understanding 27 reclaiming unused 29 service console 23 sharing 24 sharing across virtual machines 32 virtual machines 29 virtualization 23 memory affinity, NUMA nodes 76 memory idle tax 29 memory usage 33 memory virtualization hardware-assisted 25 software-based 25 migration recommendations 46 monitoring software 64 multicore processors 17 N NUMA advanced attributes 99 AMD Opteron
vSphere Resource Management Guide interactive mode command-line options 80 interactive mode single-key commands 81 interrupt panel 94 memory panel 84 network panel 93 options 79 order pages 81 performance monitoring 79 statistics column 81 storage adapter panel 87 storage device panel 89 virtual machine storage panel 91 root resource pool 35 S sched.mem.maxmemctl 29, 100 sched.mem.pshare.enable 100 sched.swap.dir 100 sched.swap.
