vSphere Resource Management Update 1 VMware vSphere 6.5 VMware ESXi 6.5 vCenter Server 6.5 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see http://www.vmware.com/support/pubs.
vSphere Resource Management You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com Copyright © 2006–2017 VMware, Inc. All rights reserved. Copyright and trademark information. VMware, Inc. 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com 2 VMware, Inc.
Contents About vSphere Resource Management 7 1 Getting Started with Resource Management 9 Resource Types 9 Resource Providers 9 Resource Consumers 10 Goals of Resource Management 10 2 Configuring Resource Allocation Settings 11 Resource Allocation Shares 11 Resource Allocation Reservation 12 Resource Allocation Limit 12 Resource Allocation Settings Suggestions 13 Edit Resource Settings 13 Changing Resource Allocation Settings—Example Admission Control 15 14 3 CPU Virtualization Basics 17 Software-B
vSphere Resource Management Memory Reclamation 35 Using Swap Files 36 Sharing Memory Across Virtual Machines 40 Memory Compression 41 Measuring and Differentiating Types of Memory Usage 42 Memory Reliability 43 About System Swap 43 7 Configuring Virtual Graphics 45 View GPU Statistics 45 Add an NVIDIA GRID vGPU to a Virtual Machine Configuring Host Graphics 46 Configuring Graphics Devices 47 45 8 Managing Storage I/O Resources 49 About Virtual Machine Storage Policies 50 About I/O Filters 50 Storage I
Contents Removing a Host from a Cluster 76 DRS Cluster Validity 77 Managing Power Resources 82 Using DRS Affinity Rules 86 12 Creating a Datastore Cluster 91 Initial Placement and Ongoing Balancing 92 Storage Migration Recommendations 92 Create a Datastore Cluster 92 Enable and Disable Storage DRS 93 Set the Automation Level for Datastore Clusters 93 Setting the Aggressiveness Level for Storage DRS 94 Datastore Cluster Requirements 95 Adding and Removing Datastores from a Datastore Cluster 96 13 Using
vSphere Resource Management Host has Insufficient Capacity for Each Virtual Machine CPU 125 The Virtual Machine Is in vMotion 125 No Active Host in Cluster 125 Insufficient Resources 125 Insufficient Resources to Satisfy Configured Failover Level for HA No Compatible Hard Affinity Host 125 No Compatible Soft Affinity Host 125 Soft Rule Violation Correction Disallowed 125 Soft Rule Violation Correction Impact 126 125 17 DRS Troubleshooting Information 127 Cluster Problems 127 Host Problems 130 Virtual M
About vSphere Resource Management ® ® vSphere Resource Management describes resource management for VMware ESXi and vCenter Server environments. This documentation focuses on the following topics.
vSphere Resource Management 8 VMware, Inc.
Getting Started with Resource Management 1 To understand resource management, you must be aware of its components, its goals, and how best to implement it in a cluster setting. Resource allocation settings for a virtual machine (shares, reservation, and limit) are discussed, including how to set them and how to view them. Also, admission control, the process whereby resource allocation settings are validated against existing resources is explained.
vSphere Resource Management Resource Consumers Virtual machines are resource consumers. The default resource settings assigned during creation work well for most machines. You can later edit the virtual machine settings to allocate a share-based percentage of the total CPU, memory, and storage I/O of the resource provider or a guaranteed reservation of CPU and memory.
Configuring Resource Allocation Settings 2 When available resource capacity does not meet the demands of the resource consumers (and virtualization overhead), administrators might need to customize the amount of resources that are allocated to virtual machines or to the resource pools in which they reside. Use the resource allocation settings (shares, reservation, and limit) to determine the amount of CPU, memory, and storage resources provided for a virtual machine.
vSphere Resource Management The following table shows the default CPU and memory share values for a virtual machine. For resource pools, the default CPU and memory share values are the same, but must be multiplied as if the resource pool were a virtual machine with four virtual CPUs and 16 GB of memory. Table 2‑1. Share Values Setting CPU share values Memory share values High 2000 shares per virtual CPU 20 shares per megabyte of configured virtual machine memory.
Chapter 2 Configuring Resource Allocation Settings In most cases, it is not necessary to specify a limit. There are benefits and drawbacks: n Benefits — Assigning a limit is useful if you start with a small number of virtual machines and want to manage user expectations. Performance deteriorates as you add more virtual machines. You can simulate having fewer resources available by specifying a limit. n Drawbacks — You might waste idle resources if you specify a limit.
vSphere Resource Management 4 5 Edit the Memory Resources. Option Description Shares Memory shares for this resource pool with respect to the parent’s total. Sibling resource pools share resources according to their relative share values bounded by the reservation and limit. Select Low, Normal, or High, which specify share values respectively in a 1:2:4 ratio. Select Custom to give each virtual machine a specific number of shares, which expresses a proportional weight.
Chapter 2 Configuring Resource Allocation Settings If you select the cluster’s Resource Reservation tab and click CPU, you should see that shares for VM-QA are twice that of the other virtual machine. Also, because the virtual machines have not been powered on, the Reservation Used fields have not changed. Admission Control When you power on a virtual machine, the system checks the amount of CPU and memory resources that have not yet been reserved.
vSphere Resource Management 16 VMware, Inc.
CPU Virtualization Basics 3 CPU virtualization emphasizes performance and runs directly on the processor whenever possible. The underlying physical resources are used whenever possible and the virtualization layer runs instructions only as needed to make virtual machines operate as if they were running directly on a physical machine. CPU virtualization is not the same thing as emulation. ESXi does not use emulation to run virtual CPUs. With emulation, all operations are run in software by an emulator.
vSphere Resource Management Hardware-Assisted CPU Virtualization Certain processors provide hardware assistance for CPU virtualization. When using this assistance, the guest can use a separate mode of execution called guest mode. The guest code, whether application code or privileged code, runs in the guest mode. On certain events, the processor exits out of guest mode and enters root mode.
Administering CPU Resources 4 You can configure virtual machines with one or more virtual processors, each with its own set of registers and control structures. When a virtual machine is scheduled, its virtual processors are scheduled to run on physical processors. The VMkernel Resource Manager schedules the virtual CPUs on physical CPUs, thereby managing the virtual machine’s access to physical CPU resources. ESXi supports virtual machines with up to 128 virtual CPUs.
vSphere Resource Management n Use advanced settings under certain circumstances. n Use the vSphere SDK for scripted CPU allocation. n Use hyperthreading. Multicore Processors Multicore processors provide many advantages for a host performing multitasking of virtual machines. Intel and AMD have developed processors which combine two or more processor cores into a single integrated circuit (often called a package or socket).
Chapter 4 Administering CPU Resources Many processors do not support hyperthreading and as a result have only one thread per core. For such processors, the number of cores also matches the number of logical processors. The following processors support hyperthreading and have two threads per core. n Processors based on the Intel Xeon 5500 processor microarchitecture.
vSphere Resource Management Using CPU Affinity By specifying a CPU affinity setting for each virtual machine, you can restrict the assignment of virtual machines to a subset of the available processors in multiprocessor systems. By using this feature, you can assign each virtual machine to processors in the specified affinity set.
Chapter 4 Administering CPU Resources Potential Issues with CPU Affinity Before you use CPU affinity, you might need to consider certain issues. Potential issues with CPU affinity include: n For multiprocessor systems, ESXi systems perform automatic load balancing. Avoid manual specification of virtual machine affinity to improve the scheduler’s ability to balance load across processors.
vSphere Resource Management Select a CPU Power Management Policy You set the CPU power management policy for a host using the vSphere Web Client. Prerequisites Verify that the BIOS settings on the host system allow the operating system to control power management (for example, OS Controlled). Note Some systems have Processor Clocking Control (PCC) technology, which allows ESXi to manage power on the host system even if the host BIOS settings do not specify OS Controlled mode.
Chapter 4 Administering CPU Resources 6 VMware, Inc. Parameter Description Power.MinFreqPct Do not use any P-states slower than the given percentage of full CPU speed. Power.UseStallCtr Use a deeper P-state when the processor is frequently stalled waiting for events such as cache misses. Power.TimerHz Controls how many times per second ESXi reevaluates which P-state each CPU should be in. Power.UseCStates Use deep ACPI C-states (C2 or below) when the processor is idle. Power.
vSphere Resource Management 26 VMware, Inc.
Memory Virtualization Basics 5 Before you manage memory resources, you should understand how they are being virtualized and used by ESXi. The VMkernel manages all physical RAM on the host. The VMkernel dedicates part of this managed physical RAM for its own use. The rest is available for use by virtual machines. The virtual and physical memory space is divided into blocks called pages. When physical memory is full, the data for virtual pages that are not present in physical memory are stored on disk.
vSphere Resource Management After a virtual machine consumes all of the memory within its reservation, it is allowed to retain that amount of memory and this memory is not reclaimed, even if the virtual machine becomes idle. Some guest operating systems (for example, Linux) might not access all of the configured memory immediately after booting.
Chapter 5 Memory Virtualization Basics virtual machine often consumes less memory than it might when running on physical machines. As a result, higher levels of overcommitment can be supported efficiently. The amount of memory saved by memory sharing depends on whether the workload consists of nearly identical machines which might free up more memory. A more diverse workload might result in a lower percentage of memory savings.
vSphere Resource Management n The dashed arrows show the mapping from guest virtual memory to machine memory in the shadow page tables also maintained by the VMM. The underlying processor running the virtual machine uses the shadow page table mappings. Software-Based Memory Virtualization ESXi virtualizes guest physical memory by adding an extra level of address translation. n The VMM maintains the combined virtual-to-machine page mappings in the shadow page tables.
Chapter 5 Memory Virtualization Basics Performance Considerations When you use hardware assistance, you eliminate the overhead for software memory virtualization. In particular, hardware assistance eliminates the overhead required to keep shadow page tables in synchronization with guest page tables. However, the TLB miss latency when using hardware assistance is significantly higher. By default the hypervisor uses large pages in hardware assisted modes to reduce the cost of TLB misses.
vSphere Resource Management 32 VMware, Inc.
Administering Memory Resources 6 Using the vSphere Web Client you can view information about and make changes to memory allocation settings. To administer your memory resources effectively, you must also be familiar with memory overhead, idle memory tax, and how ESXi hosts reclaim memory. When administering memory resources, you can specify memory allocation. If you do not customize memory allocation, the ESXi host uses defaults that work well in most situations.
vSphere Resource Management ESXi memory virtualization adds little time overhead to memory accesses. Because the processor's paging hardware uses page tables (shadow page tables for software-based approach or two level page tables for hardware-assisted approach) directly, most memory accesses in the virtual machine can execute without address translation overhead. The memory space overhead has two components. n A fixed, system-wide overhead for the VMkernel.
Chapter 6 Administering Memory Resources n Working set size — ESXi hosts estimate the working set for a virtual machine by monitoring memory activity over successive periods of virtual machine execution time. Estimates are smoothed over several time periods using techniques that respond rapidly to increases in working set size and more slowly to decreases in working set size.
vSphere Resource Management n ESXi system swaps out a page from a virtual machine to a server swap file without any involvement by the guest operating system. Each virtual machine has its own swap file. Memory Balloon Driver The memory balloon driver (vmmemctl) collaborates with the server to reclaim pages that are considered least valuable by the guest operating system.
Chapter 6 Administering Memory Resources n It is functioning properly, but maximum balloon size is reached. Standard demand-paging techniques swap pages back in when the virtual machine needs them. Swap File Location By default, the swap file is created in the same location as the virtual machine's configuration file, which may either be on a VMFS datastore, a vSAN datastore or a VVol datastore. On a vSAN datastore or a VVol datastore, the swap file is created as a separate vSAN or VVol object.
vSphere Resource Management Host-local swap is now enabled for the standalone host. Swap Space and Memory Overcommitment You must reserve swap space for any unreserved virtual machine memory (the difference between the reservation and the configured memory size) on per-virtual machine swap files. This swap reservation is required to ensure that the ESXi host is able to preserve virtual machine memory under any circumstances. In practice, only a small fraction of the host-level swap space might be used.
Chapter 6 Administering Memory Resources Setting an alternative swapfile location might cause migrations with vMotion to complete more slowly. For best vMotion performance, store the virtual machine on a local datastore rather than in the same directory as the virtual machine swapfiles. If the virtual machine is stored on a local datastore, storing the swapfile with the other virtual machine files will not improve vMotion. Prerequisites Required privilege: Host machine.Configuration.
vSphere Resource Management 4 Next to swap file location, click Edit. 5 Select where to store the swap file. 6 Option Description Virtual machine directory Stores the swap file in the same directory as the virtual machine configuration file. Datastore specified by host Stores the swap file in the location specified in the host configuration. If the swap file cannot be stored on the datastore that the host specifies, the swap file is stored in the same folder as the virtual machine. Click OK.
Chapter 6 Administering Memory Resources Memory Compression ESXi provides a memory compression cache to improve virtual machine performance when you use memory overcommitment. Memory compression is enabled by default. When a host's memory becomes overcommitted, ESXi compresses virtual pages and stores them in memory.
vSphere Resource Management Measuring and Differentiating Types of Memory Usage The Performance tab of the vSphere Web Client displays several metrics that can be used to analyze memory usage. Some of these memory metrics measure guest physical memory while other metrics measure machine memory. For instance, two types of memory usage that you can examine using performance metrics are guest physical memory and machine memory.
Chapter 6 Administering Memory Resources A similar result is obtained when determining Memory Shared and Memory Shared Common for the host. n Memory Shared for the host is the sum of each virtual machine's Memory Shared. Calculate shared memory by looking at each virtual machine's guest physical memory and counting the number of blocks that have arrows to machine memory blocks that themselves have more than one arrow pointing at them.
vSphere Resource Management ESXi determines automatically where the system swap should be stored, this is the Preferred swap file location. This decision can be aided by selecting a certain set of options. The system selects the best possible enabled option. If none of the options are feasible then system swap is not activated. The available options are: n Datastore - Allow the use of the datastore specified. Please note that a vSAN datastore or a VVol datastore cannot be specified for system swap files.
Configuring Virtual Graphics 7 You can edit graphics settings for supported graphics implementations. vSphere supports multiple graphics implementations. n VMware supports 3d graphics solutions from AMD, Intel and NVIDIA. n NVIDIA GRID support. n Allows single NVIDIA vib to support both vSGA and vGPU implementations. n Provides vCenter GPU performance charts for Intel and NVIDIA. n Enables graphics for Horizon View VDI desktops.
vSphere Resource Management Prerequisites n Verify that an NVIDIA GRID GPU graphics device with an appropriate driver is installed on the host. See the vSphere Upgrade documentation. n Verify that the virtual machine is compatible with ESXi 6.0 and later. Procedure 1 Right-click a virtual machine and select Edit Settings. 2 On the Virtual Hardware tab, select Shared PCI Device from the drop-down menu. 3 Click Add.
Chapter 7 Configuring Virtual Graphics Configuring Graphics Devices You can edit graphics type for a video card. Prerequisites Virtual machines must be powered off. Procedure 1 2 Under Graphics Devices, select a graphics card and click Edit. a Select Shared for VMware shared virtual graphics. b Select Shared Direct for Vendor shared passthrough graphics. Click OK. If you select a device, it shows which virtual machines are using that device if they are active.
vSphere Resource Management 48 VMware, Inc.
Managing Storage I/O Resources 8 vSphere Storage I/O Control allows cluster-wide storage I/O prioritization, which allows better workload consolidation and helps reduce extra costs associated with over provisioning. Storage I/O Control extends the constructs of shares and limits to handle storage I/O resources.
vSphere Resource Management About Virtual Machine Storage Policies Virtual machine storage policies are essential to virtual machine provisioning. The policies control which type of storage is provided for the virtual machine, how the virtual machine is placed within the storage, and which data services are offered for the virtual machine. vSphere includes default storage policies. However, you can define and assign new policies. You use the VM Storage Policies interface to create a storage policy.
Chapter 8 Managing Storage I/O Resources Storage I/O Control Resource Shares and Limits You allocate the number of storage I/O shares and upper limit of I/O operations per second (IOPS) allowed for each virtual machine. When storage I/O congestion is detected for a datastore, the I/O workloads of the virtual machines accessing that datastore are adjusted according to the proportion of virtual machine shares each virtual machine has.
vSphere Resource Management Set Storage I/O Control Resource Shares and Limits Allocate storage I/O resources to virtual machines based on importance by assigning a relative amount of shares to the virtual machine. Unless virtual machine workloads are very similar, shares do not necessarily dictate allocation in terms of I/O operations or megabytes per second.
Chapter 8 Managing Storage I/O Resources Set Storage I/O Control Threshold Value The congestion threshold value for a datastore is the upper limit of latency that is allowed for a datastore before Storage I/O Control begins to assign importance to the virtual machine workloads according to their shares. You do not need to adjust the threshold setting in most environments. Caution Storage I/O Control will not function correctly if you share the same spindles on two different datastores.
vSphere Resource Management Storage DRS Integration with Storage Profiles Storage Policy Based Management (SPBM) allows you to specify the policy for a virtual machine which is enforced by Storage DRS. A datastore cluster can have set of datastores with different capability profiles. If the virtual machines have storage profiles associated with them, Storage DRS can enforce placement based on underlying datastore capabilities.
Managing Resource Pools 9 A resource pool is a logical abstraction for flexible management of resources. Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources. Each standalone host and each DRS cluster has an (invisible) root resource pool that groups the resources of that host or cluster. The root resource pool does not appear because the resources of the host (or cluster) and the root resource pool are always the same.
vSphere Resource Management Why Use Resource Pools? Resource pools allow you to delegate control over resources of a host (or a cluster), but the benefits are evident when you use resource pools to compartmentalize all resources in a cluster. Create multiple resource pools as direct children of the host or cluster and configure them. You can then delegate control over the resource pools to other individuals or organizations. Using resource pools can result in the following benefits.
Chapter 9 Managing Resource Pools Create a Resource Pool You can create a child resource pool of any ESXi host, resource pool, or DRS cluster. Note If a host has been added to a cluster, you cannot create child resource pools of that host. If the cluster is enabled for DRS, you can create child resource pools of the cluster. When you create a child resource pool, you are prompted for resource pool attribute information.
vSphere Resource Management Example: Creating Resource Pools Assume that you have a host that provides 6GHz of CPU and 3GB of memory that must be shared between your marketing and QA departments. You also want to share the resources unevenly, giving one department (QA) a higher priority. This can be accomplished by creating a resource pool for each department and using the Shares attribute to prioritize the allocation of resources.
Chapter 9 Managing Resource Pools n Under Monitor, the information displayed in the Resource Reservation tab about the resource pool’s reserved and unreserved CPU and memory resources changes to reflect the reservations associated with the virtual machine (if any). Note If a virtual machine has been powered off or suspended, it can be moved but overall available resources (such as reserved and unreserved CPU and memory) for the resource pool are not affected.
vSphere Resource Management Remove a Resource Pool You can remove a resource pool from the inventory. Procedure 1 In the vSphere Web Client, right-click the resource pool and Select Delete. A confirmation dialog box appears. 2 Click Yes to remove the resource pool. Resource Pool Admission Control When you power on a virtual machine in a resource pool, or try to create a child resource pool, the system performs additional admission control to ensure the resource pool’s restrictions are not violated.
Chapter 9 Managing Resource Pools Expandable Reservations Example 2 This example shows how a resource pool with expandable reservations works. Assume the following scenario, as shown in the figure. n Parent pool RP-MOM has a reservation of 6GHz and one running virtual machine VM-M1 that reserves 1GHz. n You create a child resource pool RP-KID with a reservation of 2GHz and with Expandable Reservation selected.
vSphere Resource Management 62 VMware, Inc.
10 Creating a DRS Cluster A cluster is a collection of ESXi hosts and associated virtual machines with shared resources and a shared management interface. Before you can obtain the benefits of cluster-level resource management you must create a cluster and enable DRS. Depending on whether or not Enhanced vMotion Compatibility (EVC) is enabled, DRS behaves differently when you use vSphere Fault Tolerance (vSphere FT) virtual machines in your cluster. Table 10‑1.
vSphere Resource Management n Displays the placement recommendation, which the user can then choose to accept or override. Note No initial placement recommendations are given for virtual machines on standalone hosts or in non-DRS clusters. When powered on, they are placed on the host where they currently reside. n DRS considers network bandwidth. By calculating host network saturation, DRS is able to make better placement decisions.
Chapter 10 Creating a DRS Cluster Example: Group Power-on The user selects three virtual machines in the same data center for a group power-on attempt. The first two virtual machines (VM1 and VM2) are in the same DRS cluster (Cluster1), while the third virtual machine (VM3) is on a standalone host. VM1 is in automatic mode and VM2 is in manual mode.
vSphere Resource Management n If the cluster and virtual machines involved are all fully automated, vCenter Server migrates running virtual machines between hosts as needed to ensure efficient use of cluster resources. Note Even in an automatic migration setup, users can explicitly migrate individual virtual machines, but vCenter Server might move those virtual machines to other hosts to optimize cluster resources. By default, automation level is specified for the whole cluster.
Chapter 10 Creating a DRS Cluster DRS Cluster Requirements Hosts that are added to a DRS cluster must meet certain requirements to use cluster features successfully. Shared Storage Requirements A DRS cluster has certain shared storage requirements. Ensure that the managed hosts use shared storage. Shared storage is typically on a SAN, but can also be implemented using NAS shared storage. See the vSphere Storage documentation for information about other shared storage.
vSphere Resource Management Configure EVC from the Cluster Settings dialog box. The hosts in a cluster must meet certain requirements for the cluster to use EVC. For information about EVC and EVC requirements, see the vCenter Server and Host Management documentation. n CPU compatibility masks – vCenter Server compares the CPU features available to a virtual machine with the CPU features of the destination host to determine whether to allow or disallow migrations with vMotion.
Chapter 10 Creating a DRS Cluster 3 Enter a name for the cluster. 4 Select DRS and vSphere HA cluster features. 5 Option Description To use DRS with this cluster a b Select the DRS Turn ON check box. Select an automation level and a migration threshold. To use HA with this cluster a b c d e Select the vSphere HA Turn ON check box. Select whether to enable host monitoring and admission control. If admission control is enabled, specify a policy. Select a VM Monitoring option.
vSphere Resource Management sufficient excess capacity is found. DRS powers-on hosts if capacity is needed. Depending on the resulting host power state recommendations, virtual machines might need to be migrated to and from the hosts as well. See “Managing Power Resources,” on page 82. Affinity Rules You can control the placement of virtual machines on hosts within a cluster, by assigning affinity rules. See “Using DRS Affinity Rules,” on page 86.
Chapter 10 Creating a DRS Cluster n Specify the type of policy that admission control enforces. n Adjust the monitoring sensitivity of virtual machine monitoring. 12 If appropriate, enable Enhanced vMotion Compatibility (EVC) and select the mode it operates in. 13 Click OK. Set a Custom Automation Level for a Virtual Machine After you create a DRS cluster, you can customize the automation level for individual virtual machines to override the cluster’s default automation level.
vSphere Resource Management Disable DRS You can turn off DRS for a cluster. When DRS is disabled, the cluster’s resource pool hierarchy and affinity rules are not reestablished when DRS is turned back on. If you disable DRS, the resource pools are removed from the cluster. To avoid losing the resource pools, save a snapshot of the resource pool tree on your local machine. You can use the snapshot to restore the resource pool when you enable DRS.
Using DRS Clusters to Manage Resources 11 After you create a DRS cluster, you can customize it and use it to manage resources. To customize your DRS cluster and the resources it contains you can configure affinity rules and you can add and remove hosts and virtual machines. When a cluster’s settings and resources have been defined, you should ensure that it is and remains a valid cluster. You can also use a valid DRS cluster to manage power resources and interoperate with vSphere HA.
vSphere Resource Management 3 Select a cluster. 4 Click OK to apply the changes. 5 Select what to do with the host’s virtual machines and resource pools. n Put this host’s virtual machines in the cluster’s root resource pool vCenter Server removes all existing resource pools of the host and the virtual machines in the host’s hierarchy are all attached to the root.
Chapter 11 Using DRS Clusters to Manage Resources Adding Virtual Machines to a Cluster You can add a virtual machine to a cluster in a number of ways. n When you add a host to a cluster, all virtual machines on that host are added to the cluster. n When a virtual machine is created, the New Virtual Machine wizard prompts you for the location to place the virtual machine. You can select a standalone host or a cluster and you can select any resource pool inside the host or cluster.
vSphere Resource Management 4 Select a datastore and click Next. 5 Click Finish. If the virtual machine is a member of a DRS cluster rules group, vCenter Server displays a warning before it allows the migration to proceed. The warning indicates that dependent virtual machines are not migrated automatically. You have to acknowledge the warning before migration can proceed.
Chapter 11 Using DRS Clusters to Manage Resources n 3 If the host is part of an automated DRS cluster, virtual machines are migrated to different hosts when the host enters maintenance mode. If applicable, click Yes. The host is in maintenance mode until you select Maintenance Mode > Exit Maintenance Mode. Remove a Host from a Cluster You can remove hosts from a cluster. Procedure 1 Browse to the host in the vSphere Web Client navigator.
vSphere Resource Management When considering cluster validity scenarios, you should understand these terms. Reservation A fixed, guaranteed allocation for the resource pool input by the user. Reservation Used The sum of the reservation or reservation used (whichever is larger) for each child resource pool, added recursively. Unreserved This nonnegative number differs according to resource pool type. n Nonexpandable resource pools: Reservation minus reservation used.
Chapter 11 Using DRS Clusters to Manage Resources n RP2 was created with a reservation of 4GHz. Two virtual machines of 1GHz and 2GHz are powered on (Reservation Used: 3GHz). 1GHz remains unreserved. n RP3 was created with a reservation of 3GHz. One virtual machine with 3GHz is powered on. No resources for powering on additional virtual machines are available. The following figure shows an example of a valid cluster with some resource pools (RP1 and RP3) using reservation type Expandable. Figure 11‑2.
vSphere Resource Management Overcommitted DRS Clusters A cluster becomes overcommitted (yellow) when the tree of resource pools and virtual machines is internally consistent but the cluster does not have the capacity to support all resources reserved by the child resource pools. There will always be enough resources to support all running virtual machines because, when a host becomes unavailable, all its virtual machines become unavailable.
Chapter 11 Using DRS Clusters to Manage Resources Invalid DRS Clusters A cluster enabled for DRS becomes invalid (red) when the tree is no longer internally consistent, that is, resource constraints are not observed. The total amount of resources in the cluster does not affect whether the cluster is red. A cluster can be red, even if enough resources exist at the root level, if there is an inconsistency at a child level.
vSphere Resource Management Managing Power Resources The vSphere Distributed Power Management (DPM) feature allows a DRS cluster to reduce its power consumption by powering hosts on and off based on cluster resource utilization. vSphere DPM monitors the cumulative demand of all virtual machines in the cluster for memory and CPU resources and compares this to the total available resource capacity of all hosts in the cluster.
Chapter 11 Using DRS Clusters to Manage Resources 2 Click the Configure tab. 3 Under System, click Power Management. 4 Click Edit. 5 Enter the following information. 6 n User name and password for a BMC account. (The user name must have the ability to remotely power the host on.) n IP address of the NIC associated with the BMC, as distinct from the IP address of the host. The IP address should be static or a DHCP address with infinite lease. n MAC address of the NIC associated with the BMC.
vSphere Resource Management 4 Observe whether or not the host successfully powers back on. 5 For any host that fails to exit standby mode successfully, perform the following steps. a Select the host in the vSphere Web Client navigator and select the Configure tab. b Under Hardware > Power Management, click Edit to adjust the power management policy. After you do this, vSphere DPM does not consider that host a candidate for being powered off.
Chapter 11 Using DRS Clusters to Manage Resources The threshold is configured under Power Management in the cluster’s Settings dialog box. Each level you move the vSphere DPM Threshold slider to the right allows the inclusion of one more lower level of priority in the set of recommendations that are executed automatically or appear as recommendations to be manually executed.
vSphere Resource Management Table 11‑2. vCenter Server Events (Continued) Event Type Event Name Exiting Standby mode (about to power on the host) DrsExitingStandbyModeEvent Successfully exited Standby mode (power on succeeded) DrsExitedStandbyModeEvent For more information about creating and editing alarms, see the vSphere Monitoring and Performance documentation.
Chapter 11 Using DRS Clusters to Manage Resources 3 Under Configuration, select VM/Host Groups and click Add. 4 In the Create VM/Host Group dialog box, type a name for the group. 5 Select Host Group from the Type drop down box and click Add. 6 Click the check box next to a host to add it. Continue this process until all desired hosts have been added. 7 Click OK.
vSphere Resource Management Create a VM-VM Affinity Rule You can create VM-VM affinity rules to specify whether selected individual virtual machines should run on the same host or be kept on separate hosts. Note If you use the vSphere HA Specify Failover Hosts admission control policy and designate multiple failover hosts, VM-VM affinity rules are not supported. Procedure 1 Browse to the cluster in the vSphere Web Client navigator. 2 Click the Configure tab. 3 Under Configuration, click VM/Host Rules.
Chapter 11 Using DRS Clusters to Manage Resources Create a VM-Host Affinity Rule You can create VM-Host affinity rules to specify whether or not the members of a selected virtual machine DRS group can run on the members of a specific host DRS group. Prerequisites Create the virtual machine and host DRS groups to which the VM-Host affinity rule applies. Procedure 1 Browse to the cluster in the vSphere Web Client navigator. 2 Click the Configure tab. 3 Under Configuration, click VM/Host Rules.
vSphere Resource Management When you create a VM-Host affinity rule, its ability to function in relation to other rules is not checked. So it is possible for you to create a rule that conflicts with the other rules you are using. When two VM-Host affinity rules conflict, the older one takes precedence and the newer rule is disabled. DRS only tries to satisfy enabled rules and disabled rules are ignored.
Creating a Datastore Cluster 12 A datastore cluster is a collection of datastores with shared resources and a shared management interface. Datastore clusters are to datastores what clusters are to hosts. When you create a datastore cluster, you can use vSphere Storage DRS to manage storage resources. Note Datastore clusters are referred to as storage pods in the vSphere API. When you add a datastore to a datastore cluster, the datastore's resources become part of the datastore cluster's resources.
vSphere Resource Management Initial Placement and Ongoing Balancing Storage DRS provides initial placement and ongoing balancing recommendations to datastores in a Storage DRS-enabled datastore cluster. Initial placement occurs when Storage DRS selects a datastore within a datastore cluster on which to place a virtual machine disk.
Chapter 12 Creating a Datastore Cluster Enable and Disable Storage DRS Storage DRS allows you to manage the aggregated resources of a datastore cluster. When Storage DRS is enabled, it provides recommendations for virtual machine disk placement and migration to balance space and I/O resources across the datastores in the datastore cluster. When you enable Storage DRS, you enable the following functions. n Space load balancing among datastores within a datastore cluster.
vSphere Resource Management 5 Click OK. Setting the Aggressiveness Level for Storage DRS The aggressiveness of Storage DRS is determined by specifying thresholds for space used and I/O latency. Storage DRS collects resource usage information for the datastores in a datastore cluster. vCenter Server uses this information to generate recommendations for placement of virtual disks on datastores.
Chapter 12 Creating a Datastore Cluster 2 (Optional) Set Storage DRS thresholds. You set the aggressiveness level of Storage DRS by specifying thresholds for used space and I/O latency. n Use the Utilized Space slider to indicate the maximum percentage of consumed space allowed before Storage DRS is triggered. Storage DRS makes recommendations and performs migrations when space use on the datastores is higher than the threshold.
vSphere Resource Management Adding and Removing Datastores from a Datastore Cluster You add and remove datastores to and from an existing datastore cluster. You can add to a datastore cluster any datastore that is mounted on a host in the vSphere Web Client inventory, with the following exceptions: n All hosts attached to the datastore must be ESXi 5.0 and later. n The datastore cannot be in more than one data center in the same instance of the vSphere Web Client.
Using Datastore Clusters to Manage Storage Resources 13 After you create a datastore cluster, you can customize it and use it to manage storage I/O and space utilization resources.
vSphere Resource Management No CD-ROM image files are stored on the datastore. There are at least two datastores in the datastore cluster. Procedure 1 Browse to the datastore in the vSphere Web Client navigator. 2 Right-click the datastore and select Maintenance Mode > Enter Maintenance Mode. A list of recommendations appears for datastore maintenance mode migration. 3 (Optional) On the Placement Recommendations tab, deselect any recommendations you do not want to apply.
Chapter 13 Using Datastore Clusters to Manage Storage Resources Applying Storage DRS Recommendations Storage DRS collects resource usage information for all datastores in a datastore cluster. Storage DRS uses the information to generate recommendations for virtual machine disk placement on datastores in a datastore cluster. Storage DRS recommendations appear on the Storage DRS tab in the vSphere Web Client datastore view.
vSphere Resource Management Change Storage DRS Automation Level for a Virtual Machine You can override the datastore cluster-wide automation level for individual virtual machines. You can also override default virtual disk affinity rules. Procedure 1 Browse to the datastore cluster in the vSphere Web Client navigator. 2 Click the Configure tab and click Configuration. 3 Under VM Overrides, select Add. 4 Select a virtual machine.
Chapter 13 Using Datastore Clusters to Manage Storage Resources 5 Expand DRS Automation. a Select an automation level. b Set the Migration threshold. Use the Migration slider to select the priority level of vCenter Server recommendations that adjust the cluster's load balance. c Select whether to enable Virtual Machine Automation. Override for individual virtual machines can be set from the VM Overrides page. 6 Expand Power Managment. a Select an automation level. b Set the DPM threshold.
vSphere Resource Management n Datastore Cluster B has an inter-VM anti-affinity rule. When you move a virtual disk out of Datastore Cluster A and into Datastore Cluster B, any rule that applied to the virtual disk for a given virtual machine in Datastore Cluster A no longer applies. The virtual disk is now subject to the inter-VM antiaffinity rule in Datastore Cluster B. n Datastore Cluster B has an intra-VM anti-affinity rule.
Chapter 13 Using Datastore Clusters to Manage Storage Resources Create Intra-VM Anti-Affinity Rules You can create a VMDK anti-affinity rule for a virtual machine that indicates which of its virtual disks must be kept on different datastores. VMDK anti-affinity rules apply to the virtual machine for which the rule is defined, not to all virtual machines. The rule is expressed as a list of virtual disks that are to be separated from one another.
vSphere Resource Management n If the virtual machine's virtual disk violates the rule, Storage DRS makes migration recommendations to correct the error or reports the violation as a fault if it cannot make a recommendation that will correct the error. When you add a datastore to a datastore cluster that is enabled for Storage DRS, the VMDK affinity rule is disabled for any virtual machine that has virtual disks on that datastore if it also has virtual disks on other datastores.
Chapter 13 Using Datastore Clusters to Manage Storage Resources Storage vMotion Compatibility with Datastore Clusters ® A datastore cluster has certain vSphere Storage vMotion requirements. n The host must be running a version of ESXi that supports Storage vMotion. n The host must have write access to both the source datastore and the destination datastore. n The host must have enough free memory resources to accommodate Storage vMotion. n The destination datastore must have sufficient disk space.
vSphere Resource Management 106 VMware, Inc.
Using NUMA Systems with ESXi 14 ESXi supports memory access optimization for Intel and AMD Opteron processors in server architectures that support NUMA (non-uniform memory access). After you understand how ESXi NUMA scheduling is performed and how the VMware NUMA algorithms work, you can specify NUMA controls to optimize the performance of your virtual machines.
vSphere Resource Management The high latency of remote memory accesses can leave the processors under-utilized, constantly waiting for data to be transferred to the local node, and the NUMA connection can become a bottleneck for applications with high-memory bandwidth demands. Furthermore, performance on such a system can be highly variable.
Chapter 14 Using NUMA Systems with ESXi ESXi 5.0 and later includes support for exposing virtual NUMA topology to guest operating systems. For more information about virtual NUMA control, see “Using Virtual NUMA,” on page 110. VMware NUMA Optimization Algorithms and Settings This section describes the algorithms and settings used by ESXi to maximize application performance while still maintaining resource guarantees.
vSphere Resource Management When initial placement, dynamic rebalancing, and intelligent memory migration work in conjunction, they ensure good memory performance on NUMA systems, even in the presence of changing workloads. When a major workload change occurs, for instance when new virtual machines are started, the system takes time to readjust, migrating virtual machines and memory to new locations.
Chapter 14 Using NUMA Systems with ESXi If the virtual NUMA topology needs to be overridden, see “Virtual NUMA Controls,” on page 111. Note Enabling CPU HotAdd will disable virtual NUMA. See https://kb.vmware.com/kb/2040375. Virtual NUMA Controls For virtual machines with disproportionately large memory consumption, you can use advanced options to override the default virtual CPU settings. You can add these advanced options to the virtual machine configuration file. Table 14‑1.
vSphere Resource Management The vSphere Web Client lets you specify the following options. NUMA Node Affinity When you set this option, NUMA can schedule a virtual machine only on the nodes specified in the affinity. CPU Affinity When you set this option, a virtual machine uses only the processors specified in the affinity. Memory Affinity When you set this option, the server allocates memory only on the specified nodes.
Chapter 14 Using NUMA Systems with ESXi 4 Click Edit. 5 Select the Virtual Hardware tab, and expand Memory. 6 Under NUMA Memory Affinity, set memory affinity. Example: Binding a Virtual Machine to a Single NUMA Node The following example illustrates manually binding the last four physical CPUs to a single NUMA node for a two-way virtual machine on an eight-way server. The CPUs (for example, 4, 5, 6, and 7) are the physical CPU numbers.
vSphere Resource Management 114 VMware, Inc.
Advanced Attributes 15 You can set advanced attributes for hosts or individual virtual machines to help you customize resource management. In most cases, adjusting the basic resource allocation settings (reservation, limit, shares) or accepting default settings results in appropriate resource allocation. However, you can use advanced attributes to customize resource management for a host or a specific virtual machine.
vSphere Resource Management Advanced Memory Attributes You can use the advanced memory attributes to customize memory resource usage. Table 15‑1. Advanced Memory Attributes 116 Attribute Description Default Mem.ShareForceSalting Mem.ShareForceSalting 0: Inter-virtual machine Transparent Page Sharing (TPS) behavior is still retained. The value of VMX option sched.mem.pshare.salt is ignored even if present. Mem.ShareForceSalting 1: By default the salt value is taken from sched.mem.pshare.salt.
Chapter 15 Advanced Attributes Table 15‑1. Advanced Memory Attributes (Continued) Attribute Description Default LPage.LPageDefragRateTotal Maximum number of large page defragmentation attempts per second. Accepted values range from 1 to 10240. 256 LPage.LPageAlwaysTryForNPT Try to allocate large pages for nested page tables (called 'RVI' by AMD or 'EPT' by Intel).
vSphere Resource Management Set Advanced Virtual Machine Attributes You can set advanced attributes for a virtual machine. Procedure 1 Find the virtual machine in the vSphere Web Client inventory. a To find a virtual machine, select a data center, folder, cluster, resource pool, or host. b Click the VMs tab. 2 Right-click the virtual machine and select Edit Settings. 3 Click VM Options. 4 Expand Advanced. 5 Under Configuration Parameters, click the Edit Configuration button.
Chapter 15 Advanced Attributes Table 15‑3. Advanced Virtual Machine Attributes (Continued) Attribute Description Default sched.swap.persist Specifies whether the virtual machine’s swap files should persist or be deleted when the virtual machine is powered off. By default, the system creates the swap file for a virtual machine when the virtual machine is powered on, and deletes the swap file when the virtual machine is powered off. False sched.swap.
vSphere Resource Management Table 15‑4. Advanced NUMA Attributes (Continued) Attribute Description numa.nodeAffinity Constrains the set of NUMA nodes on which a virtual machine's virtual CPU and memory can be scheduled. Note When you constrain NUMA node affinities, you might interfere with the ability of the NUMA scheduler to rebalance virtual machines across NUMA nodes for fairness. Specify NUMA node affinity only after you consider the rebalancing issues. numa.mem.
Chapter 15 Advanced Attributes 2 Click the Configure tab and click System. 3 Select Licensing. 4 Under Licensed Features verify Reliable Memory is displayed. What to do next You can look up how much memory is considered reliable by using the ESXCLI hardware memory get command. VMware, Inc.
vSphere Resource Management 122 VMware, Inc.
Fault Definitions 16 DRS faults indicate the reasons that prevent the generation of DRS actions (or the recommendation of those actions in manual mode). The DRS faults are defined within this section.
vSphere Resource Management Virtual Machine is Pinned This fault occurs when DRS cannot move a virtual machine because DRS is disabled on it. That is, the virtual machine is "pinned" on its registered host. Virtual Machine not Compatible with any Host This fault occurs when DRS cannot find a host that can run the virtual machine.
Chapter 16 Fault Definitions Host Has Insufficient Number of Physical CPUs for Virtual Machine This fault occurs when the host hardware does not enough CPUs (hyperthreads) to support the number of virtual CPUs in the virtual machine. Host has Insufficient Capacity for Each Virtual Machine CPU This fault occurs when the host does not have enough CPU capacity for running the virtual machine. The Virtual Machine Is in vMotion This fault occurs when DRS cannot move a virtual machine because it is in vMotion.
vSphere Resource Management Soft Rule Violation Correction Impact Correcting the non-mandatory VM/Host DRS affinity rule does not occur because it impacts performance. 126 VMware, Inc.
DRS Troubleshooting Information 17 ® This information describes vSphere Distributed Resource Scheduler (DRS) problems for particular categories: cluster, host, and virtual machine problems. This chapter includes the following topics: n “Cluster Problems,” on page 127 n “Host Problems,” on page 130 n “Virtual Machine Problems,” on page 133 Cluster Problems Cluster problems can prevent DRS from performing optimally or from reporting faults.
vSphere Resource Management n vMotion is not enabled or set up for the hosts in the cluster. Solution Address the problem that is causing the load imbalance. Cluster is Yellow The cluster is yellow due to a shortage of resources. Problem If the cluster does not have enough resources to satisfy the reservations of all resource pools and virtual machines, but does have enough resources to satisfy the reservations of all running virtual machines, DRS continues to run and the cluster is yellow.
Chapter 17 DRS Troubleshooting Information No Hosts are Powered Off When Total Cluster Load is Low Hosts are not powered off when the total cluster load is low. Problem Hosts are not powered off when the total cluster load is low because extra capacity is needed for HA failover reservations. Cause Hosts might not be powered off for the following reasons: n The MinPoweredOn{Cpu|Memory}Capacity advanced options settings need to be met.
vSphere Resource Management Cause DRS never performs vMotion migrations when one or more of the following issues is present on the cluster. n DRS is disabled on the cluster. n The hosts do not have shared storage. n The hosts in the cluster do not contain a vMotion network. n DRS is manual and no one has approved the migration. DRS seldom performs vMotion when one or more of the following issues is present on the cluster: n Loads are unstable, or vMotion takes a long time, or both.
Chapter 17 DRS Troubleshooting Information Total Cluster Load Is High The total cluster load is high. Problem When the total cluster load is high, DRS does not power-on the host. Cause The following are possible reasons why DRS does not power-on the host: n VM/VM DRS rules or VM/Host DRS rules prevent the virtual machine from being moved to this host. n Virtual machines are pinned to their current hosts, so DRS cannot move these virtual machines to hosts in standby mode to balance the load.
vSphere Resource Management DRS Does Not Evacuate a Host Requested to Enter Maintenance or Standby Mode DRS does not evacuate a host requested to enter maintenance mode or standby mode. Problem When you attempt to put a host into maintenance or standby mode, DRS does not evacuate the host as expected. Cause vSphere HA is enabled and evacuating this host might violate HA failover capacity. Solution There is no solution.
Chapter 17 DRS Troubleshooting Information Cause This may be because of problems with vMotion, DRS, or host compatibility. The following are the possible reasons: n vMotion is not configured or enabled on this host. n DRS is disabled for the virtual machines on this host. n Virtual machines on this host are not compatible with any other hosts. n No other hosts have sufficient resources for any virtual machines on this host.
vSphere Resource Management Cluster is Overloaded The cluster on which the virtual machine is running might have insufficient resources. Also, the virtual machine's share value is such that other virtual machines are granted proportionally more of the resources. To determine the demand is larger than the capacity, check the cluster statistics. Host is Overloaded To determine if the host's resources are oversubscribed, check the host statistics.
Chapter 17 DRS Troubleshooting Information When a VM/VM DRS rule or VM/Host DRS rule is violated, it might be because DRS cannot move some or all of the virtual machines in the rule. The reservation of the virtual machine or other virtual machines in the affinity rule, or their parent resource pools, might prevent DRS from locating all virtual machines on the same host. Solution n Check the DRS faults panel for faults associated with affinity rules.
vSphere Resource Management n The DRS automation level of the virtual machine is manual and the user does not approve the migration recommendation. n DRS will not move fault tolerance-enabled virtual machines. Solution Address the issue that prevents DRS from moving the virtual machine. 136 VMware, Inc.
Index A admission control CPU 23 resource pools 60 with expandable resource pools 61 advanced attributes hosts 115 memory 116 NUMA 117 Storage I/O Control 53 virtual machines 118, 120 virtual NUMA 119 affinity rules creating 88, 89 intra-VM 103 Storage DRS 101 alarms 85 AMD Opteron-based systems 107, 117 anti-affinity rules 124 applications CPU-bound 18 single-threaded 18 automation level datastore clusters 93 Storage DRS 100 virtual machines 71 B ballooning, memory 36 Baseboard Management Controller (BMC
vSphere Resource Management overrides 85 threshold 84 DRS creating rules 88 disable 72 fully automated 69 group power on 64 initial placement 63 load balancing 63 manual 69 migration 63 migration recommendations 66 migration thresholds 66 partially automated 69 single virtual machine power on 64 virtual machine migration 65 vMotion network 67 DRS groups host 86 virtual machine 87 DRS clusters adding managed hosts 73 adding unmanaged hosts 74 as resource providers 9 creating 69 managing resources with 73 pr
Index I I/O filters, about 50 idle memory tax 35 iLO, configuring 82 incompatible host 124 initial placement NUMA 109 Storage DRS 92 insufficient capacity 124 insufficient resources 125 Intelligent Platform Management Interface (IPMI), configuring 82 inter-VM anti-affinity rules, creating 102 intra-VM anti-affinity rules 103 introduction 127 invalid DRS clusters 81 L Last Time Exited Standby 85 latency sensitivity 120 Latency Sensitivity setting 120 limit 12 load balancing datastores 91 Storage DRS 94 vir
vSphere Resource Management Numa.MigImbalanceThreshold 117 Numa.PageMigEnable 117 Numa.RebalanceCoresNode 117 Numa.RebalanceCoresTotal 117 Numa.RebalanceEnable 117 Numa.
Index storage requirements 67 Storage vMotion datastore cluster compatibility 105 recommendations 92 swap file deleting 40 location 37 using 36 swap files, VMX 35 swap space 38 swap to VMX 35 swapfile location 39 swapfiles 38 system resource allocation, editing 13 System Resource Allocation Table (SRAT) 108 system swap 43 system swap data 44 T threshold, Storage I/O Control 53 V valid DRS clusters 78 vCenter Server events 85 vGPU 45–47 virtual flash 68 virtual machine affinity 86–89 virtual machine anti-
vSphere Resource Management 142 VMware, Inc.
