Microsoft Exchange 2010 on a Hyper-V Virtual Infrastructure Supported by Dell EqualLogic SANs A Dell EqualLogic Best Practices Technical White Paper Dell Storage Engineering February 2013 This document has been archived and will no longer be maintained or updated. For more information go to the Storage Solutions Technical Documents page on Dell TechCenter or contact support.
© 2013 Dell Inc. All Rights Reserved. Dell, the Dell logo, and other Dell names and marks are trademarks of Dell Inc. in the US and worldwide. Intel® is a registered trademark of Intel Corporation in the U.S. and other countries. Microsoft®, Windows®, Windows Server®, and Active Directory® are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks mentioned herein are the property of their respective owners.
Table of contents Acknowledgements .......................................................................................................................................................................... 5 Feedback ............................................................................................................................................................................................ 5 Executive summary ......................................................................................
B.1 Microsoft Jetstress considerations ............................................................................................................................ 59 B.2 Microsoft Load Generator considerations............................................................................................................... 60 Additional resources ..................................................................................................................................................................
Acknowledgements This best practice white paper was produced by the following members of the Dell Storage team: Engineering: Danilo Feroce Technical Marketing: Jim Salvadore Editing: Margaret Boeneke Feedback We encourage readers of this publication to provide feedback on the quality and usefulness of this information by sending an email to SISfeedback@Dell.com. SISfeedback@Dell.
Executive summary Virtualization technologies are an optimal answer to address the need to reduce IT operational costs. These technologies were originally introduced with the aim to consolidate multiple functional services into a single physical system and to increase hardware utilization.
1 Introduction The EqualLogic PS6100 Series arrays add a heightened capacity and increased overall performance while preserving the same peer storage architecture, the ease of use and management, the broad out-of-thebox feature set, and software portfolio as its predecessors. These include scalability without fork-lift upgrades, native load balancing, point-in-time snapshots, integrated replication, and application layer software integration and monitoring.
Pool: A logical collection that each member (array) is assigned to after being added to a group and contributes its storage space to the entire pool. Hypervisor: Denotes the software layer that manages the access to the hardware resources, residing above the hardware, and in between the operating systems running as guests. Parent and Child Partitions: Identifies the logical units of isolation supported by the Hyper-V hypervisor.
2 Microsoft Exchange and the storage subsystem Microsoft Exchange Server has a diversified set of components and services that cooperate to support the disparate requirements to design and deploy a messaging infrastructure within an organization. The mailbox server role in an Exchange infrastructure has the most impact on storage resources because it ultimately governs the storage, retrieval, and availability of user data to be routed and presented to the rest of the infrastructure.
Exchange HUB server role for the queue databases. ESE technology, previously known as Jet Database Engine, has evolved through several versions of Exchange Server releases and has been a part of several Microsoft products since its inception (for example, Microsoft Access, Active Directory, File Replication Service, WINS server, and Certificate Services). The ESE is an Indexed Sequential Access Method (ISAM) technology that organizes database data in B-Tree structures.
Mailbox size: Defines the maximum size a mailbox is allowed to grow and can be enforced by a quota policy. In more general terms, it is the average mailbox size in a corporate messaging environment. It primarily affects the capacity requirement of a mailbox role server and the planned size of the databases hosted by the server. Moreover, it influences the IOPS response due to the amount of physical disk surface that must be accessed to retrieve or store the data.
2.3 Considerations for Exchange DAG A DAG is a pool of up to 16 networked servers that hosts multiple copies of the same Exchange database or databases where only one of the copies is active at a specific point-in-time within the group. The other copies are passive and contain data sourced from replicated and replayed transaction logs. When DAG is implemented, it creates some deviations in the storage access patterns and in the Exchange memory cache behavior.
3 Test topology and architecture overview This paper presents a set of findings from tests conducted on a Microsoft Windows infrastructure and an Exchange organization built on Windows Server with Hyper-V technology accessing storage resources provided by an EqualLogic SAN. Multiple simulation tools, both Microsoft Jetstress and Loadgen, were used to generate the workloads required for this study.
The main elements of the design are: • • • • Single Active Directory forest, single domain, single site (not strictly required for the tests) Centralized management and monitoring with dedicated resources Building block design approach for mailbox role servers with Jetstress Separated network design for traffic isolation between LAN and iSCSI The architecture has then been modified to include the additional elements required to support the complexity of an Exchange organization deployed with highly availa
3.2 Physical system configuration The physical components of the test infrastructure were configured as show in Figure 3. Figure 3 Physical system design The solution is deployed on Dell Blade servers with the aim of a greater datacenter density and flexibility.
• Single EqualLogic iSCSI SAN provisioned with one of the following array models (exchanged for each test): PS6100 3.5”, PS6100X, or PS6100E • Single EqualLogic iSCSI SAN with up to three PS6100XV 3.
4 Trend of Exchange deployment variables The performance trends of an EqualLogic SAN and of the Hyper-V virtual infrastructure were evaluated under the load of a simulated Exchange mailbox role server. The following sections summarize each of the variations tested to understand these trends.
Table 1 Reference configuration for Microsoft Jetstress 2010 tests Reference configuration: Test variables under study Number of simulated mailboxes/users 5000 concurrent users Number of databases 5 databases (active) Mailbox allocation 1000 mailboxes per each mailbox database Database size 1TB each RAID policy RAID 6 iSCSI initiator software collocation Guest initiator (residing in the VMs) Number of units, Array model, SAN configuration 1x PS6100XV 3.
Achieved Transactional IOPS are the amount of IOPS actually performed by the storage subsystem to address the transactional requests. The result should not diverge more than 5% from the planned IOPS to be considered a successful test iteration according to Microsoft Jetstress. LOGs IOPS are the IOPS performed against the log files during the transactional test. They are not directly taken into account as part of the transactional IOPS, but tracked separately instead.
In a typical iSCSI based SAN, the initiator is regarded as the client which is accessing the storage resources located on an iSCSI server, or target, by sending SCSI commands over an IP network. The initiator falls into two broad types: software initiator (software implementation installed within the operating system running the initiator) or hardware initiator (a dedicated hardware resource, most commonly a host bus adapter).
Figure 5 Storage trend with different iSCSI software initiator collocation Note: The Jetstress tool throttles its storage request activities during the simulations by way of two tuning parameters (threads and sluggish sessions). The planned IOPS and the achieved IOPS cannot exactly match due to the variance of the Jetstress threads efficiency during each iteration of the simulations, regardless the effort to configure them to have a perfect match.
Table 3 Test results: iSCSI software initiators collocation with KPI increase or decrease relationship Guest iSCSI initiator Host iSCSI initiator Achieved Transactional IOPS [% different vs. planned IOPS] 1031 IOPS [+3%] 1074 IOPS [+7%] Total IOPS of the SAN [% different vs. planned IOPS] 1524 IOPS [+52%] 1543 IOPS [+54%] DBs Read Latency [average] 9.9 msec 11.4 msec DBs Write Latency [average] 5.6 msec 6.8 msec LOGs Write Latency [average] 1.1 msec 2.
Table 4 Test parameters: RAID policies variation Reference configuration: Test variables under study RAID policy RAID 6 / RAID 50 Reference configuration: Consistent factors across this test Messages per day per mailbox / IOPS per mailbox 200 messages / 0.
• Conversion from RAID 10 to RAID 50 or RAID 6 is supported • Conversion from RAID 50 to RAID 6 is supported • Conversion from RAID 6 or RAID 5 is not supported, a reset and initialization of the array is required The two RAID policies selected for the test, reported in the Table 4 above, use the PS6100XV 3.5” as the reference array. The array was reinitialized to the factory level for each RAID level run, the volumes were recreated, and the Exchange Jetstress databases were redeployed.
Table 5 reports the numerical results recorded during the RAID policies test and the corresponding normalized values to match the planned and achieved IOPS. The percentages reported in the IOPS rows are calculated against the planned IOPS. The rows with the normalized values provide the relative increase or decrease of the Exchange KPI.
Table 6 Test parameters: Database deployment layout Reference configuration: Test variables under study Number of databases 5 / 10 / 20 databases (active) Mailbox allocation 1,000 / 500 / 250 mailboxes per each mailbox database Database size 1TB / 500GB / 250GB each Reference configuration: Consistent factors across this test Messages per day per mailbox / IOPS per mailbox 200 messages / 0.
Figure 7 Storage trend with increasing number of DBs/Volumes with 5,000 users in a two node DAG Table 7 reports the numerical results recorded during the increase of the number of DBs test and the corresponding normalized values to match the planned and achieved IOPS. The percentages reported in the IOPS rows are calculated against the planned IOPS. The rows with the normalized values provide the relative increase or decrease of the Exchange KPI.
Table 7 Test results: Database deployment layout with KPI increase or decrease relationship 5 DBs 10 DBs 20 DBs Achieved Transactional IOPS [% different vs. planned IOPS] 1031 IOPS [+3%] 997 IOPS [-0%] 948 IOPS [-5%] Total IOPS of the SAN [% different vs. planned IOPS] 1524 IOPS [+52%] 1686 IOPS [+69%] 1916 IOPS [+92%] DBs Read Latency [average] 9.9 msec 11.2 msec 17.9 msec DBs Write Latency [average] 5.6 msec 6.8 msec 8.7 msec LOGs Write Latency [average] 1.1 msec 1.1 msec 1.
Table 8 Test parameters: PS Series model assessment Reference configuration: Test variables under study Number of units, Array model 1x PS6100XV 3.5” or 1x PS6100X or 1x PS6100E Reference configuration: Consistent factors across this test Messages per day per mailbox / IOPS per mailbox 200 messages / 0.
Figure 8 Characterization of different PS Series models in a two node DAG Table 9 reports the numerical results recorded during the assessment of the PS Series model test and the corresponding normalized values to match the planned and achieved IOPS. The percentages reported in the IOPS rows are calculated against the planned IOPS. The rows with the normalized values provide the relative increase or decrease of the Exchange KPI.
Table 9 Test results: PS Series model assessment with KPI increase or decrease relationship PS6100XV 3.5” PS6100X PS6100E Achieved Transactional IOPS [% different vs. planned IOPS] 1031 IOPS [+3%] 1028 IOPS [+3%] 623 IOPS [+4%] Total IOPS of the SAN [% different vs. planned IOPS] 1524 IOPS [+52%] 1504 IOPS [+50%] 902 IOPS [+50%] DBs Read Latency [average] 9.9 msec 12.7 msec 19.8 msec DBs Write Latency [average] 5.6 msec 5.7 msec 4.1 msec LOGs Write Latency [average] 1.1 msec 1.
Table 10 Test parameters: Scaling up the number of concurrent mailbox users Reference configuration: Test variables under study Number of simulated mailboxes/users 5,000 / 7,000 / 8,000 concurrent users Number of databases 5 / 7 / 8 databases (active) Reference configuration: Consistent factors across this test Messages per day per mailbox / IOPS per mailbox 200 messages / 0.
Figure 9 shows the results collected from the Exchange Jetstress simulations when scaling up the number of mailbox users. Figure 9 Storage trend while scaling up the number of concurrent mailbox users in a two node DAG Table 12 reports the numerical results recorded during the increase of the number of concurrent mailbox users test and the corresponding normalized values to match the planned and achieved IOPS. The percentages reported in the IOPS rows are calculated against the planned IOPS.
Table 12 Test results: Scaling up concurrent mailbox users with KPI increase or decrease relationship 5,000 users 7,000 users 8,000 users Achieved Transactional IOPS [% different vs. planned IOPS] 1031 IOPS [+3%] 1457 IOPS [+4%] 1589 IOPS [-1%] Total IOPS of the SAN [% different vs. planned IOPS] 1524 IOPS [+52%] 2087 IOPS [+49%] 2273 IOPS [+42%] DBs Read Latency [average] 9.9 msec 16.4 msec 19.0 msec DBs Write Latency [average] 5.6 msec 8.5 msec 9.8 msec LOGs Write Latency [average] 1.
Table 13 Test parameters: Scaling out the SAN by a factor of two Series Reference configuration: Test variables under study Number of simulated mailboxes/users 10,000 / 14,000 / 16,000 concurrent users Number of databases 10 / 14 / 16 databases (active) #1 SAN Configuration one single pool (default) #2 SAN Configuration two pools (one array per each pool) #1 & #2 Reference configuration: Consistent factors across this test Messages per day per mailbox / IOPS per mailbox 200 messages / 0.
Figure 10 shows the results collected from the Exchange Jetstress simulations of the two series of incremental workloads while scaling out the SAN and the server resources. Figure 10 Storage trend while scaling up the number of concurrent users with different number of pools Table 14 reports the results recorded during the test that increased the number of concurrent mailbox users while scaling out the SAN and the server resources.
Table 14 Test results: Scaling out the SAN by a factor of two with KPI increase or decrease relationship Series #1 10,000 users 2 arrays / 1 pool 14,000 users 2 arrays / 1 pool 16,000 users 2 arrays / 1 pool Achieved Transactional IOPS [% different vs. planned IOPS] 2135 IOPS [107%] 2693 IOPS [96%] 3109 IOPS [97%] Total IOPS of the SAN [% different vs. planned IOPS] 3153 IOPS [158%] 3949 IOPS [141%] 4518 IOPS [141%] DBs Read Latency [average] 12.2 msec 17.6 msec 22.
The outcomes of the both series shows the ability of the EqualLogic SAN to accept and support the additional workload distributed across the two arrays. The distinctive difference of the gathered results underlines the sensitivity of Exchange workloads to read operations. The single pool configuration, with a shared cache across the arrays, better accommodates the write operations, which usually are the most critical I/O operations in a business environment.
Figure 11 shows the results collected from the Exchange Jetstress simulations of the two series of incremental workload while scaling out from one to three building blocks. Figure 11 Storage trend while scaling out the SAN Table 16 reports the numerical results recorded during the increase of building blocks test and the corresponding normalized values to match the planned and achieved IOPS. The percentages reported in the IOPS rows are calculated against the planned IOPS.
Table 16 Test results: Scaling out from one to three building blocks with KPI increase or decrease relationship 5000 users 1 array / 1 pool 10,000 users 2 arrays / 2 pools 15,000 users 3 arrays / 3 pools Achieved Transactional IOPS [% different vs. planned IOPS] 1031 IOPS [103%] 2178 IOPS [109%] 3276 IOPS [109%] Total IOPS of the SAN [% different vs. planned IOPS] 1524 IOPS [152%] 3168 IOPS [158%] 4766 IOPS [159%] DBs Read Latency [average] 9.9 msec 12.0 msec 12.
The final outcomes of the building blocks scaling exercise present an outstanding linear scalability of the solution proposed, shown both in the linear chart and in the percentages achieved or normalized always standing around 100%. This shows a limited or null deviation from the behavior of the single building block when we scale it out for a larger environment with heavier workloads. 4.
The configuration for the mailbox database servers and the activation of the databases are reported in Table 17.
Exchange Server cache (estimated) minimum requirement per each DAG node is calculated by 𝑀𝑒𝑚𝑜𝑟𝑦 = 𝑚𝑖𝑛𝑖𝑚𝑢𝑚 𝑅𝐴𝑀 𝑤𝑖𝑡ℎ 1𝑡𝑜10 𝐷𝐵𝑠 + (#𝑢𝑠𝑒𝑟𝑠 ∗ 𝐶𝑎𝑐ℎ𝑒 𝑝𝑒𝑟 𝑢𝑠𝑒𝑟) = 2𝐺𝐵 + (5,000 ∗ 12𝑀𝐵) = 62𝐺𝐵 and thus requires a building block server or VM with at least 64GB of RAM.
A list of failure events were identified that can affect the HA level of the messaging operations in this infrastructure and the options to address those scenarios were investigated. • In the event of a failure of either of one of the servers, all the active databases will be switched over to the remaining working server (five active), without any replicated passive copies available.
The time taken to complete these tasks corresponds to the amount of data to be replicated, though it is not duplicate or triplicate like the growth of the number of seeding operations. The first question to evaluate is if the duration of any of these sets of seeding operations is satisfactory for the service level agreement of the messaging system in the organization.
5 Best practices recommendations Refer to these best practices to plan and configure Hyper-V servers, Exchange Server 2010, and EqualLogic arrays. Storage best practices • Use Multipath I/O (MPIO) Device Specific Module (DSM) provided by EqualLogic HIT Kit to improve performance and reliability for the iSCSI connections. • Distribute network port connections on the controllers accordingly with the port failover mechanism and the redundancy implemented on the network switches.
Hyper-V and VMs best practices • The option of installing a Windows Server Core version in the root partition of the Hyper-V role server is advised when reducing the maintenance, the software attack surface, the memory, and disk space footprint are critical requirements. Otherwise, when installing a traditional Windows Server with Hyper-V technology with the GUI, minimize the use of additional software, components and/or roles in the root partition.
• Use default disk alignment provided by Windows 2008 or greater. • Use NTFS file system with 64 KB allocation unit for Exchange database and log partitions. • Evaluate the use of mount points for all the SAN volumes to increase management flexibility and database portability. Mount points are required when the number of volumes is greater than the number of available drive letters in the servers. • Deploy Windows operating system and Exchange data in physically separated disk drives.
A Configuration details A.1 Hardware components Table 18 lists the details of the hardware components used for the configuration setup. Table 18 Hardware components Test configuration – Hardware components Servers Dell PowerEdge M1000e Blade enclosure • 2x Chassis Management Controller (CMC), Firmware 4.11.A01 Dell PowerEdge M710 Blade Server • 2x Quad Core Intel® Xeon X5570 Processors, 2.
A.
Table 19 lists the details of the software components used for the configuration setup. Table 19 Software components Test configuration – Software components Operating Systems Host: Microsoft Windows Server 2008 R2 Enterprise Edition Service Pack 1 (build 7601) with Hyper-V • Dell OpenManage Server Administrator 7.1.0 • Broadcom Advanced control Suite 4 (version 15.2.20.2) • Dell EqualLogic Host Integration Toolkit 4.
• Spanning tree Portfast enabled for every switch port on 7048Rs and M6348s • Jumbo frames enabled for every switch port on 7048Rs, M6348s and M6220 Table 20, Table 21, and Table 22 summarize the different aspects of the networks implemented in the reference configuration and their usage.
Table 22 Configuration - VLANs VLAN ID Switch it is implemented on Purpose 100 PowerConnect M6220 Management 200 PowerConnect M6220 LAN traffic 1 (default) PowerConnect M6348, PowerConnect 7048R iSCSI Figure 13 illustrates the diagram of the connectivity between the two pairs of network switches and between them and the storage arrays.
Figure 14 presents the diagram of the network connections between the blade servers and the storage arrays. Figure 14 Network connectivity diagram A.4 Host hypervisor and virtual machines configuration A virtual infrastructure built upon Windows Server with Hyper-V hosted all the components of the test infrastructure.
• Guest (or host, when described) iSCSI initiator used to access volumes hosted on the EqualLogic SAN Table 23 lists the relation between the hypervisor host and each VM, with a brief summary of the virtual resources allocated for each VM.
The PowerEdge M710s used for our tests had two NUMA nodes each managing 48GB of memory. In order to run a single VM with an amount of memory greater than 48GB the NUMA activation was required (i.e. Exchange DAG nodes have 64GB each). Figure 15 Hyper-V setting for NUMA spanning Guest VMs disks The virtual disks in use to host the operating system of each VM were VHD fixed type disks, similar to the configuration shown in Figure 16 and were deployed on the second pair of RAID-1 disks on each host hypervisor.
Figure 17 Enumeration of VHD disks for the host initiator test case A.4.1 Host network adapters and Virtual networks configuration The host network adapters providing connectivity for the hosts and the VMs were configured as listed.
The virtual networks configured for the Hyper-V hypervisor servers followed the guidelines listed below.
Table 24 summarizes the assignment of virtual network adapter to each network grouped by VM.
• Is an ESE application requiring access to the ESE dynamic link libraries to perform database access. It takes advantage of the same API used by the full Exchange Server software stack and as such it is a reliable simulation application • Runs on a single server.
For additional information about Microsoft Exchange Jetstress and Exchange Loadgen 2010 refer to Microsoft documentation: Tools for performance and Scalability Evaluation, available at: http://technet.microsoft.com/en-us/library/dd335108.
Additional resources Support.dell.com is focused on meeting your needs with proven services and support. DellTechCenter.com is an IT Community where you can connect with Dell Customers and Dell employees for the purpose of sharing knowledge, best practices, and information about Dell products and your installations. Referenced or recommended Dell publications: • Dell EqualLogic Configuration Guide http://www.delltechcenter.
This white paper is for informational purposes only. The content is provided as is, without express or implied warranties of any kind.
