Dell EqualLogic Best Practices Series Sizing Microsoft Exchange 2010 on Dell EqualLogic PS6100 and PS4100 Series Arrays on VMware vSphere 5 A Dell Technical Whitepaper This document has been archived and will no longer be maintained or updated. For more information go to the Storage Solutions Technical Documents page on Dell TechCenter or contact support.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. © 2012 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
Table of Contents 1 Introduction ....................................................................................................................................................... 2 1.1 Purpose and Scope ................................................................................................................................... 2 1.2 Target audience ......................................................................................................................................... 2 1.
Acknowledgements This whitepaper was produced by the PG Storage Infrastructure and Solutions of Dell Inc. The team that created this whitepaper: Danilo Feroce, Puneet Dhawan, Suresh Jasrasaria, and Camille Daily We would like to thank the following Dell team members for providing significant support during development and review: Mark Welker Feedback We encourage readers of this publication to provide feedback on the quality and usefulness of this information by sending an email to SISfeedback@Dell.com.
Executive Summary The endless growth of storage capacity demand is showing no signs of waning. Disk drives with increasingly larger capacity are becoming available every year and the cost per gigabyte is routinely declining, and has progressed to such proportions that the terabyte is the new unit of storage capacity. A solution that provisions such large storage capacity, however, is not as simple as it appears.
1 Introduction Microsoft Exchange Server is a versatile messaging solution heavily relying on the storage subsystem. The 2010 version provides access to more data per mailbox with same or improved efficiency. The EqualLogic updated PS Series families (6100 and 4100), while maintaining the same building block approach and easy management, accommodate wider capacity needs and dispense increased performance.
Virtual Machine: Denotes an operating system implemented on a software representation of hardware resources (processor, memory, storage, network, etc.). Virtual machines are usually identified as ‘guests’ in relation with the ‘host’ operating system that execute the processes to allow them to run directly on the hardware. Exchange DAG: Database Availability Group is a pool of networked Exchange mailbox servers that hosts multiple copies of the same Exchange databases.
3 Microsoft Exchange and storage subsystem Microsoft Exchange Server product has a diversified set of components and services working together to accomplish the goal of supporting the most dissimilar requirements to design and deploy a messaging infrastructure within an organization.
Access Method (ISAM) technology, which organizes the database data in B-Tree structures, and as such the databases are populated with the effort of keeping the data together or adjacent. Considering this event does not always occur. Such structured databases benefit from external tasks directed towards the reorganization or defragmentation of the database data itself to restore the optimal data contiguity.
Number of Databases designates the database layout and mailbox distribution across the databases. Each Exchange database is managed as single administrative unit and is served by a set of services with a 1:1 ratio (defragmentation, maintenance, logs generation). Database and Log placement indicates whether the .edb and log files reside on the same volume or are deployed on isolated volumes.
Log Checkpoint depth refers to the amount of logs written to the disk and containing transactions not yet flushed to the database file. It is usually set at 20 in a standalone server install, and increased to 100 in a DAG configuration. The outcome of this change is a contraction in the write I/O for the given database, since the opportunity to combine user activity changes in memory (coalescing), and thus to reduce I/O increases.
4.2 Physical system configuration The physical components of the test infrastructure were laid out as show in Figure 2. We deployed this solution on Dell Blade servers planning for a greater datacenter density and flexibility of the solution.
• • Dual PowerConnect 7048R Ethernet switches (stacked) to support the iSCSI data storage traffic on the SAN side, and Link Aggregation Group (LAG) consisting of 4 fiber connections (2 from each switch) between the Fabric B M6348s and the top of the rack 7048s. More details of the test configuration setup, including a hardware and software list, SAN array characteristics, hypervisor and virtual machines relationship, network connections, and blade switch fabric paths are provided in Appendix A. 4.
5 Validate the trend of each factor This set of tests was run to evaluate the storage performance trends of an EqualLogic SAN under the load of a reference Exchange mailbox role server configuration while the deployment factors were altered.
Below is a list of metrics and pass/fail criteria recorded during testing. Most of this information is outlined by the Jetstress tool and the remainder is verified through Dell EqualLogic SAN Headquarters. Microsoft indications around thresholds for storage validation are reported as well. Database Reads Average Latency (msec) is the average length in time to wait for a database read operation. It should be less than 20milliseconds (random reads according to Microsoft threshold criteria).
Exchange messaging service end users retrieve and store their data on the server databases traversing the Exchange Server services. In turn, the Exchange Information Store service running on the mailbox server benefits from the database cache to perform its own storage access instructions via the ESE interface. The database cache is retained in memory, thus the access to and from it is considerably faster than if performed with the storage subsystem directly.
Figure 4 Storage trend under increasing workload per mailbox with 5,000 users distributed across 5 DBs Note: When simulating a different workload with Exchange Jetstress, the tool increases and decreases the generated IOPS via a couple of tuning parameters (threads and sluggish sessions). It is not always viable to exactly match the planned with the achieved IOPS because sometimes the number of threads used distributes more requests.
• • The IOPS differential represented in Figure 4 is almost entirely due to the database maintenance activities. The amount of IOPS generated to perform the maintenance for our database was constant. When evaluated as a percentage, the maintenance load had a heavy 29% impact in the lower workload scenario (0.06 IOPS per mailbox), but decreased to 11% and then 6% in the remaining simulations demanding more transactional IOPS to complete.
workload). The dispensed IOPS, and resultant latencies, for the second and third tests would show more inferior values if they were equated following the argument exposed and represented in Table 3. 5.2 Characterize the mailbox size The goal of the mailbox size variance analysis was to establish the storage trend, IOPS ratios, and relationship while varying the size of the user mailboxes and maintaining the remaining factors. The configuration parameters for the test are shown in Table 5.
The three mailbox storage quotas selected for the test (and listed in the Table 5) specify an average mailbox size expanded from 512 MB to 1 GB, and then 1.5 GB. The resultant database file size for our scenario of 1,000 users per database becomes respectively 512 GB, 1 TB, and 1.5 TB. The results collected from the Exchange Jetstress simulation of these three simulations are reported in Figure 5.
Table 6 Test results: improvement or decline relationship of performance when growing the mailbox size 512MB [500GB DB] 1GB [1TB DB] 1.5GB [1.5TB] IOPS planned 100% [900 IOPS] 100% [900 IOPS] 100% [900 IOPS] Total IOPS performed 100% [1530 IOPS] 90% [1383 IOPS] 92% [1413 IOPS] Read Latency DBs 100% [7.7 msec] 106% [8.2 msec] 121% [9.3 msec] Write Latency DBs 100% [5.2 msec] 91% [4.8 msec] 89% [4.7 msec] Write latency LOGs 100% [1.0 msec] 97% [0.9 msec] 107% [1.
user mailboxes hosted in a single database is not bound by a declared limit. It is an informed decision of the Exchange administrators to select how many users per database are deployed. The online mailbox move feature allows administrators to seamlessly redistribute mailboxes across databases or mailbox servers as well as postpone or lift constraints imposed by decisions made early in the design and deployment stage of the messaging infrastructure.
Table 8 lists the relative improvement or decline of performance recorded when increasing the number of databases. The percentages are calculated against the total IOPS (not just the transactional IOPS) shown in Figure 6.
5.4 Study the DAG number of database copies differential The goal of the high availability scenarios test was to establish the storage trend, and IOPS ratios, and relationship while manipulating the DAG technology through multiple database replica copies and maintaining the remaining factors. The configuration parameters for the test are shown in Table 9.
DAG (with five active database copies and 25 replicated), and eight nodes DAG (with five active database copies and 35 replicated). The results collected from the Exchange Jetstress simulation of these five scenarios are reported in Figure 7. Figure 7 Storage trend with increasing number of DBs copies with 5,000 users distributed across 5 DBs Table 10 reports the relative improvement or decline of performance recorded with an incremental number of database replica copies.
Table 10 Test results: improvement or decline relationship of performance with an incremental number of database replica copies IOPS planned Total IOPS performed Read Latency DBs Write Latency DBs Write latency LOGs Standalone 2 nodes 4 nodes 6 nodes 8 nodes 100% [900 IOPS] 100% [1530 IOPS] 100% [7.7 msec] 100% [5.2 msec] 100% [1.0 msec] 100% [900 IOPS] 94% [1435 IOPS] 106% [8.2 msec] 100% [5.2 msec] 100% [1.0 msec] 100% [900 IOPS] 94% [1432 IOPS] 107% [8.3 msec] 101% [5.3 msec] 104% [1.
• • • • RAID 10, or striping over multiple mirrored sets (RAID 1) The best performance for random access at the expense of an average capacity available. A minimal impact for RAID reconstruction in case of drive failure. A total of 22 available drives on PS6100, 10 available drives on PS4100, with two spare drives. RAID 50, or striping over multiple distributed parity sets (RAID 5) A balance between performance and capacity. A moderate impact for RAID reconstruction in case of drive failure.
Figure 8 Storage trend under RAID policies with 5,000 users distributed across 5 DBs Table 12 reports the relative improvement or decline of performance recorded while moving to a different RAID implementation on the EqualLogic array. The percentages are calculated against the total IOPS shown in Figure 8, not only the transactional IOPS.
The outcomes confirm the superior IOPS performance of RAID 10 level across the entire pool of policies, with a percentage gap of the competitors from 4% of RAID 5 to a maximum of 12% of RAID 6, with RAID 50 at 7% close to RAID 5. The achievements are influenced by the advantage to have a greater number of working spindles for RAID 5/6 (23 drives) when compared to RAID 10/50 (22 drives). The latencies recorded show a regular trend with the two limited deviations of improved RAID 10 and declined RAID 6.
The database cache evaluated for the pool of users in each series is reported in Table 14, again without considering the additional memory requirements due to other factors. The estimates for the database cache are based on Microsoft published metrics, and not on recorded values from our tests. Microsoft Exchange Jetstress resources utilization performs differently from an Exchange Server installation for both memory and processing resources, as reported in Appendix B.
Figure 9 Storage trend while scaling up the number of concurrent users (2,500 users per database) Table 15 reports the relative improvement or decline of performance recorded while incrementally adding concurrent users and mailbox databases during test series #1. The percentages are calculated against the total IOPS shown in Figure 9, not only the transactional IOPS.
Table 15 Test results: improvement or decline relationship of performance while scaling up concurrent users in test series #1 Series #1 IOPS planned Total IOPS performed Read Latency DBs Write Latency DBs Write latency LOGs 5,000 users 7,500 users 10,000 users 12,500 users 100% [900 IOps] 150% [1350 IOps] 200% [1800 IOps] 250% [2250 IOps] 100% [1247 IOps] 135% [1682 IOps] 179% [2234 IOps] 222% [2772 IOps] 100% [7.3 msec] 122% [8.8 msec] 156% [11.4 msec] 214% [15.5 msec] 100% [3.
Figure 10 Storage trend while scaling up the number of concurrent users (1,000 users per database) Table 16 reports the relative improvement or decline of performance recorded while incrementally adding concurrent users and mailbox databases during test series #2. The percentages are calculated against the total IOPS shown in Figure 10, not only the transactional IOPS.
Table 16 Test results: improvement or decline relationship of performance while scaling up concurrent users in test series #2 Series #2 IOPS planned Total IOPS performed Read Latency DBs Write Latency DBs Write latency LOGs 5,000 users 7,000 users 9,000 users 10,000 users 100% [900 IOps] 140% [1260 IOps] 180% [1620 IOps] 200% [1800 IOps] 100% [1530 IOps] 124% [1891 IOps] 168% [2564 IOps] 182% [2783 IOps] 100% [7.7 msec] 140% [10.8 msec] 219% [16.9 msec] 253% [19.5 msec] 100% [5.
The building block to scale out the SAN to support larger environments is framed around the user per database ratio of 1,000 defined in section 5.1. The three components involved are the EqualLogic SAN array, the Exchange server and the total workload per server, established from the reference workload of 0.18IOPS per mailbox and a set of 5,000 or 9,000 total users per server. The building block ratio of 1:1:1 is conserved across the three steps of the ladder to scale the SAN as summarized in Table 18.
Figure 11 Storage trend while scaling out the SAN (number of arrays and users) Table 20 reports the relative improvement or decline of performance recorded while incrementally adding array units, Exchange servers and concurrent users during test series #1 and #2. The percentages are calculated against the total IOPS shown in Figure 11, not only the transactional IOPS.
Table 20 Test results: improvement or decline relationship of performance when scaling out the SAN 5,000 users 1 array / 1 host 10,000 users 2 arrays / 2 hosts 15,000 users 3 arrays / 3 hosts IOPS planned 100% [900 IOps] 200% [1800 IOps] 300% [2700 IOps] Total IOPS performed 100% [1530 IOps] 177% [2714 IOps] 280% [4286 IOps] Read Latency DBs 100% [7.7 msec] 110% [8.5 msec] 126% [9.7 msec] Write Latency DBs 100% [5.2 msec] 76% [4.0 msec] 73% [3.8 msec] Write latency LOGs 100% [1.
5.8 Assess the 6100 and 4100 family models The goal of this test was to establish the storage trend and IOPS differential while utilizing different models of EqualLogic PS Series arrays to build the SAN, while keeping factors constant. The configuration parameters for the test are shown in Table 21. Table 21 Test parameters: RAID type variation Reference configuration: variable factor Array model, amount of units 1x PS6100XV 3.5” or 1x PS6100X or 1x PS6100E or 1x PS4100XV 3.
Note: The configuration for the physical host and the virtual machine hosting Exchange Jetstress when the PS4100 family was assessed had the number of network adapters (and port group) reduced from four to two. The decrease is matured with the aim of maintaining the 1:1 ratio of network adapters between the array and the host. The results collected from the Exchange Jetstress simulation against the different PS6100 family models and PS4100XV 3.5” are reported in Figure 12.
Table 22 reports the relative improvement or decline of performance recorded while assessing the different EqualLogic array models from the PS6100 family and the PS4100XV 3.5”model. The percentages are calculated against the total IOPS shown in Figure 12, not only the transactional IOPS. Table 22 Test results: improvement or decline relationship of performance with different PS Series model PS6100XV 3.5” PS6100X PS6100E PS4100XV 3.
Figure 13 Characterization for different workloads on PS4100 series Table 23 reports the relative improvement or decline of performance recorded while assessing the different EqualLogic array models from the PS4100 family. The percentages are calculated against the total IOPS shown in Figure 13, not only the transactional IOPS. Table 23 Test results: improvement or decline relationship of performance with different PS Series model 2,000 users [PS4100E] 2,000 users [PS4100XV 3.
The outcomes of the characterization of the PS4100 family manifest a solid relationship between workload applied to the PS4100XV 3.5”and the resultant set of key latency indicators, resulting in a favorable predictability for different loads. The PS4100E, equipped with 7.2K NL-SAS drives, performs adequately, powering 2,000 concurrent users, but suffers like the equivalent parent model on read latency because of the random nature of the workload.
Table 24 reassumes the numerical variables resulting from the Microsoft guidelines. A formula representation to provide the same outcome is reported here: 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑙𝑜𝑔𝑠 = 2.
3 Table 25 EqualLogic PS series drives and usable capacity examples (1 TB = 10 GB) PS series family PS6100 (24 disk drives) PS4100 (12 disk drives) Disk drive size & array model Capacity raw Capacity RAID 10 Capacity RAID 50 Capacity RAID 5 Capacity RAID 6 600 GB, XV 3.5” 14.4 TB 6600 GB 10800 GB 12600 GB 11400 GB 300 GB, X 2.5” 7.2 TB 3300 GB 5400 GB 6300 GB 5700 GB 600 GB, X 2.5” 14.4 TB 6600 GB 10800 GB 12600 GB 11400 GB 900 GB, X 2.5” 21.
• • Disk drives rotational speed: the more RPM are supplied by the drive platters the faster the IOPS requested from the host are served. The disk drive speeds currently are 15K SAS, 10K SAS and 7.2K NL-SAS (SSD drives are omitted from the list as they offer a price/performance ratio and level of IOPS inappropriate for Exchange 2010 requirements). RAID policy offers data protection from drives failure.
𝑊ℎ𝑖𝑡𝑒 𝑠𝑝𝑎𝑐𝑒 = (200 𝑚𝑒𝑠𝑠𝑎𝑔𝑒𝑠 𝑝𝑒𝑟 𝑑𝑎𝑦 ∗ 75 𝐾𝐵) = 15 𝑀𝐵 𝑀𝑎𝑖𝑙𝑏𝑜𝑥 𝐷𝐵 𝑠𝑖𝑧𝑒 = 6900 𝑢𝑠𝑒𝑟𝑠 ∗ (700 𝑀𝐵 + 240 𝑀𝐵 + 15 𝑀𝐵) = 6900 ∗ 955 𝑀𝐵 = 6590 𝐺𝐵 Transaction log files capacity is calculated by 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑑𝑎𝑖𝑙𝑦 𝑙𝑜𝑔𝑠 = 6900 ∗ 2.73 ∗ 200 𝑚𝑒𝑠𝑠𝑎𝑔𝑒𝑠 𝑝𝑒𝑟 𝑑𝑎𝑦 ∗ 75 𝐾𝐵 ∗ 1 = 6900 ∗ 41 = 282900 1 𝑀𝐵 We are not aware of the data protection solution implemented. We plan for three days log retention on the volume assuming log files backup and truncation happens on a daily basis.
EqualLogic SAN selection requires the evaluation of capacity and performance. Our capacity requirement is 9800 GB of total usable space for each of the four servers, and 2722 IOPS per server, while we deploy in two different physical locations. The RAID policy proposed is RAID 6 due to the requirements of high resiliency of the solution and it requires a minimal increase in read latency when compared to the other policies. According to the finding of section 5.
7 Best practices recommendations Refer to these best practices to plan and configure Exchange Server 2010 and EqualLogic arrays. Storage best practices • • • • • • Use MPIO DSM provided by EqualLogic HIT Kit to improve performance and reliability for the iSCSI connections. Maintain a 1:1 ratio between the number of network ports on the active arrays controller and the number of host network adapters to utilize the available bandwidth.
• • • Configure a dedicated Port Group for each virtual network adapter connected to the SAN traffic you plan to have in the virtual machines, and add one physical network adapter for every Port Group isolating it by the override switch failover option. To provide failover capability, add other network adapters present in the virtual switch to the standby list in the Port Group.
Number of mailboxes per mailbox database Reducing the number of users per database provides a more agile size to administer in case of using a traditional backup application or even when administrative tasks require to temporarily dismount the database causing a downtime. The balance between number of databases and users per databases must be cautiously planned according with the needs and size of the entire Exchange organization.
Appendix A Configuration details A.1 Hardware components Table 27 lists the details of the hardware components used for the configuration setup. Table 27 Hardware components Test configuration – Hardware components Servers • • • Network • • • Storage • • • • • BP1026 Dell PowerEdge M1000e Blade enclosure o 2x Chassis Management Controller (CMC), Firmware 3.21.A00 Dell PowerEdge M710 Blade Server o 2x Quad Core Intel Xeon X5570 Processors, 2.
A.2 Software components The setup of the environment required the deployment of the following software components: • • • • • • Bare-metal hypervisor: VMware 5.
A.3 Network configuration details Two physical networks were built in order to provide full isolation between regular IP traffic and iSCSI data storage traffic. Also, each IP network was segregated from the others by the use of VLANs with tagged traffic. In order to achieve network resiliency for hardware faults, we stacked at least two physical switches for each network, as well as made redundant the uplink (LAG) between the stacked switches.
Table 31 Configuration – VLANs VLAN ID Switch it is implemented on Purpose 100 PowerConnect M6220 Management (Service Console) 200 PowerConnect M6220 LAN traffic 1 (default) PowerConnect M6348, PowerConnect 7048R iSCSI Figure 14 presents the diagram of network connections between the blade servers and the storage arrays.
A.4 Host hypervisor and virtual machines configuration A virtual infrastructure built with VMware vSphere hosted all the components of the test infrastructure. Some key elements of the virtual infrastructure configuration were: • • • • • • VMware 5.
Figure 15 Configuration – vSwitch0 on ESX01 Figure 16 Configuration – vSwitch1 on ESX01 Table 33 reports the relationship between virtual switches, network adapters, and VLANs for each hypervisor host.
Table 33 Configuration – Virtual switches, port groups, and network adapters Host INFRA (M610) Virtual Switch vSwitch0 vSwitch0 ESXn (M710s) vSwitch1 Network Adapters vmnic0, vmnic1 vmnic0, vmnic1, vmnic2, vmnic3 Port Group VLAN ID Management 100 vCenter 100 LAN Traffic 200 Management 100 LAN Traffic 200 vmnic8 (vmnic4, vmnic9, vmnic5) iSCSI Traffic #1 vmnic4 (vmnic8, vmnic5, vmnic9) iSCSI Traffic #2 vmnic9 (vmnic5, vmnic8, vmnic4) iSCSI Traffic #3 vmnic5 (vmnic9, vmnic4, vmnic8)
A.4.2 Virtual Machines network configuration The assignment of the virtual network adapters for each virtual machine was configured as listed below.
Appendix B Microsoft Jetstress considerations Microsoft Exchange Server Jetstress 2010 is a simulation tool able to reproduce the database and logs I/O workload of an Exchange mailbox database role server. It is usually used to verify and validate the conformity of a storage subsystem solution before the full Exchange software stack is deployed.
Additional resources The following Dell publications are referenced in this document or are recommended sources for additional information. • • • • • • • • Dell EqualLogic Configuration Guide http://www.delltechcenter.com/page/EqualLogic+Configuration+Guide Dell EqualLogic PS Series Group Administration Guide https://support.equallogic.com/support/download_file.aspx?id=1125 Dell PowerEdge Blade Server and Enclosure Documentation http://support.dell.com/support/edocs/systems/pem/en/index.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
