HP OpenView Storage Data Protector 5.5 Advanced Backup to Disk Performance White Paper Executive Summary .............................................................................................................................. 3 Objective and Target Audience ............................................................................................................. 4 Introducing HP OpenView Storage Data Protector 5.5 .............................................................................
Results .............................................................................................................................................. 29 Tape Drive Write Test ..................................................................................................................... 29 MSA1000 Write Test ..................................................................................................................... 29 MSA1500 Write Test ........................................................
Executive Summary This white paper provides performance-related information for HP OpenView Storage Data Protector 5.5 and the Advanced Backup to Disk feature. This white paper covers HP ProLiant Windows 2003 server connected to HP StorageWorks Modular Smart Array (MSA) disk arrays and different tape drive technologies (LTO and SDLT). The proof points are all Windows-based for simplification of equipment needs, but the lessons will still hold good for heterogeneous environments.
Objective and Target Audience The main objective of this white paper is to educate and inform users of the HP OpenView Storage Data Protector 5.5 Advanced Backup to Disk feature about what levels of performance are achievable in different backup scenarios. The emphasis is in showing what is typical and not what best-case scenarios are. This white paper highlights where the current performance bottlenecks are and how these might be overcome.
Introducing HP OpenView Storage Data Protector 5.5 HP OpenView Storage Data Protector (DP) is software that manages backup and recovery from both disks and tapes, delivering maximum data protection while providing continuous 24x7 business operations. The software is designed to simplify and to centralize backup and recovery operations by integrating a variety of techniques to eliminate backup windows. These range from on-line backup, open file backup, and instant recovery or zero-downtime backups.
Tape Drives This white paper covers the Ultrium 960, Ultrium 460 and SDLT 320 tape drives. The HP StorageWorks Ultrium 960 tape drive performance is best utilized for high-performance backup in a disk and tape SAN environment, backing up directly from a high-performance disk subsystem (from primary disk storage, split mirrors, snapshots, or clones). The HP StorageWorks Ultrium 960 is not ideal (performance wise) for network backups (e.g. 100BASE-T) or backup from lower performance disk arrays.
The HP StorageWorks SDLT 320 tape drive is very useful for existing DLT and SDLT-based environments, particularly for backward-read compatibility. Figure 3. HP StorageWorks SDLT 320 Tape Drive The following table shows the major differences: Table 1.
Basics of Backup Performance Backup performance will always be limited by one or more bottlenecks in the system, of which the tape drive is just one part. The goal is to make the tape drive the bottleneck. That way the system will achieve the performance figures as advertised on the drive's specification sheet. Please note that backup jobs can stress hardware resources up to their highest limit, which would never happen during normal application load.
• Transfer Size Transfer size is the overall total size of the SCSI transfer within a single SCSI command. In some operating systems, there is limit set on this. For example, in Microsoft Windows the default transfer size is 64 KB and to increase the overall transfer size above 64 KB in Windows, a registry entry called “MaximumSGList” (associated with the HBA) must be changed. Many modern HBAs already install their drivers with this registry value set appropriately. Check the registry entries.
Disk to Disk to Tape (D2D2T) Data Protection Architecture With the high capacity and lower cost offered by SATA and Fibre Channel ATA (FATA) disk technologies, many customers are now considering implementing backup to low cost disk arrays before backup to tape. The use of secondary disk arrays for backup is best suited to environments where: • The business dictates rapid single file restore capabilities (seconds to minutes).
Figure 4. DP 5.5 Advanced Backup to Disk Example Data Protector 5.5 - Backup to Disk Disk Staging Example Client 1. Backup client to disk (File Library) Backup server 2. Backup disk to tape (object copy) Client Staging Area Client 3b. Direct restore from tape Tape 3a.
Test Data In the following proof points, two different file systems were created to cover large file servers and typical clients, so that the results shown are realistically achievable in similar situations: 1. File server data with millions of small files 2. Typical client data with fewer files and a broad range of size (KB/MB) HPCreateData The datasets were developed using the HPCreateData PAT utility, which is downloadable from http://www.hp.com/support/pat.
Small Files The “small” file system is created with file sizes between 4 KB and16 KB and the compressibility of the data 2:1. The utility creates an equal distribution of files in each directory. Finally, the file system contains 49.2 GB with 5,535,750 files in 7,380 folders. The created files have a name with maximal 16 characters for avoiding corner cases. Figure 6.
HPReadData The HPReadData PAT utility is useful in assessing the rate at which your disk subsystem can supply data, and this is ultimately what will limit the backup performance. It simulates the way DP read files. A single instance of HPReadData can read eight streams simultaneously from your array. To read more than eight streams, initiate multiple instances of HPReadData. HPReadData is available for Windows, HP-UX, Solaris, and Linux. It can be downloaded free from http://www.hp.com/support/pat.
Configuration HP ProLiant DL380 G4 Server The HP ProLiant DL380 G4 server features 2 x 3.2 GHz processors with 4 GB of RAM, 1 x 36 GB local disk, and Windows Server 2003. All file systems are configured as NTFS. HP OpenView Storage Data Protector 5.5 HP OpenView Storage Data Protector 5.5 is configured with default values if not specified in the following subchapters. When more than one LUN is backed up, the data is multiplexed with the described concurrency parameter (see result tables).
File Library The HP OpenView Storage Data Protector Advanced Backup to Disk license (terabyte-based) enables the MSA1500 to be configured as a “file library,” so the initial backup from MSA1000 to MSA1500 takes place as a normal backup job with the file library being specified as the destination. The file library can be configured in much the same way as a tape device, with block size, segment count, disk buffers, and so on.
Figure 10. File Library Configuration – Maximum File Depot Size 10 GB (50 GB Default) Only for the Ultrium 960 tape drives, the block sizes of file library drives are configured with 256 KB (64 KB default), as described in “Getting the most performance from your HP StorageWorks Ultrium 960 tape drive white paper” (http://h18006.www1.hp.com/storage/tapewhitepapers.html). Figure 11.
File System Tree Walk During runtime, Data Protector creates backup statistics by a first tree walk, which briefly scans the files selected for the backup and calculates its size, so that the progress (percentage done) can be calculated. In a second tree walk, the data is written to the backup device. Note: If millions of small files are backed up, the first tree walk could take considerable time and increase the overall runtime.
HP StorageWorks Ultrium 460 Tape Drive The Ultrium 460 tape drive is configured with default values. Figure 13. HP StorageWorks Ultrium 460 Tape Drive – Test Environment Advanced Backup to Disk Ultrium460 Tape Drives Primary Disk Array MSA1000 with 14 x 146GB/10K disks configured as RAID1+0, 8 x 100GB LUNs configured Secondary SATA-Based Disk Array MSA1500 with 12 x 250GB/7.2K disks Backup to Secondary Array 2 Gigabit Fibre Channel Backup to Tape from Secondary Array HP ProLiant DL380G4 3.
HP StorageWorks Ultrium 960 Tape Drive The high performance Ultrium 960 tape drive is configured either with the default block size = 64 KB or with the optimal block size = 256 KB. Figure 14. HP StorageWorks Ultrium 960 Tape Drive – Test Environment Advanced Backup to Disk Ultrium960 Tape Drive Primary Disk Array MSA1000 with 14 x 146GB/10K disks configured as RAID1+0, 8 x 100GB LUNs configured Secondary SATA-based Disk Array MSA1500 with 12 x 250GB/7.
HP StorageWorks Modular Smart Array 1000 (MSA1000) The MSA1000 disk array keeps the test data (application data). It is composed of 14 x 146 GB 10K rpm disks in an enclosure and driven by a MSA1000 controller with 4 Ultra3 SCSI (160 MB/s per channel) interfaces. The array is connected to the server by a single 2 Gigabit Fibre Channel link. The disk enclosure is configured as 1 logical disk array with 14 disks. Figure 16.
HP StorageWorks Modular Smart Array 1500 (MSA1500) The MSA1500 disk array keeps the file library (backup-to-disk data). It is composed of 12 x 250 GB 7.2 rpm disks in an enclosure and driven by a MSA1500 controller with 4 Ultra3 SCSI (160 MB/s per channel) interfaces. The array is connected to the server by a single 2 Gigabit Fibre Channel link.
RAID 0 and 4 Logical Arrays The disk enclosure is configured as 4 logical disk arrays with 3 disks each. Figure 20. MSA1500 Logical Array Configuration – RAID 0 and 4 Logical Arrays 4 LUNs protected by RAID 0 are created as shown in the next figure. Figure 21.
RAID 1 and 1 Logical Array The disk enclosure is configured as 1 logical disk array with 2 disks. Figure 22. MSA1500 Logical Array Configuration – RAID 1 and 1 Logical Array 1 LUN protected by RAID 1+0 is created. Due to having only 2 disks assigned, this is equivalent to RAID 1 (no striping). Figure 23.
RAID 1+0 and 3 Logical Arrays The disk enclosure is configured as 3 logical disk arrays with 4 disks each. Figure 24. MSA1500 Logical Array Configuration – RAID 1+0 and 3 Logical Arrays 3 LUNs protected by RAID 1+0 are created (only LUNs 1-2 will be utilized for tests) as shown in the next figure. Figure 25.
RAID 5 and 2 Logical Arrays The disk enclosure is configured as 2 logical disk arrays with 6 disks each. Figure 26. MSA1500 Logical Array Configuration – RAID 5 and 2 Logical Arrays 2 LUNs protected by RAID 5 are created as shown in the next figure. Figure 27.
Test Data Each LUN is loaded with 49.8 GB typical or 49.2 GB small files as previously described in chapter Test Data. Library and Tape Tools The tape drive write tests are executed with the HP industry-leading Library and Tape Tools diagnostics (downloadable from http://www.hp.com/support/tapetools). The tool is configured to create: • Zeros with 64 KB block size • 2:1 compressible data with 64 KB block size • 2:1 compressible data with 256 KB block size Figure 28.
Figure 29. Library and Tape Tools – 2:1 compressible data with 64 KB block size Figure 30.
Results Tape Drive Write Test Table 3. Tape Drive Write Test Specification HP StorageWorks Ultrium 960 Transfer Rate (MB/s) HP StorageWorks Ultrium 460 Transfer Rate (MB/s) HP StorageWorks SDLT 320 Transfer Rate (MB/s) Zeros, 64 KB 168 112 47 Compr. 2:1, 64 KB 157 60 35 Compr. 2:1, 256 KB 157 N/A N/A MSA1000 Write Test Table 4.
MSA1000 Read Test Table 6. MSA1000 Read Test Specification MSA1000 Transfer Rate (MB/s) Typical Files LUN #1 45,70 Typical Files LUN #1-4 93,58 Small Files LUN #1 8,12 Small Files LUN #1-4 14,25 MSA1500 Read Test Table 7. MSA1500 Read Test Specification MSA1500 Transfer Rate (MB/s) RAID 0 and 1 Logical Array LUN #1 (12 Phys. Disks) 29,53 RAID 0 and 1 Logical Array LUN #1-4 (12 Phys. Disks) 32,94 RAID 0 and 4 Logical Arrays LUN #1 (3 Phys.
Backup to NULL Device Table 8. Backup to NULL Device Specification Conc. NULL Device Transfer Rate (MB/s) Average CPU Load % Typical Files LUN #1 1 45,57 8 Typical Files LUN #1-4 2 67,34 13 Typical Files LUN #1-4 4 92,80 21 Small Files LUN #1 1 7,35 9 Small Files Initial Run LUN #1-4 4 13,99 27 Small Files LUN #1-4 4 14,23 23 Specification Conc.
Table 10. Backup to Ultrium 460 Specification Conc. Ultrium 460 Transfer Rate (MB/s) Average CPU Load % Typical Files LUN #1 1 39,39 7 Typical Files LUN #1-4 2 N/A N/A Typical Files LUN #1-4 4 N/A N/A Small Files LUN #1 1 N/A N/A Small Files LUN #1-4 4 N/A N/A Table 11. Backup to Ultrium 960 Specification Conc.
Advanced Backup to Disk MSA1000 to File Library (MSA1500) Table 12. MSA1000 to File Library (MSA1500) Specification Conc. File Library 2 Drives Transfer Rate (MB/s) Average CPU Load % Typical Files LUN #1-4 Block Size 256 KB File Depot Size 50 GB 4 19,47 12 Typical Files LUN #1-4 Block Size 256 KB File Depot Size 10 GB 4 19,24 4 Small Files LUN #1-4 Block Size 256 KB File Depot Size 50 GB 4 12,22 28 File Library (MSA1500) to Tape Table 13.
Restore File Library (MSA1500) to MSA1000 Table 16. Restore from File Library (MSA1500) to MSA1000 Specification Conc. File Library Transfer Rate (MB/s) Average CPU Load % Typical Files LUN #1-4 4 42,52 9 Small Files Single 8 KB File 4 N/A < 30 s N/A Tape to MSA1000 Table 17. Restore from SDLT320 to MSA1000 Specification Conc.
Observations HP ProLiant DL380 G4 Server The highest average CPU load (84%) occurred during the small file restore (22 million) from Ultrium 960, which was caused by the slow Windows NTFS. Please check the figure below and Table18. Figure 31. HP ProLiant DL380 G4 Server - CPU Utilization during Small File Restore from Ultrium 960 The highest memory usage occurred during the Advanced Backup to Disk of small files. One backup process (bma.
Compressible (2:1) test data was saved with expected transfer rates as described in the drive’s specifications: • 35 MB/s with SDLT 320 • 60 MB/s with Ultrium 460 • 157 MB/s with Ultrium 960 For other test data, the Ultrium 460 tape drive showed exceptional results. The “NULL” test data was saved with 112 MB/s which is very good compared to 60 MB/s for compressible and 30 MB/s for native data. Please check Table1 and Table3.
RAID 1 was only tested in the 1 logical array configuration, which resulted in poor performance values (write 9,24 MB/s and read 15,72 MB/s). Furthermore, it must be considered that RAID 1 reduces the usable disk space by 50%. RAID 1+0 achieved better results than RAID 1 (write 16,96 MB/s and read 29,26 & 48,30 MB/s) due to striped disk data but the problem of inefficient disk usage remains.
Figure 33. Backup to Ultrium 960 - Average CPU Utilization with 64 KB Block Size Figure 34. Backup to Ultrium 960 - Average CPU Utilization with 256 KB Block Size These results confirm the block size recommendation of the Ultrium 960 white paper “Getting the most performance from your HP StorageWorks Ultrium 960 tape drive white paper” (downloadable from http://h18006.www1.hp.com/storage/tapewhitepapers.html).
and its SATA disks have also advantages. It enables you to improve small file backups and fast single file restores. If typical files were saved from MSA1000 to MSA1500, the maximum transfer rate was 19,47 MB/s. If saved from MSA1500 to Ultrium 960, the transfer rate was 45,06 MB/s. The transfer rate was higher, if the same files were directly saved from the MSA1000 to 2 SDLT 320 (62,87 MB/s) or 1 Ultrium 960 (91,97 MB/s). In this case, tape was faster than disk.
Figure 36. Advanced Backup to Disk - Average CPU Utilization with 10 GB File Depot Size Restore The data was restored from the file library (MSA1500) or tape (SDLT/Ultrium) to the original location (MSA1000). The typical file restore of four LUNs (concurrency = 4) showed a transfer rate of 42,52 MB/s for the file library, which was beaten by double SDLT 320 (43,12 MB/s) and one Ultrium 960 (51,00 MB/s).
Tuning Recommendations General • Ensure the Ultra320 SCSI HBA for the tape drive is placed on a dedicated 133-MHz PCI-X bus and is not sharing the bus with other HBAs. • For all HP StorageWorks Ultrium 960 backups, HP recommends the Tape Block Size be configured to 256 KB. • Match file library block size to tape drive block size.
Fragmentation of File Library File Systems Fragmentation of Windows NTFS file systems can decrease the file library performance. For best performance, each file system should belong to one file library writer only. No other writer, process or application should write to it. The following examples show one optimal and one problematic configuration. The optimal configuration with one writer results in very little fragmentation and good read performance as shown in the next three figures. Figure 37.
Figure 39. Optimal File Library Configuration – Good Read Performance The problematic configuration with multiple writers assigned to one single directory results in large fragmentation and poor read performance. Figure 40.
Figure 41. Problematic File Library Configuration – Large Fragmentation Figure 42.
Summary and Conclusions This white paper briefly describes performance-related information for HP OpenView Storage Data Protector 5.5 and the Advanced Backup to Disk feature: • The test environment is able to provide a good performance with a low usage of CPU and memory resources. Only a high number of small files cause I/O problems. • Disk staging acts as a buffer allowing media drives to operate at maximum speeds and provide the option to do automatic data replication during off-peak hours.
For more information HP OpenView Storage Data Protector http://www.hp.com/go/dataprotector HP OpenView Storage Data Protector 5.5 Guides http://ovweb.external.hp.com/lpe/doc_serv HP OpenView Storage Data Protector 5.5 Support Matrices http://www.openview.hp.com/products/datapro/spec_0001.html HP OpenView Storage Data Protector 5.5 White Papers http://www.openview.hp.com/sso/searchdocs?prod=DATAPRO&ct=TWP HP OpenView Storage Data Protector 5.5 Disk-Assisted Backup White Paper http://www.openview.hp.
