Storage Mirroring for Linux User’s Guide Third Edition (April 2009) Part Number: T2558-96317
© Copyright 2009 Hewlett-Packard Development Company, L.P. Hewlett-Packard Company makes no warranty of any kind with regard to this material, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Table of Contents Chapter 1 Introduction .........................................................1-1 Storage Mirroring operations ............................................................ 1-3 Mirroring ............................................................................................................. 1-3 Replication ........................................................................................................... 1-4 Failure monitoring and failover ....................................
Chapter 5 Establishing a Connection Using the Connection Wizard ..................................................................................5-1 Chapter 6 Establishing a Connection Using the Connection Manager ...............................................................................6-1 Chapter 7 Establishing a Connection Across a NAT or Firewall ................................................................................7-1 Verifying the Storage Mirroring port settings .......................
Chapter 13 Failover and Failback ........................................13-1 Configuring failover ........................................................................ 13-1 Editing failover configuration ......................................................... 13-3 Removing failover configuration ..................................................... 13-4 Monitoring failover ......................................................................... 13-4 Testing failover ..................................
Introduction Welcome to Storage Mirroring® for Linux®! Storage Mirroring is a real-time data replication and failover software product. Storage Mirroring augments your existing data protection strategy by reducing downtime and dataloss, and it provides these services with minimal impact on existing network and communication resources.
The following diagram is one common Storage Mirroring configuration.
Storage Mirroring operations Storage Mirroring performs four basic types of operations.
Replication Replication is the real-time transmission of file changes. Unlike other related technologies, which are based on a disk driver or a specific application, the Storage Mirroring replication process operates at the file system level and is able to track file changes independently from the file’s related application. In terms of network resources and time, replicating changes is a more efficient method of maintaining a real-time copy of data than copying an entire file that has changed.
Failure monitoring and failover Failover is the process in which a target stands in for a failed source. As a result, user and application requests that are directed to the failed source are routed to the target. Storage Mirroring monitors the source status by tracking network requests and responses exchanged between the source and target. When a monitored source misses a user-defined number of requests, Storage Mirroring assumes that the server has failed.
Restoration Restoration provides an easy method for copying replicated data from the target back to its original location on the source. The process only requires you to select the source, target, and the appropriate replication set. There is no need to select files or to remember where the data came from on the source since that information is maintained by Storage Mirroring.
Sample configurations Storage Mirroring is an exceptionally flexible product that can be used in a wide variety of network configurations. To implement Storage Mirroring effectively, it is important to understand the possible configuration options and their relative benefits. Storage Mirroring configuration options can be used independently or in varying combinations. NOTE: The Storage Mirroring replication and failover features have different server requirements.
One-to-one, active/active Data from first source Data from second source Source and Target (Active) Target and Source (Active) Description Each server acts as both a source and target actively replicating data to each other. Applications This configuration is appropriate for failover and critical data backup. This configuration is more cost-effective than the Active/Standby configuration because there is no need to buy a dedicated target server for each source.
Many-to-one Source Source Target Source Description Many source servers are protected by one target server. Application This configuration is appropriate for offsite disaster recovery. This is also an excellent choice for providing centralized tape backup because it spreads the cost of one target server among many source servers. Considerations The target server must be carefully managed. It must have enough disk space and RAM to support replication from all of the source systems.
One-to-many Target Source Target Target Description One source server sends data to multiple target servers. The target servers may or may not be accessible by one another. Applications This configuration provides offsite disaster recovery, redundant backups, and data distribution. For example, this configuration can replicate all data to a local target server and separately replicate a subset of the mission-critical data to an offsite disaster recovery server.
Chained Source Target and Source Target Description The source servers sends replicated data to a target server, which acts as a source server and sends data to a final target server, which is often offsite. Applications This is a convenient approach for integrating local high availability with offsite disaster recovery. This configuration moves the processing burden of WAN communications from the source server to the target/source server.
Single server C:\Data Replication set data D:\Data Copy of replication set data on specified target path Source and Target Description Source and target Storage Mirroring components are loaded on the same server allowing data to be replicated from one location to another on the same volume or to a separate volume on the same server. These could be locally attached SCSI drives or Fibre Channel based SAN devices.
Virtual systems Guest Guest Guest Guest Guest Guest Guest Guest Host Host Source Target Virtual to virtual configuration Source Guest Guest Guest Guest Source Host Target Source Physical to virtual configuration Source Description Source and/or target Storage Mirroring components are loaded on the guest operating system allowing data to be replicated between virtual systems. One Storage Mirroring for Virtual Machines license can be used on up to five virtual servers.
Resources You have many resources available to you when using Storage Mirroring. Operating System and application documentation—Make sure that you have complete documentation for your operating system and your applications. Storage Mirroring documentation—The complete set of Storage Mirroring documentation includes the manuals listed below. Each manual is available in the \DblTake\docs directory on the product CD and in the root of the installation directory you selected during the installation.
Installation Review System requirements on page 2-1 and then use the appropriate instructions from the following list to meet your goal. Installing or upgrading Storage Mirroring for Linux on page 2-3—Use these instructions if you are installing or upgrading on a Linux server. Configuring your servers using DTSetup on page 2-4—Use these instructions to configure your Storage Mirroring servers.
Physical Servers and Virtual Machines Guest Operating Systems SUSE Linux Enterprise Server 10.2 for 32-bit architecture SUSE Linux Enterprise Server 10.2 for 64-bit architecture NOTE: Kernel Version 2.6.16.x.
Installing or upgrading Storage Mirroring for Linux Use these instructions if you are installing or upgrading Storage Mirroring. 1. Determine the installation package that is appropriate for your operating system. Operating System Installation File 32-Bit Operating System StorageMirroring-versionnumbers.i386.rpm 64-Bit Operating System StorageMirroring-versionnumbers.x86_64.
Configuring your servers using DTSetup DTSetup is a menu-driven application to provide easy access to Storage Mirroring server configuration. 1. Run the following command from the shell prompt to start DTSetup. DTSetup NOTE: Do not run DTSetup using the sudo command. Use a real root shell to launch DTSetup instead, either by logging in as root on the console or by using the login session of a non-privileged user to run su - to start a root shell. 2.
1.3. Configure File System or Block Device Replication—Storage Mirroring hooks operations for replication by interjecting itself between the kernel VFS layer and the underlying file system. This must be done on a case by case basis for each block device, mount point, or parent directory that needs to be replicated. NOTE: When making changes replication configuration changes, stop any applications that may be running and restart them after the replication changes have been made.
1.3.2. Configure Block Device Replication setup file—In order to use Storage Mirroring replication, data that will be replicated on a block device must be accessed through a loop device which is specially attached using the Storage Mirroring loop driver (DTLOOP). DTLOOP allows the loop device to serve as a monitoring point for operations performed on the block device so that data changes can be captured.
1.3.3.3. Manually attach DTLOOP to a block device—Enter the path to the block device that is to be replicated and press Enter. Then, enter the path to the loop device to use (/dev/loop#). DTLOOP can also use the first one available. After pressing Enter, you will be asked if you want to attach at an offset into the block device and if you want to use an encrypted loop device. In general, these options can be left blank. See the documentation for losetup for more information on using encryption. 1.3.3.4.
4.1.3. View /var/log/messages in less—This option uses the less file viewer program to view the system log messages. 4.1.4. Follow the output of /var/log/messages—This option uses tail -f to watch the output of the system log messages in real-time. 4.2. Collect and package diagnostic info—You may need to collect configuration data when reporting problems to technical support.
Installing or Upgrading the Storage Mirroring Client for Windows Use the instructions to install the Storage Mirroring clients on a Microsoft Windows machine. Your Windows version can be Windows 2003, Windows XP Service Pack 2 or later, or Windows Vista on 32-bit or 64-bit platforms. 1. Close any open applications. 2. Start the installation program using the appropriate instructions, depending on your media source. CD—Load the Storage Mirroring CD into the local CD-ROM drive.
Removing (Erasing) Storage Mirroring Use these instructions if you want to remove (erase) an existing Storage Mirroring installation. 1. Go to a shell prompt by launching a terminal application from your graphical user interface (GUI) or logging in via the Linux virtual console. 2. If you are not already, become the superuser ("root") by typing su -. 3. Erase Storage Mirroring by using the command rpm - e StorageMirroring. A successful removal returns you to the shell prompt.
Clients Storage Mirroring has two clients, the Management Console and the Failover Control Center that control and manage your connections and failover. Both clients can be started from the Windows Start menu. You can also launch the Failover Control Center from the Tools menu in the Management Console. NOTE: Storage Mirroring also has a scripting language which can be used in an interactive client or in scripts. For more information, see the Scripting Guide.
Logging on and off of Storage Mirroring To ensure protection of your data, Storage Mirroring offers multi-level security using native operating system security features. Privileges are granted through membership in user groups defined on each machine running Storage Mirroring. To gain access to a particular Storage Mirroring source or target, the user must provide a valid operating system user name and password and the specified user name must be a member of one of the Storage Mirroring security groups.
Storage Mirroring workspaces The Management Console workspace contains the display of the panes of the Management Console and any servers that may have been inserted. Multiple workspaces can be used to help organize your environment or to view settings from another machine. Saving a workspace As you size, add, or remove windows in the Management Console, you can save the workspace to use later or use on another Storage Mirroring client machine. Select File and one of the following options.
Failover Control Center From the Failover Control Center, you can manage, monitor, and control failover for your Storage Mirroring servers. The Failover Control Center displays a main window for monitoring failover activity. Control buttons to the right allow you to configure and manage your servers. Configuring communication ports The Failover Control Center use two ports for various Storage Mirroring communications.
4 Replication Sets A replication set defines the data on a source machine that Storage Mirroring protects. Replication sets are defined by volumes, directories, files, or wild card combinations. Creating multiple replication sets allows you to customize sets of data that need to be protected. When a replication set is created, a series of rules are defined that identify the volumes, directories, files, and/or wild card combinations that will be replicated to the target.
Storage Mirroring will not replicate the same data from two different replication sets on your source. The data will only be replicated from one of the replication sets. If you need to replicate the same data more than once, connect the same replication set to multiple targets. If you rename the root folder of a connected replication set, Storage Mirroring interprets this operation as a move from inside the replication set to outside the replication set.
Replication capabilities Storage Mirroring replicates all file and directory data in the supported Linux file systems. (See System requirements on page 2-1 for the file systems supported). It does not replicate items that are not stored on the file system, such as pseudo-file systems like /proc and /sys. In addition, note the following. Data that will be replicated must reside on a Storage Mirroring partition, which is specially mounted with the Storage Mirroring file system (DTFS) driver.
If you are using hard links, keep in mind the following. If a hard link exists (or is created) only inside the replication set on the source, having no locations outside the replication set, the linked file will be mirrored to the target for all locations and those locations will be linked if all link locations on the target exist on the same partition.
Creating a replication set Before you can establish a connection, you must create a replication set. 1. Highlight a source in the left pane of the Management Console and select Insert, Replication Set from the menu bar. You can also right-click on the source name and select New, Replication Set. 2. A replication set icon appears in the left pane under the source. By default, it is named New Replication Set.
Creating or modifying replication rules manually There may be times when you cannot browse for data when creating a replication set. For example, you can create a replication set rule for a directory or file that does not exist. Since you cannot browse for the location, you have to create replication set rule manually. At other times, the data you want to replicate cannot be easily selected from the Management Console. For example, you may want to select all .db files from a specific volume or directory.
Selecting a block device for replication Storage Mirroring allows you to select block devices for replication. 1. In the left pane, right-click the replication set that should include the block device and select Add Device. 2. The block devices configured for Storage Mirroring replication are shown by default. Highlight the device to include in the replication set and click OK.
Calculating replication set size While Storage Mirroring is mirroring, the right pane of the Management Console displays statistics to keep you informed of its progress. If the size of the replication set is determined before the mirror is started, Storage Mirroring can display the percentage of the replication set that has been mirrored in the Mirror Status column. If the size was not calculated prior to starting the mirror, the column displays Mirroring. 1.
Establishing a Connection Using the Connection Wizard 5 The Connection Wizard guides you through the process of protecting your data. It helps you select a source, identify the data from your source that will be included in the replication set, and select a target. The connection itself provides disaster recovery protection. If you want to extend the protection and provide high availability, you can add failover monitoring. 1.
7. Select the location on the target where the data will be stored. Send all data to a single path on the target—This option sends all selected volumes and directories to the same location on the target. The default location is /source_name/replication_set_name/volume_name. Send all data to the same path on the target—This option sends all selected volumes and directories to the same directories on the target.
Establishing a Connection Using the Connection Manager 6 After you have created a replication set, you can establish a connection through the Connection Manager by connecting the replication set to a target. 1. Open the Connection Manager to establish the connection. Highlight the replication set and select Tools, Connection Manager. Right-click on the replication set and select Connection Manager. Drag and drop the replication set onto a target.
All To One—This option replicates data from the source to a single volume on the target. The pre-defined path is /source_name/replication_set_name/volume_name. If you are replicating from multiple volumes on the source, each volume would be replicated to the same volume on the target. One To One—This option replicates data from the source to the same directory structure on the target.
Use checksum comparison to send minimal blocks of data—For those files flagged as different, the mirror performs a checksum comparison and only sends those blocks that are different. NOTE: Remirror on Auto-Reconnect—In certain circumstances, for example if the disk-based queues on the source are exhausted, Storage Mirroring will automatically disconnect connections (called auto-disconnect) and then automatically reconnect them (called auto-reconnect).
Establishing a Connection Across a NAT or Firewall 7 If your source and target are on opposite sides of a NAT or firewall, you will need special configurations to accommodate the complex network environment. Additionally, you must have the hardware already in place and know how to configure the hardware ports. If you do not, see the reference manual for your hardware.
Port Description Default Setting Storage Mirroring Server Service Listen Port The Service Listen Port sends and receives TCP communication between Storage Mirroring servers and between Storage Mirroring servers and Storage Mirroring clients. 1500 Storage Mirroring Server Heartbeat Transmit Port The Heartbeat Advertisement port sends and receives broadcast UDP communications to populate the Management Console tree with Storage Mirroring servers.
8 Simulating a Connection Storage Mirroring offers a simple way for you to simulate a connection in order to generate statistics that can be used to approximate the time and amount of bandwidth that the connection will use when actively established. This connection uses the TDU (Throughput Diagnostics Utility), which is a built-in null (non-existent) target to simulate a real connection. No data is actually transmitted across the network.
Establishing the simulated connection After you have created a replication set, you can establish the simulated connection through the Connection Manager by connecting the replication set to the TDU. 1. Opening the Connection Manager to establish the connection. Highlight the replication set and select Tools, Connection Manager. Right-click on the replication set and select Connection Manager. 2. The Connection Manager opens to the Servers tab.
Managing Connections 9 A unique connection ID is associated with each Storage Mirroring connection. The connection ID provides a reference point for each connection. The connection ID is determined by sequential numbers starting at one (1). Each time a connection is established, the ID counter is incremented. It is reset back to one each time the Storage Mirroring daemon is restarted.
Monitoring connections When a source is highlighted in the left pane of the Management Console, the connections and their statistics are displayed in the right pane. Additionally, colors and icons are used for the connections, and the Storage Mirroring servers, to help you monitor your connections. Review the Connection Statistics and Connection and Server Display tables for statistics and status descriptions. (Statistics marked with an asterisk (*) are not displayed, by default.
Statistic Mirror Status Description Idle—Data is not being mirrored to the target machine. Paused—Mirroring has been paused. Percentage Complete—If the file size of the replication set has been calculated and the data is being mirrored to the target machine, the Mirror Status will display the percentage of the replication set that has been sent. Waiting—Mirroring is complete, but data is still being written to the target. Restoring—Data is being restored from the target to the source.
Statistic Description Sent Compressed Mirror (Bytes) * The sent compressed mirror (bytes) statistic is the total number of compressed mirror bytes that have been transmitted to the target. If compression is disabled, this statistic will be the same as sent mirror (bytes). Skipped Mirror (Bytes) The skipped mirror (bytes) statistic is the total number of bytes that have been skipped when performing a difference or checksum mirror.
Connection and Server Display Icon or Color Location Icons and colors displayed in the right pane when a server is highlighted in the left pane Left pane icon Description A green checkmark on a connection indicates the connection is working properly. A red X on a connection indicates a connection error. For example, an error may be caused by broken transmission or pending replication. To determine the exact problem, locate the connection data item that appears in red.
Monitoring the message window In addition to the statistics and status shown in the Management Console, you can also open a message window to view Storage Mirroring processing alerts. These alerts include notifications, warnings, and errors. NOTE: The standard appearance of the message window is a white background. If your message window has a gray background, the window is inactive.
Message Window Control Pause/ Resume Description Toolbar Icon Pauses and resumes the message window. Pausing prevents new messages from being displayed in the message window so that you are not returned to the bottom of the message window every time a new message arrives. The messages that occur while the window is logged are still logged to the Storage Mirroring log file. Resuming displays the messages that were held while the window was paused and continues to display any new messages.
Queuing Storage Mirroring data During the Storage Mirroring installation, you identified the amount of disk space that can be used for Storage Mirroring queuing. Queuing to disk allows Storage Mirroring to accommodate high volume processing that might otherwise fill up system memory. For example, on the source, this may occur if the data is changing faster than it can be transmitted to the target, or on the target, a locked file might cause processing to backup.
6. Like the source, system memory on the target contains the oldest data so when data is applied to the target, Storage Mirroring pulls the data from system memory. Storage Mirroring automatically moves operations from the oldest transaction log file to system memory. As a transaction log is depleted, it is deleted. When all of the transaction log files are deleted, data is again written directly to system memory (step 4).
Configuring queuing You should configure queuing on both the source and target. 1. Right-click the server on the left pane of the Management Console and select Properties. 2. Select the Queue tab. 3. Specify the Folder where the disk queue will be stored. Storage Mirroring displays the amount of free space on the volume selected. Any changes made to the queue location will not take effect until the Storage Mirroring daemon has been restarted on the server.
Since the source is typically running a production application, it is important that the amount of memory Storage Mirroring and the other applications use does not exceed the amount of RAM in the system. If the applications are configured to use more memory than there is RAM, the system will begin to swap pages of memory to disk and the system performance will degrade.
Configuring automatic reconnections While disk queues are user configurable and can be extensive, they are limited. If the amount of disk space specified for disk queuing is met, additional data could not be added to the queue and data would be lost. To avoid any data loss, the auto-disconnect and auto-reconnect processes occur. Exhausted queues on the source—If disk queuing is exhausted on the source, Storage Mirroring will automatically start disconnecting connections.
Configuring compression To help reduce the amount of bandwidth needed to transmit Storage Mirroring data, compression allows you to compress data prior to transmitting it across the network. In a WAN environment this provides optimal use of your network resources. If compression is enabled, the data is compressed before it is transmitted from the source. When the target receives the compressed data, it uncompresses it and then writes it to disk.
Mirroring and Replication Mirroring and replication are the key components of Storage Mirroring. This chapter contains information on the following mirroring and replication capabilities. Controlling mirroring on page 10-1 Controlling an automatic mirror on page 10-3 Configuring and removing orphan files on page 10-5 Starting replication on page 10-6 Controlling mirroring After a connection is established, you need to be able to control the mirroring.
File Differences Mirror Options Compared Server Properties Source Taba Checksum All ( ) ( ) ( ) Connection Manager Mirroring Tab or Start Mirror Dialog File Differences Only if Source is Newer Action Performed Checksum Any file that is different on the source and target based on the date, time, and/or size is transmitted to the target. The mirror sends the entire file. Any file that is newer on the source than on the target based on date and/or time is transmitted to the target.
Controlling an automatic mirror In certain circumstances, for example if the disk-based queues on the source are exhausted, Storage Mirroring will automatically disconnect connections (called auto-disconnect) and then automatically reconnect them (called auto-reconnect). In order to ensure data integrity on the target, Storage Mirroring will perform an automatic mirror (called an auto-remirror) after an auto-reconnect. NOTE: Auto-remirror is a per source option.
Full—All files are sent to the target. NOTE: Database applications may update files without changing the date, time, or file size. Therefore, if you are using database applications, you should use the File Differences with checksum or Full option. See the table File Differences Mirror Options Compared on page 10-2 for a comparison of how the file difference remirror settings work together, as well as how they work with the global checksum setting on the Source tab of the Server Properties. 5.
Configuring and removing orphan files An orphan file is a file that exists in the target’s copy of the replication set data, but it does not exist in the source replication set data. An orphan file can be created when you delete a file contained in the source replication set while there is no Storage Mirroring connection. For example, if a connection was made and a mirror was completed and then the connection was stopped and a file was deleted on the source, an orphan file will exist on the target.
3. Specify if you want to log the name of the orphan files to the Storage Mirroring log file on the target by marking Log Orphaned Files to Target Log. 4. By default, the orphan files feature is disabled. To enable it, markMove/Delete Orphan Files. 5. Specify if you want to Delete Orphaned Files or Move Orphaned Files to a different location. If you select the move option, identify the location where these orphan files will be located.
Verification Verification is the process of confirming that the data on the target is identical to the data on the source. Verification creates a log file detailing what was verified as well as which files are not synchronized. If the data is not the same, Storage Mirroring can automatically initiate a remirror. The remirror ensures data integrity between the source and target. Manual verification A manual verification can be run anytime a mirror is not in progress. 1.
Scheduled verification Verification can be scheduled to occur automatically at periodic intervals. 1. Right-click the connection on the right pane of the Management Console and select Connection Manager. 2. Select the Verify tab. 3. Specify when you want to start the initial verification. Select the immediate date and time by clicking Now, or enter a specific Date and Time. The down arrow next to Date displays a calendar allowing easy selection of any date. Time is formatted for any AM or PM time. 4.
Verification log A verification log is created on the source during the verification process. The log identifies what is verified as well as which files are not synchronized. 1. Right-click the source server on the left pane of the Management Console and select Properties. 2. Select the Logging tab. 3. At the top of the window, Folder identifies the location where the log files identified on this tab are stored.
Data Transmission 12 Storage Mirroring data is continuously transmitted to the target machine. Although the data may be queued if the network or target machine is slow, the default transmission setting is to transmit the data as soon as possible. You can modify the tranmission to suit your environment. Manual transmission control—You can maintain the source/target connection, but still control the transmission of data across the network by using the manual transmission controls.
Scheduling transmission criteria and bandwidth limiting Using the Connection Manager Transmit tab, you can set start and stop criteria along with a schedule window. NOTE: Storage Mirroring checks the schedule once every second, and if a user-defined criteria is met, transmission will start or stop, depending on the option specified. Any replication sets from a source connected to the same IP address on a target will share the same scheduled transmission configuration. 1.
4. When you schedule transmission start criteria, transmission will start when the criteria is met and will continue until the queue is empty or a transmission stop criteria is met. Select the Start option in the Limit Type box. Define the start options for Storage Mirroring transmission by using any combination of the following options. Transmission session start—This option establishes a date and time of the day to begin transmitting data.
5. Schedule any desired stop criteria to stop transmission after a transmission start criteria has initiated the transmission. If you do not establish a stop criteria, transmission will end when the queue is empty. Select the Stop option in the Limit Type box. Define the stop options to stop Storage Mirroring transmissions by using either or both of the following options. Time Limit—The time limit specifies the maximum length of time for each transmission period.
6. Schedule a transmission window to establish a period of availability for all Storage Mirroring transmissions. You can specify a begin and end time for all Storage Mirroring transmissions. When a transmission window is in effect, all other start and stop criteria are bound by this window. This means that Storage Mirroring will never transmit data outside of an established window, regardless of other transmission settings. For example, if you set a window of availability from 9 p.m. to 4 a.m.
Failover and Failback Failover is the process in which a target stands in for a failed source. As a result, user and application requests that are directed to the failed source are routed to the target. Storage Mirroring monitors the source status by tracking network requests and responses exchanged between the source and target. When a monitored source misses a user-defined number of requests, Storage Mirroring assumes that the machine has failed.
Click Browse to search for a machine. Select a domain from the list box at the top of the Select Machine dialog box to list the available machines for that domain. Highlight a source to be monitored and click OK. Click Custom. Enter the name of the server and click Add. Specify the IP address and subnet mask of the specified server and click OK. Click OK again. The Insert Source Machine dialog closes and the Monitor Settings dialog remains open with your source listed in the Names to Monitor tree. 6.
Include Unmonitored—All of the IP address(es) will be failed over. NOTE: If you are monitoring multiple IP addresses, IP address conflicts may occur during failover when the number of IP addresses that trigger failover is less than the number of IP addresses that are assumed by the target during failover.
Removing failover configuration If you want to discontinue monitoring a source, highlight that machine on the Monitored Machines tree on the main Failover Control Center screen and click Remove Monitor. No additional dialog boxes will open. Monitoring failover Since it can be essential to quickly know the status of failover, Storage Mirroring offers various methods for monitoring the state of failover.
The following table identifies how the visual indicators change as the status of failover changes. Time to Fail Countdown Source is Online Source Fails and Failover is Initiated Failover is Complete Status Bar Colored Bullets Desktop Icon Tray The Time to Fail counter is counting down and resetting each time a response is received from the source machine. The status bar indicates that the target machine is monitoring the source machine. The bullets are green.
Initiating failback When failover occurs, a source machine has failed. The steps below must be completed in order to complete failback without IP address or name conflicts. 1. Verify that your source machine is not connected to the network. If it is, disconnect it. 2. Resolve the source machine problem that caused the failure. WARNING: Do not connect the source machine to the network at this time. 3.
5. You will be prompted to determine if you want to continue monitoring the source. Bring the source online and select Continue or Stop to indicate if you want to continue monitoring the source. After you have selected whether or not to continue monitoring the source machine, the source post-failback script, if configured, will be started. NOTE: At this time, the source machine must be online and Storage Mirroring must be running to ensure that the source post-failback script can be started.
Restoration The Management Console provides an easy method for restoring replicated data from the target back to the original source or to a new source server. You are only required to input the original source, the target, and the name of the replication set you want to restore. Storage Mirroring handles the rest, including selecting the files in the replication set and restoring them to the correct location. 1. From the Management Console, select Tools, Restoration Manager. 2.
7. Select the Use Backup Replication Set check box to use the target’s copy of the replication set database for the restoration. If this check box is not marked, you will be accessing the replication set information from the source. 8. Select the Restore Replication Set check box to restore the target’s copy of the replication set database to the source during the restoration process. 9. Select the restoration conditionals that you want to use.
Monitoring Tools In addition to the monitoring capabilities within the Storage Mirroring clients, you can monitor Storage Mirroring processing using other tools. Log files are generated for various Storage Mirroring components recording alerts (notifications, warnings, and errors). These log files can be viewed using a standard text viewer. See Log files on page 15-2. The Windows Event Viewer also receives alerts (notifications, warnings, and errors), but only from the Storage Mirroring service.
Log files Various Storage Mirroring components (Storage Mirroring daemon, Management Console, Failover Control Center, and the Command Line Client) generate a log file to gather alerts, which are notification, warning, and error messages. The log files are written to disk. They can be viewed, from the location where Storage Mirroring is installed, with a standard text viewer. Each log file consists of a base name, a series number, and an extension.
Viewing the log files The log files can be viewed, from the location where Storage Mirroring is installed, with a standard text viewer. Sample Storage Mirroring Daemon Log File 01/15/2007 14:14:18.3900 01/15/2007 14:14:18.4200 01/15/2007 14:14:18.5350 01/15/2007 14:14:18.6760 01/15/2007 14:14:18.9870 01/15/2007 14:24:15.2070 ip://206.31.4.305 01/15/2007 14:24:16.3090 01/15/2007 14:24:40.9680 01/15/2007 14:25:22.4070 01/15/2007 14:25:22.5030 ip://206.31.4.305. 01/15/2007 14:25:22.6060 ip://206.31.4.
Log messages The following table describes some of the standard Storage Mirroring alerts that may be displayed in the log files. The ID appears in column 7 of the log file, and the message appears in column 8. NOTE: In the following table, con_id refers to the unique connection ID assigned to each connection between a source replication set and a target. ID Message 0 N/A 7 Synchronous ioctl returned STATUS_ PENDING Failed to reset Replication Flags. Replication may not be performed correctly.
ID Message Description 78 Auto-disconnect threshold has been reached. The Storage Mirroring queue has exceeded its limit, and the auto-disconnect process will disconnect the source and target connection. The auto-reconnect process will automatically reestablish the connection if the auto-reconnect feature is enabled.
ID Message Description 97 Restore started con_id The restoration process of copying the up-to-date data from the target back to the original source machine has started. 98 Restore completed con_id The restoration process of copying the up-to-date data from the target back to the original source machine has been completed. 99 RepSet Modified: repset_ name This message means that the specified replication set has been modified.
ID 52000 Message Unfinished Op error Description This error message contains various Microsoft API codes. The text Code - Internal appears at the end of this message. The code value indicates why the operation failed, and the internal value indicates the type of operation that failed. The most common code values that appear in this error message are: (5) Permission denied: The account running the Storage Mirroring daemon does not have permission to update the file specified.
ID 600002 700000 800000 Message Unified login provides ADMIN access User user has level access (x) The source machine source_machine is not responding to a ping. Description Using the current login grants ADMIN access. The listed user has listed access level and access level ID. This occurs when all monitored IP addresses on the source machine stop responding to pings.
Linux System Log An event is a significant occurrence in the system or in an application that requires administrators to be notified. The operating system writes notifications for these events to the Linux system log. The location of the log file depends on the configuration of /etc/syslog.conf, however, by default, it is /var/log/messages. The following table identifies the events generated by Storage Mirroring.
ID Category Severity Event Message Required Response 4007 Service Warning Auto-disconnecting from %1 (%2) for Replication Set %3, ID: %4 due to %5 The connection is auto-disconnecting because the disk-based queue on the source has been filled, the daemon has encountered an unknown file ID, the target server has restarted, or an error has occurred during disk queuing on the source or target (for example, Storage Mirroring cannot read from or write to the transaction log file). 4008 Service Info.
ID Category Severity Event Message Required Response 4023 Service Info. Service has completed a mirror to %1 %2 for Replication Set %3, ID: %4, %5 No action required. 4024 Service Info. Service has started Replication to %1 (%2) for Replication Set %3, ID: %4 No action required. 4025 Service Info. Service has stopped Replication to %1 (%2) for Replication Set %3, ID: %4 No action required. 4026 Service Info. The target has been paused due to user intervention. No action required.
ID Category Severity Event Message Required Response 4034 Service Error Error verifying the service name. The product functionality has been disabled. Reinstall the software, using the installation Repair option, to install a new copy of the RSResource.dll. Contact technical support if this error persists. 4035 Service Error Error verifying the product name. The product functionality has been disabled.
ID Category Severity Event Message Required Response 4043 Service Error This product can only be run on a limited number of processors and this server exceeds the limit. The product functionality has been disabled. Reinstall the software, using the installation Repair option, to install a new copy of the RSResource.dll. Contact technical support if this error persists. 4044 Service Error An error was encountered and replication has been stopped.
ID Category Severity Event Message Required Response 4056 Service Info. Service has stopped a restore task to %1 (%2) for Replication Set %3, ID: %4 No action required. 4057 Service Info. Service has completed a restore task to %1 (%2) for Replication Set %3, ID: %4 No action required. 4058 Service Info. Service has started a verification task to %1 (%2) for Replication Set %3, ID: %4 No action required. 4059 Service Info.
ID Category Severity Event Message Required Response 4150 Service Error Target write %1 failed after %2 retries and will be discarded. See the event log or log files for error conditions. After correcting the problem, you should re-mirror or run a verify to resynchronize the changes. The operation has been retried according to the TGExecutionRetryLimit setting but was not able to be written to the target and the operation was discarded. Correct the problem and remirror the files.
ID Category Severity Event Message Required Response 4303 Service Warning Blocking of target path: %1 failed. Error Code: %2 If you need to block your target paths, contact technical support. 4304 Service Info. Target Path %1 has been successfully unblocked No action required. 4305 Service Warning Unblocking of target path: %1 failed. Error Code: %2 If you need to unblock your target paths, contact technical support.
ID Category Severity Event Message Required Response 5108 Failover Error The source machine %1 is not responding to a ping. This occurs when all monitored IP addresses on the source machine stop responding to pings. Countdown to failover will begin at the first occurrence and will continue until the source machine responds or until failover occurs. 5109 Failover Error The public NIC on source machine %1 is not responding to a ping.
ID Category Severity Event Message Required Response 5502 Service Warning E-mail notification could not be initialized. Check to make sure Internet Explorer 5.0 or later is installed. E-mail notification requires Internet Explorer 5.0 or later. Verify that you have this version or later installed on the Storage Mirroring server. 5503 Service Warning E-mail notification could not be processed. Check to make sure the correct version of SMTPMail.DLL is registered on the system (error code: %1).
DTStat Statistics logging is the process of taking snapshots of Storage Mirroring statistical data. The data can be written to a file for future use. Changes to the DTStat file configuration are detected and applied immediately without restarting the Storage Mirroring daemon. The statistics log file created is a binary file. To view the log file, you must run DTStat from the command prompt.
Configuring the statistics file 1. Right-click a machine in the left pane of the Management Console and select Properties. 2. Select the Logging tab. 3. At the top of the tab, specify the Folder where the log files for messages, alerts, verification, and statistics will be saved. 4. Under Statistics, specify the following information: Filename—The name of the statistics log file. The default file name is statistic.sts. Maximum Length—The maximum length of the statistics log file.
Running DTStat from the command prompt From a command prompt, use the DTStat command from the location where Storage Mirroring is installed.
DTStat statistics The following statistics appear in the default statistic.sts file. NOTE: The categories you see will depend on the function of your server (source, target, or both). If you have multiple IP addresses connected to one target server, you will see multiple Target sections for each IP address. If you convert your statistics output to an ASCII, comma-delimited file using the dtstat -s option, keep in mind the following differences.
Category Kernel Target Statistic Description MirOps Generated The number of mirror operations transmitted to the target. Mirroring is completed by transmitting the file system operations necessary to generate the files on the target. This statistic indicates the number of file system operations that were transmitted during the initial mirror. It will continue to increase until the mirror is complete. Any subsequent remirrors will reset this field to zero and increment from there.
Category Target Source Connection Statistic Description Bytes In Target Disk Queue The number of bytes currently in the disk queue on the target Tasks Succeeded The number of task commands that have succeeded on the target TasksFailed The number of task commands that have failed on the target Tasks Ignored The number of task commands that have been ignored on the target auto DisConnects The number of automatic disconnects since starting Storage Mirroring.
Category Statistic Description Connection conOpsInMir Queue The number of mirror operations currently waiting to be executed on the target conBytesIn RepQueue The number of replication bytes remaining to be transmitted to the target conOpsTx The number of operations transmitted to the target. This is the total number of operations that Storage Mirroring has transmitted as a source. In other words, the cumulative number of operations transmitted by this source to all connected targets.
SNMP SNMP, Simple Network Management Protocol, is the Internet's standard for remote monitoring and management of hosts, routers and other nodes and devices on a network. Storage Mirroring provides an SNMP sub-agent that monitors Storage Mirroring and can be managed from an SNMP Management Console. Storage Mirroring installs two components to work with SNMP. The sub-agent is a program that installs and runs on the same machine as Storage Mirroring and gathers statistics, data, and traps.
SNMP statistics The following tables identify the Storage Mirroring statistics provided through SNMP.
Object Type Connection Statistic dtconState Description The state of the active connection 0—None. This indicates a connection has not been established. Statistics are still available for the source and target machines. 1—Active. This indicates that the connection is functioning normally and has no scheduling restrictions imposed on it at this time. (There may be restrictions, but it is currently in a state that allows it to transmit.) 2—Paused. This indicates a connection that has been paused.
SNMP traps The following tables identify the Storage Mirroring traps provided through SNMP.
Object Type Trap Connection dttrapAuto DisconnectPause Transmission Auto-disconnect requested that the source pause any operation (create, modify, or delete) sending dttrapAuto DisconnectEnd Connection Auto-disconnect has intentionally dropped the connection dttrapAuto Disconnect Shutdown Auto-disconnect forced Storage Mirroring to shutdown dttrapReplication Start Replication has started dttrapReplication Stop Replication has stopped dttrapMirrorStart Mirroring has started dttrapMirrorStop M
16 Server Settings Most of the Storage Mirroring server settings are located in the Management Console Server Properties dialog box. To access this dialog box, right-click a server in the left pane of the Management Console and select Properties. The Server Properties dialog box contains multiple tabs with the Storage Mirroring server settings. For information on the server settings not available through the Management Console, see the Scripting Guide.
General The General tab identifies the server. Nickname—A nickname is saved in the Management Console workspace, therefore, it only appears in the Management Console on this server. It is not communicated across the network. If you export a workspace and use it on another Storage Mirroring server, the server nickname will appear there also. Machine—This is the actual server name. This field is not modifiable. Addresses—The IP address(es) for this server are listed in this field.
Activation Code—This is the Storage Mirroring license which is required on every Storage Mirroring machine. There are three licenses available. Evaluation—A license that has an end date built into the activation code. When the license expires, the software will no longer function. The same evaluation licenses can be used on multiple machines on a network. Single—A license that is available on a per machine basis.
Setup The Setup tab contains various setup options. Log Statistics Automatically—If enabled, Storage Mirroring statistics logging will start automatically when Storage Mirroring is started. Enable Task Command Processing—Task command processing is a Storage Mirroring feature that allows you to insert and run tasks at various points during the replication of data. Because the tasks are user-defined, you can achieve a wide variety of goals with this feature.
Network The Network tab contains settings that control network communications. Any changes made to this tab will not take effect until the Storage Mirroring daemon has been restarted on the server. Default Address—On a machine with multiple NICs, you can specify which address Storage Mirroring traffic will use. It can also be used on machines with multiple IP addresses on a single NIC. Default Protocol—The default protocol for all Storage Mirroring communications is the TCP/IP protocol.
Queue The Queue tab contains settings for Storage Mirroring queue usage. Folder—This is the location where the disk queue will be stored. Storage Mirroring displays the amount of free space on the volume selected. Any changes made to the queue location will not take effect until the Storage Mirroring daemon has been restarted on the server. Select a location on a volume that will have minimal impact on the operating system and applications being protected.
application. In this case, you would need to configure the applications so that they collectively do not exceed the amount of RAM on the server. Perhaps on a server with 1 GB of RAM running the application and Storage Mirroring, you might configure the application to use 512 MB and Storage Mirroring to use 256 MB, leaving 256 MB for the operating system and other applications on the system.
Source The Source tab contains settings specific to the source module of Storage Mirroring. Replication Packets to 1 Mirror Packet—You can specify the ratio of replication packets to mirror packets that are placed in the source queue. Specify a larger number if you have a busy network that has heavy replication. Also, if you anticipate increased network activity during a mirror, increase this number so that the replication queue does not get too large.
Use Checksum on All blocks of data during a Difference Mirror—This option allows a file difference mirror to check each block of data, regardless of the file attributes. If this option is not marked, Storage Mirroring will assume files are synchronized if their attributes match. NOTE: Database applications may update files without changing the date, time, or file size. Therefore, if you are using database applications, you should use the Block Checksum All option to ensure proper file comparisons.
Target The Target tab contains settings specific to the target module of Storage Mirroring. Target Mirror Capacity High Percentage—You can specify the maximum percentage of system memory that can contain mirror data before the target signals the source to pause the sending of mirror operations. The default setting is 20.
Database The Database tab identifies the Storage Mirroring database files. Folder—Specify the directory where each of the database files on this tab are stored. The default location is the directory where the Storage Mirroring program files are installed. Replication Set—This database file maintains which replication sets have been created on the server along with their names, rules, and so on. The default file name is DblTake.db.
Logging The Logging tab contains settings for various Storage Mirroring log files. Folder—Specify the directory where each of the log files on this tab are stored. The default location is the directory where the Storage Mirroring program files are installed. Messages & Alerts Maximum Length—Specify the maximum length of the client and daemon log files. The default size is 1048576 bytes and is limited by the available hard drive space.
E-mail notification The E-mail Notification tab allows you to specify e-mail notification settings. When enabled, the messages logged to the Linux System Log will be sent as e-mail messages to the addresses specified. The subject of the e-mail will contain an optional prefix, the server name where the message was logged, the message ID, and the severity level (information, warning, or error). The text of the message will be displayed in the body of the e-mail message.
Subject Prefix and Add event description to subject—The subject of each e-mail notification will be in the format Subject Prefix : Server Name : Message Severity : Message ID : Message Description. The first and last components (Subject Prefix and Message Description) are optional. The subject line is limited to 150 characters. If desired, enter unique text for the Subject Prefix which will be inserted at the front of the subject line for each Storage Mirroring e-mail message.
17 Security To ensure protection of your data, Storage Mirroring offers multi-level security using native operating system security features. Privileges are granted through membership in user groups defined on each machine running Storage Mirroring. To gain access to a particular Storage Mirroring source or target, the user must provide a valid operating system user name and password and the specified user name must be a member of one of the Storage Mirroring security groups.
Clearing maintained security credentials To remove cached credentials, follow these steps. 1. To access the credentials security option, select File, Options and select the Security tab. 2. To remove the security credentials, click Clear Cached Security Credentials. 3. Click OK.
Index A H O Active Directory alternate data streams 4-3 hard links 4-3 heartbeats 16-5 high availability overview 1-1 one-to-many configuration 1-10 one-to-one configuration 1-7, 1-8 orphan files 10-5 overview operations 1-3 product 1-1 B block target paths 16-10 C I ignore delete operations 16-8 centralized backup 1-1 in-band control chained configuration 1-11 See task command processing P client logs 16-12 insert servers Management ports client overview 3-1 Console 7-2 Failover Control Center 3-4
source settings 16-8 sparse files 4-3 statistics DTStat 15-19 log 16-12 SNMP 15-26 system information 8-1 system log 15-9 system requirements 2-1 T target controls 12-1 target settings 16-10 task command processing 16-4 TDU 8-1 Throughput Diagnostics Utility 8-1 transmission controls 12-1 manual controls 12-1 overview 12-1 scheduling 12-2 V verification log 11-3, 16-12 manual control 11-1 overview 11-1 scheduling 11-2 virtual server configuration 1-13 virus protection 4-2 W workspaces 3-3 II
