Building Disaster Recovery Serviceguard Solutions Using Metrocluster with Continuous Access XP P9000 for Linux B.01.00.
Legal Notices © Copyright 2012 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. The information contained herein is subject to change without notice.
Contents 1 Introduction...............................................................................................6 Overview of Continuous Access P9000 and XP concepts...............................................................6 PVOLs and SVOLs................................................................................................................6 Device groups.....................................................................................................................
Configuring the remote array RAID Manager instance for automatic startup..........................32 Restrictions...................................................................................................................33 4 Using XP or P9000 features in a Metrocluster environment.............................34 Metrocluster with Continuous Access XP P9000 for Linux and thin provisioning volumes...................34 HP storage works smart tiers........................................................
C Sample configuration files for Metrocluster with Continuous Access XP P9000 for Linux B.01.00.00....................................................................................64 Sample Raid Manager configuration file....................................................................................64 Sample file for configuring automatic Raid Manager startup.........................................................66 Sample RAID Manager configuration file for the Remote Command Device................
1 Introduction This document describes the Continuous Access P9000 and XP software and the additional files that integrate the XP or P9000 Disk Arrays with Metrocluster. The document explains how to configure Metrocluster using Continuous Access XP or P9000.
Continuous Access Synchronous Replication In Continuous Access Synchronous replication, all write operations on the primary volume are replicated to the secondary volume before the write is acknowledged to the host. This synchronous replication mode ensures the highest level of data currency possible. Host I/O performance is directly impacted by the distance between the primary and secondary volumes.
(I/O ordering). A CT group is equal to a device group in the Raid Manager configuration file. A consistency group ID (CTGID) is assigned automatically during pair creation. NOTE: Different P9000 and XP models support different maximum numbers of Consistency Groups. For details, see the P9000 user guide or XP user guide.
Figure 2 Journal Volume Primary Host Primary Data Volume Secondary Host Secondary Data Volume Issuing Read Journal Command Journal obtain function Journal restore function Journal Copy Function Master Journal Volume Restore Journal Volume Primary Storage System Secondary Storage System Site A Site B By writing the records to journal disks instead of keeping them in cache, Continuous Access Journal overcomes the limitations of earlier asynchronous replication methods.
mode, noting changed tracks in a bitmap for future resynchronization. Recovery typically involves a destructive process such as rewriting all the changed tracks, with possible loss of data consistency for ordered writes. In contrast, Continuous Access Journal logs every change to the journal disk at the primary site, including the metadata required to apply the changes consistently.
The XP or P9000 RAID Manager allows you to run XP or P9000 Business Copy software and XP or P9000 Continuous Access software commands from a host. Every execution of XP or P9000 RAID Manager is known as a RAID Manager instance. RAID Manager instances running on the local nodes communicate with the RAID Manager instances running on the remote nodes to get the status of the device group pair.
Figure 3 Sample configuration of Metrocluster with Continuous Access XP P9000 for Linux Quorum Server A B Node 1 Node 2 Node 3 Node 4 Metrocluster Router Router Synchronous / Asynchronous / Journal XP Array XP Array Site 1 NOTE: Site 2 The P9000 disk array family does not support asynchronous replication using side file.
NOTE: If the monitor is configured to automatically resynchronize the data from PVOL to SVOL upon link recovery, a Business Copy (BC) volume of the SVOL must be configured as another mirror. In the case of a rolling disaster and the data in the SVOL becomes corrupt due to an incomplete resynchronization, the data in the BC volume can be restored to the SVOL.
2 Configuring an application in a Metrocluster environment Installing the necessary hardware and software When you complete the following procedures, an adoptive node is able to access the data belonging to a package after it fails over. Setting up the storage hardware 1. 2. Before configuring Metrocluster with Continuous Access XP P9000 for Linux, the XP or P9000 arrays must be correctly cabled with redundant paths to every node in the cluster that will run packages accessing data on the array.
Site Aware Failover Configuration Serviceguard cluster allows sites to be configured in a Metrocluster environment. The Serviceguard cluster configuration file includes the following attributes shown in Table 1 (page 15) to define sites. This enables to use the package failover policies site_preferred_manual or site_preferred for Metrocluster package. Table 1 Site Aware Failover Configuration Attributes Description SITE_NAME To define a unique name for a site in the cluster.
1. 2. Install the Raid Manager software for either XP or P9000 on every host system depending on the disk array used in your environment. Edit the /etc/services file, to add an entry for the Raid Manager instance to use with the cluster. Use the following format to add an entry to the /etc/services file. horcm /udp For Example, horcm0 11000/udp #Raid Manager instance 0 3.
6. If the Raid Manager protection facility is enabled, set the HORCPERM environment variable to the pathname of the HORCM permission file, then export the variable. For example: # export HORCMPERM=/etc/horcmperm0.conf The RAID Manager protection facility restricts volume control operations to volumes that the host is allowed to see, with or without host-based LUN security and to volumes that are listed in the configuration file.
perspective, there is no difference between a pair created from internal devices and a pair created using external devices. See the HP StorageWorks XP or P9000 documentation for information about the configuration requirements of external storage devices attached to XP or P9000 arrays and supported external storage devices. 9. Determine the devices used by the application package. Define a device group that contains all of these devices.
Creating Continuous Access Journal Pair To create a journal group pair, use the paircreate command: # paircreate -g -f async -vl -c 15 -jp -js Similar to Continuous Access Asynchronous, the fence “async ” must be assigned to the command with two additional options -jp and -js.
1. Create LVM physical volumes on each LUN. # pvcreate -f /dev/sda 2. Create the volume group on the source volume. # vgcreate --addtag $(uname -n) /dev/ /dev/sda 3. Create the logical volume. (XXXX indicates size in MB). # lvcreate -L XXXX /dev/ 4. Create a file system on the logical volume. # mke2fs -j /dev//lvol1 5. If required, deactivate the Volume Groups on the primary system and remove the tag. # vgchange -a n # vgchange --deltag $(uname -n) 6.
Configuring a Metrocluster package To configure a Modular package: 1. Run the following command to create a Metrocluster Modular package configuration file: # cmmakepkg –m dts/mcxpca temp.config In this command, dts/mcxpca is the Metrocluster module that needs to be included to create a Metrocluster with Continuous Access XP P9000 for Linux package configuration file. By default, the mcxpca module includes only the Serviceguard volume group and service modules.
3. If you want to monitor the status of the device group pairs, configure Device Group Monitor (DGM). Otherwise, skip this step. a. Configure the parameters for the Device Group Monitor in the package configuration file. Specify the polling interval for the DGM. If the parameter is not defined (commented out), the default value is 10 minutes. Otherwise, the value is set to the desired polling interval in minutes.
# touch /MON_RESYNC CAUTION: If the Continuous Access links are still down while the monitor is trying to do the resynchronization and another failure occurs that causes a remote failover to the secondary site, the SVOL’s BC volumes will remain split from its mirror group. This will only occur if the monitor is configured to perform automatic resynchronization using AUTO_RESYNC=1. b.
1. 2. 3. Access one of the node's in System Management Home Page, at http://:2301. Log in using the root user’s credentials of the node. Click Tools, if Serviceguard is installed, one of the widgets will have Serviceguard as an option. Click Serviceguard Manager link within the widget. On the Cluster’s Home Page, click the Configuration Tab, and then select Create A Modular Package option. Figure 4 Creating modular package 4.
Figure 6 Configuring package name and type 7. 8. Select additional modules depending on the application. For example, if the application uses LVM volumegroups or VxVM diskgroups, select the volume_group module. Click Next. Review the node order in which the package will start, and modify other attributes, if required. Click Next. Figure 7 Failover module attributes 9. You are prompted to configure the attributes for a Metrocluster package.
Figure 8 Configuring Metrocluster P9000/XP parameters 10. Enter the values for other modules selected in step 7. 11. After you enter the values for all the modules, review all the inputs given to the various attributes in the final screen. If you want to validate the package configuration click on Check Configuration, else click on Apply Configuration.
3 Metrocluster features Data replication storage failover preview In an actual failure, packages are failed over to the standby site. As part of the package startup, the underlying storage is failed over based on the parameters defined in the package configuration. The storage failover can fail due to many reasons, and can be categorized as the following: • Incorrect configuration or setup of Metrocluster and data replication environment.
Live Application Detach There may be circumstances in which you want to do maintenance that involves halting a node, or the entire cluster, without halting or failing over the affected packages. Such maintenance might consist of anything short of rebooting the node or nodes, but a likely case is networking changes that will disrupt the heartbeat. New command options in Serviceguard for Linux A.11.20.
HP recommends configuring a remote array RAID Manager instance in all Metrocluster configurations, especially in a 2-node configuration, as it is more robust. For more information about configuring a remote array RAID Manager instance in Metrocluster, see “Configuring a remote array RAID Manager instance” (page 29). NOTE: • Remote command device can be configured in Continentalclusters. • Remote command device need not be configured when XP or P9000 Continuous Access Journal replication is used.
Configuring the command device to manage the remote XP or P9000 arrays A command device to manage the remote XP or P9000 arrays can be configured using one of the following methods: • Configure a command device (remote command device) in a Metrocluster site by mapping a command device in the remote XP or P9000 array to a local device in the local XP or P9000 array using the XP or P9000 external storage feature. Present this remote command device to all Metrocluster nodes in the local site.
command devices. The remote array RAID Manager instance at both sites use the remote command devices. • Configure a dedicated command device in the XP or P9000 array at the remote site that is directly presented to the nodes in the local site over an extended SAN without using the XP or P9000 external storage feature. Similarly, configure a command device in the remote site XP or P9000 array and present it to the nodes of the local site over an extended SAN. Figure 11 illustrates this configuration.
3. Create a RAID Manager configuration file for the remote array RAID Manager instance using the RAID Manager configuration file template. For Example, # cp /etc/horcm.conf /etc/horcm1.conf 4. Configure the following parameters in the RAID Manager configuration file: • HORCM_MON Enter the host name of the system on which you are editing the file and the TCP/IP port number specified for the RAID Manager instance in the /etc/services file.
To configure the remote array RAID Manager instance for automatic startup: 1. Edit the following parameters in the configuration file /etc/mcxpca/raidmgr: • START_RAIDMGR Set this parameter to 1. • RAIDMGR_INSTANCE Specify all the RAID Manager instances that must be started during node boot-up. Include the remote array RAID Manager instance number as the value for this parameter.
4 Using XP or P9000 features in a Metrocluster environment Metrocluster with Continuous Access XP P9000 for Linux and thin provisioning volumes This product support for Thin Provisioning Volumes (TPVOLs) only on arrays XP20000, XP24000, P9500 and beyond, and P9000 disk array family. When TPVOLs are configured, there is a possibility that the TPVOL utilization will exceed the specified capacity or threshold of the pool. When this threshold is exceeded, the state of all pool volumes are marked as BLOCKED.
5 Understanding failover/failback scenarios Metrocluster package by default fails to start if the data is not current or if it is not able to determine the status of the device group. In such situations, the user has options to start a package either by setting the value of Metrocluster package AUTO parameters to “1” or by using FORCEFLAG. To use FORCEFLAG option, you must create a FORCEFLAG file in the package directory (/FORCEFLAG).
Table 5 Failover/failback scenarios (continued) Failover/Failback Scenarios Fence Level Link failure followed by application failure of all nodes in the primary site. Failback to primary site after link restoration Metrocluster Behavior (By default) AUTO parameters or FORCEFLAG set data or never Metrocluster package fails to start as AUTO_NONCURDATA is set to “0” and FORCEFLAG is not present . Metrocluster package checks for AUTO_NONCURDATA setting and FORCEFLAG presence.
Table 5 Failover/failback scenarios (continued) Failover/Failback Scenarios Fence Level Metrocluster Behavior (By default) AUTO parameters or FORCEFLAG set FORCEFLAG is present, Metrocluster package starts and issues horctakeover -S which results in SVOL takeover. async (journal) Failback to primary site after link restoration and recovery of primary site nodes.
Table 5 Failover/failback scenarios (continued) Failover/Failback Scenarios Fence Level Metrocluster Behavior (By default) AUTO parameters or FORCEFLAG set package starts and issues horctakeover -S which results in SVOL takeover. Any Metrocluster package fails to start as AUTO_PSUSSSWS is set to “0” and FORCEFLAG is not present. If AUTO_PSUSSSWS is set to “1” or FORCEFLAG is present, • Metrocluster package Resynchronizes data from the recovery site to primary site by issuing pairesync swapp command.
6 Administering Metrocluster Administering a Metrocluster using Continuous Access XP/P9000 replication Adding a node to Metrocluster To add a node to Metrocluster with Continuous Access XP P9000 for Linux, use the following procedure: 1. Add a node in the cluster by editing the Serviceguard cluster configuration file and applying the configuration: # cmapplyconf -C cluster.config 2. Create the RAID Manager configuration on the newly added node.
Maintaining a cluster that uses Metrocluster with Continuous Access XP P9000 for Linux In normal operation, the Metrocluster with Continuous Access for P9000 and XP starts like any other cluster, runs and halts packages in the same way as a standard cluster. However, startup time for packages might be considerably slower because of the, must verify, disk status on both disk arrays.
The pairdisplay -fe is primarily used for the following: Continuous Access Journal device group consistency set, Journal group ID (JID), and Continuous Access link status (AP).
Figure 12 Q-Marker and Q-CNT • U(%): Displays the usage rate of the journal data. • D-SZ: Displays the capacity for the journal data on the journal group. • Seq#: Displays the serial number of the XP12000. • Num: Displays the number of LDEV (journal volumes) configured for the journal group. • LDEV#: Displays the first LDEV number of journal volumes. Resynchronizing the device groups After certain failures, data is no longer remotely protected.
the volumes, with the PVOL becoming the SVOL and SVOL becoming the PVOL. With the personalities swapped, any data that has been written to the volume on the recovery site (now PVOL) are then copied back to the SVOL (now running on the primary site). During this time the package continues running on the recovery site. Failback to the primary datacenter After resynchronization is complete, you can halt the package on the recovery site, and restart it on the primary site.
Takeover timeout for Continuous Access journal mode A takeover timeout occurs when a package failover to the secondary site (SVOL) and Metrocluster Continuous Access issues takeover (either swap or SVOL takeover) command on SVOL. If the journal group pair is flushing the journal data from PVOL to SVOL and takeover timeout occurs, the package does not start and the following situations occur: • The device group pair state remains in PVOL-PAIR/SVOL-PAIR. • The journal data is being transferred to the SVOL.
on completing a rolling upgrade of HP Serviceguard, see the latest edition of Managing HP Serviceguard A.11.20.10 for Linux available at http://www.hp.com/go/linux-serviceguard-docs. Upgrading Metrocluster software To perform a rolling upgrade of Metrocluster software: 1. Disable package switching for all Metrocluster packages. 2. Install the new Metrocluster software on all the nodes. 3. Enable package switching for all Metrocluster packages.
• pairsplit for asynchronous mode might take a long time depending on how long the synchronization takes. There is a potential for the Continuous Access link to fail while pairsplit is in progress. If this happens, pairsplit will fail with a return code of EX_EWSUSE. • In most cases, Metrocluster/Continuous Access in asynchronous mode behaves the same as when the fence level is set to NEVER in synchronous mode.
7 Troubleshooting Troubleshooting Metrocluster Analyse Metrocluster and Raid Manager log files to understand the problem in the respective environment and follow a recommended action based on the error or warning messages. Metrocluster log file Regularly review the following files for messages, warnings, and recommended actions. It is good to review these files after every system, data center, and/or application failures: • View the system log at /var/log/messages.
Table 6 Metrocluster Log Messages and their resolution (continued) Log Messages Cause Local disk status: SVOL_PAIR Remote disk status: ERROR: CA link is down; SVOL may not contain current data. The package is currently configured not to start under these circumstances. Create the file if you really want to startup the package on this host. Exiting.
Table 6 Metrocluster Log Messages and their resolution (continued) Log Messages Cause AUTO_SVOLPSUS is set to 1 or FORCEFLAG was present. The package startup resulted in SVOL takeover. Resolution # pairresync –g • If you want the data on SVOL side, then issue the following command • Package is failing back to the PVOL on PVOL side to resynchronize the side. data from SVOL to PVOL.
Troubleshooting the Metrocluster with Continuous Access XP P9000 for Linux device group monitor The following is a guideline to help identify the cause of potential problems with the Metrocluster with Continuous Access XP P9000 for Linux device group monitor. • Problems with email notifications: Metrocluster with Continuous Access XP P9000 for Linux device group monitor uses SMTP to send out email notifications. All email notification problems are logged in the package log file.
A Checklist and worksheet for configuring a Metrocluster with Continuous Access XP P9000 for Linux Disaster recovery checklist Use this checklist to ensure you have adhered to the disaster tolerant architecture guidelines for two main data centers and a third location configuration. Data centers A and B have the same number of nodes to maintain quorum in case an entire data center fails.
Timing Parameters _____________________________________________________ Member Timeout: ___________________________________________________ Network Polling Interval: ____________________________________________ AutoStart Delay: ____________________________________________________ Package configuration worksheet Use this package configuration worksheet either in place of, or in addition to the worksheet provided in the latest version of the Managing HP Serviceguard A.11.20.
__________________________________________________________________________ Device Group: _____________ Fence Level: _____________ Raid Manager Instance#: _____________ Package configuration worksheet 53
B Package attributes for Metrocluster with Continuous Access XP P9000 for Linux This appendix lists all Serviceguard package attributes for Metrocluster with Continuous Access XP P9000 for Linux. HP recommends that you use the default settings for most of these variables, so exercise caution when modifying them.
NOTE: When a device group state is SVOL_PAIR on the local site and EX_ENORMT (Raid Manager or node failure) or EX_CMDIOE (disk I/O failure) on the remote site (this means it is impossible for Metrocluster/Continuous Access to determine if the data on the SVOL site is current), Metrocluster/Continuous Access conservatively assumes that the data on the SVOL site might by non-current and uses the value of AUTO_NONCURDATA to determine whether the package is allowed to automatically start up.
PSUS(SSWS). When failing back to the primary site, any differential data that was on the PVOL before to failover is lost during resynchronization. NOTE: This variable is also used for the combination of PVOL_PFUS and SVOL_PSUS. When the side file or journal volumes reach threshold timeout, the PVOL becomes PFUS. If there is a Continuous Access link, or some other hardware failure, and we fail over the secondary site, the SVOL becomes PSUS(SSWS) but the PVOL will remain PFUS.
1 – Startup the package after making the SVOL writable. The risk of using this option is that the SVOL data might actually be non-current and the data written to the PVOL side after the hardware failure might be loss. AUTO_SVOLPSUE (Default = 0) This parameter is applicable when the PVOL and SVOL both have the state of PSUE.
If a Raid Manager device group contains multiple items where either the PVOL or SVOL devices reside on more than a single P9000 Series array, then the Fence level must be set to “data” in order to prevent the possibility of inconsistent data on the remote side if an Continuous Access link or an array goes down.
For Continuous Access Journal mode package, journal volumes in PVOL might contain a significant amount of journal data to be transferred to SVOL. Also, the package startup time might increase significantly when the package fails over and waits for all of the journal data to be flushed. The HORCTIMEOUT must be set long enough to accommodate the outstanding data transfer from PVOL to SVOL.
Table 11 AUTO_NONCURDATA (continued) Local State Remote State Fence Level SVOL_PFUL PVOL_PSUE EX_ENORMT EX_CMDIOE ASYNC PVOL-PAIR; MINAP>0, MINAP=0 (Continuous Access-Journal) SVOL_PFUL SVOL_PFUL SVOL-PAIR;MINAP=0 AUTO_NONCURDATA AUTO_NONCURDATA =1 or =0 (Default) FORCEFLAG=yes non-current data in the package log file. Do not start with exit 1 EX_ENORMT, EX_CMDIOE Perform SVOL takeover, which changes SVOL to PSUS(SSWS).
Table 15 AUTO_SVOLPSUE (continued) Local State Remote State Fence Level AUTO_SVOLPSUE =0 (Default) AUTO_SVOLPSUE =1 or FORCEFLAG=yes starts with a warning message about non-current data in the control log file of the package. Table 16 AUTO_SVOLPSUS Local State Remote State Fence Level AUTO_SVOLPSUS =0 (Default) AUTO_SVOLPSUS =1 or FORCEFLAG=yes SVOL_PSUS PVOL_PSUS NEVER/ SVOL_PSUS EX_ENORMT DATA/ SVOL_PSUS EX_CMDIOE ASYNC NOTE: Do not start with exit 2. Perform a SVOL to PSUS (SSWS).
This parameter defines whether the monitor sends console notifications. When the parameter is set to 0, the monitor will NOT send console notifications. When the parameter is set to 1, the monitor sends console notifications. If the parameter is not defined (commented out), the default value is 0.
dts/xpca/AUTO_PSUSSSWS 0 dts/xpca/AUTO_NONCURDATA 0 dts/xpca/MULTIPLE_PVOL_OR_SVOL_FRAMES_FOR_PKG 0 dts/xpca/HORCMPERM MGRNOINST dts/xpca/HORCMINST 0 dts/xpca/HORCTIMEOUT 360 dts/xpca/WAITTIME 300 dts/xpca/FENCE never dts/xpca/DEVICE_GROUP mcxpdg dts/xpca/MON_POLL_INTERVAL 10 dts/xpca/MON_NOTIFICATION_FREQUENCY 0 dts/xpca/MON_NOTIFICATION_EMAIL admin@company.
C Sample configuration files for Metrocluster with Continuous Access XP P9000 for Linux B.01.00.00 Sample Raid Manager configuration file The following is an example of a Raid Manager configuration file for one node (ftsys1). ## horcm0.conf.ftsys1- This is an example Raid Manager configuration file for node ftsys1.Note that this configuration file is for Raid Manager instance 0, which can be determined by the "0" in the filename "horcm0.conf".
/dev/sda /dev/sdf #/************************* HORCM_DEV *************************************/ # # The HORCM_DEV parameter is used to define the addresses of the physical # volumes corresponding to the paired logical volume names. every group # name is a unique name used by the hosts which will access the volumes. # # The group and paired logical volume names defined here must be the same for # all other (remote) hosts that will access this device group.
Sample file for configuring automatic Raid Manager startup --------------------------RAID MANAGER -----------------------------------# Metrocluster with Continuous Access Toolkit script for configuring the # startup parameters for a HP Disk Array XP Raid Manager # instance. The Raid Manager instance must be running before any # Metrocluster package can start up successfully. # # @(#) $Revision: 1.
# run the raidscan command on ftsys1a or ftsys2a that is connected to the # remote XP array. # # # # # # # # # # # # # # # # The HORCM_LDEV parameters are used to describe stable LDEV# and Serial# as another way of HORCM_DEV used 'port#,Target-ID,LUN'. [Note]: This feature depends on the micro version of RAID or kind of RAID. This parameter is used to define the device group name for paired logical volumes.
D Sample output of the cmdrprev command The following procedure shows you how to use the cmdrprev command to preview the data replication preparation for a package in an MC with P9000/XP environment. Run the following command on : $> $SGBIN/cmdrprev -p demopkg1 Following is the output that is displayed: Oct 24 Oct 24 Oct 24 Oct 24 Oct 24 $4}' | Oct 24 Oct 24 21:23:41 - Node 21:23:41 - Node 21:23:41 - Node 21:23:41 - Node 21:23:41 - Node sed 's/[-,.
Glossary A arbitrator Nodes in a disaster tolerant architecture that act as tie-breakers in case all of the nodes in a data center go down at the same time. These nodes are full members of the Serviceguard cluster and must conform to the minimum requirements. The arbitrator must be located in a third data center to ensure that the failure of an entire data center does not bring the entire cluster down.
E, F ESCON Enterprise Storage Connect. A type of fiber-optic channel used for inter-frame communication between EMC Symmetrix frames using EMC SRDF or between HP Storage E series disk array units using Continuous Access . failover The transfer of control of an application or service from one node to another node after a failure. Failover can be manual, requiring human intervention, or automated, requiring little or no human intervention.
Q, R remote failover Failover to a node at another data center or remote location. resynchronization The process of making the data between two sites consistent and current once systems are restored following a failure. Also called data resynchronization.
Index A network problems, 9 module, 24 asynchronous mode, 8 replication, 7 P B bi-directional configurations, 14 C cluster Serviceguard, 14 cmviewcl command, 15 Consistency Group, 7 Continuous Access journal, 8 link timeout, 7 P9000 and XP concepts, 6 D device group monitor, 12 Device groups, 6 E extended SAN, 28 F failover_policy site_preferred, 15 fence levels, 6 DATA, 7 NEVER, 7 H hardware, 14 storage, 14 J journal group, 10 pair state, 10 volume, 8 L LVM volume group, 19 exporting, 19 M Metr
