User Guide HP StorageWorks File System Extender 3.
Legal and notice information © Copyright 2005 Hewlett-Packard Development Company, L.P. Hewlett-Packard Company makes no warranty of any kind with regard to this material, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Contents Intended audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Related documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hardware requirements . . . . . . . . . . . . . . . . . . . . . . . . . .
Media pool types and their characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Configuring a media pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Determining the configured media pools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Checking the status of a particular media pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Monitoring the FSE implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Managing FSE partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Monitoring FSE partition status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Disabling an FSE partition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FSE error codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 A Directory layout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 B FSE configuration templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
About this guide This guide provides information about: • Preparing your environment prior to installing software • Installing the software • Verifying successful installation and troubleshooting problems Intended audience This guide is intended for use by system administrators who are experienced with the following (depending on the type of installation): • SUSE LINUX Enterprise Server 8 (SLES) or Red Hat Enterprise Linux 3 (RHEL) operating systems (abbreviated to Linux throughout this guide) NOTE: You mu
Prerequisites Prerequisites for setting up product include: Software requirements For information on software requirements, see the FSE Release Notes and the FSE Installation Guide. Hardware requirements For information on hardware requirements, see the FSE Release Notes. Document conventions and symbols Document conventions Table 1 Convention Element Medium blue text: Figure 1 Cross-reference links and e-mail addresses Medium blue, underlined text (http://www.hp.
HP technical support Telephone numbers for worldwide technical support are listed on the HP support web site: http://www.hp.com/support/.
1 Introducing HP StorageWorks File System Extender This chapter includes the following topics: • What is FSE?, page 11 • FSE basics, page 11 • FSE architecture, page 12 What is FSE? HP StorageWorks File System Extender (FSE) is a mass storage oriented software product, based on client-server technology.
Library - tape compatibility FSE software is a part of HP StorageWorks File System Extender, and can be used with supported SCSI tape libraries with supported drive types. FSE software supports LTO drive technologies for LTO Ultrium 1, LTO Ultrium 2 and LTO Ultrium 3 media. For a list of supported tape libraries, tape drives, and tape media, see the FSE Release Notes. Automatic error detection FSE automatically detects drive and media hardware errors.
FSE operations FSE main operations are migration, release, recall, and deletion. They are configured with user-defined policies. For more information on configuring policies, see ”Migration, release, recall, and deletion” on page 109. After they are configured, the policies enable automatic, unattended execution of the corresponding operations. However, these operations can also be triggered manually using the fsefile command. For details on manual triggering, see the FSE Command-Line Reference.
For further information on release process and its configuration, see ”Migration, release, recall, and deletion” on page 109. Recall Recall is a process of copying file data from FSE media back to an HSM file system. It is the reverse process to migration. Recall is done in two steps: 1. Data is copied from the FSE media to a temporary location (disk buffer). 2. Data from this temporary location is copied back to the HSM file system, thus recreating file contents.
In a distributed FSE implementation, HSM file system, its HSM, and the File System Event Manager are located on a separate FSE client. FSE partition FSE uses FSE partitions to split the storage into multiple entities, meaning that one FSE implementation can have more than one FSE partition. This way FSE provides the possibility for complex organization of data management. Each FSE partition is related to a single HSM file system and has its own policies and its own File System Catalog (FSC).
FSE medium FSE supports different types of tape media and also disk media for data storage. Each FSE medium, either a medium cartridge or disk medium has an entry and properties written in the Resource Management Database (RMDB). Each FSE medium is assigned to one FSE media pool. For more information on media types, see ”FSE media types” on page 87. An FSE medium is usually formatted and further divided into partitions that hold medium volumes. The Resource Manager identifies the medium with a barcode.
Consolidated implementation The simplest application is the “all in a box” configuration, where all FSE components reside on the same machine. Figure 3 Consolidated FSE implementation This consolidated FSE implementation integrates the functionality of both FSE server and FSE client in a single machine. It has a direct connection to secondary media and hosts an arbitrary number of HSM file systems. User data from local HSM file systems are stored on the media in the attached tape library or on disk media.
Distributed implementation In this implementation, FSE clients connect to one FSE server and are physically separated. External clients connect to the server through either a LAN or a private network, and host HSM file systems, while the FSE server does not need to host any HSM file systems. Figure 4 Distributed FSE implementation The clients manage the HSM file systems and communicate with major services running on the server.
Client-server communication in a distributed FSE implementation An external FSE client communicates with an FSE server through the network using CORBA services. An external client can connect to only one FSE server. Figure 5 Client-server communication The network must provide a full-duplex connection between the FSE server and each of the external FSE clients. You can verify the network connection using the FSE tool fse_net. For more information about fse_net, see ”FSE troubleshooting tools” on page 223.
connections or private network connections for FSE client-server communication. You can retrieve the parameters that you will use for network configuration using the FSE fse_net tool. If you are about to configure the interprocess communication in a newly installed FSE implementation, the configuration must be performed before the first startup of FSE.
Generally, FSE daemons/services are the processes that are running most of the time FSE is operating. An FSE daemon/service is usually a server process with several sophisticated services. FSE agents are software agents, and are designed to perform defined tasks required by each specific job. Location of the binaries for FSE daemons/services and FSE agents is platform-dependent. Linux specific FSE daemon/service and agent binaries are located in the directory /opt/fse/lbin.
FSE daemon/service Binary Running on... Description Library Agent fse-la-s FSE server Handles actions on FSE libraries, such as loading and unloading media, library inventory rescan, and so on Back End Agent fse-bea FSE server Handles data transfer between FSE drives and FSE media, executes FSE media formatting and initialization or FSE agent The following figure demonstrates the interaction of the main FSE components.
Linux specific To use FSE commands, you must be logged on to the system as “root”. Windows specific To use FSE commands, you need administrative privileges. These privileges are granted only if you are either locally logged to the system as Administrator (recommended) or using a remote access software that supports logging in with the real Administrator account.
FSE command list The following table lists all FSE commands with a brief description. For a detailed description of command functionality and invocation syntax, see the FSE Command-Line Reference, or use the online FSE command-line reference as described in ”Accessing the online FSE command-line reference” on page 24. FSE command Description fse Initiates or terminates the FSE server or FSE client operation. Shows current status of the FSE daemons. Prepares FSE system for the restore process.
Windows specific On Windows platform, FSE command-line reference is available online as Windows/HTML help. To view HTML help for FSE commands, perform the following steps: 1. Click on the Start menu. 2. Expand Programs, Hewlett-Packard, and FSE. 3. To view the HTML help compiled in a single CHM file, click Docs (CHM file). To view the HTML help as a series of linked HTML files in your system default web browser, click Docs (index.html).
Introducing HP StorageWorks File System Extender
2 Configuring, starting, and stopping the FSE Management Console This chapter includes the following topics: • Configuring the FSE Management Console, page 27 • Starting and stopping the FSE Management Console server, page 31 • Starting and stopping the FSE Management Console client, page 33 Configuring the FSE Management Console This chapter provides instructions for initial configuration of the FSE Management Console server. The FSE Management Console client has no specific configuration files.
# GUI Server debug level # DEBUG_LEVEL = -d4 -DAll # path to OMNIORB configuration file OMNIORB_CONFIG = /etc/opt/fse/omniORB.
Parameter Description MIF_MAP Specifies the location of the MIF-to-MC translation file which contains mappings for values returned by the Management Interface process Linux specific On Linux platform, the default full pathname of the file is /etc/opt/fse/gui/guisrv.mif.conf. Windows specific On Windows platform, the default full pathname of the file is %InstallPath%\var\cfg\gui\guisrv.mif.conf. In normal circumstances, you should not change this parameter.
Parameter Description ORB_ARGS Specifies the command line parameters that are passed to the omniORB binary. The default value is: -ORBendPoint giop:tcp::4055 Specifies the CORBA Naming Service URL of the Management Interface process, which is used by the FSE Management Console server to interact with FSE. MIF_URL The default value is: FSE/mgr/%(SERVER)%/mif In normal circumstances, you should not change this parameter.
Linux specific On Linux platform, the full pathname of the FSE Management Console server log file is /var/opt/fse/log/guisrv.log. Windows specific On Windows platform, the full pathname of the FSE Management Console server log file is %InstallPath%\var\log\guisrv.log. The value of %InstallPath% depends on the choice made in the FSE installation process. It defaults to C:\Program Files\Hewlett-Packard\FSE.
-F - startup in foreground mode -d -D
In this mode, FSE Management Console server is running as foreground console application. Use either command-line interface or graphical user interface to run the FSE Management Console server in this mode: • If you choose to use CLI: At the command line, specify the command-line option -F to run the FSE Management Console server in the foreground mode. • If you choose to use GUI: follow the procedure described in ”Starting the FSE Management Console server using GUI” on page 33.
Windows specific On Windows platform, you can also start or stop the FSE Management Console client using graphical user interface. When starting up, the FSE Management Console client reads the initialization file for the current user. This file contains all the current settings made by this user during the last session, for example, defined systems with connection details and settings from the Settings dialog.
Starting and stopping the FSE Management Console client on Windows systems On Windows platform, the FSE Management Console client can be started using command-line interface or graphical user interface. Starting the FSE Management Console client using CLI To start the FSE Management Console client, proceed as follows: 1. Change the current directory as shown: C:\>cd %InstallPath%\bin The value of %InstallPath% depends on the choice made in the FSE installation process.
Configuring, starting, and stopping the FSE Management Console
3 Configuring FSE This chapter includes the following topics: • Before you start, page 37 • Configuration basics, page 38 • Configuration procedure, page 59 • Reconfiguring FSE interprocess communication, page 75 • Changing the IP address of an FSE host, page 80 Before you start FSE provides effective management of your data with its extended virtual disk storage, which is controlled by user-defined policies.
• Define your migration and release policies regarding the characteristics of files on each individual file system, such as how much are the files in use, are these “working” files or files that simply need to be retained safely, how often do you want these files to be migrated, and so on. • Consider the effect of your policies on your FSE implementation and tune it accordingly.
BlockSize = 32KB <---------------------------- qualifier NumOfVolumes = 1 SysVolLocation = NONE # NONE | FIRST | MIDDLE | LAST SizeOfSysVol <- = 0MB ----------------------------- variable Templates For each resource type, a default configuration template is available so that you do not need to create configuration files from scratch. The templates already contain most of the required parameters with remarks on the appropriate use of configuration variables, and details of maximum and minimum values.
Windows specific %InstallPath%\var\cfg Resource Management Database Once FSE resources are configured, the data from the configuration files are written to the Resource Management Database (RMDB). This database keeps all relevant information on FSE resources, such as the current status, usage information, and characteristics. When an FSE resource is removed from the configuration, all data referring to it is removed from Resource Management Database accordingly.
NOTE: While modifying the template, preserve the double quote characters (") that surround the values of some variables. Removing these characters might lead to problems in certain cases. 3. To configure a new partition, apply the modified template using the following: fsepartition --add PartitionCfgFileName You have configured a new FSE partition. This FSE partition configuration is stored in the configuration database as the first respective configuration revision.
NOTE: The FSE Management Console client can connect to several FSE Management Console servers simultaneously. Thus, if your environment includes more than one FSE implementation, you can configure and monitor all FSE implementations from the same FSE Management Console client session running on a single computer. Operational modes of the FSE Management Console client The FSE Management Console client supports two operational modes.
Choosing online operational mode Prerequisites The prerequisites for using online operational mode for a particular FSE implementation are the running FSE implementation and a running FSE Management Console server for this FSE implementation. To choose the online operational mode, perform the following steps: 1. Click Start > Programs > Hewlett-Packard > FSE > FSE GUI-Client to start the FSE Management Console client. 2.
Graphical user interface layout Figure 10 GUI layout The object tree contains hierarchically organized objects. You can browse the hierarchy by clicking the expand/collapse buttons besides objects. Expanding a subtree does not select the object at the subtree origin. To select an object, click it. NOTE: If there are problems with particular object, the belonging icon displayed in the object tree changes into a question mark (“?”), similarly as in the Windows-native Device Manager.
This is the default update mode. • On-request update mode: In this update mode, when the properties of a particular object are displayed in the FSE Management Console client GUI, the FSE Management Console server immediately queries the Management Interface and retrieves the requested data. The object tree The object tree lists all manageable objects in the FSE implementation.
When this object is selected, the General tab is displayed on the right pane. The tab shows the current status and the configuration variables of the FSE library. When this object is right-clicked, the following actions are available in the pop-up menu: • Enable or Disable: makes the library available to the FSE implementation or puts the library offline. • Rescan: triggers inventory update on the library. • Update: triggers inventory update with a physical slot rescan on the library.
When this object is right-clicked, the actions New, New Window from this place, and Properties are available in the pop-up menu. For action descriptions, see ”Actions associated to objects” on page 52. • Media Pools > (virtual object) An FSE media pool configured in the FSE implementation. Holds a list of media assigned to this FSE media pool. When this object is selected, the same parameters are displayed for each medium as with the All Media Pools container.
When this object is right-clicked, the actions New, Configuration, Delete object, Configuration File, New Window from this place, and Properties are available in the pop-up menu. For action descriptions, see ”Actions associated to objects” on page 52. • Partitions > > All Media (container) This container holds a list of all media belonging to the FSE partition PartitionName. When this object is selected, the same parameters are displayed for each medium as with the All Media Pools container.
IMPORTANT: displayed. Only the jobs that are currently running and up to 25 most recently finished jobs are When this object is right-clicked, the action New Window from this place is available in the pop-up menu. For action description, see ”Actions associated to objects” on page 52. • Log (physical object) This object holds a list of the error and warning messages stored in the FSE error log.
When this object is right-clicked, the following actions are available in the pop-up menu, depending on the current state of the medium volume: • Reorganize: starts the media reorganization job for the medium volume. • Close: marks the medium volume as full. • (physical object) An FSE job that was recently running or is still running in the FSE implementation. Objects of this type are displayed in the right pane when the Jobs container is selected in the object tree.
Creation, Configuration, and Properties dialog boxes The Creation, Configuration, and Properties dialog boxes have identical contents for a particular object. The only difference is in the protection of the object properties (read-only, editable).
Figure 13 Properties dialog box Actions associated to objects When you right-click a particular object in the object tree, a pop-up menu is displayed. The set of the available menu items depends on the object type and the current object status. The actions (pop-up menu items) that are available for the majority of object and container types are the following: • New: creates a new object of the selected type, for example: new FSE library, new FSE media pool, and so on.
The Server menu The menu items of the Server menu are listed and described in the following table. Menu item (action) Action description New Connection Opens the Connection window where you can specify connection settings and create a new connection to an FSE implementation: • • • • • • • • Name: specifies a descriptive name for the connection. This name identifies the remote FSE Management Console server system in the object tree.
Menu item (action) Applicable objects Action description Library Library objects. Displays a submenu with the menu items: • • Partition Partition objects. Displays a submenu with the menu items: • • • • • • Job Jobs objects. Change Priority Abort Medium volume objects. Displays a submenu with the menu items: Configuration Library, drive, partition, and media pool objects. Opens the Configuration window for the selected object, in which you can modify its configuration.
The Windows menu The menu items of the Windows menu are listed and described in the following table. Menu item (action) Action description New Instance Opens a new FSE Management Console client instance in which the initial GUI view is displayed. New Window from this place Opens a new FSE Management Console client instance in which the properties of the currently selected object are displayed in the right pane.
The status bar The status bar provides current status of the FSE Management Console client, and the object count and the status information for the item that is currently selected in the object tree. The status information is only available for the following containers and objects: • containers: Libraries, Slots, Drives, Media Pools, Partitions, All Media, Unassigned Media, Jobs. • objects (virtual and physical): , Log.
The status bar displays the number of configured FSE media in the selected FSE media pool. Additional numbers represent FSE media pool counts according to their states. The status icons have the following meaning: Status icon Current status good unusable unreliable uninitialized • Partitions (container) The status bar displays the number of configured FSE partitions in the selected FSE implementation. Additional numbers represent FSE partition counts according to their states.
Keyboard shortcuts The following keyboard shortcuts are available for the most common actions. Menu item (action) Keyboard shortcut New Connection Alt + N Connect Alt + C Disconnect Alt + D Remove Server Alt + R Exit Alt + X Settings Alt + O Properties Alt + Enter FSE Management Console users Access to an FSE implementation from the FSE Management Console client is allowed only to configured users that are stored in the FSE Management Console user database.
Removing FSE Management Console users To remove an FSE Management Console user, follow the steps: 1. In the FSE Management Console client GUI, in the object tree, select Users. 2. In the property page, right-click on an existing user name and click Delete object. FSE Management Console user properties Each configured FSE Management Console user has the following properties, which are shown in the User Details window: • User name: unique alphanumeric string for identifying the user.
Linux specific NOTE: You must be logged on to the system as root in order to execute FSE commands. Windows specific NOTE: You need administrative privileges to perform the configuration tasks. These privileges are granted only if you are either locally logged to the system as Administrator (recommended) or using a remote access software that supports logging in with the real Administrator account.
Linux specific C:\Program Files\Hewlett-Packard\FSE\bin>fse_scsi --scan ** /dev/sg0 id:0:0:6:0 SCSI device ID: port 0, ch 0, id 6, lun 0, type Disk 0 Device: HP ** /dev/sg1 OPEN-V 5003 ser #:10804 std :3 id:0:0:7:0 SCSI device ID: port 0, ch 0, id 7, lun 0, type Disk 0 Device: HP ** /dev/sg2 OPEN-V 5003 ser #:10804 std :3 id:0:0:8:0 SCSI device ID: port 0, ch 0, id 8, lun 0, type Disk 0 Device: HP ** /dev/sg3 OPEN-V 5003 ser #:10804 std :3 id:0:0:8:0 SCSI device ID: port 0, ch 0, id 8, lun
** /dev/sg7 id:0:0:10:3 SCSI device ID: port 0, ch 0, id 10, lun 3, type Storage Array 12 Device: HP NS E1200-160 5678 ser #:P088LB std :4 Done! NOTE: If the Library Agent is running, the device for the library used by the agent is not seen in the list. TIP: You can also check the /proc/scsi/scsi file for a list of available SCSI devices. Windows specific C:\Program Files\Hewlett-Packard\FSE\bin>fse_scsi --scan ** \\.
2. Enter the values you get from the output of the above command to the configuration files for library and drives, and check that all other parameters match the characteristics of your library and drives. See ”Library and tape drive configuration examples” on page 63 for reference. You will configure your media pool according to your drive configuration. 3. Configure the FSE library using the command: fselibrary --add LibraryCfgFile 4. Once the library is configured, a new Library Agent is started.
# You can specify ranges of slot numbers and individual # slot numbers.
Windows specific [FSE Library] [Version = 1] [Date = "2005/04/11 15:24:32"] [User = "fse-administrator@company.com"] # # Specify a name for the tape library and a hostname, where it # resides. Use full host name. # Name = "HP MSL5000 #2 LTO" Host = "fse_server.company.com" # # Control device is identification of library's # (media changer's) on the specified system.
[User = "fse-administrator@company.com"] Name = "WORM 01" Library = "GTL340i-AIT" MediaFamily = AIT DriveIndex = "1" Capabilities = (WORM) Host "fse_server.company.com" { ControlDevice = "/dev/sg3" } Windows specific [FSE Drive] [Date = "2005/04/11 15:30:12"] [User = "fse-administrator@company.com"] Name = "WORM 01" Library = "GTL340i-AIT" MediaFamily = AIT DriveIndex = "1" Capabilities = (WORM) Host "fse_server.company.
{ ControlDevice = "/dev/sg4" } Windows specific [FSE Drive] [Date = "2005/04/11 15:30:12"] [User = "fse-administrator@company.com"] Name = "LTO 01" Library = "MSL6000-LTO" MediaFamily = LTO DriveIndex = "1" Capabilities = () Host "fse_server.company.com" { ControlDevice = "id:0:2:0:0" } Step 2: configuring FSE media pools For detailed information on creating a specific media pool type, see ”Configuring a media pool” on page 82. Generally, you configure an FSE media pool as follows: 1.
Preparing HSM file systems on Windows platform To create a separate volume (disk partition) for an HSM file system, proceed as follows: 1. In Control Panel, double-click Administrative Tools, and then double-click Computer Management. 2. Click Disk Management, right-click an unallocated region of a basic disk, and then click Create Partition, or right-click free space in an extended partition, and then click Create Logical Drive. Note that dynamic disks are also supported. 3.
For details on how to integrate existing file systems on a particular platform, see the FSE Installation Guide, appendix ”Integrating existing file systems in the FSE implementation”, section ”Integrating existing file systems on Windows platform”. The next step, configuring an FSE partition, is described in the FSE User’s Guide, section ”Configuring a regular FSE partition on Windows platform” on page 70. For a WORM FSE partition configuration, see ”Configuring a WORM FSE partition” on page 71.
4. Add the new FSE partition: # fsepartition --add PartitionCfgFile Once the partition is configured, a new Partition Manager is started for the newly configured partition. 5.
Example of the required format of the volume ID is shown in this excerpt of the partition configuration file: # Where is the HSM file system? # Client "windows_fse_host.company.com" { # FileSystemID = _HSM_FS_MAJOR_:_HSM_FS_MINOR_ # e.g. 58:1, Linux client FileSystemID="Volume{f6f3e1b6-b2ac-11d7-9c99-00b0d02fadef}" # Windows client Type = Regular # Regular - regular read/write file system # WORM - Write-Once-Read-Many file system # uncomment and configure if Type = WORM : # WORMTimeout = 450s } 4.
There is also a timeout parameter (WORMTimeout) that defines the interval during which it is possible to modify or delete a file after the file is closed. During this interval, the file contents and its metadata can be modified; the file can be renamed, moved, or removed. Once the interval has expired, the file, its metadata, and location can never be modified again. This parameter is commented out in the template file. It must be uncommented if the file system type is “WORM”.
NOTE: A particular FSE medium is related to a single FSE media pool, single FSE partition, and single FSE library. 1. Create an entry for the FSE medium and add it to the already configured media pool: fsemedium --add Barcode --pool PoolName 2. Physically place the FSE medium into a free library slot, and then trigger a physical inventory update (barcode rescan): fselibrary --update-inventory LibraryName --rescan 3. Format the medium using the command below.
1. Check the status of the FSE partitions using the following command: # fsesystem --status The status for all configured partitions must be “ready”. 2. Create a directory that will serve as a mount point for the HSM file system: # mkdir /fse/fsefs_01 3. Add the corresponding file system entry to the local file /etc/fstab.This step is platform dependent.
1. At the command prompt, use the cd command to change the current directory to the mount point of the HSM file system, and run the dir command. 2. Invoke the following command: C:\>fsepartition --list In the command output, the corresponding FSE partition should have its status set to mounted.
4. Modify the services.cfg or omniORB.cfg file on all FSE hosts according to the changes in your FSE implementation or changes of the network type employed. The required modifications are described in the following sections. You can use FSE proprietary fse_net tool to retrieve the parameters required for the configuration. For more information on fse_net, see ”FSE troubleshooting tools” on page 223.
The following is an example of a properly configured services.cfg file in FSE implementations using a private network. The server variable is redundant in such FSE implementation. hostname = fse-server.fsenet server = private-server.company.com 2. Rename the omniORB.cfg backup file (for example, omniORB.bak) back to omniORB.cfg.
Private network connection If the external (distributed) Linux FSE clients will communicate through a private network to the FSE server, you must modify both configuration files, services.cfg and omniORB.cfg, on each external Linux client, as follows: 1. Modify the value of the server variable in services.cfg to contain the fully-qualified domain name (FQDN) that identifies the consolidated FSE implementation or the FSE server inside the private network. 2.
1. Ensure that the value of the server variable in services.cfg equals the fully-qualified domain name (FQDN) that identifies the consolidated FSE implementation or the FSE server inside the LAN. The following is an example of a properly configured services.cfg file in FSE implementations using an ordinary LAN. The file can also include the hostname variable, but this variable is redundant in such FSE implementation. server = lan-server.company.com 2. Rename the omniORB.cfg file (for example, to omniORB.
serverTransportRule = 192.168.240.0/255.255.255.0 tcp serverTransportRule = * none Changing the IP address of an FSE host When changing the IP address of a host of the FSE implementation, proceed as follows: 1. Stop the FSE processes with fse --stop. 2. Stop the omniORB naming service with omniNames --stop. 3. Remove all files in the directory for temporary omniORB files. This directory is platform-dependent. Linux specific The temporary omniORB directory is /var/opt/fse/omniorb.
4 Managing media The way you manage FSE media can significantly increase the stability and performance of your FSE implementation. Managing media consists of tasks that are related to the media life cycle, from formatting, initialization and optional duplication for data vaulting, to replacing worn-out media and media reorganization. FSE supports several media types, which are grouped into appropriate media pools.
This media pool type is the most common one since it is used for regular archiving of the migrated FSE implementation data. It contains FSE media that can be overwritten, reorganized, reinitialized, and reformatted. • WORM media pool Used for WORM media only. On these media, data can be recorded only once, therefore WORM media pool type is particularly suitable for storing permanent archives. It is used to migrate files stored on an WORM HSM file system accordingly. Note that FSE 3.
Example of an LTO media pool configuration file [FSE Pool] [Version = 1] [Date = "2005/04/10 16:11:26"] [User = "fse-administrator@company.com"] # # Specify the name of the media pool. Media pools are # used by FSE partitions. # # NOTE: You have to assign media to pools with fsemedium command.
# # Specify the name of the media pool. Media pools are # used by FSE partitions. # # NOTE: You have to assign media to pools with fsemedium command. # Name = "My_DM_pool" MediaFamily = DISK PoolType = Regular # # Change the values of the on-tape format if the defaults don't suit you. # # NOTE: Some of the parameters will be used only if the tapes are # (re)formatted.
• the system volume size. You can specify most of these attributes in the media pool configuration file. If you use the --detail option, fsepool formats the output in a different way, and displays additional FSE media pool attributes, as follows: • the UUID of the pool, • the UUID of the owning partition, • the media pool’s RMDB entry key. These attributes are assigned automatically by FSE during the FSE media pool (or FSE partition) configuration.
In the fsepool command output, locate the name of the owning partition and supply it as an argument to the fsepartition command as follows: fsepartition --status PartitionName The output of this command lists the names of all media pools that are currently assigned to the selected partition, and their current status, as follows: • the total and available capacity of all data volumes on media belonging to the media pool, • the percentage of data volume usage, • the total and available capacity of all system
• Data volume Used to store regular migration data. Each FSE medium must have at least one data volume; usually, an FSE medium is partitioned into several data volumes. LTO Ultrium media do not support multiple medium volumes, and they do not need to be formatted. FSE supports up to 255 data volumes on disk media. FSE media types The following is a list of the supported media types and their characteristics: • LTO Ultrium 1, LTO Ultrium 2, LTO Ultrium 3 Used in regular and backup media pools.
The main purpose of a capacity disk in FSE is its availability as a secondary storage space. A capacity disk should be dedicated to the FSE disk media and the FSE disk buffer. Generally, it is used as a large archive storage, meaning that the frequency of user access to archived files is relatively low compared to that of the HSM file system on the performance disk.
Disk media specifics The main difference between tape and disk media is in preparing media for use by FSE. A disk medium must be mounted to a dmNumber mount point in the disk media location before it can be used. The location is platform-dependent. Linux specific On Linux platform, disk media are located in the directory /var/opt/fse/dm. Windows specific On Windows platform, disk media are located in the directory %InstallPath%\var\dm.
• medium volume size identifier (vol.info) • Data compression is not supported for disk media. • Location information, which you get by running the fsemedium --list command, is not valid for disk media. For details, see ”Checking the status of media” on page 104. Adding media to an FSE media pool By adding media to an FSE media pool, you prepare them so they can be used by FSE.
Medium contents Type of formatting Regular (--format) Forced (--format --force) FSE volumes with no valid FSE user data NO YES FSE volumes with valid FSE user data NO NO Skipping media formatting In certain cases, a medium only needs to be initialized in order to be used in the FSE implementation, and the formatting step can be skipped. This can be done for the following media: • new or already used LTO media These media support only one medium volume and do not need partitioning.
Notes FSE medium volumes that store valid FSE user data are the volumes which: • store some data that is referenced by the belonging File System Catalog (FSC). • belong to an FSE media pool which is currently assigned to an FSE partition. • have the parameter Used[%] greater than zero in the output of the command fsemedium --list --volume. FSE medium volumes that do not store valid FSE user data are the volumes for which any of the above criteria is not met.
NOTE: To replace an original medium with its duplicate, you should label the duplicate medium with the original barcode, remove the original medium, and use the duplicate instead. When a medium copy is first used in place of the original, a warning is written to the fse.log file as the FSE implementation recognizes that the medium for the particular barcode has been switched.
Forced duplication When any of the last three duplication prerequisites is not met and normal duplication fails, you can still duplicate to the target medium if you use forced duplication functionality. You can perform a forced medium duplication using the following command: fsemedium --duplicate SourceBarcode TargetBarcode [--no-monitor] --force CAUTION: Since forced duplication overwrites any data that is stored on the target medium, data loss may occur on the target medium.
Over time, the situation can arise where a large proportion of space on FSE media is occupied by old, obsolete generations of files. FSE enables you to reclaim media space by reorganizing the data on the media. The media reorganization process frees space on media by preserving only selected file generations and removing the others. FSE media space can then be reused for newly migrated data. Media reorganization is a medium-volume-based process.
The following table depicts the reorganization steps, a description of their tasks, the volume status indicators that are set, and the corresponding FSE jobs. Reorganization step Step tasks Step 1: Scanning Status of medium volumes Job type [open/full] FSE reorganizational scan job 1. Change the status of the FSE partition to in scan. 2. Scan medium volumes for obsolete data. 3. Write the scan results to the RMDB. [open/full], scanned 4.
CAUTION: With either parameter, only the migrated generations containing file contents count as file generations. The migrated generations that contain only file headers (name, standard and extended attributes, ACLs, ...) or changes to a file alternate data stream do not count as migrated file generations. Generally, these generations are not preserved during media reorganization.
Example output of the fsepartition --reorg-scan command fsepartition --reorg-scan my_part "2005/11/26 15:25:13" 3 [11:42:09] Started (JobID: 20050121000001). [11:42:10] Scanning FSC for reorganization info. [11:42:10] Scan Parameters: "2005/11/26 15:25:13", generations: 3. [11:55:22] Scanning complete. Scanned 33.005.664 file generation copies. [11:55:23] Use 'fsepartition --reorg-stat' to see the details. [11:55:25] Finished.
Med003 5 0 1000 100% Med004 5 100 1000 90% The meaning of the columns is the following: VolNum Number of the FSE medium volumes on the medium. ValidData Amount of space on the FSE medium occupied by valid file generations. If the FSE medium has not yet been scanned for reorganization, it has 0 (zero) displayed in place of the ValidData amount. Total Total amount of space on the FSE medium. SlackSpace Percentage of FSE slack space.
[12:49:58] Reading finished, releasing resources. [13:00:58] Medium 050010, volume 5: Data written to medium volume. [13:01:58] Medium 050010, volume 6: Writing… [13:10:50] Medium 050010, volume 6: Data written to medium volume. [13:11:50] Writing finished, releasing resources. [13:12:38] Finished. If you disabled monitoring, you can still check the status of the FSE reorganization job using the fsejob --list command.
which multiple copying was configured. This process is called recreating a redundant copy and is available for FSE tape media and FSE disk media. By recreating the redundant copy of the damaged migrated data, you can restore the availability and safety of the migrated data. The redundant copy recreation process Redundant copy recreation is started by the fsemedium command and performed by an FSE maintenance job.
CAUTION: Before treating an FSE medium as unreadable, make sure it is not loaded in a faulty tape drive. In certain circumstances, an FSE medium may acquire “unusable” status because of problems with the tape drive in which it is loaded. In this situation, the drive itself gets the “error” status.
[11:42:09] Request resources (bc: 050004) ... [11:42:09] Resources allocated. [11:42:09] Medium 050004 is already present in drive drv3. [11:42:09] Starting Back End Agent. [12:21:02] Medium 050004, volume 2: Reading files. [12:45:58] Medium 050004, volume 2: Files written to disk buffer. [12:46:02] Reading finished, releasing resources. [12:47:12] Requesting resources... [12:47:12] Resources allocated. [11:42:09] Medium 050010 is already present in drive drv2. [12:49:09] Starting Back End Agent.
Checking the status of media Run fsemedium --list [--detail] to get information on media status for all media configured in the FSE implementation. As shown in the example that follows, the output is the following: • the medium barcode, • the name of the media pool, • the medium status, • the physical location of the medium, • the number of all medium volumes (data- and system volumes), • the system volume number according to its location on the medium.
SysVolNo: n/a SysVolSize: n/a CurVol: 1 Key: 5 Barcode: 000046 Type: LTO-1 ... For disk media, the Location information is not applicable (n/a). The Location value indicates the current location of the tape medium in an FSE library slot (library name and slot number) or in an FSE drive (drive name). The status of the medium can be one of the following: free Medium volumes are full and Fast Recovery Information is already written on the medium. in use The medium is currently in use.
Example output of the fsemedium --list --volume command fsemedium --list 000045 --volume Barcode VolNum Type Size[MB] Avail[MB] Used[%] Status 000045 1 System 1138 1126 1 open 000045 2 Data 1138 0 95 full 000045 3 Data 48172 42391 12 open Changing the condition status of media The Resource Management Database recognizes three kinds of statuses which address a medium condition: good, unreliable, and unusable.
To close a medium volume, specify the medium barcode and volume number that you want to mark as full: fsemedium --close-volume Barcode VolumeNumber Recreating Fast Recovery Information from media Recreating Fast Recovery Information (FRI) reads the complete contents of a medium volume and creates a new FRI file. The new file contains the same information but is usually more compact because it is created in a single pass.
Managing media
5 Migration, release, recall, and deletion This chapter includes the following topics: • FSE operation basics, page 109 • Migration, page 112 • Release, page 114 • Recall, page 121 • Deletion, page 122 • Resource allocation, page 125 • System allocation and job priority policy, page 127 • Partition allocation and job priority policy, page 128 • Priority calculation, page 130 • Multiple copying, page 131 • HSM file system access modes, page 132 FSE operation basics How does FSE function? FSE keeps track of
simpler process that does not require an FSE job. It is managed and executed locally by the Hierarchical Storage Manager (HSM). FSE disk buffer Files that are either waiting for migration and are not yet written to the FSE media, or have been recalled and are not yet recreated on the HSM file system, are copied to the FSE disk buffer.
The way transitions between HSM lists and other aspects of migration, recall, release, and deletion are controlled is defined by parameters in the configuration file for the FSE partition. These parameters are described in ”Migration” on page 112, ”Release” on page 114, ”Recall” on page 121, ”Deletion” on page 122, and in ”System allocation and job priority policy” on page 127 and ”Partition allocation and job priority policy” on page 128.
Migration The relevant configuration parameters for the migration policy are specified in the FSE partition configuration file. It is very important that you create a migration policy appropriate for your environment. You should check the default migration policy and adjust settings according to the characteristics of your files. FSE migration policies are based on file age, wait time for migration, number of files waiting for migration, and the migration volume.
Maximum wait time for migration (MaxWaitTime) Definition Maximum wait time for migration; if reached, migration is started immediately. The interval starts with the first file being added to the empty migration candidate list. Impact Without this parameter, if new files keep arriving to the migration list fast enough, the MinWaitTime would never be reached and the files would never get migrated. MaxWaitTime ensures this does not happen.
The period of time since the first file was put on the list is greater than MaxWaitTime. • Situation 3 There are at least MaxNumMigFiles on the list. • Situation 4 The total size of files to be migrated is equal or greater than MaxMigSize. Default migration policy Default migration policy is defined in the FSE configuration template for FSE partition configuration. For details, see ”Template for FSE partitions - Migration part” on page 248.
Note that the regular release job starts immediately after the forced release job has ended (because the system is still filled to the high watermark). Figure 19 Reaching the critical watermark on an HSM file system High watermark High watermark is usually reached when new files are created or copied to the HSM file system. Regular release is started for all files on the release candidate list that have passed their migration or recall retention period, and are larger than MinFileSize.
system is reached or when there are no further valid files on the release candidate list. Files not released remain on the release candidate list. Figure 20 Reaching the high watermark on an HSM file system Low watermark Low watermark is considered during a regular release triggered by reaching the high watermark on the HSM file system. The low watermark is the parameter that defines to what percentage volume the disk space should be freed, and stops the release when it has achieved this level.
NOTE: Zero-sized files remain permanently online. They cannot be migrated nor released. Limitations • Only sets of files can be excluded from the release process. You cannot exclude directories. • When an FSE partition is reconfigured by modifying the ExcludeFromRelease variable, the contents of the corresponding release candidate list is not affected.
MinFileSize = "2kB" ExcludeFromRelease = () NOTE: When configuring the ExcludeFromRelease variable, filenames and filename patterns must be specified in the UTF-8 format. Special pattern-matching characters The filename pattern-matching mechanism built into the FSE conforms to that used in command shells on UNIX systems, with a few differences. Special characters are permitted to simplify the definition of sets of files. The following table lists the special characters, meanings, and usage examples.
Configuring exclusion from release After you have modified the FSE partition configuration template or an already used FSE configuration file with a configured ExcludeFromRelease variable, you have to apply it by running the fsepartition --add or the fsepartition --modify command, respectively. Potential syntax errors in the ExcludeFromRelease parameter specification are immediately reported to the standard command output. For information on the fsepartition command, see the FSE Command-Line Reference.
Migration retention time (MigRetentionTime) Definition A period of time during which a certain file is going to be kept online after migration before it will be released. Impact Consider for how long a migrated file should stay online. Adjust the value according to your data characteristics. If you use the partition for archiving purposes, the value should be low. Partitions with frequently used files should have a higher value set, so that the files do not get released and recalled all the time.
regular release. The release job is stopped when the low watermark is reached or if there are no more files on the release candidate list that match the specified criteria. • Situation 2 A critical watermark is reached on the HSM file system. In this case, all files on the release candidate list (regardless of their size and retention time) are scheduled for forced release. The release job is stopped when the high watermark is reached or if there are no more files to release.
NOTE: Having multiple media copies can speed up the recall of files, since any of the copies can be used for recall. For further information on multiple copying, see ”Multiple copying” on page 131. Default recall policy Default recall policy is defined in the FSE configuration template for FSE partition configuration. For details, see ”Template for FSE partitions - Recall and Allocation part” on page 251. Deletion FSE provides the ability to delete files automatically on HSM file systems.
You define a deletion policy with pairs of FileExpiration and Path variables. Each specified FileExpiration variable requires a corresponding Path variable. File expiration time (FileExpiration) Definition A period of time that determines which files in the specified Path directories are to be considered for deletion. The following units are available for specifying the time period: s (seconds), m (minutes), h (hours), D (days), M (months), and Y (years).
Windows specific [Delete] FileExpiration "1M" { Path = (\Months\Current, "\Arch\June 2005") } FileExpiration "7D" { Path = (\Weeks\Current, "\Arch\Week 24") } This deletion policy will have the following effect: • All files in the directories \Months\Current and "\Arch\June 2005" on the owning HSM file system are to be considered for deletion as soon as one month has passed since they were last changed.
Note that files that were deleted from an HSM file system and whose file data was stored on FSE medium volumes that were subsequently reorganized can no longer be recalled. This happens when all migrated generations of the deleted files were considered obsolete (outdated) by the FSE media reorganization process and all medium volumes that stored these file generations were reorganized.
job requires new resources. Typically, the Resource Manager locks the use of resources in the Resource Management Database for jobs that have requested allocation. Media selection by media location Allocation selects a medium according to its location. Possible locations of a medium are: • Online: Medium is loaded into a drive. • Nearline: Medium is in a library slot. • Offline: Medium location is unknown.
System allocation and job priority policy The system allocation and job priority parameters are read by the Resource Manager upon its startup. System allocation and job priority policy parameters The configuration parameters for the system along with their definition and impact are listed below. Maximum number of drives for migration (SystemMaxNumDrivesMigration) Definition Maximum number of drives that can be used for migration by all FSE partitions.
Time step (TimeStep) Definition System-wide factor by which the priority of the queued jobs is increased due to the age of the job. Impact The purpose of this parameter is to leverage the elapsed time (the time since the job was started) and increase the priority of jobs that are waiting in the queue. The time factor is multiplied by the number of seconds since the job started, and by the JobTypePriority divided by 100.
Maximum number of drives (MaxNumDrives) Definition Maximum number of all drives that can be used for all jobs by the partition. Impact With this parameter you can prevent a single partition from using all the drives and not leaving any drives for other partitions and administrative jobs. You should carefully balance resources. Note that all other jobs are queued if the Partition Manager uses the maximum number of drives.
Recovery priority (RecoveryPriority) Definition The priority for recovery jobs on a scale of 1 to 100. Impact This priority is multiplied by the partition priority in order to calculate the initial priority of a recovery job. See ”Priority calculation” on page 130. Maintenance priority (MaintenancePriority) Definition The priority for maintenance jobs on a scale of 1 to 100. Impact This priority is multiplied by the partition priority in order to calculate the initial priority of a maintenance job.
Administration and backup job priority calculation For administration and backup jobs, the priority is calculated by the following formula: Priority = JobTypePriority + (ElapsedTime * TimeStep * JobTypePriority/10000) + (Phase * PhaseFactor) Migration, recall, recovery, and maintenance job priority calculation For migration, recall, recovery, and maintenance jobs, the priority is calculated by the following formula: Priority = PartitionPriority * JobTypePriority + (ElapsedTime * TimeStep * JobTypePriority/
The number of FSE media pools assigned to the same FSE partition equals the number of copies the data from that FSE partition will have on FSE media. If you have more copies of file data on FSE media, you are able to recall your data from any available copy, thus benefiting on recall speed and data safety. Depending on the configuration of your FSE implementation and availability of required resources, you can have copies made in parallel or sequentially.
Once mounted, each HSM file system remains mounted all the time, even after the FSE client host is restarted. A mounted HSM file system, however, can operate in Limited Access Mode (LAM) or Full Access Mode (FAM). The operational mode depends on whether the FSE processes are running and whether an FSE administrator decided to switch between the two modes manually.
Migration, release, recall, and deletion
6 Monitoring and maintaining FSE The procedures in this chapter provide instructions for: • About monitoring and maintaining in FSE, page 135 • Monitoring the FSE implementation, page 135 • Managing FSE partitions, page 136 • Monitoring FSE jobs, page 139 • Checking the consistency of the File System Catalog, page 141 • Recalling older generations of a file, page 146 • Viewing logs, page 148 • Retrieving detailed information, page 158 • Determining total offline storage capacity, page 168 About monitoring
Windows specific C:\Program Files\Hewlett-Packard\FSE\bin>fsesystem --status # of Partitions: 2 # of Admin Jobs: 1 Partition Status part1 ready part2 mounted #Jobs Host Mountpoint 0 fse_host.company.com 0 fse_host.company.com I:\fse2\part2 Managing FSE partitions An FSE implementation can be configured to include several FSE partitions. FSE partitions can be located on consolidated FSE systems and on external FSE clients.
Windows specific C:\Program Files\Hewlett-Packard\FSE\bin>fsepartition --status part2 Partition: part2 Status: mounted Host: fse_host.company.com Mountpoint: I:\fse2\part2 # of Migration Jobs: 0 # of Recall Jobs: 0 # of Maintenance Jobs: 0 # of Recovery Jobs: 0 Pool Size[MB] Avail[MB] pooldm1 49152 46632 Used[%] SysSize[MB] 6 0 #Unreliable #Unusable #Uninit 0 0 0 SysAvail[MB] SysUsed[%] #Good 0 0 1 Disabling an FSE partition To disable an FSE partition, proceed as follows: 1.
Invoke the following command: C:\>fse --mount MountPoint VolumeName The command starts the appropriate HSM process and switches the HSM FS filter operation for managing accesses to the FSE partition to Full Access Mode (FAM). Removing an FSE partition CAUTION: When an FSE partition is removed from the FSE system, all files that have been released from the corresponding HSM file system (offline files) will be lost. To remove an FSE partition, proceed as follows: 1.
Switching to Limited Access Mode Prerequisites • FSE processes on the FSE server and the FSE client must be running. • The HSM file system must be operating in FAM. To switch the HSM file system access to LAM, use the following command. Use the name of the corresponding disk volume: C:\>fse --limit-access VolumeName Switching to Full Access Mode Prerequisites • FSE processes on the FSE server and the FSE client must be running. • The HSM file system must be operating in LAM.
Example output of the fsejob --list command fsejob --list JobId Partition Type Prio Started Status 20051202000082 resources video_captures Recall 8478 2005/12/02 10:22:57 Waiting for 20051202000083 DiskBuffer video_captures Migration 7436 2005/12/02 10:23:07 Writing 20051202000084 Medium scanned_img Migration 4439 2005/12/02 10:24:34 Writing to Example output of the fsejob --list --detail command fsejob --list --detail JobID: 20051202000082 Partition: video_captures Type: Recall
Example output of the fsejob --status command fsejob --status 20051202000173 JobId: 20051202000173 Partition: part2 Type: Migration Priority: Started: 2005/12/02 14:32:54 Copy Phase Drive Medium Volume 1 1 n/a n/a n/a Status Writing Disk Buffer Checking the consistency of the File System Catalog The File System Catalog (FSC) consists of a Data Location Catalog and a Name Space Catalog. It is related to a single partition. FSC location is platform-dependent.
• not in NSC A file is recorded as missing in NSC because the FSE implementation has been upgraded to release 2.1 in the past, but recovery of FSC has not been performed yet. In such case, you need to run fserecover --fsc PartitionName to update NSC. For details, see section ”File System Catalog recovery” on page 186. The following mismatches can also be a result of upgrading to release 2.1 without performing FSC recovery afterwards.
[2005/12/03 15:11:51] 578672 273989682176 /mnt/fsefs_01/A1/linux-2.4.19-ea/something DUMP: FSC HSMFS Notes MigID 273989682176 273989682176 FileSize 6 0 MISMATCH FileName something something OwnerID 31 31 [2005/12/03 15:12:29] Finished (checked: 578684, missing: 4, suspicious: 0, prolonged:0). [2005/12/03 15:12:29] Finished (JobID: 20051203000011). Windows specific C:\Program Files\Hewlett-Packard\FSE\bin>fsecheck --fsc-hsmfs FS04-CLIENT01 [2005/03/23 08:15:09] Started. (JobID: 20050323000011).
Windows specific %InstallPath%\var\log\check_media_fsc_PartitionName.log When used with the --autocorrect and --match-media options, the FSC vs. media check invalidates the results of a reorganization scan.
[2005/12/03 15:29:18] Started FSC chk vol num: 4, medium: dm002 [2005/12/03 15:29:18] 154340 274037495040 MAINTAINERS Missing in FSC. DUMP: FRI MigID 274037495040 MediaKey 4 MedVolNum 4 MedVolOffset 472 FileType 1 FileSize 41643 MigSize 41643 MigFlags HD SplitFlags L SplitOffset 0 SplitSize 41643 FileName MAINTAINERS OldFileName OwnerID 154336 OldOwnerID 0 [2005/12/03 15:29:18] 578684 274037495040 NewFile Missing in FSC.
[2005/12/03 15:29:38] Finished FSC chk vol num: 2 (checked: 72400, missing in FSC: 0, missing on media: 0, mismatch: 0, no NSC: 0) [2005/12/03 15:29:38] Started FSC chk vol num: 3, medium: dm001 [2005/12/03 15:29:47] Finished FSC chk vol num: 3 (checked: 73599, missing in FSC: 0, missing on media: 0, mismatch: 0, no NSC: 0) [2005/12/03 15:29:47] Started FSC chk vol num: 4, medium: dm001 [2005/12/03 15:29:55] Finished FSC chk vol num: 4 (checked: 67326, missing in FSC: 0, missing on media: 0, mismatch: 0, no
/fse/part01/20050328/root/gmon.out 1023249 268530735104 2005/03/29 15:29:44 5512 /fse/part01/20050328/root/hsmdb.out 1023249 268509426688 2005/03/28 16:22:28 4684 /fse/part01/20050328/root/hsmdb.out 1023249 268509288448 2005/03/28 16:13:28 2014 /fse/part01/20050328/root/hsmdb.
Example of a recall by date Linux specific # fsefile --recall /fse/part01/20050328/root/hsmdb.out --date "2005/03/28 16:22:28" --into /tmp Example of a recall by file migration ID Windows specific C:\Program Files\Hewlett-Packard\FSE\bin>fsefile --recall i:\fse\part01\20050328\mylittledoc.doc --migid 268509426688 --into D:\tmp Each file must be recalled separately. You define a recall by specifying a filename or by using the argument combination --id PartitionName FileID, as shown in the examples below.
This log records all events that occur on an HSM file system: file creation, rename, move, deletion, and so on. Each configured FSE partition has its own HSM file system log. The log file is located on the FSE client that hosts the FSE partition. For information on the locations of FSE log files, see appendix ”Directory layout” on page 239. The FSE event log The FSE event log records specific information on events occurring in FSE processes during the operation of FSE.
[2005/06/17 13:57:25, linux_fse_server, 009799, MIF:DRIVE, , 0, drv01] Updated (rev. 2). [2005/06/17 13:57:37, linux_fse_server, 009817, LAS, lib01, 0, ] Inventory [2005/06/17 13:57:37, linux_fse_server, 009799, MIF:DRIVE, , 0, drv02] Added new. [2005/06/17 13:58:34, linux_fse_server, 009799, MIF:POOL, , 0, pool01] Added new. [2005/06/17 14:03:47, linux_fse_server, 009799, MIF:POOL, , 0, pool02] Added new. [2005/06/17 14:11:31, linux_fse_server, 010321, PMG, part01, 0, ] Started (3.0.145 "Beta_02").
[2005/06/17 14:21:25, linux_fse_server, 010445, BEA, /dev/sg2, 20050617000002, 000007] Vol 2: Changing to volume 4... [2005/06/17 14:21:48, linux_fse_server, 009799, MIF:ADMIN, , 20050617000003, 000008] Started. [2005/06/17 14:21:48, linux_fse_server, 009799, MIF:ADMIN, , 20050617000004, 000009] Started. [2005/06/17 14:21:48, linux_fse_server, 009799, MIF:ADMIN, , 20050617000005, 000010] Started. [2005/06/17 14:22:24, linux_fse_server, 010445, BEA, /dev/sg2, 20050617000002, 000007] Vol 4: Rewind...
[2005/07/06 15:20:09, windows_fse_server, 002260, HSM, c:\fsefs2\, 20050706000005, ] Migration (#files: 39, size: 278 KB).
[2005/07/08 16:04:20, windows_fse_server, 001980, PMG:MAINTENANCE, part2, 20050708000002, ] Started. [2005/07/08 16:04:20, windows_fse_server, 001980, PMG:MAINTENANCE, part2, 20050708000002, ] Finished. [2005/07/08 16:06:06, windows_fse_server, 001704, MIF:PARTITION, , 0, part2] Updated (rev. 4). [2005/07/08 16:10:25, windows_fse_server, 001704, MIF:PARTITION, , 0, part2] Updated (rev. 5). [2005/07/08 16:10:42, windows_fse_server, 001980, PMG:MAINTENANCE, part2, 20050708000003, ] Started.
The value of %InstallPath% depends on the choice made in the FSE installation process. It defaults to C:\Program Files\Hewlett-Packard\FSE. Example FSE error log Linux specific ============================================= FSE Error log file Create time : 2005/06/17 14:11:05 ============================================= [W MIF() ,009799.010245 2005/06/17 14:11:05.045; root@linux_fse_server.company.com; mif_CfgRep::ReplaceClients, mif_cfgrep.cpp,v 5.
[E MIF() ,009799.016390 2005/06/17 14:17:56.861; root@linux_fse_server.company.com; i_AdminJob_i::StartBEA, i_adminjob_impl.cpp,v 5.10 :328] [20050617000001] Caught error when starting BEA.fse_Error: (6004: Caught Corba exception.) CORBA::COMM_FAILURE#IDL:omg.org/CORBA/COMM_FAILURE:1.0 Execution stack dump: 00000004. i_AdminJob_i::StartBEA i_adminjob_impl.cpp,v 5.10:328 00000003. i_AdminJob_i::Execute i_adminjob_impl.cpp,v 5.10:1351 00000002. job_Executor::Run job_executor.cpp,v 5.5:62 00000001.
00000001. cmn_Thread::run cmn_thread.cpp,v 5.3:355 End of stack dump. [W LAS(lib01) ,009817.006149 2005/06/17 14:17:57.203; root@linux_fse_client.company.com; la_SCSILibrary::EjectMedium, la_scsi.cpp,v 5.4 :507] Medium eject attempted on non-configured drive index: 2 Windows specific ============================================= FSE Error log file Create time : 2005/07/06 15:20:09 ============================================= [W RMG() ,001696.001912 2005/07/06 15:20:09.156; SYSTEM@windows_fse_server.
End of stack dump. [W RMG() ,001696.001912 2005/07/06 15:20:09.171; SYSTEM@windows_fse_server.company.com; log_EventLogger::Write, log_eventlogger.cpp,v 5.7 :219] [2005/07/06 15:20:09, gtl-host, 001696, RMG, , 0, ] WARNING: Medium (BC:PFY229L2) has no usable volumes for migration Execution stack dump: 00000006. log_EventLogger::Write log_eventlogger.cpp,v 5.7:219 00000005. rm_Operator::SelectMediumVolumeMig librmdb.exx,v 5.11:10301 00000004. rm_Operator::SelectMediumForMig librmdb.exx,v 5.
00000002. rm_DBThread::Run rm_dbthread.cpp,v 5.0:99 00000001. cmn_Thread::run cmn_thread.cpp,v 5.3:355 End of stack dump. [E MIF() ,001704.002232 2005/07/08 16:05:10.171; SYSTEM@windows_fse_server.company.com; ::yylex, cfg_l.cpp,v 5.3 :120] invalid string") Execution stack dump: 00000005. ::yylex cfg_l.cpp,v 5.3:120 00000004. pf_File::DoParse pf.cpp,v 5.4:450 00000003. pf_File::Parse pf.cpp,v 5.4:417 00000002. mif_CfgRep::ParseToFile mif_cfgrep.cpp,v 5.3:137 00000001.
FSE monitoring commands Table 4 FSE entities Monitoring command Files on HSM file systems fsefile --list FSE jobs fsejob --list Additionally, FSE extends each of these monitoring commands with the --detail option, which has two benefits: • It causes the monitoring commands to format their output in a different way, where each retrieved parameter of the queried entity is displayed in a separate line in the form of ParameterName: ParameterValue.
SerialNo: 0000000360 VendorID: HP FirmwareRev: 1.5 ScsiID: id:0:0:4:0 LoadNr: 141 UnloadNr: 138 Key: 2 Name: libLTO01 Family: LTO ... Excerpt from an example output of the fsedrive --list --detail command Name: drv01 Family: LTO Library: lib01 Host: fse_server.company.
Key: 2 Name: drv_lto01 Family: LTO ...
DrvName: Status: free Location: lib:lib01:10 SerialNo: 10000000000000000000000001985033 BlockSize: 65536 #Vol: 1 SysVolNo: n/a SysVolSize: n/a CurVol: 1 Key: 9 Barcode: 000007 Type: LTO-1 Pool: pool01 LibName: lib01 SlotAddr: 7 SlotType: repository DrvName: Status: free Location: lib:lib01:7 SerialNo: 10000000000000000000000001987376 BlockSize: 65536 #Vol: 1 SysVolNo: n/a SysVolSize: n/a CurVol: 1 Key: 6 Barcode: 656701 Type: UNKNOWN Pool: LibName: lib01 Slo
Barcode: dm01 Type: DISK Pool: pool02 LibName: SlotAddr: SlotType: DrvName: Status: open Location: n/a SerialNo: 58:4 BlockSize: 65536 #Vol: 50 SysVolNo: 0 SysVolSize: 0 CurVol: 33 Key: 11 Barcode: dm02 Type: DISK Pool: pool02 LibName: SlotAddr: SlotType: DrvName: Status: open Location: n/a SerialNo: 58:5 BlockSize: 65536 #Vol: 50 SysVolNo: 0 SysVolSize: 0 CurVol: 12 Key: 12 ...
Status: empty AccessNum: 0 WriteNum: 0 OwriteNum: 0 InitTime: 2005/06/18 12:00:09 AccessTime: 2005/09/14 01:00:00 WriteTime: 2005/09/14 01:00:00 OwriteTime: 1970/03/01 17:41:05 LastVer: 1970/01/01 01:00:00 ValidData: 0 TotalData: 0 Barcode: 000009 VolNum: 2 UUID: eca50ba5-f068-4650-a587-945fcffda287 Type: Data Size[MB]: 15815 Avail[MB]: 15815 Used[%]: 0 Status: empty AccessNum: 0 WriteNum: 0 OwriteNum: 0 InitTime: 2005/06/18 12:01:12 AccessTime: 2005/09/14 01:00:00
WriteTime: 2005/09/14 01:00:00 OwriteTime: 1970/03/01 17:41:05 LastVer: 1970/01/01 01:00:00 ValidData: 0 TotalData: 0 Barcode: 000009 VolNum: 4 UUID: b8e4d73b-890c-41d0-bb20-eefe19d3f057 Type: Data Size[MB]: 17391 Avail[MB]: 17391 Used[%]: 0 Status: empty AccessNum: 0 WriteNum: 0 OwriteNum: 0 InitTime: 2005/06/18 12:01:47 AccessTime: 2005/09/14 01:00:00 WriteTime: 2005/09/14 01:00:00 OwriteTime: 1970/03/01 17:41:05 LastVer: 1970/01/01 01:00:00 ValidData: 0 TotalData: 0
... Excerpt from an example output of the fsepartition --list --detail command Partition: part01 UUID: 891af3c2-0aa9-4cc2-bd57-b16140691f31 Status: mounted Host: fse_host.company.com Mountpoint: /mnt/fsefs01 FSID: 1088756689 Options: Regular MigIdTime: 0 MigIdCount: 58:3 Activity: scan in progress Partition: part2 UUID: bfca4697-a1e0-4881-ba12-953acbb72a02 Status: ready Host: fse_host.company.
FileName: bigfile1 FileID: 42110 Status: online FileName: dir18 FileID: 42111 Status: online FileName: dir19 FileID: 12 Status: offline FileName: linux-2.4.19-64GB-SMP.tar.bz2 FileID: 1 Status: online FileName: lost+found FileID: 2 Status: online FileName: new_dir01 FileID: 11 Status: online FileName: new_dir010 FileID: 3 Status: online FileName: new_dir02 ...
Status: Reading from medium JobID: 20051225000003 Partition: partLTO Type: Migration Priority: 4001 Started: 2005/12/25 18:21:00 Status: Waiting for resources JobID: 20051225000004 Partition: partLTO Type: Recall Priority: 4001 Started: 2005/12/25 18:21:00 Status: Waiting for resources ... Determining total offline storage capacity You can determine the total amount of offline storage capacity in an FSE implementation using Licence Checker.
Active HSM client(s): 3 fse_client_04.company.com, fse_client_04 , 192.168.0.84 192.168.12.72 fse_client_09.company.com, fse_client_09 , 192.168.0.93 fse_client_12.company.com, fse_client_12 , 192.168.0.110 Total storage capacity maintained by the HSM server: 12.00 TiB You can store the report into a file for later use using standard output redirection technique: checklic >checklic-output.
Monitoring and maintaining FSE
7 Backup, restore, and recovery FSE enables you to have several migrated copies of the same file data, thus ensuring availability of files in case one of the copies is damaged. For details on configuring FSE to create multiple migrated copies, see ”Multiple copying” on page 131. To make your FSE environment as safe and as stable as possible, you need to plan ahead and regularly copy FSE databases and system files (FSE system data) onto different storage media. This copying process is called FSE backup.
FSE component Description Recovery method File System Catalog (FSC) Contains information about the location of migrated data on FSE media as well as the Name Space Catalog with metadata information on files, and is therefore needed to access files on FSE media. • Resource Management Database (RMDB) Contains records of all configured FSE resources. These are library, drive, pool, and media.
Server backup is the preferred backup type because it is the fastest one and includes all data needed to restore your FSE implementation in combination with recommended HSM file system recovery which recovers data on clients. • full backup Backup of all required databases that are stored on the FSE server as well as on FSE clients. You are able to restore the whole FSE implementation (FSE server and FSE clients) from a full backup.
Backup Before you configure an automatic backup, you must know how it works, select the backup type that is appropriate for your environment, and ensure that the prerequisites are met. What is FSE backup? FSE backup creates copies of data on backup media for emergency purposes. It saves a consistent state of information stored in FSE databases, FSE system files, and user data on HSM file systems. FSE backup is performed using the fsebackup command.
The backup progress is recorded in the backup.log file. Once backup is finished, it reports barcodes of the FSE backup media where backed up data is stored. Backup start and stop times are written to the fse.log file and can help estimate the time needed for backup. Backup media management FSE backup requires special media management to ensure safety of the backed up data and to prevent data loss.
Linux specific On Linux platform, DeviceFile is represented as the string /dev/sgX. Windows specific On Windows platform, DeviceFile is represented as the string id:P:B:T:L. For details on how to identify SCSI device addresses or IDs, see ”About SCSI identification” on page 60 in chapter ”Configuring FSE” on page 37. The LTO Ultrium medium is now ready to be used in the FSE backup pool. Estimating backup frequency You need to estimate how often you should run an FSE backup.
Example of a full backup The whole FSE implementation (the FSE server and all FSE clients) is going to be backed up to the FSE media with barcodes 100190, 100191, 100192: # fsebackup --media 100190 100191 100192 --client Example of an FSE client backup The FSE server and a specified FSE client are going to be backed up to the FSE medium with barcode 001243: # fsebackup --media 001243 --client fse_client.company.com Note that the FSE client must be specified by its fully-qualified domain name (FQDN).
2005/12/16 10:54:20 fse_host.company.com lv_snap_size = 200 MB 2005/12/16 10:54:21 fse_host.company.com Mounting vgraid5/lvstrato1_snap... 2005/12/16 10:54:21 fse_host.company.com Mounted. 2005/12/16 10:54:21 fse_host.company.com Dump EA to file... 2005/12/16 10:54:23 fse_host.company.com EA written to file. 2005/12/16 10:54:23 fse_host.company.com Mounting vgraid5/lvstrato2_snap... 2005/12/16 10:54:23 fse_host.company.com Mounted. 2005/12/16 10:54:23 fse_host.company.com Dump EA to file...
2005/12/16 10:58:02 fse_host.company.com Writing client data of partition strato1 on host fse_host.company.com... 2005/12/16 10:58:06 fse_host.company.com Client fse_host.company.com done for component hsmdb. 2005/12/16 10:58:06 fse_host.company.com Writing FSC of partition strato1... 2005/12/16 10:58:07 fse_host.company.com Writing client data of partition strato2 on host fse_host.company.com... 2005/12/16 10:58:50 fse_host.company.com Client fse_host.company.com done for component hsmfs.
2005/04/14 10:20:29 fse_host.company.com Executing: fse-arch --backup "" 32 "\\?\Volume{cfe0dd5f-3155-11d8-a16f-806d6172696f}"\ part2| fse-dd --write id:2:0:2:0 "" 32 --eom "CHANGE_MEDIUM_CMD" 2005/04/14 10:20:32 fse_host.company.com Warning: Skipping \\?\Volume{cfe0dd5f-3155-11d8-a16f-806d6172696f}\part2 Backup (#files: 0, #failed files: 1, size: 0 KB). 2005/04/14 10:20:32 fse_host.company.com Writing RMDB... 2005/04/14 10:20:32 fse_host.company.com Executing: fse-arch --backup "" 32 "C:\WINNT\Temp\FSE\.
2005/04/14 10:19:42 fse_host.company.com ...partitions suspended: Suspend performed for 2 FSC of 2 0 HSM of 2 2005/04/14 10:19:42 fse_host.company.com Make snapshots for fse_host.company.com... 2005/04/14 10:19:45 fse_host.company.com Client-side script initialized, making snapshots. 2005/04/14 10:19:45 fse_host.company.com Creating snapshot of \\?\Volume{cfe0dd5f-3155-11d8-a16f-806d6172696f}... 2005/04/14 10:19:47 fse_host.company.com Mounting \\?\Volume{cfe0dd5f-3155-11d8-a16f-806d6172696f}...
Restore Contrary to FSE backup, restoring FSE data is an interactive process. Before performing a restore, you must know how to restore your data and be fully aware of the FSE concepts and the meaning of restore. You need to be familiar with the FSE directory layout. For more information, see appendix ”Directory layout” on page 239. NOTE: FSE also provides recovery of the File System Catalog and the HSM file system as an alternative way of recreating data to backup and restore.
In order to successfully restore the FSE implementation, all data that was backed up must be restored to its original location. You may want to back up current data before performing a restore (in case the restore fails). • Restoring to different devices Since restore is often needed when the hard disk has failed, it is very common to make a restore to devices different than the ones used for backup.
Example FSE restore log file Linux specific 2005/12/16 11:22:57 fse_host.company.com Started on host fse_host.company.com 2005/12/16 11:23:01 fse_host.company.com Restore will use tape device /dev/sg5 2005/12/16 11:25:39 fse_host.company.com Rewind tape & check medium header... 2005/12/16 11:25:39 fse_host.company.com Restoring client fse_host.company.com... 2005/12/16 11:25:40 fse_host.company.com Attempting restore of client fse_host.company.com 2005/12/16 11:25:40 fse_host.company.
2005/12/16 11:28:58 fse_host.company.com Reading HSMFS of partition strato2... 2005/12/16 11:29:57 fse_host.company.com Setting extended attributes... 2005/12/16 11:29:57 fse_host.company.com Unmounting partition filesystem... 2005/12/16 11:29:59 fse_host.company.com Reading HSM DB of partition strato2... 2005/12/16 11:29:59 fse_host.company.com Restore complete for partition strato2 on client fse_host.company.com. 2005/12/16 11:29:59 fse_host.company.com Reading FSC of partition strato2...
Recovery FSE provides functionality for recovering from disasters that can happen to the FSE implementation, and which can be used as an alternative or as a supplement to the conventional FSE restore process. HSM file systems and File System Catalogs (FSCs) can be restored using specially tailored recovery methods. If one or more HSM file systems are damaged, you can use HSM file system recovery to recreate an HSM file system from the Name Space Catalogs, as an alternative to FSE backup and restore.
HSM file system recovery is necessary in cases where the HSM file system is damaged and has not been backed up, therefore it cannot be restored. How it works Limitations • Recovery cannot properly handle user changes on an HSM file system that occurred after the last migration is finished. To prevent this situation, you need to format the HSM file system. • Symbolic links cannot be recovered because they are not migrated to FSE media.
Windows specific On Windows platform, the full pathname of the recovery progress status file is %InstallPath%\var\tmp\PartitionName–_hsmfs_recover. 6. The value of %InstallPath% depends on the choice made in the FSE installation process. It defaults to C:\Program Files\Hewlett-Packard\FSE. 7. It unmounts the temporarily mounted HSM FS and removes temporary mount points and device files.
8 Troubleshooting In this chapter, the most common errors regarding configuration and use of FSE are presented, including instructions on how to collect debug traces for different kinds of FSE activities. It also contains a list of all errors that can be issued by the FSE implementation, with a one-line explanation for each.
FSE startup and shut-down problems Description After running the fse --stop command to shut down FSE, you may experience the following situation: - The fse command does not finish (does not return to the command prompt) in a reasonable time. - Local HSM file systems are still mounted. - Local FSE processes are still running. Explanation If a pre-stop script has been configured on this system, the above symptoms may indicate problems with commands in the script.
Description After invocation of the fse command on a particular FSE system host, some FSE processes that should be started on that host (consolidated FSE implementation, FSE server, or external FSE client) are not running (SUSE LINUX system-specific). Explanation If the required FSE packages were installed on the host using a single rpm command, some of the packages may not have been installed correctly, due to unresolved dependencies.
Description Each time an FSE command is invoked, the following error message is displayed: ERROR 6008: Cannot connect to Management Interface - probably not running after system reboot. Requested name does not yet exist in the NameService. Giving up. Explanation The local services.cfg file contains the fully qualified domain name (FQDN) of the FSE server host, where MIF is running. If the primary DNS suffix of this host is not set appropriately, the full computer name may contain only the host name.
Description If a consolidated FSE system or FSE server has a power failure, external FSE clients are not automatically reconnected to the server after it is restarted. Explanation n/a Workaround Stop all FSE processes on the FSE server and all external FSE clients. Restart the services on the FSE server first, then restart the processes on the external FSE clients.
Firebird server problems Description While FSE daemons/processes are starting up, the following error is reported: ERROR: Firebird SuperServer (Interbase) is not up and running! Explanation Firebird is a third-party software used by the Resource Manager for managing resources in FSE implementations. Firebird software must be installed on the system before you can start FSE processes. For instructions on how to install the Firebird package, see the FSE Installation Guide.
Communication problems Description CORBA cannot be initialized. After invocation of an FSE command, the following error is reported: FSE server specific 6000 Cannot initialize CORBA ORB. 'CORBA::COMM_FAILURE' FSE client specific Failed to initialize CORBA Explanation Possible reason for this problem may be that omniNames is not running on the FSE server and, consequentially, FSE server is not accessible to the FSE client. Workaround Proceed as follows: 1.
Description When inspecting the FSE error log of an improperly operating FSE implementation, the following error entry is detected: This machine (localhost.localdomain) has only localhost IP. Misconfigured DNS on host: localhost.localdomain. Official name (hostent::h_name) is NULL. Misconfigured DNS on host: localhost.localdomain. Official name (hostent::h_name) should contain FQDN.
SCSI problems Description On a consolidated FSE implementation or an FSE server, major problems with FSE drives are experienced (Windows-system specific). Explanation See Workaround. Workaround Check the FSE error log for the following message: Access to SCSI device is prevented. Make sure that the device driver is disabled. If this error message is logged, the cause of the problem with the FSE drive is the operating system driver for the tape drive, which has not been disabled.
Description Tape drives or other SCSI devices may not appear at boot time in the /proc/scsi/scsi file although they are connected properly (Linux-system specific). Explanation This problem is caused by the current generic SCSI driver in both supported Linux distributions, Red Hat Enterprise Linux and SUSE LINUX Enterprise Server. Workaround Proceed as follows: 1. Add a call to the HP utility hp_rescan by adding the following line to the file /etc/rc.d/rc.
Tape library problems Description A tape library reports the same barcode in different locations. Explanation It could happen that the tape library reports the same medium barcode in a drive and in a slot at the same time. In this case FSE writes an error message to the FSE error log file, and marks both the medium and the drive as unusable. This happens when you manually move the medium from the drive to the slot, and then start the library inventory rescan.
Description A bad drive cannot be removed if a medium is still in it. Explanation When trying to remove a drive from the FSE implementation, the command fsedrive --remove Name reports a problem when trying to unload the medium, and the drive cannot be removed from the system. Workaround If the medium could not be unloaded, you must identify the reason and unload the medium manually.
Media management problems Description Initialization of media with large block size fails with the following error: ERROR: Medium 052007, volume 3: Init failed (12053: SCSI driver error. 'Buffer too large. See error.log for details.'). (Windows-system specific) Explanation This problem occurs because some Windows SCSI or FS host bus adapter (HBA) drivers (particularly Adaptec and QLogic) do not support large block sizes by default.
Description If you remove an FSE medium from its FSE media pool without physically removing it from an FSE library, then add it to whichever FSE media pool, and finally start an FSE administrative job to format the medium, the job seems to be blocked before formatting actually begins. Explanation When an FSE administrative job commands the Resource Manager (RM) to allocate a particular FSE medium, the RM cannot execute the allocation unless it knows the medium’s physical location in the FSE library.
Description Though a reorganizational scan fails, the volume status is changed from 'full, scanned' to 'full' Explanation In cases when the reorganization fails while reading data from a source medium volume (due to a medium or hardware error, for example), the medium volume's flag "scanned" is cleared even though the reorganizational scan results are still valid. Workaround Run the reorganizational scan again after the cause of the failure is resolved.
HSM file system mounting problems Description On a Linux FSE system with an entry in the /etc/fstab file for the /fse/fsefs_01 file system with type hsmfs, running the mount /fse/fsefs_01 command produces the following error: # mount /fse/fsefs_01/ mount: wrong fs type, bad option, bad superblock on /dev/vg_fse/fsefs_01, or too many mounted file systems (Linux system-specific) Explanation This error is reported if any of the following is true: • FSE daemons are not running.
Description Output of the command fsepartition --list shows a partition as ‘ready’ or ‘mounted’ but the belonging HSM file systems cannot be accessed (Linux system-specific). Explanation Regardless of the operating system, the output of the fsepartition --list command shows the status of the corresponding fse-pm and fse-hsm processes, and not the status of the corresponding HSM file system. If the fse-pm process is running, the status is ready. If the fse-hsm process is running, the status is mounted.
Description rmdir also deletes mount points (Windows system-specific). Explanation If an FSE volume is mounted to the directory, for example: fse --mount I:\fse\fsefs_01 \\?\Volume{...}\ and then rmdir is executed, for example: rmdir /S /Q I:\fse\fsefs_01 it deletes all files and directories together with the mount point. Workaround You need to redefine the FSE volume and mount it using the fse --mount command. For details, see chapter ”Configuring FSE” on page 37.
Description If an FSE partition is removed from the FSE implementation configuration, with the assigned HSM file system left intact, this HSM file system cannot be mounted again even it the FSE partition is configured with the same parameters (External FSE client-specific). Explanation The HSM DB, located on the external FSE client, is not removed during removal of the FSE partition.
HSM file system filter problems Description All ordinary file system operations can be performed on an HSM file system that is currently operating in Limited Access Mode. No file or directory creation, move, deletion, rename, and similar operations are prevented (Windows system-specific). Explanation Most probably the HSM file system filter cannot attach to the HSM file system, because the system native NTFS has already attached itself.
Description Free space on an HSM file system runs out and the following error is written to the kernel ring buffer (displayed by the dmesg command): [CRITICAL] FSE-HSM not listening for FS events! Unmount FS and check FSE logs for more details! (Linux system-specific) Explanation n/a Workaround To solve this critical situation, proceed as follows: 1. Stop the FSE implementation using the fse --stop command. 2. Extend the size of the volume that stores the HSM file system.
Migration problems Description All migrations are waiting for disk buffer and data transfer to FSE media does not start. Explanation Regardless of the operating system, if jobs are aborted, killed or they die for reasons unknown, it can happen that the disk buffer file they created will remain on the system, using up disk space, and will never be removed by FSE. The file fse.log will show jobs reporting FSE disk buffer as being full.
Description Files are not migrated because the file system ran out of free space. Explanation The FSE implementation stops migrating files if a file system for FSE databases and system files runs out of free space. Linux specific Mount point of the file system for FSE databases and system files is /var/opt/fse. Windows specific Mount point of the file system for FSE databases and system files is %InstallPath%\var. The value of %InstallPath% depends on the choice made in the FSE installation process.
Description After a migrated file is modified using a third-party application, its new generation is migrated using a different file ID. Explanation The cause for this problem is the way the third-party application is handling the modification process. Such application creates a copy of the current file first and then modifies the copy instead of the original. After the copy is saved, the application renames the copy with the original filename and thus replaces the original.
Recall problems Description Recall jobs do not continue if an FSE medium is marked unusable during the job. Explanation If several recalls are waiting for the same resource (medium), which in the meantime has been marked as unusable, all recall jobs associated to this medium will permanently wait for resources. This is because required FSE media for recall jobs are determined before the resource allocation is requested.
Description A recall job does not search for alternative copies if the FSE medium status is set to unusable after the allocation. Explanation This may happen in the following situation: • A running FSE recall job has determined from which FSE medium the data will be recalled but is still waiting for that medium to become available (probably because another FSE job has allocated the same medium).
Description After an internet shortcut located on an HSM file system has been released, it cannot be opened. The following error message is displayed after clicking on such a shortcut: on Windows Server 2003 systems: The target “TargetObject” of this Internet Shortcut is not valid. Go to the Internet Shortcut property sheet and make sure the target is correct. on Windows 2000 Server systems: The Internet shortcut cannot be opened because failed to run.
Backup, restore, and recovery problems Description FSE backup is aborted, but the backup job hangs. Explanation Backup uses an archiving tool, depending on the platform, to pack all data that needs to be backed up. If the archiving tool has already been started and in the meantime the FSE backup job is aborted, the job hangs until the tool completes the archiving. This may be time consuming, especially if FSE implementation contains a lot of data to be backed up.
Description FSE backup problems (Linux FSE server-specific). Explanation Backup uses tar to pack all data that needs to be backed up. If a tar command has already been started and in the meantime a backup job is aborted, the job hangs until the tar command completes. This might be time consuming if FSE contains a lot of data to be backed up. Workaround In this case, the FSE server system must be cleaned up after the abort command. To clean up the FSE server, proceed as follows: 1.
Description Backup process fails and leaves snapshots of the backed-up file systems (and HSM file systems) on the FSE server (and the FSE clients). Explanation Backup failure prevents the clean-up code to unmount and remove the snapshots that were created during backup. These snapshots occupy space on the FSE server (and the FSE clients), and also prevent you from running subsequent backups. Workaround On the consolidated FSE implementation or the FSE sever, run the release_snap.
Other problems Description Problems with detecting hardware such as SCSI cards, LAN cards (Linux system-specific). Explanation Problems when using an ASUS PR-DL533 motherboard: Linux does not detect hardware correctly. It is a known problem of the Linux ACPI subsystem that it does not work correctly with some motherboards. If specifying the general acpi=off option, it is possible that the ACPI subsystem does not detect the hardware (on-board network card, SCSI card) correctly.
Description Execution of the fsecheck --fsc-media command seems to complete successfully, however, the following message is written to the FSE event log: ERROR: FAILED creating new FRI... ...No space left on device. Explanation The file system that stores the temporary Fast Recovery Information files ran out of free space. Linux specific Fast Recovery Information is located in the directory /var/opt/fse/fri. Windows specific Fast Recovery Information is located in the directory %InstallPath%\var\fri.
Description After an FSE partition is removed from the FSE implementation configuration, the directory /var/opt/fse/part/ and its contents are not deleted (External FSE client-specific). Explanation The HSM DB, located on the external FSE client, is not removed during removal of the FSE partition. Workaround After removing the FSE partition from the FSE implementation configuration, manually delete the corresponding directory and its contents from the external FSE client.
Description The command fsefile cannot parse filenames and directories with special characters. Explanation The command fsefile cannot parse and handle filenames and directories with some special characters, for example path separator characters: backslash ('\') or slash ('/'). Workaround Use wildcards in the command line, for example: fsefile -l .
FSE troubleshooting tools FSE provides a set of commands and scripts called FSE tools, designed to perform advanced low-level FSE servicing tasks. These commands communicate directly with FSE daemons/services and FSE agents, and are usually used for low-level modifications and more extensive monitoring or troubleshooting tasks.
Tool name Description fse_fsc Navigates the File System Catalog (FSC) database. This tool is available on consolidated FSE implementation and FSE server only. dumprmdb.py Dumps the Resource Management Database into the file rmdb_dump_HostName_DateTime.txt in the current directory on the consolidated FSE implementation or the FSE server. This tool is available on consolidated FSE implementation and FSE server only.
The following table represents the values the HSM_DEBUG_LEVEL variable can take, their corresponding levels, and their descriptions. Option value Debugging level Description L 1 — Low Very little trace information will be logged in files, producing less debug overhead and small debug files. N 2 — Normal The usual level of detail in debug trace files. A 3 — API Trace third-party API calls (CORBA, Firebird…). D 4 — Detail Trace all process details, which can produce a lot of debug output.
Linux specific On Linux platform, if you need more disk space than available in the /var/opt/fse/log/debug directory, change your working directory to /var/opt/fse/log and create a symbolic link to a location with more disk space.
Linux specific Flag value Description 0x00000001 main filter execution flow 0x00000002 file system filter 0x00000004 tracing of VFS methods 0x00000008 pseudo device communication 0x00000010 logging of events 0x00000020 HSM attributes 0x00000040 locking 0x00040000 tracing of VFS methods (page cache) 0x00080000 pseudo device communication - detail tracing 0x00100000 logging of events - detail event structure dump 0x00200000 inode info/HSM attributes - detail tracing 0x00400000 locking
Example export HSM_DEBUG_DISKBUF=corrupt HSM_MEDIA_LOGGING This debugging variable cannot be used with the fsesystem --enable-dbg command. It can only be specified in the trace.cfg file. Specifying this variable, you can prevent FSE from logging the actions performed on the FSE media, which are normally logged into the FSE event log. Use the following line to disable media logging: export HSM_MEDIA_LOGGING=no NOTE: Excessive logging may result in an FSE system performance becoming noticeably degraded.
Example export HSM_BEA_VERIFY_POSITION=ALWAYS HSM_LA_SERIAL_UNLOAD This variable makes unloads from drives to library slots serialized. By default the unloads (medium ejects) are made in parallel. Example export HSM_LA_SERIAL_UNLOAD=yes HSM_FRI_SAFETYBUF_TAPE When migrating data, BEA checks for free space on tape to be able to copy the FRI file to the end of the medium. Since the drives only estimate the free space, there should be a safety buffer. This ensures that FRI fits into the available space.
1. Add the following lines to the file /etc/syslog.conf: # Log all kernel messages to file kern.* /var/log/kernel 2. Restart the syslogd daemon: # kill -HUP SyslogdProcessID The file /var/log/kernel will track the complete dmesg history. NOTE: If HSM file system generates too many kernel messages, they may not all be tracked in the file and some of them may be lost. For more information on logging system messages, see the sysklogd man page.
The HSM file system filter debugging level is always set to 0x0000 after restarting the system Linux specific If the HSM file system partition is not listed as noauto in the /etc/fstab file, the HSM file system filter will be loaded at boot time with the debugging level set to 0x0000 regardless of the settings specified in the trace.cfg file. Resolve the problem as follows: 1. Add the following line into the /etc/modules.
/var/opt/fse/log/Hostname_info.log. Send this file and the contents of the directory /var/opt/fse/cfg to technical support. Windows specific Use System Information to collect data about the system. Click Start > Programs > Accessories > System Tools > System Information to open System Information. From the Action menu, select the Save As System Information option and send the file to technical support. One archive package Pack everything collected in previous steps into a .tar file (on Linux platform) or .
Code Description 24 Not found 25 Network connectivity problem 26 Internationalization problem 999 FATAL ERROR 1000 Resource database error 1001 Invalid foreign key 1002 Duplicate unique value 1003 Problem with connection 1004 Not found 1005 Duplicate drive index 1010 Disk buffer allocation error 2000 Cannot insert object 2001 Cannot find object 2002 List is empty 2003 Wrong method call 2004 Event is empty or bad 2005 Object exists 2006 Object is NULL or is not created 2
Code Description 2027 Job index exceeds vector size 2028 Job element index exceeds vector size 2029 HSM is not ready to handle events from filter 2030 Premount event is waiting for same device 2031 No premount event is waiting for device 2032 Mount point is already defined 2033 Invalid job status 2034 HSM FS recovery error 2035 Volume UUID not found. 2036 HSM already started. Unmount first.
Code Description 4012 Invalid alter structure version 4013 Record is not end transaction structure 4014 Record is not start transaction structure 4015 Record is not alter structure 4016 Invalid transaction 4017 Relative file transaction must have pointer to DB 4018 Too many transaction instances 4019 Invalid DB info header 4020 Reserved space in journal file was filled by a transaction 4021 Unused vector is not in buffer 4022 Invalid unused vector structure 4023 Invalid header posit
Code Description 7010 Invalid record sequence number 7011 Byte stream in invalid format 7012 Corrupted volume header 7013 BLKMGR: Writer already running No more readers allowed 7014 Byte stream size mismatch 7015 Rethrown error from other thread 7016 Limited-stream missing 7017 Broken pipe 8004 Problems with Library Agent of this library 8005 Problems with Back End Agent 8006 Problems with Resource Manager 8007 Packer error 8008 Problems starting/stopping partition managers from M
Code Description 9005 Job not found 9006 Table size not valid 9007 Migration ID generator should be initialized first 10000 Job ID does not exist 10001 Configuration error 10002 FSC did not return any set of copies for specific file ID 10003 Job was removed. 10004 Job failed.
Code Description 13001 Split information is wrong 13002 Too many copies in the same generation 13003 No copies found 13004 Media volume number exceeds limit 13005 No media volume file found 13006 Successive change of data 13007 Number of generations does not match 13008 No data 13009 No NSC element object 13010 FSC is busy Possible: Reply from journal in progress 13011 FSC DB error 14000 Invalid volume position 14001 Invalid volume format (not FSE) 14002 Invalid volume ID 14003
A Directory layout Directories created during the installation of FSE are presented in the following tables. Windows specific The value of %InstallPath% depends on the choice made in the FSE installation process. It defaults to C:\Program Files\Hewlett-Packard\FSE.
Path on Linux platform Path on Windows platform Contents /var/opt/fse/log %InstallPath%\var\log All FSE log files /var/opt/fse/log/debug %InstallPath%\var\log\debug All FSE debug files /var/opt/fse/omniorb %InstallPath%\var\omniorb Data files used by the CORBA/omniNames service /var/opt/fse/part %InstallPath%\var\part File System Catalog and Hierarchical Storage Management database and associated journals for each configured FSE partition /var/opt/fse/rmdb %InstallPath%\var\rmdb The Resourc
B FSE configuration templates This appendix provides contents of default templates for the FSE configuration. The template location is platform-dependent. Linux specific Configuration templates are located in the directory /opt/fse/newconfig/. Windows specific Configuration templates are located in the directory %InstallPath%\newconfig. The value of %InstallPath% depends on the choice made in the FSE installation process. It defaults to C:\Program Files\Hewlett-Packard\FSE.
Template for FSE libraries [FSE Library] [Version = 1] [Date = ""] [User = ""] # # Specify a name for the tape library and a hostname, where # it resides. Use full host name. # Name = "USER_DEFINED_NAME" Host = "LIBRARY_HOST_NAME" # # Control device is identification of library's (media changer's) # on the specified system.
Template for FSE drives [FSE Drive] [Version = 1] [Date = ""] [User = ""] # # Specify a name for the tape drive. # Name = "DRIVE_NAME" # # Specify the tape library name where the drive is placed # and the index of the drive in the library (IMPORTANT!) # Index of the first drive in the library always starts with 1.
Host "HOST_NAME" { ControlDevice = "CONTROL_DEVICE_FILE" } 244 FSE configuration templates
Template for an LTO media pool [[FSE Pool] [Version = 1] [Date = ""] [User = ""] # # Specify the name of the media pool. Media pools are # used by FSE partitions. # # NOTE: You have to assign media to pools with fsemedium command. # Name = "MEDIA_POOL_NAME" MediaFamily = LTO # Regular - regular media pool for FSE partition # Backup - media pool for backup media PoolType = Regular # # Change the values of the on-tape format if the defaults don't suit you.
Template for a disk media pool [FSE Pool] [Version = 1] [Date = ""] [User = ""] # # Specify the name of the media pool. Media pools are # used by FSE partitions. # # NOTE: You have to assign media to pools with fsemedium command. # Name = "MEDIA_POOL_NAME" MediaFamily = DISK PoolType = Regular # # Change the values of the on-tape format if the defaults don't suit you. # # NOTE: Some of the parameters will be used only if the tapes are # (re)formatted.
Template for FSE partitions - General part [FSE Partition] [Version = 1] [Date = ""] [User = ""] # # User defined name for an FSE partition. FSE partition # consists of: # * HSM file system # * media pool(s) # Name = "PARTITION_NAME" # # Where is the HSM file system? # Client "HSM_FS_HOST_NAME" { FileSystemID = _HSM_FS_MAJOR_:_HSM_FS_MINOR_ # e.g.
Template for FSE partitions - Migration part [Migration] # # The amount of time that a file must inactive to # become a candidate for migration. # MinFileAge = 10m # min=1s, max=2D [RegularMigration] # # Parameters that trigger a migration.
Template for FSE partitions - Release part [Release] # # Parameters that control the release of files on HSM file system: # # Release starts: # # * on HighWaterMark for files that expire retention and are # larger than MinFileSize # # * on CriticalWaterMark for all already migrated files # # * ReleaseInterval means how often the release conditions are checked.
# If the closing bracket ']' character is to be included, # it must be preceded by a quote '`' # [a-z] Range. # Matches a single character in the range 'a' to 'z' # Ranges and sets may be combined within the same set of # brackets (e.g. [a-c123] matches any character in the set # {a, b, c, 1, 2, 3}) # \ Quote (linux). # ` Quote (Windows). # character Makes the next character a regular (nonspecial) # Note that to match the quote character itself, it must # be quoted.
Template for FSE partitions - Recall and Allocation part [Recall] # # Recall will be cancelled if it can't be performed in RecallTimeOut.
Template for FSE partitions - Deletion part [Delete] # # FileExpiration age: min=1D max=UNLIMITED default=UNLIMITED # # Example: # FileExpiration 7D # { # # Path = (/usr/tmp, "\Temp\Temporary Internet Files") } # 252 FSE configuration templates
Template for FSE system configuration [FSE System] [Version = 1] [Date = "2005/01/01 01:01:01"] [User = ""] # # Resource Manager tries to allocate resources to # all jobs that have priorities in the range of # ResourceAllocationThreshold (in percentage) from # the job with the top priority. # ResourceAllocationThreshold = 50% # min=0%, max=100% # # The influence of time factor on job allocation. # TimeStep = 2 # min=0, max=10000 # # Influence of retries on the job priorities.
FSE configuration templates
Glossary This glossary defines terms used in this guide or related to this product and is not a comprehensive glossary of computer terms. administrative job (admin job) A process, executed by the Management Interface, which performs formatting and initialization of the FSE media. administrator (FSE administrator) A system administrator who installs, configures, monitors, and maintains an FSE implementation.
command (FSE command) An FSE command-line interface command that communicates with FSE system components through the Management Interface. FSE commands are used to configure, monitor, and maintain the FSE implementation. command-line interface (FSE command-line interface (FSE CLI)) A set of FSE commands that are available to FSE users.
deletion policy A set of rules that define expiration periods for groups of directories on an HSM file system. Files which are older than the expiration period defined by the corresponding deletion policy can be automatically deleted using the fsefile command. Configuration parameters for the deletion policy are specified in the FSE partition configuration file. Each FSE partition has its own deletion policy.
event log (FSE event log) A file that records relevant information on events happening in the FSE processes during the operation of the FSE implementation. FSE event log, named fse.log, is located in the directory /var/opt/fse/log (on Linux platform) or in the directory %InstallPath%\var\log (on Windows platform). explicit release Unconditional release, started for a file or set of files on the release candidate list, specified by an FSE user.
File System Event Manager A daemon/service on the FSE client which receives notification about mount events for the HSM file system and triggers the startup of the appropriate Hierarchical Storage Manager. The File System Event Manager process name is fse-fsevtmgr. forced release Release of all files on an HSM file system, which are on the release candidate list, regardless of their retention time and file size.
HSM file system filter The function of this module is to intercept relevant file system events for the FSE implementation and to report them to the Hierarchical Storage Manager. On Linux platform, the filter is a kernel module providing the HSM file system that is built on top of the Linux native Ext3 file system. On Windows platform, the filter is a kernel driver and intercepts all accesses to the Windows native NTFS file system. implementation HP StorageWorks File System Extender.
medium (FSE medium) A high-capacity magnetic tape, or a disk medium used by the FSE implementation. FSE medium volumes are stored on tape partitions, in case of tape medium, or in file system subdirectories, in case of disk medium. Tape medium is further divided into tape partitions and disk medium is divided into subdirectories. Tape partitions store FSE medium volumes on a tape medium and subdirectories store FSE medium volumes on a disk medium.
partition (FSE partition) A configurable entity used by the FSE implementation to split the storage and configure it according to user requirements. Each FSE partition is related to a single HSM file system and one or more FSE media pools. It has its own policies, File System Catalog, and Hierarchical Storage Management Database.
release candidate An online file that is scheduled for release and that has either been migrated or was brought online by recall. release candidate list A list of online files that have already been migrated or that have recently been recalled and are waiting to be released. The list is maintained by the Hierarchical Storage Manager. release policy A set of rules by which the release process for files on a release candidate list is initiated.
software (FSE software) The programs and data files that are included in the FSE package. Once installed, they actively control the FSE implementation and provide functionality to its users. See also ”implementation (FSE implementation)”. system file (FSE system file) A temporary file, created by the FSE software, which contains information about the current status of the FSE implementation.
Index A access modes HSM file system 132 switching 138 switching, limitations 138 adding FSE Management Console users 58 adding media FSE media pools 90 agents 20 allocating media 175 architecture 12 audience 7 authorized resellers 9 automatic deletion 122 automatic error detection 12 B backup 174 backup configuration modifying, on Linux 176 backup frequency estimating 176 backup media management 175 backup media pools 82 configuring 175 backup strategy defining 171 C calculating administration and backup
total offline storage 168 determining configured media pools 84 directory layout 239 dirty files lists 112 disabling FSE partitions 137 disk media 87 disk volumes, Windows 70 displaying FSE Management Console users 58 distributed implementation 18 client-server communication 19 onfiguring the communications settings 19 document conventions 8 documentation 7 duplicating media concepts 92 prerequisites 93 E elements FSE configuration files 38 enabling FSE partitions 137 error messages Limited Access Mode 139
FSE disk buffer 110 FSE disk media 87 local and remote file systems 88 performance disk 87 specifics 89 FSE documentation 7 FSE drives 16 FSE error log examples 154 FSE log files 148, 153 FSE event log examples 149 FSE log files 148, 149 FSE hosts changing IP address 80 FSE implementation consolidated implementation 17 distributed implementation 18 maintaining 135 monitoring 135 multiple libraries implementation 20 FSE interprocess communication configuring 75 configuring, consolidated FSE implementation 76
full backup FSE backup types 173 G GUI FSE Management Console 43 H hardware requirements 8 Help 9 Hierarchical Storage Manager (HSM) lists 109, 110 high watermarks 115 HSM file system 14 access modes 132 Full Access Mode (FAM) 132 Limited Access Mode (LAM) 132 HSM file system log FSE log files 148 HSM file system recovery 186, 187 limitations 187 operation 187 HSM file systems preparing 67, 68 I inconsistency types File System Catalog (FSC) 141, 144 infinite file system 11 initializing media 91 introduci
N NFS support 12 O object tree FSE Management Console GUI 45 object types FSE Management Console 45 offline operational modes FSE Management Console client 42 FSE Management Console client, limitations 43 offline storage, total determining 168 online operational modes FSE Management Console client 42 FSE Management Console client, prerequisites 43 operation HSM file system recovery 187 operations deletion 14 migration 13 recall 14 release 13 P parallel copying 132 parameters deletion policies 122 FSE migr
S scanning media 97 checking results 98 example 97 SCSI identification 60 sequential copying 132 server backup FSE backup types 172 services 20 single client backup FSE backup types 173 software requirements 8 special pattern-matching characters 118 starting deletion jobs 124 FSE Management Console client 33 FSE Management Console server 31 starting FSE backup 176 status bar FSE Management Console GUI 56 status check media pools 85 stopping FSE Management Console client 33 FSE Management Console server 31 s
