Dell EMC SC Series with Red Hat Enterprise Linux 7x Abstract This paper provides guidelines for volume discovery, multipath configuration, file system, queue depth management, and performance tuning of Red Hat® Enterprise Linux® (RHEL) 7.x with Dell™ Storage Center Operating System (SCOS) 7.x.x.
Revisions Revisions Date Description July 2014 Initial release May 2015 Introduce connectivity to Dell Storage SCv2000 July 2017 Refresh for RHEL 7.3 January 2018 Minor updates Acknowledgements Author: Steven Lemons The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Table of contents Table of contents Revisions.............................................................................................................................................................................2 Acknowledgements .............................................................................................................................................................2 Table of contents .................................................................................................
Table of contents 5.3 The dmsetup command ....................................................................................................................................52 5.4 The dmesg command .......................................................................................................................................53 5.5 The /proc/scsi/scsi file ......................................................................................................................................
Executive summary Executive summary Red Hat® Enterprise Linux® (RHEL) is an extremely robust and scalable enterprise-class operating system. Correctly configured using the best practices presented in this paper, the RHEL operating system provides an optimized experience with Dell EMC™ SC Series storage. These recommendations include guidelines for volume discovery, multipath configuration, file system, queue depth management, and performance tuning. This paper presents features of RHEL 7.
Overview 1 Overview SC Series storage provides Linux-compatible and SCSI-3 compliant disk volumes that remove the complexity of allocating, administering, using, and protecting mission-critical data. A properly configured SC Series array removes the need for cumbersome physical disk configuration exercises and management along with complex RAID configuration practices.
Volume management 2 Volume management Understanding how volumes are managed in Linux requires basic understanding of the /sys pseudo filesystem. The /sys filesystem is a structure of files that allow interaction with various elements of the kernel and modules. While the read-only files store current values, read/write files trigger events with the correct commands. Generally, the cat and echo commands are used with a redirect as STDIN instead of being opened with a traditional text editor.
Volume management else echo "WARN: OSMajor parameter of unknown value, exit 1" exit 1 fi The following shows the sample output from this script: # ./scan_bus.sh INFO: OS Major Rev. el7 detected! Alternatively, the installation of the sg3_utils package would provide a native Red Hat command rescanscsi-bus.sh located in the /usr/bin folder. # /usr/bin/rescan-scsi-bus.sh --alltargets Scanning SCSI subsystem for new devices [snip] 0 new or changed device(s) found. 0 remapped or resized device(s) found.
Volume management As a best practice, use SC Series volumes as whole drives. However, LVM may still be used if it can provide features or benefits that are not provided by the storage layer. 2.3.1 LVM configuration and SCSI UNMAP/TRIM If LVM is used with SC Series volumes, LVM can be configured to respect SCSI UNMAP/TRIM commands and pass these commands back to the SC Series storage. This scenario applies where logical volumes are removed from the member volume group.
Volume management There are multiple ways to ensure that these volumes are assigned and referenced by a persistent naming scheme. This section presents using volume labels or universally unique identifiers (UUIDs) with /dev/sdX referenced volumes. Volume labels are exceptionally useful when scripting SC Series snapshot recovery. An example involves mounting a snapshot view volume of a production SC Series snapshot to a backup server.
Volume management 2.5.2 Universally unique identifier UUID values exist and persist at multiple layers of volume metadata. UUIDs are created and used by the device-mapper drivers to uniquely identify and manage each multipath device in Linux. UUIDs are created within LVM metadata (if the volumes are managed by LVM). UUIDs are also created and managed within the filesystem layer to uniquely identify each filesystem volume.
Volume management 2.5.4 Volume label and UUID script This sample script parses all devices identified as vendor=”COMPELNT”, determines if any filesystem has been applied to the volume, then extracts and displays any discovered disk label values, mpath-layer UUID, and disk-layer UUID values.
Volume management Sample output from this script as shown in the following: # ./get_UUID.sh [snip] INFO: /dev/mapper/vol_001 Multipath Info: UUID: mpath-36000d3100000650000000000000017f2 Filesystem Details: Type: xfs label = "TestVol" UUID = fbef5bfb-94b9-4827-a103-516045b9b608 INFO: /dev/mapper/vol_002 Multipath Info: UUID: mpath-36000d3100000650000000000000017f3 Filesystem Details: Type: LVM2_member WARN: Filesystem unknown [snip] 2.
Volume management Note: The attempt to mount /dev/mapper/vol_004 as /vol_004 fails because the UUID is not unique and already in use on the Linux host. # df -k Filesystem [snip] rhevm:/nfsroot /dev/mapper/vol_001 [snip] 1K-blocks Used Available Use% Mounted on 103213056 54833152 52418560 1939052 43137024 50479508 56% /Tools 4% /vol_001 Attempting to mount /dev/mapper/vol_004 to /vol_004 returns the following error.
Volume management Note: The following volume expansion steps are only for those volumes which have an XFS filesystem applied. This is not applicable to volumes which have already been partitioned or LVM enabled. 1. Expand the SC Series volume. a. b. c. d. Right-click the SC Series volume. Select Expand Volume. Enter a new size for the volume. Click OK to execute the expansion. 2. Rescan the drive geometry for each path of the multipath device by executing: # echo 1 >> /sys/block/sdX/device/rescan 3.
Volume management The following shows the sample, truncated output from the rescan-scsi-bus.sh script showing a rescan of the SCSI bus against two host adapter’s (adapter 10 and adapter 8) with no new device found or devices removed: # rescan-scsi-bus.sh Host adapter 10 (iscsi_tcp) found. Host adapter 8 (iscsi_tcp) found. Scanning SCSI subsystem for new devices Scanning host 8 for all SCSI target IDs, all LUNs Scanning for device 8 0 0 1 ...
Volume management 4. Remove any mappings to the volume(s) from SC Series storage. 2.9 Boot from SAN The ability to use SC Series volumes as bootable volumes in Linux allows system and storage administrators to further leverage the strengths of SC Series snapshot and snapshot view volume technologies. Two uses for snapshots of Linux boot volumes are: a backup/recovery mechanism, or to preserve the state of an operating system at a point in time prior to upgrades.
Volume management 3. Click Advanced Options. 4. Select the Map volume using LUN 0 check box and click Finish to map this volume. During the Linux host boot, the HBA BIOS boot process is interrupted and instructed to identify this SC Series volume as the preferred boot device in the boot device order.
Volume management During the Linux installation process, this SC Series volume is identified and selected as the installation target shown as follows. 1. In the Installation main menu, click Installation Destination. 2. Click Add a disk….
Volume management 3. Identify the SC Series volume in the Multipath Devices tab, and select the check box accordingly. 4. Click Done. The SC Series volume shown is identified and selected as the single installation target for the Linux operating system. 5. Click Done to return to the installation main menu.
Volume management 2.9.1 Capturing a boot from SAN snapshot volume Capturing a snapshot volume of a boot from SAN volume is straightforward in SC Series storage. However, it is recommended to freeze any non-LVM-bound XFS volumes on the Linux host in order to quiesce any I/O before the snapshot volume is captured. In this scenario (with a default Linux installation), the / and /boot filesystems are XFS-based, and frozen with the following commands.
Volume management 3. After the snapshot has been captured, the / and /boot filesystems are XFS-based and should be unfrozen with the following commands. # xfs_freeze -u / # xfs_freeze -u /boot A snapshot view volume from this captured snapshot is created as shown in step 4. 4. Right-click the desired snapshot in the volume Snapshots tab, and click Create Volume from Snapshot.
Volume management 5. Associate this snapshot view volume to the appropriate server host object and click OK. The snapshot view volume is shown in the Volumes tree, beneath the volume where it was created.
Volume management 2.9.2 Recovering from a boot from SAN snapshot view volume Snapshot view volumes of a boot from SAN volume, when presented back to the same host, are automatically bootable. This is because the volumes are now tracked using their unique and identifying UUID values inside of /etc/fstab and /boot/grub2/grub.cfg instead of using their system device file names.
Server configuration 3 Server configuration This section discusses the various configuration aspects of the I/O stack of a Linux host. The Linux I/O stack can be precisely configured to adapt to the needs of the environment in which the Linux host operates. The configuration aspects include tuning the HBA port retry count, queue depth, SCSI and iSCSI device timeout values, and more. 3.
Server configuration Note: It is no longer recommended to manually edit the /boot/grub/grub.cfg file. Use the grubby command line interface to achieve boot-level configuration changes. The grubby tool manual pages and help are available with the following commands, respectively: man grubby or grubby --help. 3.2 iSCSI RHEL 7.x introduces an updated software implementation of the open-iSCSI stack (based on RFC3720). This new iSCSI stack resides in the kernel memory space instead of user space.
Server configuration In this demonstration, only the em2 interface is configured to the iSCSI VLAN as shown in the following ifconfig and netstat output. In production use, it is recommended to configure more than one iSCSI interface for each Linux host, using dm-multipath for path, redundancy, and I/O queue management. 1. Configure the network interface. # ifconfig -a [snip] em2: flags=4163 mtu 1500 inet 10.10.83.21 netmask 255.255.0.0 broadcast 10.10.255.
Server configuration 3. Issue the following commands to request SC Series target IQNs and request the Linux host to login to these targets. # iscsiadm -m discovery -t sendtargets -p 10.10.97.2:3260 10.10.97.1:3260,0 iqn.2002-03.com.compellent:5000d3100002bb2f 10.10.97.1:3260,0 iqn.2002-03.com.compellent:5000d3100002bb30 10.10.97.1:3260,0 iqn.2002-03.com.compellent:5000d3100002bb32 10.10.97.1:3260,0 iqn.2002-03.com.compellent:5000d3100002bb3b # iscsiadm -m discovery -t sendtargets -p 10.10.130.2:3260 10.10.
Server configuration 4. On the SC Series storage, set up the new iSCSI server object. a. Right-click the appropriate Servers tree folder and click Create Server. b. Name the server object, set the Operating System to Red Hat Linux 6.x, select the IQN checkbox that corresponds to the Linux host IP address (in this case, 10.10.83.21), and click OK.
Server configuration The new LinuxRHEL-iscsi server object is shown in the LinuxRHEL folder and is ready to have volumes mapped to it. 3.2.2 iSCSI timeout In single-path environments, configure the iSCSI daemon to wait and queue I/O for an amount of time sufficient to accommodate proper failure recovery. For example, a SAN fabric failure or SC Series storage array failover event can take between 5 and 60 seconds to complete.
Server configuration To adjust the following timeout values, for a multipath enabled environment, open /etc/iscsi/iscsid.conf and configure the following parameters: To control how often a NOP-Out request is sent to each target, configure the following parameter to 5 seconds instead of the default value of 10 seconds. node.conn[0].timeo.noop_out_interval = 5 To control the timeout value for the NOP-Out request, configure the following parameter to 5 seconds instead of the default value of 15 seconds.
Server configuration 3.2.3 Configuring /etc/fstab iSCSI is dependent on an operational network, and volumes added to /etc/fstab need to be designated as network-dependent. In other words, do not attempt to mount an iSCSI volume until the network layer services have completed the startup and the network is operational. The example below demonstrates how to create this network dependency to the iSCSI mount using the _netdev mount option in the /etc/fstab file.
Server configuration 3.3.2 SAS /etc/multipath.conf Add the following device schema to the /etc/multipath.conf file that is used exclusively for SAS-connected Linux hosts. This schema defines the configuration parameters for all devices identified by vendor=”COMPELNT” and product=”Compellent Vol”. This schema is typically added after the defaults schema and before the blacklist_exceptions schema.
Server configuration 3.3.5 Identify SAS devices on Dell SCv2000 SAS devices are exposed to the Dell SCv2000 through their singular device names instead of their multiple SAS World-Wide Port Names (WWPNs). The following sequence of screenshots outlines the procedure to create a new server object on a Dell SCv2000 storage array. 1. Right-click in the Servers folder tree and select Create Server. 2. Name the new server object, select the proper Operating System (Red Hat Linux 6.x or 7.
Server configuration 3. The new server object (tssrv303) is created in the Servers folder tree, with the Connectivity pane displaying that the Linux host has established connectivity to both bottom and top controllers of the SC Series array. 4. Select the SAS controller ending with *FEB315 in the Hardware pane to display the actual SAS WWPN ending with *2800 in the Connectivity pane. 5.
Server configuration 3.3.6 Configured multipath A properly configured SAS volume will return the following multipath –ll output. This SC Series volume is discovered as a multipath ALUA-capable volume, where each path is capable of I/O. The path represented by prio=50 (active/optimized path) is used for all active I/O requests. The path represented by prio=1 (standby path) is a highly available, redundant path and is used when the active/optimized path becomes unavailable.
Server configuration For QLogic: options qla2xxx ql2xmaxqdepth= For Emulex: options lpfc lpfc_hba_queue_depth= lpfc_lun_queue_depth= The values inside these files can be stacked with other additional key value pairs as needed to define a complete configuration set. 3.5 SCSI device timeout The SCSI device timeout of discovered SC Series storage volumes defaults to 30 seconds. Verify the setting with the following command.
Server configuration 3.6 /etc/multipath.conf and SC Series volume definition The SC Series storage device definitions are incorporated into the kernel. Use /etc/multipath.conf within its native syntax constraints as shown in the following. In this example, all other vendor devices are blocked while the SC Series storage devices are identified in the blacklist_exceptions clause by means of their unique WWID values. This /etc/multipath.
Server configuration vendor PLDS product DVD-ROM_DS-8D3SH } #wwid "*" } blacklist_exceptions { device { vendor "COMPELNT" product "Compellent Vol" } } multipaths { multipath { wwid "36000d3100000650000000000000017f2" alias "vol_001" uid 0 gid 0 mode 0600 } multipath { wwid "36000d3100000650000000000000017f3" alias "vol_002" uid 0 gid 0 mode 0600 } multipath { wwid "36000d3100000650000000000000017f4" alias "vol_003" uid 0 gid 0 mode 0600 } } 39 Dell EMC SC Series with Red Hat Enterprise Linux 7x | CML1071
Server configuration An example of the multipath -ll command output is shown as follows. [root@dean ~]# multipath -ll boot_drive (36000d31000035e000000000000000059) dm-0 COMPELNT,Compellent Vol size=146G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 1:0:1:0 sdc 8:32 active ready running |- 1:0:2:0 sde 8:64 active ready running |- 4:0:1:0 sdb 8:16 active ready running `- 4:0:2:0 sdd 8:48 active ready running [snip] Note: With RHEL 7.
Server configuration 3.6.2 Multipath queueing The compiled defaults of presented and discovered SC Series storage volumes feature the 1 queue_if_no_path option that instructs multipath to automatically queue and retry all I/O requests should all paths become unavailable. This is observed from the following sample output, and should be left in its default state.
Server configuration In either failover scenario, the SC Series storage may take anywhere from five up to 60 seconds to propagate these changes through the SAN fabric. In order to mitigate I/O disruption in multipath scenarios, it is recommended to instruct the HBA code to wait up to 5 seconds before marking a port as down or failed. This minimizes I/O latency to the I/O or application stack above it by quickly relocating I/O requests to alternate and still active HBA paths through the SAN fabric.
Server configuration ports experiencing failure will be relocated to the active NPIV ports within the same fault domain on the alternate active controller. In order to mitigate I/O disruption in single-pathed scenarios, instruct the HBA driver code to wait up to 60 seconds before marking a port as down or failed. This allows the SC Series storage sufficient time to relocate the WWPN of the failed ports to active ports and to propagate the changes through the SAN fabric.
Performance considerations 4 Performance considerations This section provides general information and guidance pertaining to some of the more common performance tuning options and variables available to Linux. This information is not intended to be all encompassing and the values used should not be considered final. This section provides a starting point that Linux and storage administrators can use to fine tune a Linux installation and achieve optimal performance.
Performance considerations 4.1 Tuning profiles RHEL 7.x introduced a new set of tools to assist administrators and storage professionals with tuning RHEL hosts. This is achieved using two new commands, tuned and tuned-adm, which manage a set of predefined, performance-tuned profiles. The following output presents the active tuning profile (default) as well as the list of alternative tuning profiles that can be applied.
Performance considerations 1. Click the Advanced Options link. 2. Uncheck Restrict Mapping Paths and select the controller to pin the volume to for mapping the server object. 4.3 Understanding HBA queue depth Queue depth refers to the number of pending I/O requests. Modifying this value can lead to an improvement in I/O performance in certain workloads.
Performance considerations 4.4 SCSI UNMAP/TRIM The application and use of SCSI UNMAP/TRIM in filesystem use cases can be very effective towards storage and cost management in the business enterprise. However, consideration should also be made in regard to how this function is implemented. The discard mount parameter would enable the filesystem to perform real-time, on-demand SCSI UNMAP commands to the SC Series array.
Performance considerations This parameter can be dynamically changed by performing the following command. However, it is recommended to leave the scheduler as deadline with all SC Series storage implementations. The scheduler value is also dynamically adjusted according to the use of RHEL 7.x tuning profiles as previously discussed.
Performance considerations The following example is applied to all SC Series storage devices. # cat >> /etc/udev/rules.d/50-compelnt.rules <
Useful tools 5 Useful tools The following native Linux tools can be used to identify and correlate volumes to their respective Storage Center devices. 5.1 The lsscsi command The lsscsi command is a tool that parses information from the /proc and /sys pseudo filesystems into human readable output. The lsscsi tool is installed using the following command. # yum –y install lsscsi The lsscsi output is shown in the following example, using the grep command to display only SC Series devices.
Useful tools This script applies the scsi_id command and correlates Linux device names (/dev/sdX) to their respective WWID values. #!/bin/bash SCSIID=/usr/lib/udev/scsi_id #OSMajor=`cat ${ReleaseFile} | awk '{print $7}' | cut -d. -f1` OSMajor=`uname -r | awk -F. '{print $4}'` echo "INFO: OS Major Rev.
Useful tools These WWID values in turn, correlate to the Serial Number value of the SC Series storage devices. Serial Number field 5.3 The dmsetup command The dmsetup command can be used to view, manage, and change the metadata of logical volumes that are managed by the device-mapper driver. In the following example, the dmsetup command is used to display the Name and UUID values of this SC Series volume (the UUID value is a concatenation of the mpath- prefix to the volume serial number value).
Useful tools 5.4 The dmesg command The dmesg command is useful for discovering which device names are assigned to recently discovered volumes. The following output demonstrates the discovery and assignment of a new SC Series volume to the /dev/sdh device file. # dmesg [snip] [1203830.267568] scsi 4:0:0:1: Direct-Access COMPELNT Compellent Vol 0605 PQ: 0 ANSI: 5 [1203830.267996] sd 4:0:0:1: Attached scsi generic sg8 type 0 [1203830.268089] sd 4:0:0:1: [sdh] 20971520 512-byte logical blocks: (10.7 GB/10.
Dell Storage REST API 6 Dell Storage REST API The Dell Storage REST API interface is available with the installation of Dell Storage Manager (DSM) 2015 R3 or newer. The REST API is recommended when intending to interact with SC Series storage managed by the DSM installation, either programmatically or in a command line interface. For more information, refer to the Dell Storage Representational State Transfer (REST) API Cookbook. The use of the Compellent Command Utility (CompCU) has been deprecated.
Configuration details A Configuration details Component Description Operating system Red Hat Enterprise Linux 7.3 (Maipo) Driver version Driver version = 8.04.00.04.06.3-k-debug BIOS version = 3.00 Firmware version Firmware version = 5.06.05 (90d5) Application NA Cabling QLE2562 8Gb Dual Port PCIe FC HBA Dell EMC 12Gbps SAS HBA Server Dell PowerEdge™ R630 x 2 Storage Dell SC8000, SCOS 6.6.x, virtual port mode Dell SCv2x00, SCOS 6.6.
Technical support and resources B Technical support and resources Dell.com/support is focused on meeting customer needs with proven services and support. Dell TechCenter is an online technical community where IT professionals have access to numerous resources for Dell EMC software, hardware, and services. Storage Solutions Technical Documents on Dell TechCenter provide expertise that helps to ensure customer success on Dell EMC storage platforms. B.
