53-1001766-01 30 March 2010 Fabric OS FCIP Administrator’s Guide Supporting Fabric OS v6.4.
Copyright © 2009-2010 Brocade Communications Systems, Inc. All Rights Reserved. Brocade, the B-wing symbol, BigIron, DCX, Fabric OS, FastIron, IronPoint, IronShield, IronView, IronWare, JetCore, NetIron, SecureIron, ServerIron, StorageX, and TurboIron are registered trademarks, and DCFM, Extraordinary Networks, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.
Document History Title Publication number Summary of changes Date Fabric OS FCIP Administrator’s Guide 53-1001349-01 New document. July 2009 Fabric OS FCIP Administrator’s Guide 53-1001349-02 Various changes and corrections. October 2009 Fabric OS FCIP Administrator’s Guide 53-1001755-01 New document for Fabric OS January 2010 version 6.3.1. Fabric OS FCIP Administrator’s Guide 53-1001766-01 New document for Fabric OS March 2010 version 6.4.0.
iv Fabric OS FCIP Administrator’s Guide 53-1001766-01
Contents Chapter 1 FCIP overview In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 FCIP platforms and supported features . . . . . . . . . . . . . . . . . . . . . . . 1 FCIP concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 IP WAN network considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Chapter 2 FCIP on the 7800 Switch and FX8-24 Blade In this chapter . . . . . . . . . .
Open Systems Tape Pipelining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 FCIP Fastwrite and OSTP configurations . . . . . . . . . . . . . . . . . .23 Support for IPv6 Addressing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 IPv6 with Embedded IPv4 Addresses . . . . . . . . . . . . . . . . . . . . .25 Configuration preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26 Configuration steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
IPSec implementation over FCIP . . . . . . . . . . . . . . . . . . . . . . . . . . . .45 IPsec configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 IPsec parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Creating an IKE and IPsec policy . . . . . . . . . . . . . . . . . . . . . . . .48 Displaying IKE and IPsec policy settings . . . . . . . . . . . . . . . . . .49 Deleting an IKE and IPsec policy . . . . . . . . . . . . . . . . . . . . .
Portshow command usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Displaying IP interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Displaying IP routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Displaying FCIP tunnel information. . . . . . . . . . . . . . . . . . . . . . . 81 Displaying FCIP tunnel information (7800 switch and FX8-24 blade) . . . . . . . . . . . . . . . . . . . . . . . .
About This Document In this chapter • How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix • Supported hardware and software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix • What’s new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x • Document conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x • Notice to the reader . . . . . . . . . . . .
What’s new in this document This manual applies to FCIP support in Fabric OS version 6.4.0 and later releases. New features include support for IPSec, support for VLAN and DSCP tagging, and support for VEX ports on the FX8-24 blade. Document conventions This section describes text formatting conventions and important notice formats used in this document.
Command syntax conventions Command syntax in this manual follows these conventions: command Commands are printed in bold. --option, option Command options are printed in bold. -argument, arg Arguments. [] Optional element. variable Variables are printed in italics. In the help pages, variables are underlined or enclosed in angled brackets < >. ... Repeat the previous element, for example “member[;member...]” value Fixed values following arguments are printed in plain font.
For definitions of SAN-specific terms, visit the Storage Networking Industry Association online dictionary at: http://www.snia.org/education/dictionary Notice to the reader This document may contain references to the trademarks of the following corporations. These trademarks are the properties of their respective companies and corporations. These references are made for informational purposes only.
Getting technical help Contact your switch support supplier for hardware, firmware, and software support, including product repairs and part ordering. To expedite your call, have the following information available: 1.
Document feedback Quality is our first concern at Brocade and we have made every effort to ensure the accuracy and completeness of this document. However, if you find an error or an omission, or you think that a topic needs further development, we want to hear from you. Forward your feedback to: documentation@brocade.com Provide the title and version number of the document and as much detail as possible about your comment, including the topic heading and page number and your suggestions for improvement.
Chapter 1 FCIP overview In this chapter • FCIP platforms and supported features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 • FCIP concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 • IP WAN network considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 FCIP platforms and supported features There are five Brocade platforms that support FCIP: • The Brocade 7800 switch.
1 FCIP platforms and supported features TABLE 1 FCIP capabilities by platform Capabilities 7800 switch FX8-24 blade 7500/7500E switch FR4-18i blade QoS • • Marking DSCP Yes Yes Yes Yes Marking 802.1P - VLAN tagging Yes Yes Yes Yes • Enforcement 802.
FCIP concepts 1 FCIP concepts Fibre Channel over IP (FCIP) enables you to use existing IP wide area network (WAN) infrastructure to connect Fibre Channel SANs. FCIP supports applications such as remote data replication (RDR), centralized SAN backup, and data migration over very long distances that are impractical or very costly using native Fibre Channel connections. FCIP tunnels are used to pass Fibre Channel I/O through an IP network.
1 4 IP WAN network considerations Fabric OS FCIP Administrator’s Guide 53-1001766-01
Chapter FCIP on the 7800 Switch and FX8-24 Blade 2 In this chapter • 7800 switch hardware overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 • 7800 switch license options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 • FX8-24 blade hardware overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 • FX8-24 blade licensing options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 • FCIP trunking .
2 7800 switch hardware overview 7800 switch hardware overview Figure 2 shows the FC ports and GbE ports on the 7800 switch. There are sixteen FC ports, numbered 0 through 15. The FC ports can operate at 1, 2, 4, or 8 Gbps. There are six GbE ports. Ports 0 and 1 are available as either RJ-45 ports or SFP ports. Only six total GbE ports may be used. The GbE ports provide up to 6 Gbps of bandwidth. 1 3 2 FIGURE 2 4 7800 switch FC and GbE ports 1 FC ports 0 through 3. 2 FC ports 4 through 15.
7800 switch license options 2 VE_Ports and FCIP tunnels on the 7800 switch A 7800 switch can support eight VE_Ports. VE_Ports are numbered from 16 to 23. Each FCIP tunnel is identified with a VE_port number. Up to eight FCIP tunnels may be created. The 7800 switch supports VEX_ports to avoid the need to merge fabrics. FCIP trunking capacity on the 7800 switch FCIP trunks are built by creating a set of FCIP circuits.
2 FX8-24 blade hardware overview FX8-24 blade hardware overview Figure 3 shows the FC ports, GbE port, and 10GbE ports on the FX8-24 blade. There are twelve FC ports, numbered 0 through 11. The FC ports can operate at 1, 2, 4, or 8 Gbps. There are ten GbE ports, numbered 0 through 9. Ports XGE0 and XGE1 are 10GbE ports. The FX8-24 blade provides a maximum of 20 Gbps of bandwidth for connections, and can operate in one of three different modes: • 1 Gbps mode - you can use all ten GbE ports (0 through 9).
FX8-24 blade hardware overview 2 2 3 1 5 FIGURE 3 4 FX8-24 blade FC and GbE ports 1 10GbE ports. (labeled XGE0 and XGE1 on the sticker). 2 GbE ports 0 through 3. 3 GbE ports 4 through 9. 4 FC ports 6 through 11. 5 FC ports 0 through 5.
2 FX8-24 blade licensing options FX8-24 blade licensing options Some of the capabilities of the FX8-24 blade require slot-based feature licenses. These include the following: • • • • 10GbE support. Advanced FICON acceleration. The IR license is required for FCR. The IR license is required to configure VEX_ports. The Advanced Extension License is required for FCIP trunking and Adaptive Rate Limiting (ARL).
FCIP trunking 2 r 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5 FIGURE 4 FCIP Tunnel 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 FCIP tunnel and FCIP circuits Design for redundancy and fault tolerance Multiple FCIP tunnels can be defined between pairs of 7800 switches or FX8-24 blades, but doing so defeats the concept of a multiple circuit FCIP tunnel. Defining two tunnels between a pair of switches or blades is not as redundant or fault tolerant as having multiple circuits in one tunnel.
2 FCIP trunking • In a scenario where a FCIP tunnel has multiple circuits of different metrics, circuits with higher metrics are treated as standby circuits, and are not used until all lower metric circuits fail. Refer to “FCIP circuit failover capabilities” for a more detailed description. • An FCIP tunnel can have up to four circuits when using the 1GbE interfaces, They may be on the same 1GbE interface or spread out over up to four 1GbE interfaces.
FCIP trunking 2 In Figure 6, circuit 1 is assigned a metric of 0, and circuit 2 is assigned a metric of 1. Both circuits are in the same FCIP tunnel. In this case, circuit 2 is a standby that is not used unless there are no lowest metric circuits available. If all lowest metric circuits fail, then the pending send traffic is retransmitted over any available circuits with the higher metric Failover between like metric circuits or between different metric circuits is lossless.
2 Adaptive Rate Limiting Adaptive Rate Limiting Adaptive Rate Limiting (ARL) is performed on FCIP tunnel circuits to change the rate in which the FCIP tunnel transmits data through the IP network. ARL uses information from the TCP connections to determine and adjust the rate limit for the FCIP circuit dynamically. This allows FCIP connections to utilize the maximum available bandwidth while providing a minimum bandwidth guarantee.
QoS SID/DID priorities over an FCIP trunk 2 QoS SID/DID priorities over an FCIP trunk QoS SID/DID traffic prioritization is a capability of Brocade Fabric OS Adaptive Networking licensed feature. This feature allows you to prioritize FC traffic flows between initiators and targets. Each circuit has four internal TCP connections that manage QoS SID/DID priorities over an FCIP tunnel, as illustrated in Figure 7.
2 QOS, DSCP, and VLANs QOS, DSCP, and VLANs Quality of Service (QoS) refers to policies for handling differences in data traffic. These policies are based on data characteristics and delivery requirements. For example, ordinary data traffic is tolerant of delays and dropped packets, but voice and video data are not. QoS policies provide a framework for accommodating these differences in data as it passes through a network. QoS for Fibre Channel traffic is provided through internal QoS priorities.
QOS, DSCP, and VLANs TABLE 2 2 Default Mapping of DSCP priorities to L2Cos Priorities (Continued) DSCP priority/bits L2CoS priority/bits Assigned to: 11 / 001011 3 / 011 Medium QoS 15 / 001111 3 / 011 Medium QoS 19 / 010011 3 / 011 Medium QoS 23 / 010111 3 / 011 Medium QoS 27 / 011011 0 / 000 Class 3 Multicast 31 / 011111 0 / 000 Broadcast/Multicast 35 / 100011 0 / 000 Low Qos 39 / 100111 0 / 000 Low Qos 43 / 101011 4 / 100 High QoS 47 / 101111 4 / 100 High QoS 51 / 1100
2 QOS, DSCP, and VLANs The following example creates an additional FCIP circuit with a different VLAN tag. switch:admin> portcfg fcipcircuit 16 create 1 192.168.2.21 192.168.2.11 100000 -v 200 Operation Succeeded The following example shows a fcipcircuit modify command that changes the vlan tag and l2cos levels for circuit 0. Parameters are the same for both the create and modify options.
QOS, DSCP, and VLANs 2 Keepalive Timeout: 10000 Path MTU Disc: 0 VLAN ID: 300 L2CoS: F: 7 H: 5 M: 3 L: 1 DSCP: F: 32 H: 16 M: 8 L: 4 Flags: 0x00000000 ------------------------------------------Circuit ID: 16.1 Circuit Num: 1 Admin Status: Enabled Oper Status: In Progress Remote IP: 192.168.2.21 Local IP: 192.168.2.
2 Compression options ipif_addr The locally defined IP address. vlan_id The VLAN tag used for this tag (range 1-4094). L2CoS Layer 2 class of service (range 0-7) dst_IP_addr The destination IP address. All frames destined for this IP address will be tagged with the specified vlan_id and L2 CoS. If a destination IP address is not specified, all frames not already tagged will be tagged. The following example adds an entry that tags all frames from IP address 192.168.10.1 destined for IP address 192.
IPSec implementation over FCIP tunnels 2 • Jumbo frames are not supported for IPsec. • There is no RAS message support for IPsec. • IPsec can only be configured on IPv4 based tunnels. IPSec for the 7800 and FX8-24 blade AES-GCM-ESP is used as a single, pre-defined mode of operation for protecting all TCP traffic over an FCIP tunnel. AES-GCM-ESP is described in RFC-4106. Key features are listed below: • Encryption is provided by AES with 256 bit keys.
2 IPSec implementation over FCIP tunnels portcfg fciptunnel 17 create 192.168.0.91 -K12345678901234567890123456789012 portcfg fcipcircuit 17 create 1 192.168.1.91 portcfg fcipcircuit 17 create 2 192.168.2.91 portcfg fcipcircuit 17 create 3 192.168.3.91 portcfg fcipcircuit 17 create 4 192.168.4.91 portcfg fcipcircuit 17 create 5 192.168.5.91 22 192.168.0.81 50000 -x 0 -d c0 -i 192.168.1.81 192.168.2.81 192.168.3.81 192.168.4.81 192.168.5.
Open Systems Tape Pipelining 2 Open Systems Tape Pipelining Open Systems Tape Pipelining (OSTP) can be used to enhance open systems SCSI tape write I/O performance. When the FCIP link is the slowest part of the network, OSTP can provide accelerated speeds for tape read and write I/O over FCIP tunnels. To use OSTP, you need to enable both FCIP Fastwrite and Tape Pipelining.
2 Support for IPv6 Addressing FIGURE 9 Multiple tunnels to multiple ports, Fastwrite and OSTP enabled on a per-tunnel/per-port basis In some cases, traffic isolation zoning TI or LS/LF configurations may be used to control the routing of SID/DID pairs to individual tunnels and provide deterministic flows between the switches, allowing the use of multiple equal cost tunnels. Refer to the Fabric OS Administrator’s Guide for more information about traffic isolation zoning.
Support for IPv6 Addressing 2 • The IPv6 8-bit Traffic class field will be defined by the configured Differentiated Services field for IPv6 (RFC 2474). The configuration of this will be done on the FCIP circuit using the Differentiate Services Code Point (DSCP) parameters to fill the 6-bit DSCP field. • Flow labels are not supported on this IPv6 implementation. The 20-bit Flow Label field will be defaulted to all zeros. • The IPv6 optional Extension Headers will not be supported.
2 Configuration preparation Configuration preparation Before you begin to configure FCIP, do the following: • Determine the amount of bandwidth that will be required for the RDR, FICON or tape application to be deployed. • • • • • • The WAN link has been provisioned and tested for integrity. Cabling within the data center has been completed. Equipment has been physically installed and powered on. Make sure you have admin access to all switches and blades you need to configure.
Configuration steps 2 Setting VE_ports to persistently disabled state VE_Ports used on an FCIP tunnel must be persistently disabled before you can configure FCIP tunnels. You must change their state from persistently enabled to persistently disabled. Once the FCIP tunnels have been fully configured on both ends of the tunnel, you can persistently enable the ports. 1. Enter the portCfgShow command to view ports that are persistently disabled. 2.
2 Configuration steps Setting the GbE port operating mode (FX8-24 blade only) The GbE ports on an FX8-24 blade can operate in one of three ways: • GbE ports 0 through 9 may be enabled as GbE ports, with the XGE ports disabled (the 10GbE license is not required). • 10GbE ports XGE0 and XGE1 may be enabled, with GbE ports 0 through 9 disabled The 10GbE license is required and must be assigned to the slot in which the FX8-24 blade resides.
Configuration steps 2 Configuring a GbE or XGE port IP address You must configure an IP address, netmask, and an MTU size for each GbE port that you intend to use. This is done using the portCfg ipif create command. The following examples create the addressing needed for the basic sample configuration in Figure 10. The following command creates an IP interface for port ge0 on the FX8-24 blade in slot 8 of the Brocade DCX-4S. switch:admin> portcfg ipif 8/ge0 create 192.168.1.24 255.255.255.
2 Configuration steps Configuring an IP route Routing is based on the destination IP address presented by an FCIP circuit. If the destination address is not on the same subnet as the GbE port IP address, you need to configure an IP route with an IP gateway as the destination, using the portCfg iproute create command. Up to 32 IP routes may be defined for each GbE port. Figure 11 adds an IP route for the basic sample configuration. The following command creates an IP route to destination network 192.168.
Configuration steps 2 Creating an FCIP tunnel FCIP tunnels are created using the portCfg fciptunnel create command. The following command creates the FX8-24 end of the tunnel. VE_port 12 is specified. Circuit parameters are included to create circuit 0. The 7800 switch destination address is specified first, followed by the FX8-24 source address. ARL minimum and maximum committed rates are specified for circuit 0. switch:admin> portcfg fciptunnel 8/12 create 192.168.11.78 192.168.1.
2 Configuration steps TABLE 4 Tunnel options Option Argument Compression -c 0|1|2|3 Disruptive Description Enables compression on an FCIP tunnel. Compression is set by the portCfg fciptunnel create or modify command, and applies to traffic over all circuits in the tunnel. Compression cannot be set or modified by the portCfg fcipcircuit create or modify command. A value of 1 enables hardware compression. A value of 0 disables compression.
Configuration steps 2 . TABLE 5 Circuit options Option Argument Committed rate Disruptive Description This option may be used on a portcfg fciptunnel create command or on the portcfg fcipcircuit create command to set a committed rate for an FCIP circuit. When this option is used on the portcfg fciptunnel create command, the committed rate applies only to circuit 0.
2 Configuration steps TABLE 5 34 Circuit options Option Argument Disruptive Description Minimum retransmit time -m The minimum retransmit time, in milliseconds. The range of valid values is 20 through 5,000 ms and the default is 100 ms. failover/standby metric -x You can configure standby circuits by assigning a metric. Refer to “FCIP circuit failover capabilities” on page 12 for a description of circuit failover and the use of standby circuits.
Configuration steps 2 Creating additional FCIP circuits If the Advanced Extension license is enabled, additional FCIP circuits can be created and added to an FCIP tunnel using the portCfg fcipcircuit create command. The following examples adds a circuit to the tunnel in the basic sample configuration (refer to Figure 12). The following command creates circuit 1 on the FX8-24 end of the tunnel. switch:admin> portcfg fcipcircuit 8/12 create 1 192.168.11.79 192.168.1.
2 Configuration steps Enabling persistently disabled ports Ports must be disabled while they are being configured. Before an FCIP tunnel can be used, the associated ports must be persistently enabled. 1. Connect to the switch and log in using an account assigned to the admin role. 2. Enter the portCfgShow command to view ports that are persistently disabled. 3. After identifying the ports, enter the portCfgPersistentEnable command to enable the ports. 4.
Modifying an FCIP tunnel 2 Modifying an FCIP tunnel FCIP tunnel characteristics and options can be modified as needed, using the portCfg fcipTunnel command with the modify option. The command syntax is as follows: portCfg fciptunnel ve_port modify Where: ve_port Each tunnel is assigned to a specific VE_port. The VE_port number serves as the tunnel ID. The range is 16 through 23. Options are as listed and described in Table 4 on page 32, and Table 5 on page 33.
2 Deleting an IP route Deleting an IP route You can delete an IP route to a gateway destination IP address using the portcfg iproute with the delete option. The command syntax is as follows: portcfg iproute ge delete dest_IPv4_addr netmask Deleting an FCIP tunnel When you delete an FCIP tunnel, you also delete all associated FCIP circuits. Use the portCfg fciptunnel command with the delete option to delete FCIP tunnels.
Chapter 3 FCIP on the 7500 Switch and FR4-18i Blade In this chapter • The 7500 switch and FR4-18i blade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • FCIP services license. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • QoS implementation over FCIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • IPSec implementation over FCIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Virtual Fabrics and FCIP .
3 The 7500 switch and FR4-18i blade The 7500 switch and FR4-18i blade Fabric OS supports SAN extension between Brocade 7500 switches, or between FR4-18i blades installed on Brocade 48000 directors or Brocade DCX Data Center Backbone directors. The Brocade 7500 and the FR4-18i blade both have 16 physical Fibre Channel ports and 2 physical GbE ports as illustrated in Figure 13 and Figure 14.
FCIP Design Considerations for the 7500 switch and FR4-18i blade 3 7500 switch and FR4-18i blade ports Each Brocade 7500 Extension Switch with an upgraded license and FR4-18i blade presents 16 FC ports and 16 virtual ports. The Brocade 7500E Extension Switch presents only two active FC ports and one virtual port per GbE interface. Each GbE interface can support up to eight FCIP tunnels which are represented as eight virtual ports on ge0 and 8 virtual ports on ge1.
3 FCIP Design Considerations for the 7500 switch and FR4-18i blade Virtual ports and FCIP tunnels Each Brocade 7500 Extension Switch with an upgraded license and FR4-18i blade presents 16 FC ports and 16 virtual ports. The Brocade 7500E Extension Switch presents only 2 active FC ports and 1 virtual port per GE interface. Each GbE interface can support up to 8 FCIP tunnels which are represented as 8 virtual ports on ge0 and 8 virtual ports on ge1.
FCIP Design Considerations for the 7500 switch and FR4-18i blade Fibre Channel initiator Fibre Channel initiator Office FC SAN Data Center FC SAN IP WAN Network VE_Port VE_Port Brocade 7500 VE_Port Brocade 48000 with FR4-18i Blade Office FC SAN Brocade 7500 VE_Port Office FC SAN Fibre Channel Target FIGURE 15 3 Brocade 48000 with FR4-18i Blade Fibre Channel Target Network using FCIP Compression on FCIP tunnels Data compression can be enabled or disabled on FCIP tunnels.
3 FCIP services license FCIP services license Most of the FCIP extension services described in this chapter require the Brocade High Performance Extension over FCIP/FC license. Use the licenseShow command to verify the license is present on the hardware used on both ends of the FCIP tunnel. QoS implementation over FCIP Quality of Service (QoS) refers to policies for handling differences in data traffic. These policies are based on data characteristics and delivery requirements.
IPSec implementation over FCIP 3 When both DSCP and L2CoS are used If an FCIP tunnel is not VLAN tagged, only DSCP is relevant. If the FCIP tunnel is VLAN tagged, both DSCP and L2CoS are relevant, unless the VLAN is end-to-end, with no intermediate hops in the IP network. The following table shows the default mapping of DSCP priorities to L2Cos priorities per tunnel ID. This may be helpful when consulting with the network administrator. These values may be modified per FCIP tunnel.
3 IPSec implementation over FCIP IPsec uses some terms that you should be familiar with before beginning your configuration. These are standard terms, but are included here for your convenience. TABLE 9 IPsec terminology Term Definition AES Advanced Encryption Standard. FIPS 197 endorses the Rijndael encryption algorithm as the approved AES for use by US Government organizations and others to protect sensitive information. It replaces DES as the encryption standard. AES-XCBC Cipher Block Chaining.
IPSec implementation over FCIP 3 • Secure Tunnels cannot be defined with VLAN Tagged connections. IPsec configuration IPsec requires predefined configurations for IKE and IPsec. You can enable IPsec only when these configurations are well-defined and properly created in advance. The following describes the sequence of events that invokes the IPsec protocol. 1. Traffic from an IPsec peer with the lower local IP address initiates the IKE negotiation process. 2.
3 IPSec implementation over FCIP The parameters listed inTable 11 can be modified. TABLE 11 Modifiable policy parameters Parameter Description Encryption Algorithm 3DES—168-bit key AES-128—128-bit key (default) AES-256—256-bit key Authentication Algorithm SHA-1—Secure Hash Algorithm (default) MD5—Message Digest 5 AES-XCBC—Used only for IPsec Security Association lifetime in seconds Security association lifetime in seconds. A new key is renegotiated before seconds expires.
IPSec implementation over FCIP 3 IKE Policy 10 ----------------------------------------Authentication Algorithm: MD5 Encryption: 3DES Perfect Forward Secrecy: on Diffie-Hellman Group: 1 SA Life (seconds): 28800 Operation Succeeded Displaying IKE and IPsec policy settings 1. Connect to the switch and log in using an account assigned to the admin role. 2.
3 IPSec implementation over FCIP where type is the policy type and number is the number assigned. For example, to delete the IPsec policy number 10: switch:admin> policy --delete ipsec 10 The policy has been successfully deleted. Viewing IPsec information for an FCIP tunnel 1. Connect to the switch and log in using an account assigned to the admin role. 2. Enter the portShow fcipTunnel command.
Virtual Fabrics and FCIP 3 Virtual Fabrics and FCIP Any GigE_Port and all of its associated FCIP tunnels on a chassis can be assigned to any Logical Switch. As with the current Fabric OS, the port types supported by FCIP are either VE_ or VEX_Port. When a GigE port is moved to a logical switch, all eight VE_ and VEX_Ports are automatically moved. There is no interaction required to assign or move them.
3 Options for enhancing tape I/O performance Options for enhancing tape I/O performance There are two options available for enhancing open systems SCSI tape write I/O performance: • FCIP Fastwrite and Open Systems Tape Pipelining (OSTP) • FC Fastwrite FCIP Fastwrite and OSTP are implemented together.
Options for enhancing tape I/O performance TABLE 12 3 Using FCIP Fastwrite and OSTP (Continued) FCIP Fastwrite OSTP Class 3 traffic is accelerated with Fastwrite. Class 3 traffic is accelerated between host and sequential device. With sequential devices (tape drives), there are 1024 initiator-tape (IT) pairs per GbE port, but 2048 initiator-tape-LUN (ITL) pairs per GbE port. The ITL pairs are shared among the IT pairs. For example: Two ITL pairs for each IT pair as long as the target has two LUNs.
3 Options for enhancing tape I/O performance FIGURE 17 Multiple tunnels to multiple ports, Fastwrite and OSTP enabled on a per-tunnel/per-port basis Unsupported configurations for Fastwrite and OSTP The following configurations are not supported with Fastwrite and OSTP. These configurations use multiple equal-cost paths.
Options for enhancing tape I/O performance 3 VE-VE or VEX-VEX FIGURE 18 Fabric OS FCIP Administrator’s Guide 53-1001766-01 Unsupported configurations with Fastwrite and OSTP 55
3 FCIP services configuration guidelines FCIP services configuration guidelines There are multiple configuration requirements and options associated with FCIP services. The following general guidelines may be helpful. The steps are presented in an order that minimizes the number of times ports need to be disabled and enabled. In practice, the steps do not have to be taken in this order. 1. Determine if you are implementing IPsec.
Setting persistently disabled ports 3 Setting persistently disabled ports Ports used on an FCIP tunnel must be persistently disabled before you can configure FCIP tunnels. You must change their state from persistently enabled to persistently disabled. Once the FCIP tunnels have been fully configured on both ends of the tunnel, you can persistently enable the ports. 1. Enter the portCfgShow command to view ports that are persistently disabled. 2.
3 Creating IP interfaces and routes Creating IP interfaces and routes The IP network connection between two Brocade 7500 Extension switches or two FC4-18i blades or one Brocade 7500 Extension switch and one FC4-18i blade is configured by defining IP interfaces for origin and destination virtual ports, and then defining one or more IP routes to connect them. 1. Define the IP interface of each virtual port, using the portCfg command. You can define up to eight IP interfaces per GbE port.
Creating IP interfaces and routes 3 The destination IPv6 address of the virtual port, if IPv6 is used. The address must be an IPv6 global, unicast address. Optionally specify the prefix length. This is used for IPv6 addresses instead of a netmask. If prefix_len is not specified, the prefix length learned from the Neighbor Discovery protocol will be used. dest_IPv4_addr netmask The destination IPv4 address of the virtual port, if IPv4 is used.
3 Creating IP interfaces and routes The following example shows two routes being added to an interface: switch:admin06> portcfg iproute 8/ge0 create 192.168.11.0 255.255.255.0 192.168.100.1 1 switch:admin06> portcfg iproute 8/ge0 create 192.168.12.0 255.255.255.0 192.168.100.
Creating an FCIP tunnel 3 -q type-of-service The DiffServ QoS. The default is 0 (zero). The value must be an integer in the range from 0 through 255. -t ttl The time to live. The default value is 100. -v vlan tag The vlan tag for a VLAN tagged IP connection. -w wait-time The time to wait for the response of each ping request. This parameter is specified in milliseconds and the default value is 5000 milliseconds (5 sec). The maximum allowed wait time for ping is 29000 milliseconds (29 sec).
3 Creating an FCIP tunnel portCfg fciptunnel [slot/]ge0|ge1 create tunnel_id remote_ip_addr local_ip_addr comm_rate [-c] [-s] [-f] [-t] [-M] [-n remote_wwn] [-k timeout] [-r retransmissions] [-m time] [-q control_dscp] [-Q data_dscp] [-v vlan_id] [-p control_L2CoS] [-P data_L2CoS] [-ike ike_number] [-ipsec ipsec_number] [-key preshared_key] [-d FCIP_tunnel_description] [-bstr 0|1 TCP Byte Streaming] Where: slot The number of a slot in a Brocade 48000, a Brocade DCX or DCX-4S enterprise-class platform tha
Verifying the FCIP tunnel configuration -v vlan_id 3 The number used as the VLAN ID. This number is used in the IP frame to route the frame to a specific VLAN. -p control_L2Cos The layer 2 class of service used for control traffic. -P data_L2Cos The layer 2 class of service used for data traffic. -ike ike_number The IKE policy number to be used for this FCIP tunnel. -ipsec ipsec_number The IPsec policy number to be used for this FCIP tunnel.
3 Verifying the FCIP tunnel configuration tunnel_id Displays the specified FCIP tunnel. The following example shows an active tunnel FCIP Fastwrite and OSTP (tape pipelining) enabled. If TCP Byte Streaming were enabled, then FCIP Fastwrite and OSTP would be disabled. SP3:admin> portshow fciptunnel ge1 1 Port: ge1 ------------------------------------------Tunnel ID 1 Tunnel Description Not Configured Remote IP Addr 192.168.15.2 Local IP Addr 192.168.15.
Enabling persistently disabled ports 3 IKE Policy 1 IPSec Policy 1 Pre-Shared Key qbcdefghijklmnopqrstuvwxyz123456 After FCIP tunnels are created, the configuration is saved in a persistent database. At this point, all configured FCIP tunnels now appear in the fabric as VE_Ports. 3. Verify that the VE_Port or VEX_Port is online, use the switchShow command to view and verify that the FCIP tunnel is online.
3 Enabling persistently disabled ports 1. Connect to the switch and log in using an account assigned to the admin role. 2. Enter the portCfgShow command to view ports that are persistently disabled. 3. After identifying the ports, enter the portCfgPersistentEnable command to enable the ports. 4.
Managing FCIP tunnels 3 Managing FCIP tunnels CAUTION Using the modify option disrupts traffic on the specified FCIP tunnel for a brief period of time. NOTE IPsec-enabled tunnels cannot be modified, they can only be deleted and then recreated with new options. This is because IPsec key negotiation uses many of the parameter values during secure tunnel initialization. 1. Connect to the switch and log in using an account assigned to the admin role. 2.
3 Managing FCIP tunnels The maximum number of retransmissions on the existing FCIP tunnel. The range of valid values is 1 through 16. If OSTP is enabled, the number of retransmissions is calculated based on the minimum retransmit time to ensure that the tunnel does not time out before the host times out (approximately 80 seconds). If you change this value, the value specified must be greater than the calculated value. -q control_dscp The DSCP marking for the FCIP tunnel’s TCP control connection.
Managing FCIP tunnels 3 portCfg fciptunnel [Slot/]ge0|ge1 qosmap tunnel_id -default|-delete|vc_num -Q dscp -P L2cos Where: tunnel_id The tunnel_id. Range is 0-7. -default Resets or sets the virtual channel QoS map to default values. -delete Deletes associated QoS map configuration file. Delete QoS mappings before downgrading to pre-v6.0.0 firmware versions that do not support QoS mapping. It removes the file from the config flash memory only.
3 Managing the VLAN tag table Deleting an IP interface (IPIF) The following command deletes an IP interface. portcfg ipif [slot/]ge0|ge1 delete ipaddr NOTE You cannot delete an IP interface until after the tunnel and route have been removed, Managing the VLAN tag table The VLAN tag table is used by ingress processing to filter inbound VLAN tagged frames. If a VLAN tagged frame is received from the network and there is no entry in the VLAN tag table for the VLAN ID, the frame is discarded.
Chapter 4 FCIP Management and Troubleshooting In this chapter • WAN performance analysis tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • FCIP tunnel issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • FCIP links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • FTRACE concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 WAN performance analysis tools Tperf will test single and multiple circuit tunnels. Tperf also tests the different priority connections that are provided by an FCIP Tunnel. When a Tperf--enabled tunnel is operative, it is not an active VE port. Fabrics will not merge over an operative FCIP Tperf tunnel.
WAN performance analysis tools -time 4 Specifies the duration of the Tperf traffic flow in seconds. If a duration is not specified, the process continues to run until it is terminated with Ctrl + C. -unidirectional Generates traffic in one direction only. The default is round-trip. -random Specifies a random protocol data unit (PDU) size between 1 and the size of the send request, as set by -size. -crc Specifies cyclic redundancy check (CRC) to be performed on the payload.
4 WAN performance analysis tools ********************************************************************** ********************************************************************** Tunnel ID: 16 High Priority Medium Priority Low Priority bytes tx 4288640 3669640 2828040 bytes rx 1088456832 931354632 717756552 PDUs tx 107216 91741 70701 PDUs rx 107216 91741 70701 bad CRC headers rx 0 0 0 bad CRC payloads rx 0 0 0 out of seq PDUs rx 0 0 0 flow control count 0 0 0 packet loss (%) 0.2159 0.0957 0.
WAN performance analysis tools 4 The ipperf option NOTE The ipperf option is for 7500 switches and FR4-18i blades. It does not work with 7800 switches and FX8-24 blades. The ipperf option allows you to specify the slot and port information for displaying performance statistics for a pair of ports. For this basic configuration, you can specify the IP addresses of the endpoints, target bandwidth for the path, and optional parameters such as the length of time to run the test and statistic polling interval.
4 WAN performance analysis tools Ipperf performance statistics The following table lists the end-to-end IP path performance statistics that you can display using the portCmd ipperf command and option. TABLE 13 WAN tool performance characteristics Characteristic Description Bandwidth Indicates the total packets and bytes sent. Bytes/second estimates are maintained as a weighted average with a 30 second sampling frequency and also as an average rate over the entire test run.
WAN performance analysis tools 4 2. Configure the sender test endpoint using a similar CP CLI. The syntax for invoking the sender test endpoint using --ipperf for slot8, port ge0 on an FR4-18i is as follows: portcmd --ipperf 8/ge0 -s 192.168.255.100 -d 192.168.255.10 –S The following example shows the results of the performance analysis for slot 8, port ge0: ipperf to 192.41.70.43 from IP interface 192.41.70.42 on 0/1:3227 Sampling frequency(30s) Total time(30s) BW:112.73MBps WBW:55.57MBps Loss(%):0.
4 WAN performance analysis tools Ipperf options Please refer to Brocade Fabric OS Command Reference Manual or the man pages for definitive command syntax and option descriptions. Ipperf options are repeated here for convenience. portCmd --ipperf [slot]/ge0|ge1 -s source_ip -d destination_ip -S|-R [-r rate] [-z size] [-t time] [-i interval] [-p port] [-q diffserv] [-v vlan_id] [-c L2_Cos] Where: 78 -s source_ip The source IP address. -d destination_ip The destination IP address.
WAN performance analysis tools 4 Using ping to test a connection The portCmd ping command tests the connection between the IP address of a local Ethernet port and a destination IP address. If you want to use this command to test a VLAN connection when you do not have an active FCIP tunnel, you must manually add entries to the VLAN tag table on both the local and remote sides of the route, using portCfg vlantag command.
4 WAN performance analysis tools Using Traceroute The portCmd traceroute command traces routes from a local Ethernet port to a destination IP address. If you want to use this command to trace a route across a VLAN when you do not have an active FCIP tunnel, you must manually add entries to the VLAN tag table on both the local and remote sides of the route using portCfg vlantag command.
Portshow command usage 4 Portshow command usage The portshow command can be used to display operational information for 7800 switches, FX8-24 blades, 7500 switches, and FR4-18i blades. The Fabric OS Command Reference Manual and the man pages provide complete descriptions of portshow command syntax and options. The following sections identify a few specific outputs that may be useful for maintenance and troubleshooting.
4 Portshow command usage 23 0 ge0 Disable ----s 0s 0.00 0.00 0 200/1000 0 23 1 ge1 Disable ----s 0s 0.00 0.00 0 200/1000 0 23 2 ge5 Disable ----s 0s 0.00 0.00 0 200/1000 0 23 3 ge4 Disable ----s 0s 0.00 0.
Portshow command usage 4 Oper Status: Up Remote IP: 100.83.0.100 Local IP: 100.80.0.100 Metric: 0 Min Comm Rt: 150000 Max Comm Rt: 150000 SACK: On Min Retrans Time: 100 Max Retransmits: 8 Keepalive Timeout: 5000 Path MTU Disc: 0 VLAN ID: 0 L2CoS: F: 0 H: 0 M: 0 L: 0 DSCP: F: 0 H: 0 M: 0 L: 0 Flags: 0x00000000 ------------------------------------------Circuit ID: 16.1 Circuit Num: 1 Admin Status: Enabled Oper Status: Up Remote IP: 100.83.0.101 Local IP: 100.80.0.
4 Portshow command usage Metric: 0 Min Comm Rt: 150000 Max Comm Rt: 150000 SACK: On Min Retrans Time: 100 Max Retransmits: 8 Keepalive Timeout: 5000 Path MTU Disc: 0 VLAN ID: 0 L2CoS: F: 0 H: 0 M: 0 L: 0 DSCP: F: 0 H: 0 M: 0 L: 0 Flags: 0x00000000 Displaying FCIP tunnel performance (7800 switch and FX8-24 blade) The following example shows performance statistics for a tunnel on a 7800 switch.
Portshow command usage 4 switch:admin>portshow fciptunnel 17 -c --tcp ------------------------------------------Tunnel ID: 17 Tunnel Description: Admin Status: Enabled Oper Status: Up Compression: On (Moderate) Fastwrite: Off Tape Acceleration: Off TPerf Option: Off IPSec: Disabled Remote WWN: Not Configured Local WWN: 10:00:00:05:1e:55:59:e9 Peer WWN: 10:00:00:05:1e:55:68:05 Circuit Count: 4 Flags: 0x00000000 FICON: Off ------------------------------------------Circuit ID: 17.
4 Portshow command usage Retransmit Timeout: 0 ms, Duplicate ACKs 0 Retransmits: 0, max: 0 Fast ReTx: 0, HWM 0, Slow ReTx: 0 Receiver Statistics: Bytes Received: 19624 Packets Received: 450 Receive Window: 25165824 Bytes, max: 0 negotiated window scale: 9 RecvQ Packets: 0 RecvQ Next: 0x494df9a6 Min: 0x494df9a6 Max: 0x494df9a6 Out Of Sequence Pkts: 0, HWM 0, Total 0 Keepalive: Keepalive Timeout: 3600000 ms Keepalive Interval: 37500 ms Inactivity: 300000 ms ------------------------------------------TCP Conn
Portshow command usage 4 Displaying a single circuit The following example shows information for a single FCIP circuit on a 7800 switch. switch:admin> portshow fcipcircuit 20 1 ------------------------------------------Circuit ID: 20.1 Circuit Num: 1 Admin Status: Enabled Oper Status: Up Remote IP: 104.83.0.141 Local IP: 104.80.0.
4 Portshow command usage 0 pkt/s 30s Avg, 0 pkt/s Lifetime Avg 118180 Input Bytes 0 Bps 30s Avg, 4 Bps Lifetime Avg 757 Input Packets 0 pkt/s 30s Avg, 0 pkt/s Lifetime Avg Displaying QoS prioritization for a circuit The following example shows QoS prioritization for an FCIP circuit on a 7800 switch. switch:admin> portshow fcipcircuit 20 1 --perf --qos Circuit ID: 20.1 Circuit Num: 1 Admin Status: Enabled Oper Status: Up Remote IP: 104.83.0.141 Local IP: 104.80.0.
Portshow command usage 4 98 Input Packets 0 pkt/s 30s Avg, 0 pkt/s Lifetime Avg Performance Statistics - Priority: Low 0 Output Bytes 0 Bps 30s Avg, 0 Bps Lifetime Avg 0 Output Packets 0 pkt/s 30s Avg, 0 pkt/s Lifetime Avg 0 Input Bytes 0 Bps 30s Avg, 0 Bps Lifetime Avg 0 Input Packets 0 pkt/s 30s Avg, 0 pkt/s Lifetime Avg Displaying FCIP tunnel information (7500 switch/FR4-18i blade) You can use the portShow fcipTunnel command to view the performance statistics and monitor the behavior of an online FCIP
4 Portshow command usage Data transfer TCP connection: Local 192.175.4.100:4140, Remote 192.175.4.200:3226 Performance stats: 12899612 output packets 34508 pkt/s 30s avg, 30495 pkt/s lifetime avg 14499127648 output Bytes 38787792 Bps 30s avg, 34276897 Bps lifetime avg 0 packets lost (retransmits) 0.
Portshow command usage 4 timeout 10 s Data transfer TCP connection: Local 192.175.4.100:4140, Remote 192.175.4.200:3226 Performance stats: 12899612 output packets 34508 pkt/s 30s avg, 30495 pkt/s lifetime avg 14499127648 output Bytes 38787792 Bps 30s avg, 34276897 Bps lifetime avg 0 packets lost (retransmits) 0.
4 FCIP tunnel issues FCIP tunnel issues The following are common FCIP tunnel issues and recommended actions for you to follow to fix the issue. NOTE The portshow -perf and - params options can be applied only to the 7500 switch and FR4-18i blade. Symptom FCIP tunnel does not come Online. Probable cause and recommended action Confirm the following steps. 1. Confirm GE port is online. portshow ge1 Eth Mac Address: 00.05.1e.37.93.
FCIP tunnel issues 4 Refer to the Fabric OS Administrator’s Guide to review the setup of the ipRoute. 6. Confirm FCIP tunnel is configured correctly. The Compression, Fastwrite, and Tape Pipelining settings must match the opposite endpoint or the tunnel may not come up. Remote and local IP and WWN should be opposite each other. portshow fciptunnel ge1 all Port: ge1 ------------------------------------------Tunnel ID 0 Tunnel Description Not Configured Remote IP Addr 20.24.60.164 Local IP Addr 20.23.70.
4 FCIP links • Confirm that traffic shaping is configured to limit the bandwidth to available using the portShow fciptunnel all -perf –params command. Examine data from both routers. This data is not in the supportshow output and shows retransmissions indicating, input and output rates on the tunnels. Gather this information for both data and management TCP connections. 8. Run tperf for 7800 switches and FX8-24 blades, or ipperf for 7500 switches and FR4-18i blades to gather WAN performance data.
FCIP links 4 Gathering additional information The following commands should be executed and their data collected before a supportsave is run. A supportsave can take 10 minutes or more to run, and some of the information is time critical. NOTE The portshow -perf and - params options can be applied only to the 7500 switch and FR4-18i blade.
4 FTRACE concepts FTRACE concepts FTRACE is a support tool used primarily by your switch support provider. FTRACE can be used in a manner similar to that of a channel protocol analyzer. FTRACE may be used to troubleshoot problems using a Telnet session rather than sending an analyzer or technical support personnel to the site. CAUTION FTRACE is meant to be used solely as a support tool and should be used only by Brocade support personnel, or at the request of Brocade support personnel.
FTRACE concepts 4 Displaying the trace for a tunnel 1. Log on to the switch as admin. 2. Enter the portShow ftrace command. - For the 7500 and FR4-18i blade, the format is as follows: portshow ftrace geX tunnel_ID stats - For the 7800 and FX8-24 blade, the format is as follows: portshow vePortNumber stats Include the slot number for the blades. The FTRACE structures for a 7800 all come from the same trace pool.
4 FTRACE concepts +-----+-------------+--------+------------+------------+------+------+-------------+----------+ | | | |Trace Header| Wrap | In | Out | Switch | Switch | | Id | State | Size | Address | Count | OXID | OXID | Date | Time | +-----+-------------+--------+------------+------------+------+------+-------------+----------+ | 0 | Current | 100000 | 0x0019a980 | 92 | FFFF | FFFF | | | | 1 | unused | 100000 | 0x0019aa80 | 0 | FFFF | FFFF | | | | 2 | unused | 100000 | 0x0019ab80 | 0 | FFFF | FFFF |
FTRACE concepts 4 Example of capturing an FTRACE on a tunnel This process defines how to capture an FTRACE buffer for a 7800 switch or FR4-18i blade, save it, and then enter the supportSave command that includes that data. NOTE For the 7800 and FX8-24, any triggered or checked out and current non-empty trace buffers are captured in a supportSave automatically. There is no user command to force the saving of a trace buffer.
4 FTRACE concepts Write Write Write Write Write Write Write Write Write Write Write Write Write Write Write Write Write Write Progress: 5964096 of 16000320 bytes sent Progress: 6799680 of 16000320 bytes sent Progress: 7078208 of 16000320 bytes sent Progress: 7700800 of 16000320 bytes sent Progress: 8520000 of 16000320 bytes sent Progress: 9355584 of 16000320 bytes sent Progress: 10174784 of 16000320 bytes sent Progress: 10338624 of 16000320 bytes sent Progress: 10846528 of 16000320 bytes sent Progress: 1
Index Numerics 7500 switch and FR4-18i blade, 41 7800 switch, 6 configuring a GbE port, 29 configuring an IP route, 30 creating an FCIP tunnel, 31 creating and FCIP circuit, 35 A Adaptive Rate Limiting (ARL), 14 F Fastwrite, 52 Fabric OS FCIP Administrator’s Guide 53-1001766-01 FCIP configuration guidelines, 56 configuring VEX_Ports, 27, 57 creating a tunnel, 61 creating interfaces, 58 creating routes, 58 DSCP, 16, 44 Fastwrite, 52 gathering additional information, 95 IP compression, 45 IPsec changeabl
G gathering FCIP information, 95 GbE port mode on the FX8-24 blade, 28 I ipperf, 75 IPsec FCIP, 45 FCIP changeable parameters, 48 FCIP configuration, 47 FCIP fixed parameters, 47 L tape read and write acceleration, 52 tperf, 71 tunnel goes on- and offline, 93 V VE_Ports, 42 VEX_Port, 42 Virtual Fabrics FCIP, 51 W WAN, 71 WAN analysis tools, 71 License requirements 7800 switch, 6 FX8-24 blade, 10 Load leveling and failover, 12 M Media type for 7800 GbE ports, 27 O Open Systems Tape Pipelining (OSTP),
