TRex i TRex
TRex ii REVISION HISTORY NUMBER DATE 2.
TRex iii Contents 1 2 Introduction 1 1.1 A word on traffic generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Current Challenges: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.2 Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Overview of TRex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
TRex 4 iv Basic usage 18 4.1 DNS basic example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.2 DNS, take flow IPG from pcap file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.3 DNS, Set one server ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.4 DNS, Reduce the number of clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
TRex 7 v Appendix 7.1 Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 7.1.1 7.2 7.3 72 Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 firmware update to XL710/X710 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 7.2.1 Download the driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
TRex vi 7.8.7 Performance Cycles/Packet ConnectX-4 vs Intel XL710 . . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.8.8 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 7.8.9 Limitations/Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 7.8.10 Build with native OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 7.
TRex 1 / 113 Chapter 1 Introduction 1.1 A word on traffic generators Traditionally, routers have been tested using commercial traffic generators, while performance typically has been measured using packets per second (PPS) metrics. As router functionality and services have become more complex, stateful traffic generators have become necessary to provide more realistic traffic scenarios. Advantages of realistic traffic generators: • Accurate performance metrics.
TRex 1.2 2 / 113 Overview of TRex TRex addresses the problems associated with commercial stateful traffic generators, through an innovative and extendable software implementation, and by leveraging standard and open software and x86/UCS hardware. • Generates and analyzes L4-7 traffic. In one package, provides capabilities of commercial L7 tools. • Stateful traffic generator based on pre-processing and smart replay of real traffic templates. • Generates and amplifies both client- and server-side traffic.
TRex 3 / 113 Chapter 2 Download and installation 2.1 Hardware recommendations TRex operates in a Linux application environment, interacting with Linux kernel modules. TRex curretly works on x86 architecture and can operate well on Cisco UCS hardware. The following platforms have been tested and are recommended for operating TRex. Note A high-end UCS platform is not required for operating TRex in its current version, but may be required for future versions.
TRex 4 / 113 Table 2.2: (continued) Components Memory RAID Details 2x4 banks f.or each CPU. Total of 32GB in 8 banks. No RAID. Table 2.3: High-end C240 Mx - Internal components Components CPU PCIe CPU Configuration Memory RAID Riser 1/2 Details 2x E5-2667 @ 3.20 GHz. 1x Riser PCI expansion card option A PID UCSC-PCI-1A-240M4 enables 2 PCIex16. 2-Socket CPU configurations (also works with 1 CPU). 2x4 banks for each CPU. Total of 32GB in 8 banks. No RAID. Both left and right should support x16 PCIe.
TRex 5 / 113 Table 2.4: (continued) Chipset Bandwidth (Gb/sec) 10 TSO LRO Example + - 40 + - Napatech SmartNICs NT200A01 10/25/40/50/100 + - Mellanox ConnectX4/Lx 25/40/50/56/100 + + Mellanox ConnectX5 Cisco 1300 series VMXNET3 25/40/50/56/100 + + ./b configure --with-ntacc to build the library, it is not part of our regression so it might be broken .
TRex 6 / 113 Table 2.6: XL710 NIC base QSFP+ support QSFP+ QSFP+ SR4 optics QSFP+ LR-4 Optics QSFP Active Optical Cables (AoC) QSFP+ Intel Ethernet Modular Supported QSFP+ DA twin-ax cables Active QSFP+ Copper Cables Intel Ethernet Converged XL710-QDAX Supported: APPROVED OPTICS. Not supported: Cisco QSFP-40G-SR4-S Supported: APPROVED OPTICS.
TRex 7 / 113 Table 2.9: FM10K QSFP28 support QSFP28 todo Example todo Important • Intel SFP+ 10Gb/sec is the only one supported by default on the standard Linux driver. TRex also supports Cisco 10Gb/sec SFP+. • For operating high speed throughput (example: several Intel XL710 40Gb/sec), use different NUMA nodes for different NICs.
TRex 8 / 113 • Ubuntu 14.04.1 LTS, 64-bit kernel (not 32-bit) • Ubuntu 16.xx LTS, 64-bit kernel (not 32-bit) — Not fully supported. • CentOs/RedHat 7.2 LTS, 64-bit kernel (not 32-bit) — This is the only working option for ConnectX-4. Note Additional OS versions may be supported by compiling the necessary drivers. To check whether a kernel is 64-bit, verify that the ouput of the following command is x86_64. [bash]>uname -m x86_64 2.2.
TRex 9 / 113 Note • Requirement for using TRex: sudo or root password for the machine. • Upgrading the linux Kernel using yum upgrade requires building the TRex drivers. • In Ubuntu 16, auto-updater is enabled by default. It is recommended to turn it off. Updating kernel requires re-compiling the DPDK .ko file. To disable auto-updater: > sudo apt-get remove unattended-upgrades 2.2.4 Verify Intel NIC installation Use lspci to verify the NIC installation.
TRex 10 / 113 Bleeding edge version: [bash]>wget --no-cache $WEB_URL/release/be_latest To obtain a specific version: [bash]>wget --no-cache $WEB_URL/release/vX.XX.tar.gz # 1v v 1 X.
TRex 11 / 113 Chapter 3 First time Running 3.1 Configuring for loopback Before connecting TRex to your DUT, it is strongly advised to verify that TRex and the NICs work correctly in loopback. Note 1. For best performance, loopback the interfaces on the same NUMA (controlled by the same physical processor). If you are unable to check this, proceed without this step. 2.
TRex 12 / 113 0000:03:00.1 ’82599ES 10-Gigabit SFI/SFP+ Network 0000:13:00.0 ’82599ES 10-Gigabit SFI/SFP+ Network 0000:13:00.1 ’82599ES 10-Gigabit SFI/SFP+ Network 0000:02:00.
TRex 3.2.1 13 / 113 Interactive mode The script provides a list of available interfaces with interface-related information. Follow the instructions to create a basic config file. [bash]>sudo ./dpdk_setup_ports.py -i 3.2.2 Command line mode Run the following command to display a list of all interfaces and interface-related information: [bash]>sudo ./dpdk_setup_ports.py -t • In case of Loopback and/or only L1-L2 Switches on the way, IPs and destination MACs are not required.
TRex 3.3 14 / 113 TRex on ESXi General recommendation: For best performance, run TRex on "bare metal" hardware, without any type of VM. Bandwidth on a VM may be limited, and IPv6 may not be fully supported. In special cases, it may be reasonable or advantageous to run TRex on VM: • If you already have VM installed, and do not require high performance. • Virtual NICs can be used to bridge between TRex and NICs not supported by TRex. 3.3.1 Configuring ESXi for running TRex 1.
TRex 15 / 113 2. Mark the desired NICs. 3. Reboot the ESXi to apply. 4. Right-click the guest machine. Edit settings → Add → PCI device → Select the NICs individually. 3.4 Configuring for running with router (or other L3 device) as DUT You can follow this presentation for an example of how to configure the router as a DUT. 3.
TRex 16 / 113 -----------link : link : Link Up - speed 10000 Mbps - full-duplex promiscuous : 0 -Per port stats table ports | 0 | 1 | 2 | 3 ------------------------------------------------------------------------------------opackets | 1003 | 1003 | 1002 | 1002 obytes | 66213 | 66229 | 66132 | 66132 ipackets | 1003 | 1003 | 1002 | 1002 ibytes | 66225 | 66209 | 66132 | 66132 ierrors | 0 | 0 | 0 | 0 oerrors | 0 | 0 | 0 | 0 Tx Bw | 217.09 Kbps | 217.14 Kbps | 216.83 Kbps | 216.
TRex v 9 17 / 113 Number of TRex active "flows". Could be different than the number of router flows, due to aging issues. Usualy the TRex number of active flows is much lower than that of the router because the router ages flows slower. v Total number of TRex flows opened since startup (including active ones, and ones already closed). v Drop rate. v Rx and latency thread CPU utilization. v Tx_ok on port 0 should equal Rx_ok on port 1, and vice versa. 10 11 12 13 3.5.
TRex 18 / 113 Chapter 4 Basic usage 4.1 DNS basic example The following is a simple example helpful for understanding how TRex works. The example uses the TRex simulator. This simulator can be run on any Cisco Linux including on the TRex itself. TRex simulates clients and servers and generates traffic based on the pcap files provided. Clients/Servers The following is an example YAML-format traffic configuration file (cap2/dns_test.yaml), with explanatory notes. [bash]>cat cap2/dns_test.
TRex 19 / 113 udp_aging : 1 cap_info : - name: cap2/dns.pcap cps : 1.0 ipg : 10000 rtt : 10000 w : 1 v v 5v 6v 3 4 v Range of clients (IPv4 format). v Range of servers (IPv4 format). v pcap file, which includes the DNS cap file that will be used as a template. v Number of connections per second to generate. In the example, 1.0 means 1 connection per secod. v Inter-packet gap (microseconds). 10,000 = 10 msec. v Should be the same as ipg.
TRex 16 17 18 19 20 20 / 113 ,8.020000,8,0x9054760,2,93,0,0,1,0,0,10000008,30000008,1024 ,9.010000,9,0x9055598,1,77,0,1,0,0,0,10000009,30000009,1024 ,9.020000,9,0x9054760,2,93,0,0,1,0,0,10000009,30000009,1024 ,10.010000,a,0x9055598,1,77,0,1,0,0,0,1000000a,3000000a,1024 ,10.020000,a,0x9054760,2,93,0,0,1,0,0,1000000a,3000000a,1024 file stats ================= m_total_bytes m_total_pkt m_total_open_flows m_total_pkt m_total_open_flows m_total_close_flows m_total_bytes v 1 v 2 : : : : : : : 1.
TRex 21 / 113 Table 4.1: (continued) pkt time sec fid 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2.020000 3.010000 3.020000 4.010000 4.020000 5.010000 5.020000 6.010000 6.020000 7.010000 7.020000 8.010000 8.020000 9.010000 9.020000 10.010000 10.020000 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 a a flowpkt-id 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 where: fid:: Flow ID - different IDs for each flow. low-pkt-id Packet ID within the flow. Numbering begins with 1. client_ip Client IP address.
TRex 22 / 113 cps ipg rtt w : : : : v Duration is 1 second. v CPS is 10.0. v IPG is 50 msec. 1 2 3 v v 10.0 50000 50000 1 2 3 Running this produces the following output: [bash]>./bp-sim-32-debug -f cap2/dns_test.yaml -o my.erf -v 3 Table 4.2: Formated results pkt time sec template fid 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0.010000 0.060000 0.210000 0.260000 0.310000 0.360000 0.410000 0.460000 0.510000 0.560000 0.610000 0.660000 0.710000 0.760000 0.810000 0.860000 0.
TRex 23 / 113 - duration : 1.0 generator : distribution : "seq" clients_start : "16.0.0.1" clients_end : "16.0.0.255" servers_start : "48.0.0.1" servers_end : "48.0.0.255" clients_per_gb : 201 min_clients : 101 dual_port_mask : "1.0.0.0" tcp_aging : 0 udp_aging : 0 mac : [0x00,0x00,0x00,0x01,0x00,0x00] 1v cap_ipg : true #cap_ipg_min : 30 #cap_override_ipg : 200 cap_info : - name: cap2/dns.pcap cps : 10.0 ipg : 10000 rtt : 10000 w : 1 v 1 IPG is taken from pcap. Table 4.
TRex 24 / 113 #cap_override_ipg v 1 v 2 4.3 v : 200 2 Sets the minimum IPG (microseconds) which should be override : ( if (pkt_ipg
TRex 25 / 113 Table 4.4: (continued) 4.4 pkt time sec fid 13 14 15 16 17 18 19 20 7.010000 7.030944 8.010000 8.030944 9.010000 9.030944 10.010000 10.030944 7 7 8 8 9 9 a a flowpkt-id 1 2 1 2 1 2 1 2 client_ip 16.0.0.7 16.0.0.7 16.0.0.8 16.0.0.8 16.0.0.9 16.0.0.9 16.0.0.10 16.0.0.10 client_port 1024 1024 1024 1024 1024 1024 1024 1024 server_ip desc 48.0.0.7 48.0.0.7 48.0.0.7 48.0.0.7 48.0.0.7 48.0.0.7 48.0.0.7 48.0.0.7 → ← → ← → ← → ← server_ip desc 48.0.0.1 48.0.0.1 48.0.0.2 48.0.0.2 48.
TRex 26 / 113 Table 4.5: (continued) pkt time sec fid 10 11 12 13 14 15 16 17 18 19 20 5.030944 6.010000 6.030944 7.010000 7.030944 8.010000 8.030944 9.010000 9.030944 10.010000 10.030944 5 6 6 7 7 8 8 9 9 a a flowpkt-id 2 1 2 1 2 1 2 1 2 1 2 client_ip 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 client_port 1028 1029 1029 1030 1030 1031 1031 1032 1032 1033 1033 server_ip desc 48.0.0.5 48.0.0.6 48.0.0.6 48.0.0.7 48.0.0.7 48.0.0.8 48.0.0.
TRex v 1 27 / 113 Two clients will be allocated from the same template.
TRex 28 / 113 Table 4.6: DNS ipg from pcap file 4.6 pkt time sec fid 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0.010000 0.030944 2.010000 2.030944 3.010000 3.030944 4.010000 4.030944 5.010000 5.030944 6.010000 6.030944 7.010000 7.030944 8.010000 8.030944 9.010000 9.030944 10.010000 10.030944 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 a a flowpkt-id 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 client_ip 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.1 16.0.0.2 16.0.0.2 16.0.0.2 16.0.0.2 16.0.0.3 16.0.0.3 16.0.0.
TRex 29 / 113 v, 2v Same CPS for both templates. 1 This creates the following output: Table 4.7: DNS ipg from pcap file pkt time sec template fid 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 0.010000 0.030944 0.093333 0.104362 0.115385 0.115394 0.126471 0.126484 0.137530 0.148609 0.148621 0.148635 0.159663 0.170750 0.170762 0.170774 0.176667 0.181805 0.181815 0.192889 0.
TRex tcp_aging : 0 udp_aging : 0 mac : [0x0,0x0,0x0,0x1,0x0,0x00] cap_ipg : true cap_info : - name: avl/delay_10_http_get_0.pcap cps : 404.52 ipg : 10000 rtt : 10000 w : 1 - name: avl/delay_10_http_post_0.pcap cps : 404.52 ipg : 10000 rtt : 10000 w : 1 - name: avl/delay_10_https_0.pcap cps : 130.8745 ipg : 10000 rtt : 10000 w : 1 - name: avl/delay_10_http_browsing_0.pcap cps : 709.89 ipg : 10000 rtt : 10000 w : 1 - name: avl/delay_10_exchange_0.pcap cps : 253.
TRex 31 / 113 - - - - - - - one_app_server : false plugin_id : 1 name: avl/delay_10_smtp_0.pcap cps : 7.3369 ipg : 10000 rtt : 10000 w : 1 name: avl/delay_10_smtp_1.pcap cps : 7.3369 ipg : 10000 rtt : 10000 w : 1 name: avl/delay_10_smtp_2.pcap cps : 7.3369 ipg : 10000 rtt : 10000 w : 1 name: avl/delay_10_video_call_0.pcap cps : 11.8976 ipg : 10000 rtt : 10000 w : 1 one_app_server : false name: avl/delay_10_sip_video_call_full.pcap cps : 29.
TRex 32 / 113 [bash]>sudo ./t-rex-64 -f cap2/simple_http.yaml -c 4 -m 100 -d 100 Simple HTTP 1Gb/sec with latency for 100 sec [bash]>sudo ./t-rex-64 -f cap2/simple_http.yaml -c 4 -m 100 -d 100 -l 1000 SFR 35Gb/sec traffic [bash]>sudo ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -c 4 -m 35 -d 100 -p SFR 20Gb/sec traffic with latency [bash]>sudo ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -c 4 -m 20 -d 100 -l 1000 SFR ipv6 20Gb/sec traffic with latency [bash]>sudo ./t-rex-64 -f avl/sfr_delay_10_1g_no_bundeling.
TRex 4.10 33 / 113 Mimicking stateless traffic under stateful mode Note TRex supports also true stateless traffic generation. If you are looking for stateless traffic, please visit the following link: TRex Stateless Support [?] With this feature you can "repeat" flows and create stateless, IXIA like streams. After injecting the number of flows defined by limit, TRex repeats the same flows.
TRex 34 / 113 - - - - - cps ipg rtt w limit name: cps ipg rtt w limit name: cps ipg rtt w limit name: cps ipg rtt w limit name: cps ipg rtt w limit name: cps ipg rtt w limit : 90615 : 10000 : 10000 : 1 : 199 cap2/udp_576B.pcap : 64725 : 10000 : 10000 : 1 : 199 cap2/udp_1500B.pcap : 12945 : 10000 : 10000 : 1 : 199 cap2/udp_64B.pcap : 90615 : 10000 : 10000 : 1 : 199 cap2/udp_576B.pcap : 64725 : 10000 : 10000 : 1 : 199 cap2/udp_1500B.
TRex 35 / 113 - - - - ipg rtt w limit name: cps ipg rtt w limit name: cps ipg rtt w limit name: cps ipg rtt w limit name: cps ipg rtt w limit : 10000 : 10000 : 1 : 16666 cap2/udp_1500B.pcap : 12945 : 10000 : 10000 : 1 : 16667 cap2/udp_64B.pcap : 90615 : 10000 : 10000 : 1 : 16667 cap2/udp_576B.pcap : 64725 : 10000 : 10000 : 1 : 16667 cap2/udp_1500B.pcap : 12945 : 10000 : 10000 : 1 : 16667 The following example of a simple simulation includes 3 flows, with CPS=10. $more cap2/imix_example.
TRex 36 / 113 Table 4.8: IMIX example limit=3 pkt time sec template fid 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 0.010000 0.210000 0.310000 0.310000 0.510000 0.610000 0.610000 0.810000 0.910000 0.910000 1.110000 1.210000 1.210000 1.410000 1.510000 1.510000 1.710000 1.810000 1.810000 2.010000 2.110000 2.110000 2.310000 2.410000 2.410000 2.610000 2.710000 2.710000 2.910000 3.010000 3.
TRex 37 / 113 servers_start : "48.0.0.1" servers_end : "48.0.0.255" dual_port_mask : "1.0.0.0" tcp_aging : 0 udp_aging : 0 v 1 v 1 Offset to add per port pair. The reason for the “dual_port_mask” is to make static route configuration per port possible. With this offset, different ports have different prefixes. For example, with four ports, TRex will produce the following ip ranges: port pair-0 (0,1) --> C (16.0.0.1-16.0.0.128 ) <-> S( 48.0.0.1 - 48.0.0.128) port pair-1 (2,3) --> C (17.0.0.129-17.0.
TRex 38 / 113 v Connected to TRex port 0 (client side) v Connected to TRex port 1 (server side) 1 2 3v v 4 Connected to TRex port 2 (client side) Connected to TRex port 3(server side) One server: To support a template with one server, you can add “server_addr” keyword. Each port pair will be get different server IP (According to the “dual_port_mask” offset). - name: cap2/dns.pcap cps : 1.0 ipg : 10000 rtt : 10000 w : 1 server_addr : "48.0.0.1" one_app_server : true wlength : 1 v Server IP.
TRex 4.11.2 39 / 113 How to determine the packet per second(PPS) and Bit per second (BPS) • Let’s look at an example of one flow with 4 packets. • Green circles represent the first packet of each flow. • The client ip pool starts from 16.0.0.1, and the distribution is seq. TotalPPS = ∑nk=0 (CPSk × f low_pktsk ) Concurrent f low = ∑nk=0 CPSk × f low_durationk The above formulas can be used to calculate the PPS.
TRex 40 / 113 generator : distribution : "seq" clients_start : "16.0.0.1" clients_end : "16.0.1.255" servers_start : "48.0.0.1" servers_end : "48.0.20.255" clients_per_gb : 201 min_clients : 101 dual_port_mask : "1.0.0.0" tcp_aging : 0 udp_aging : 0 generator_clients : - name : "c1" distribution : "random" ip_start : "38.0.0.1" ip_end : "38.0.1.255" clients_per_gb : 201 min_clients : 101 dual_port_mask : "1.0.0.
TRex 41 / 113 distribution : "random" clients_start : 26.0.0.1" clients_end : 26.0.1.255" tcp_aging : 0 udp_aging : 0 - name : "c" pools_list : - name:"a" probability: 0.8 - name:"b" probability: 0.2 4.12 Measuring Jitter/Latency To measure jitter/latency using independent flows (SCTP or ICMP), use -l [Hz] where Hz defines the number of packets to send from each port per second. This option measures latency and jitter.
TRex 42 / 113 Chapter 5 Advanced features 5.1 VLAN (dot1q) support To add a VLAN tag to all traffic generated by TRex, add a “vlan” keyword in each port section in the platform config file, as described in the YAML Configuration File [trex_config_yaml_config_file] section. You can specify a different VLAN tag for each port, or use VLAN only on some ports.
TRex 43 / 113 Problem definition: Scenario: TRex with two ports and an SFR traffic profile. Without VLAN/sub interfaces, all client emulated traffic is sent on port 0, and all server emulated traffic (example: HTTP response) on port 1. TRex port 0 ( client) <-> [ DUT ] <-> TRex port 1 ( server) Without VLAN support, the traffic is asymmetric. 10% of the traffic is sent from port 0 (client side), 90% from port 1 (server). Port 1 is the bottlneck (10Gb/s limit).
TRex 44 / 113 v arp 11.77.11.12 0000.0001.0000 ARPA arp 22.77.11.12 0000.0001.0000 ARPA 6 route-map vlan_100_p1_to_p2 permit 10 set ip next-hop 22.77.11.12 ! route-map vlan_100_p2_to_p1 permit 10 set ip next-hop 11.77.11.12 ! 7 v route-map vlan_200_p1_to_p2 permit 10 set ip next-hop 22.88.11.12 ! route-map vlan_200_p2_to_p1 permit 10 set ip next-hop 11.88.11.12 ! v Main interface must not have IP address.
TRex 5.4 45 / 113 IPv6 support Support for IPv6 includes: 1. Support for pcap files containing IPv6 packets. 2. Ability to generate IPv6 traffic from pcap files containing IPv4 packets. The --ipv6 command line option enables this feature. The keywords src_ipv6 and dst_ipv6 specify the most significant 96 bits of the IPv6 address.
TRex 46 / 113 v Enable IPv6 v Add pbr v Enable IPv6 routing v MAC address setting. Should be TRex MAC. v PBR configuraion 1 2 3 4 5 5.5 Client clustering configuration TRex supports testing complex topologies with more than one DUT, using a feature called "client clustering". This feature allows specifying the distribution of clients that TRex emulates.
TRex 47 / 113 Topology example There are two clusters of DUTs. Using the configuration file, you can partition TRex emulated clients into groups, and define how they will be spread between the DUT clusters. Group configuration includes: • IP start range.
TRex 48 / 113 • IP end range. • Initiator side configuration: Parameters affecting packets sent from client side. • Responder side configuration: Parameters affecting packets sent from server side. Note It is important to understand that this is complimentary to the client generator configured per profile. It only defines how the clients will be spread between clusters. In the following example, a profile defines a client generator. [bash]>cat cap2/dns.yaml - duration : 10.
TRex 49 / 113 initiator : vlan : 100 dst_mac : "00:00:00:01:00:00" responder : vlan : 200 dst_mac : "00:00:00:02:00:00" count - : 4 ip_start : 16.0.0.205 ip_end : 16.0.0.255 initiator : vlan : 101 dst_mac : "00:00:01:00:00:00" responder: vlan : 201 dst_mac : "00:00:02:00:00:00" count : 3 The above configuration divides the generator range of 255 clients to two clusters.
TRex 50 / 113 responder : vlan : 200 next_hop : 2.2.2.1 src_ip : 2.2.2.100 count : 4 In this case, TRex attempts to resolve the following addresses using ARP: 1.1.1.1, 1.1.1.2, 1.1.1.3, 1.1.1.4 (and the range 2.2.2.1-2.2.2.4) If not all IPs are resolved, TRex exits with an error message. src_ip is used to send gratuitous ARP, and for filling relevant fields in ARP request. If no src_ip is given, TRex looks for the source IP in the relevant port section in the platform configuration file (/etc/trex_cfg.
TRex 5.5.1 51 / 113 Latency with Cluster mode Latency streams client IP is taken from the first IP in the default client pool. Each dual port will have one Client IP. In case of cluster configuration this is a limitation as you can have a topology with many paths. Traffic profile - duration : 10.0 generator : distribution : "seq" clients_start : "16.0.0.1" clients_end : "16.0.0.255" servers_start : "48.0.0.1" servers_end : "48.0.0.255" dual_port_mask : "1.0.0.0" cap_info : - name: cap2/dns.pcap cps : 1.
TRex 52 / 113 Cluster example For this topology the following traffic and cluster configuration file were created Traffic profile - duration : 10.0 generator : distribution : "seq" clients_start : "12.1.1.1" clients_end : "12.1.1.1" servers_start : "13.1.1.1" servers_end : "13.1.1.1" dual_port_mask : "0.1.0.0" generator_clients : - name : "c1" distribution : "seq" ip_start : "12.1.1.2" ip_end : "12.1.1.255" - name : "c2" distribution : "seq" ip_start : "12.1.2.1" ip_end : "12.1.2.
TRex 53 / 113 client_pool : "c2" server_pool : "s2" cps : 1 ipg : 20000 rtt : 20000 w : 1 - name: file10k.pcap client_pool : "c1" server_pool : "s1" cps : 1 ipg : 20000 rtt: 20000 w : 1 v Latency flow will be IP 12.1.1.1<→13.1.1.1/vlan=4050/next-hop=11.10.0.1 v First clients pool v Second clients pool 1 2 3 Cluster profile lan: true groups: - - v 1 ip_start : 12.1.1.1 ip_end : 12.1.1.255 initiator : vlan : next_hop : src_ip : responder : vlan : next_hop : src_ip : count : 1 ip_start : 12.1.
TRex 54 / 113 Cluster example Note Latency stream will check only 12.1.1.1/4050 path (one DUT intertface) there is no way to check latency on all the ports in current version Note DUT should have a static route to move packets from client to server and vice versa, as traffic is not in the same subnet of the ports An example of one flow generation 1. next hop resolotion. TRex resolve all the next hop option e.g. 11.10.0.1/4050 11.11.0.
TRex 55 / 113 2. Choose template by CPS, 50% probability for each. take template #1 3. SRC_IP=12.1.1.2, DEST_IP=13.1.1.2 4. Allocate src_port for 12.1.1.2 =⇒src_port=1025 for the first flow of client=12.1.1.2 5. Associate the next-hop from cluster pool. In this case 12.1.1.2 has the following information 5.1 client side : VLAN=4050 and MAC of 11.10.0.1 (Initiator) 5.2 server side : VLAN=4054 and MAC of 10.10.0.1 (Responder) to run this using FirePower and NAT learning and checksum offload [bash]>sudo .
TRex 56 / 113 - name : "c1" distribution : "seq" ip_start : "12.1.1.2" ip_end : "12.1.1.255" - name : "c2" distribution : "seq" ip_start : "12.1.2.1" ip_end : "12.1.2.255" generator_servers : - name : "s1" distribution : "seq" ip_start : "13.1.1.2" ip_end : "13.1.1.255" - name : "s2" distribution : "seq" ip_start : "13.1.2.1" ip_end : "13.1.2.255" cap_ipg : true cap_info : - name: file10k.pcap client_pool : "c2" server_pool : "s2" cps : 1 ipg : 20000 rtt : 20000 w : 1 - name: file10k.
TRex 57 / 113 count - - v 1 5.6 vlan : 4055 next_hop : 10.10.0.2 src_ip : 10.10.0.11 : 1 ip_start : 12.2.1.1 ip_end : 12.2.1.255 initiator : vlan : next_hop : src_ip : responder : vlan : next_hop : src_ip : count : 1 ip_start : 12.2.2.1 ip_end : 12.2.2.255 initiator : vlan : next_hop : src_ip : responder : vlan : next_hop : src_ip : count : 1 v 1 4060 11.10.0.3 11.10.0.11 4064 10.10.0.3 10.10.0.11 4061 11.10.0.4 11.10.0.11 4065 10.10.0.4 10.10.0.
TRex 5.6.1 58 / 113 Examples simple HTTP traffic [bash]>sudo ./t-rex-64 -f cap2/http_simple.yaml -c 4 1 -l 1000 -d 100000 -m 30 --learn-mode ←- SFR traffic without bundling/ALG support [bash]>sudo ./t-rex-64 -f avl/sfr_delay_10_1g_no_bundling.yaml -c 4 10 --learn-mode 2 -l 1000 -d 100000 -m ←- NAT terminal counters: -Global stats enabled Cpu Utilization : 0.6 % 33.4 Gb/core Platform_factor : 1.0 Total-Tx : 3.77 Gbps NAT time out : Total-Rx : 3.77 Gbps NAT aged flow id: Total-PPS : 505.
TRex 59 / 113 mtu 4000 ip address 11.11.11.11 255.255.255.0 ip policy route-map p1_to_p2 ip nat outside load-interval 30 ip v 3 nat pool my 200.0.0.0 200.0.0.255 netmask 255.255.255.0 ip nat inside source list 7 pool my overload access-list 7 permit 16.0.0.0 0.0.0.255 ip nat inside source list 8 pool my overload access-list 8 permit 17.0.0.0 0.0.0.
TRex 60 / 113 This feature ensures that: • Packets get out of DUT in order (from each flow perspective). • There are no packet drops (no need to wait for the end of the test). Without this flag, you must wait for the end of the test in order to identify packet drops, because there is always a difference between TX and Rx, due to RTT. Full example [bash]>sudo ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30 check 128 Cpu Utilization : 0.
TRex 61 / 113 high_cnt : 2 max_d_time : 1041 usec sliding_average : 1 usec precent : 100.
TRex 62 / 113 Chapter 6 Reference 6.1 6.1.1 Traffic YAML (parameter of -f option) Global Traffic YAML section 1v - duration : 10.0 2v generator : distribution : "seq" 3v clients_start : "16.0.0.1" clients_end : "16.0.0.255" servers_start : "48.0.0.1" servers_end : "48.0.0.255" 4v clients_per_gb : 201 5v min_clients : 101 dual_port_mask : "1.0.0.
TRex v 6 v 7 6.1.2 63 / 113 time in sec to linger the deallocation of TCP flows (in particular return the src_port to the pool). Good for cases when there is a very high socket utilization (>50%) and there is a need to verify that socket source port are not wrapped and reuse. Default value is zero. Better to keep it like that from performance point of view. High value could create performance penalty same as #11 for UDP flows Timer Wheel section configuration (from v2.
TRex 6.2.1 64 / 113 Basic Configuration - port_limit : 2 #mandatory 1v version : 2 #mandatory 2v interfaces : ["03:00.0", "03:00.1"] #mandatory 3v #enable_zmq_pub : true #optional 4v #zmq_pub_port : 4500 #optional 5v #zmq_rpc_port : 4501 #optional 6v #prefix : setup1 #optional 7v #limit_memory : 1024 #optional 8v c : 4 #optional 9v port_bandwidth_gb : 10 #optional 10v port_info : # set eh mac addr mandatory - default_gw : 1.1.1.
TRex 65 / 113 v, 17v Old MAC address format. New format is supported since version v2.09. 16 v 6 Stateless ZMQ RPC port number. Default value is good. If running two TRex instances on the same machine, each should be given distinct number. Otherwise, can remove this line. Note If you use version earlier than 2.10, or choose to omit the “ip” and have mac based configuration, be aware that TRex will not send any gratitues ARP and will not answer ARP requests.
TRex 66 / 113 traffic_mbuf_64 : traffic_mbuf_128 : traffic_mbuf_256 : traffic_mbuf_512 : traffic_mbuf_1024 : traffic_mbuf_2048 : dp_flows : 1048576 global_flows : 10240 16380 8190 8190 8190 8190 4096 v 3 v v 4 5 v Memory section header v Numbers of memory buffers allocated for packets in transit, per port pair. Numbers are specified per packet size. 1 2 v 3 v 4 v 5 6.2.3 Numbers of memory buffers allocated for holding the part of the packet which is remained unchanged per template.
TRex CPU utilization was very high ~100%, with c=2 and c=4 the results were same.
TRex 68 / 113 + We added configuration to the /etc/trex_cfg.
TRex 69 / 113 - socket threads : 1 : [9, 10, 11, 12, 13, 14, 15] This gave best results: with ~98 Gb/s TX BW and c=7, CPU utilization became ~21%! (40% with c=4) 6.2.4 Timer Wheeel section configuration The memory section is optional. It is used when there is a need to tune the amount of memory used by TRex packet manager. Default values (from the TRex source code), are usually good for most users. Unless you have some unusual needs, you can eliminate this section. 6.2.
TRex 70 / 113 -d Duration of the test in seconds. -e Same as -p, but change the src/dst IP according to the port. Using this, you will get all the packets of the same flow from the same port, and with the same src/dst IP. It will not work good with NBAR as it expects all clients ip to be sent from same direction. -f Specify traffic YAML configuration file to use. Mandatory option for stateful mode. --hops Provide number of hops in the setup (default is one hop).
TRex 71 / 113 --no-key Daemon mode, don’t get input from keyboard. --no-watchdog Disable watchdog. -p Send all packets of the same flow from the same direction. For each flow, TRex will randomly choose between client port and server port, and send all the packets from this port. src/dst IPs keep their values as if packets are sent from two ports. Meaning, we get on the same port packets from client to server, and from server to client.
TRex 72 / 113 Chapter 7 Appendix 7.1 Simulator The TRex simulator is a linux application (no DPDK needed) that can run on any Linux (it can also run on TRex machine itself). you can create output pcap file from input of traffic YAML. 7.1.1 Simulator [bash]>./bp-sim-64-debug -f avl/sfr_delay_10_1g.yaml -v 1 -- loading cap file avl/delay_10_http_get_0.pcap -- loading cap file avl/delay_10_http_post_0.pcap -- loading cap file avl/delay_10_https_0.pcap -- loading cap file avl/delay_10_http_browsing_0.
TRex 73 / 113 06, avl/delay_10_mail_pop_1.pcap , 0.48 , 1 , 543 , 07, avl/delay_10_mail_pop_2.pcap , 0.07 , 1 , 143 , 08, avl/delay_10_oracle_0.pcap 35.62 , 4.45 , 544 , 23954 , 09, avl/delay_10_rtp_160k_full.pcap , 3.42 , 170 , 3759 , 10, avl/delay_10_rtp_250k_full.pcap , 3.81 , 122 , 4101 , 11, avl/delay_10_smtp_0.pcap , 0.04 , 1 , 161 , 12, avl/delay_10_smtp_1.pcap , 0.13 , 2 , 257 , 13, avl/delay_10_smtp_2.pcap , 0.71 , 2 , 807 , 14, avl/delay_10_video_call_0.pcap 241.05 , 30.
TRex 74 / 113 [bash]>sudo ./dpdk_nic_bind.py --status # show the ports Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 ’Device 1583’ drv=igb_uio unused= 0000:02:00.1 ’Device 1583’ drv=igb_uio unused= 0000:87:00.0 ’Device 1583’ drv=igb_uio unused= 0000:87:00.1 ’Device 1583’ drv=igb_uio unused= [bash]>sudo dpdk_nic_bind.py -u 02:00.0 # 1v # 2v # 3v 02:00.1 [bash]>sudo dpdk_nic_bind.py -b i40e 02:00.0 02:00.
TRex 7.3 75 / 113 TRex with ASA 5585 When running TRex aginst ASA 5585, you have to notice following things: • ASA can’t forward ipv4 options, so there is a need to use --learn-mode 1 (or 3) in case of NAT. In this mode, bidirectional UDP flows are not supported. --learn-mode 1 support TCP sequence number randomization in both sides of the connection (client to server and server client).
TRex no arp permit-nonconnected route management 0.0.0.0 0.0.0.0 10.56.216.1 1 route inside 16.0.0.0 255.0.0.0 15.0.0.2 1 route outside 48.0.0.0 255.0.0.0 40.0.0.
TRex 77 / 113 inspect ip-options ! service-policy global_policy global service-policy icmp_policy interface outside prompt hostname context ! jumbo-frame reservation ! no call-home reporting anonymous : end ciscoasa# 7.3.2 TRex commands example Using these commands the configuration is: 1. NAT learn mode (TCP-ACK) 2. Delay of 1 second at start up (-k 1). It was added because ASA drops the first packets. 3. Latency is configured to ICMP reply mode (--l-pkt-mode 2). Simple HTTP: [bash]>sudo .
TRex 78 / 113 Expected-PPS Expected-CPS Expected-BPS : : : 539.85 Kpps 8.29 Kcps 2.90 Gbps Active-flows Open-flows drop-rate current time test duration : 7860 Clients : : 3481234 Servers : : 0.00 bps # 4v : 425.1 sec : 574.9 sec 255 5375 Socket-util : 0.0489 % Socket : 7860 Socket/Clients : 30.8 -Latency stats enabled Cpu Utilization : 0.
TRex 7.5 79 / 113 Configure Linux host as network emulator There are lots of Linux tutorials on the web, so this will not be full tutorial, only highlighting some key points. Commands were checked on Ubuntu system. For this example: 1. TRex Client side network is 16.0.0.x 2. TRex Server side network is 48.0.0.x 3. Linux Client side network eth0 is configured with IPv4 as 172.168.0.1 4. Linux Server side network eth1 is configured with IPv4 as 10.0.0.1 TRex-0 (16.0.0.1->48.0.0.1 ) <--> ( 172.168.0.
TRex 7.5.3 80 / 113 Add static ARP entries [cli]>sudo arp -s 10.0.0.100 [cli]>sudo arp -s 172.168.0.100
TRex 81 / 113 The above diagram is an example of one server with two NICS. TRex VMs can be allocated on one NIC while the DUTs can be allocated on another. Following are some links we used and lessons we learned while putting up an environment for testing TRex with VF interfaces (using SR-IOV). This is by no means a full toturial of VF usage, and different Linux distributions might need slightly different handling. 7.6.
TRex 82 / 113 Kernel params: # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-514.el7.x86_64 root=/dev/mapper/cl-root ro crashkernel=auto rd. ←lvm.lv=cl/root rd.lvm.lv=cl/swap nomodeset rhgb quiet intel_iommu=on iommu=pt isolcpus ←=2-23 nohz_full=2-23 selinux=0 audit=0 82599 and Mellanox VFs: # cat /etc/rc.local ############## # 82599 ############## ifconfig enp2s0f0 up ifconfig enp2s0f1 up echo 9500 > /sys/bus/pci/devices/0000:02:00.0/net/*/mtu echo 9500 > /sys/bus/pci/devices/0000:02:00.
TRex 83 / 113 [bash]>sudo ./dpdk_nic_bind.py -s Network devices using DPDK-compatible driver ============================================ 0000:03:10.0 ’82599 Ethernet Controller Virtual Function’ drv=igb_uio unused= 0000:03:10.1 ’82599 Ethernet Controller Virtual Function’ drv=igb_uio unused= Network devices using kernel driver =================================== 0000:01:00.0 ’I350 Gigabit Network Connection’ if=eth0 drv=igb unused=igb_uio *Active* 0000:03:00.
TRex 84 / 113 Cpu Utilization : 23.4 % if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter | , , , , average , max , (usec) ------------------------------------------------------------------------------0 | 5233, 5233, 0, 0, 19 , 580, 5 | 37 37 37 4 1 | 5233, 5233, 0, 0, 22 , 577, 5 | 38 40 39 3 TRex stateless performance [bash]>sudo .
TRex 85 / 113 oerrors ierrors 7.6.5 | | 0 | 0 | 0 | 15,539 | 0 15,539 Performance See the performance tests we did here 7.7 Napatech support Napatech SmartNIC family support 1/10/25/40/50/100 Gb/s Ethernet speeds on various cards. The NT200A01 card can potentially support all speeds within the one card. The Napatech SmartNICs operate using a bifurcated driver, the main driver resides in userspace and DPDK run on top of that driver.
TRex Installing documentation 86 / 113 [Done] ###################################################################### # Installation completed successfully # # See /tmp/nt_tools_3gd.log for a full install log # ###################################################################### [ Extract and build driver ] nt_driver_3gd_linux_3.6.8.20-6cb89.run readme_driver_3gd_linux_3.6.8.20-6cb89.txt Verifying archive integrity... All good. Uncompressing Napatech A/S driver package v 3.6.8.20-6cb89 ........... ....
TRex 87 / 113 Installing driver source Compiling the driver Installing driver Compiling the netdev driver Installing netdev driver Installing libraries Installing header files Installing binaries Installing config files Installing examples Installing documentation Checking dynamic linker bindings: Regenerate dynamic linker bindings Installing symbolic links [Done] [Done] [Done] [Done] [Done] [Done] [Done] [Done] [Done] [Done] [Done] [Done] [Done] #########################################################
TRex 88 / 113 Loading nt3gd driver Creating driver device file Loading nt3gd_netdev driver Creating driver device file Starting NTService (this may take a while) [bash]>/opt/napatech3/bin/ntstop.sh Stopping NTService (this may take a while) NTService stopped [bash]> 7.7.1.4 [Done] [Done] [Done] [Done] [Done] [Done] Change the amount of Rx/Tx queues The /opt/napatech3/config/ntservice.ini contain the following settings: ... HostBuffersRx = HostBuffersTx = ... ---They need to be [source,bash] ---...
TRex 89 / 113 [bash]>./b Waf: Entering directory ‘/usr/builds/trex-core.github.rsync/linux_dpdk/build_dpdk’ Info: Using external libverbs. update version files ... [1206/1210] Compiling version.c [1207/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-debug-o [1208/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-debug [1209/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-o [1210/1210] Linking build_dpdk/linux_dpdk/_t-rex-64 Waf: Leaving directory ‘/usr/builds/trex-core.github.
TRex 7.7.1.8 90 / 113 Config file example ### Config file generated by dpdk_setup_ports.py ### - port_limit: 2 version: 2 interfaces: [’82:00.0/0’, ’82:00.0/1’] port_info: - ip: 1.1.1.1 default_gw: 2.2.2.2 - ip: 2.2.2.2 default_gw: 1.1.1.1 platform: master_thread_id: 0 latency_thread_id: 1 dual_if: - socket: 1 threads: [8,9,10,11,12,13,14] 7.8 Mellanox ConnectX-4/5 support Mellanox ConnectX-4/5 adapter family supports 100/56/40/25/10 Gb/s Ethernet speeds.
TRex 91 / 113 Important Make sure you have an internet connection without firewalls for HTTPS/HTTP - required by yum/apt-get Verify md5 [bash]>md5sum md5sum MLNX_OFED_LINUX-4.0.0.0.0-rhel7.2-x86_64.tgz 58b9fb369d7c62cedbc855661a89a9fd MLNX_OFED_LINUX-4.0-xx.0.0.0-rhel7.2-x86_64.tgz Open the tar [bash]>tar -xvzf MLNX_OFED_LINUX-4.0-x.0.0.0-rhel7.2-x86_64.tgz [bash]>cd MLNX_OFED_LINUX-4.0-x.0.0.0-rhel7.2-x86_64 Run Install script [bash]>sudo ./mlnxofedinstall Log: /tmp/ofed.build.
TRex 92 / 113 infiniband-diags infiniband-diags-compat mft kernel-mft-dkms libibcm1 libibcm-dev perftest ibutils2 libibdm1 ibutils cc-mgr ar-mgr dump-pr ibsim ibsim-doc knem-dkms mxm fca sharp hcoll openmpi mpitests knem rds-tools libdapl2 dapl2-utils libdapl-dev srptools mlnx-ethtool libsdp1 libsdp-dev sdpnetstat This program will install the MLNX_OFED_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.
TRex Installing mlnx-nfsrdma-dkms-3.4... Installing libibverbs1-1.2.1mlnx1... Installing ibverbs-utils-1.2.1mlnx1... Installing libibverbs-dev-1.2.1mlnx1... Installing libibverbs1-dbg-1.2.1mlnx1... Installing libmlx4-1-1.2.1mlnx1... Installing libmlx4-dev-1.2.1mlnx1... Installing libmlx4-1-dbg-1.2.1mlnx1... Installing libmlx5-1-1.2.1mlnx1... Installing libmlx5-dev-1.2.1mlnx1... Installing libmlx5-1-dbg-1.2.1mlnx1... Installing libibumad-1.3.10.2.MLNX20150406.966500d... Installing libibumad-static-1.3.10.2.
TRex 94 / 113 Preparing to unpack .../mlnx-fw-updater_3.4-1.0.0.0_amd64.deb ... Unpacking mlnx-fw-updater (3.4-1.0.0.0) ... Setting up mlnx-fw-updater (3.4-1.0.0.0) ... Added RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf Attempting to perform Firmware update... Querying Mellanox devices firmware ...
TRex 95 / 113 board_id: phys_port_cnt: Device ports: port: MT_2150110033 1 1 state: max_mtu: active_mtu: sm_lid: port_lid: port_lmc: link_layer: hca_id: mlx5_0 transport: fw_ver: node_guid: sys_image_guid: vendor_id: vendor_part_id: hw_ver: board_id: phys_port_cnt: Device ports: port: 1 state: max_mtu: active_mtu: sm_lid: port_lid: port_lmc: link_layer: PORT_DOWN (1) 4096 (5) 1024 (3) 0 0 0x00 Ethernet InfiniBand (0) 12.17.
TRex 96 / 113 +----+------+---------++-------------------------------+-----------+| 9 | 1 | 87:00.0 || MT27700 Family [ConnectX-4] | mlx5_core | # 1v +----+------+---------++-------------------------------+-----------+| 10 | 1 | 87:00.1 || MT27700 Family [ConnectX-4] | mlx5_core | # 2v +----+------+---------++--------------------------------------------- v ConnectX-4 port 0 v ConnectX-4 port 1 1 2 Config file example ### Config file generated by dpdk_setup_ports.
TRex 7.8.7 97 / 113 Performance Cycles/Packet ConnectX-4 vs Intel XL710 For TRex version v2.11, these are the comparison results between XL710 and ConnectX-4 for various scenarios.
TRex 98 / 113 1. MLX5 can reach 50MPPS while XL710 is limited to 35MPPS. (With potential future fix it will be 65MPPS) 2. with IMIX you should reach ~90Gb/sec (not 100Gb/sec) with one port (Total bandwidth for both ports is 100Gb/Sec) 3. For Stateless/Stateful 256B profiles, ConnectX-4 uses half of the CPU cycles per packet. ConnectX-4 probably can handle in a better way chained mbufs (scatter gather). 4. In the average stateful scenario, ConnectX-4 is the same as XL710. 5.
TRex [bash]>sudo lspci | grep Mellanox 87:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] 87:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] 99 / 113 # 1v [bash]>sudo lspci -vv -s 87:00.0 # 2v 87:00.
TRex 100 / 113 ECRC- UnsupReq- ACSViolUEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ←ECRC- UnsupReq- ACSViolUESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ←ECRC- UnsupReq- ACSViolCESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEnCapabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Ne
TRex 101 / 113 Address: fee00138 Data: 0000 Masking: 00000002 Pending: 00000000 Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoopMaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPendLnkCap: Port #7, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <512ns, ←L1 <16us ClockPM- Surp
TRex 102 / 113 v Check parent PCI speed to be find where is the problem-→ in this case it is 0000:80:03.0 v Look for the speed 5 6 7.8.9 Limitations/Issues • CX5 require the latest OFED (4.1>) 7.8.10 Build with native OFED In some case there is a need to build the dpdk-mlx5 with different OFED (not just 4.0 maybe newer) to do so run this on native machine [bash]> .
TRex 103 / 113 v change to new version v change to new version 1 2 7.9 Cisco VIC support • Supported from TRex version v2.12. • Since version 2.21, all VIC card types supported by DPDK are supported by TRex, using “--software” command line argument. Notice that if using “--software”, no HW assist is used, causing supported packet rate to be much lower. Since we do not have all cards in our lab, we could not test all of them. Will be glad for feedback on this (good or bad).
TRex 104 / 113
TRex 105 / 113 In case it is not configured correctly, this error will be seen VIC error in case of wrong RQ/WQ Starting TRex v2.15 please wait ...
TRex 106 / 113 Server: CPU: RAM: NICs: QSFP: OS: Switch: TRex: 7.10.2 UCSC-C240-M4SX 2 x Intel® Xeon® CPU E5-2667 v3 @ 3.20GHz 65536 @ 2133 MHz 2 x Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) Cisco QSFP-H40G-AOC1M Fedora 18 Cisco Nexus 3172 Chassis, System version: 6.0(2)U5(2). v2.13/v2.12 using 7 cores per dual interface. Traffic profile cap2/cur_flow_single.yaml - duration : 0.1 generator : distribution : "seq" clients_start : "16.0.0.1" clients_end : "16.0.0.
TRex 107 / 113 threads: [8,9,10,11,12,13,14] memory : dp_flows : 1048576 v 1 v 1 add memory section with more flows 7.10.4 Traffic command command [bash]>sudo ./t-rex-64 -f cap2/cur_flow_single.yaml -m 30000 -c 7 -d 40 -l 1000 --active- ←flows 5000000 -p --cfg cfg/trex_08_5mflows.yaml The number of active flows can be change using --active-flows CLI. in this example it is set to 5M flows 7.10.
TRex 108 / 113 max_flows=8000000; min_flows=100; active_flow=min_flows; num_point=10 factor=math.exp(math.log(max_flows/min_flows,math.e)/num_point); for i in range(num_point+1): print("=====================",i,math.floor(active_flow)) minimal_stateful_test(args.server,test_file,math.floor(active_flow)) active_flow=active_flow*factor test_file.
TRex 109 / 113 MPPS/core • TW0 - v2.14 default configuration • PQ - v2.12 default configuration • To run the same script on v2.12 (that does not support active_flows directive) a patch was introduced. Observation • TW works better (up to 250%) in case of 25-100K flows • TW scale better with active-flows 7.10.7 Tunning let’s add another modes called TW1, in this mode the scheduler is tune to have more buckets (more memory) TW1 cap2/cur_flow_single_tw_8.
TRex 110 / 113 - duration : 0.1 generator : distribution : "seq" clients_start : "16.0.0.1" clients_end : "16.0.0.255" servers_start : "48.0.0.1" servers_end : "48.0.255.255" clients_per_gb : 201 min_clients : 101 dual_port_mask : "1.0.0.0" tw : 1v buckets : 16384 2v levels : 2 bucket_time_usec : 20.0 cap_info : - name: cap2/udp_10_pkts.
TRex • TW0 - v2.14 default configuration • TW1 - v2.14 more buckets 16K • TW2 - v2.
TRex MPPS/core Factor relative to v2.
TRex Extrapolation Total GbE per UCS with average packet size of 600B Observation: • TW2 (two flows) almost does not have a performance impact • TW1 (more buckets) improve the performance up to a point • TW is general is better than PQ 113 / 113
