**************************************** README **************************************** Chelsio Unified Wire for Linux Version : 3.0.0.1 (Alpha) Date : 10/14/2016 Overview ================================================================================ Chelsio Unified Wire software for Linux is an easy to use utility developed to provide installation of 64-bit Linux based drivers and tools for Chelsio's Unified Wire Adapters. The Chelsio Unified Wire Package provides an interactive installer to install various drivers and utilities. It consists of the following components: a. Network (NIC/TOE) b. iWARP (RDMA) c. iSCSI PDU Offload Target d. iSCSI PDU Offload Initiator e. Offload Bonding driver f. Offload Multi-Adapter Failover (MAFO) g. Offload IPv6 driver h. Classification and Filtering feature i. Traffic Management feature (TM) j. Utility Tools (cop,cxgbtool,t4_perftune,benchmark tools,sniffer & tracer) k. libs (iWARP and WD-UDP libraries) ================================================================================ CONTENTS ================================================================================ - 1. Requirements - 2. Supported Operating Systems - 3. Supported Cards - 4. How To Use - 5. Support Documentation - 6. Customer Support 1. Requirements ================================================================================ 1.1. Network (NIC/TOE) ====================== - Chelsio adapter and system with supported platforms mentioned in section 2. 1.2. iWARP (RDMA) ================= - Chelsio adapter and system with supported platforms mentioned in section 2. - If you are planning to upgrade OFED on one member of the cluster, the upgrade needs to be installed on all the members. - If you want to install OFED with NFS-RDMA support, please see "Setting up NFS-RDMA" in iWARP (RDMA) chapter in the User's Guide. 1.3. iSCSI PDU Offload Target ============================= - Chelsio adapter and system with supported platforms mentioned in section 2. - iSCSI PDU offload target driver (chiscsi_t4.ko) requires NIC(cxgb4), TOE(t4_tom & toecore) and iSCSI non-offload(chiscsi_base.ko) modules to work. Whereas the iSCSI non-offload target driver requires only NIC module. 1.4. iSCSI PDU Offload Initiator ================================ - Chelsio adapter and system with supported platforms mentioned in section 2. - The iSCSI PDU Offload Initiator driver (cxgb4i) runs on top of NIC module(cxgb4) and open-iscsi-2.0-872/873 only, on a T5/T4 card. - If you're planning to install iSCSI PDU Offload Initiator, please install openssl-devel package. 1.5. Offload Bonding driver =========================== - Chelsio adapter and system with supported platforms mentioned in section 2. 1.6. Offload Multi-Adapter Failover ==================================== - Chelsio adapter and system with supported platforms mentioned in section 2. 1.7. Offload IPv6 driver ========================= - Chelsio adapter and system with supported platforms mentioned in section 2. 1.8. Classification and Filtering ================================== - Chelsio adapter and system with supported platforms mentioned in section 2. 1.9. Traffic Management ======================== - Chelsio adapter and system with supported platforms mentioned in section 2. 2. Supported Operating Systems ================================================================================ The Chelsio Unified Wire software has been developed to run on 64-bit Linux based platforms. Following is the list of Drivers/Software and supported Linux distributions. 2.1. POWERPC64 Architecture =========================== |########################|#####################################################| | Linux Distribution | Driver/Software | |########################|#####################################################| |RHEL 7.1 LE (POWER8) |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator,IPv6, | |3.10.0-229.ael7b.ppc64le|Bonding,MAFO,Filtering,TM | |------------------------|-----------------------------------------------------| NOTE: Other kernel versions have not been tested and are not guaranteed to work. 3. Supported Hardware ================================================================================ 3.1. Chelsio Drivers/Software and supported adapters ===================================================== |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T62100-LP-CR |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator, | | |Bonding,MAFO,IPv6,Filtering,TM | |------------------------|-----------------------------------------------------| |T6225-CR |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator, | | |Bonding,MAFO,IPv6,Filtering,TM | |------------------------|-----------------------------------------------------| |T580-OCP-SO |NIC,Filtering | |------------------------|-----------------------------------------------------| |T520-OCP-SO |NIC,Filtering | |------------------------|-----------------------------------------------------| |T520-BT |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator, | | |Bonding,MAFO,IPv6,Filtering,TM | |------------------------|-----------------------------------------------------| |T580-CR |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator, | | |Bonding,MAFO,IPv6,Filtering,TM | |------------------------|-----------------------------------------------------| |T580-SO-CR |NIC,Filtering | |------------------------|-----------------------------------------------------| |T580-LP-CR |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator, | | |Bonding,MAFO,IPv6,Filtering,TM | |------------------------|-----------------------------------------------------| |T520-LL-CR |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator, | | |Bonding,MAFO,IPv6,Filtering,TM | |------------------------|-----------------------------------------------------| |T520-SO-CR |NIC | |------------------------|-----------------------------------------------------| |T520-CR |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator, | | |Bonding,MAFO,IPv6,Filtering,TM | |------------------------|-----------------------------------------------------| |T540-CR |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator, | | |Bonding,MAFO,IPv6,Filtering,TM | |------------------------|-----------------------------------------------------| 4. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.1. Chelsio Unified Wire ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.1.1. Installing Chelsio Unified Wire ====================================== The following table describes the various "configuration tuning options" available during installation and drivers/software installed with each option by default: |#######################################|######################################| | Configuration Tuning Option | Driver/Software installed | |#######################################|######################################| |Unified Wire |NIC/TOE,vNIC,iWARP,WD-UDP, | | |iSCSI Target,iSCSI Initiator,IPv6, | | |Sniffer & Tracer,FCoE Initiator,DCB, | | |Lustre,Bonding,MAFO, | | |UM(Agent,Client,WebGUI),Filtering,TM | |---------------------------------------|--------------------------------------| Note: Only "Unified Wire" supported with current reelase. Follow the steps mentioned below for installation using CLI. a. From source -------------- i. Download the tarball ChelsioUwire-x.xx.x.x.tar.gz ii. Untar the tarball [root@host~]# tar zxvfm ChelsioUwire-x.xx.x.x.tar.gz iii. Change your current working directory to Chelsio Unified Wire package directory. Build the source: [root@host~]# make iv. Install the drivers, tools and libraries: [root@host~]# make install v. The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= install NOTE: To view the different configuration tuning options, view help by typing [root@host~]# make help vi. Reboot your machine for changes to take effect. Installation on updated kernels ---------------------------------------- If the kernel version on your Linux distribution is updated, please execute the following command to install the Unified Wire package: [root@host~]# make UNAME_R= Where kernel_version is the nearest supported kernel version. For example, if you want to install the package on a RHEL 6 distribution updated to 2.6.32-431.20.3.el6 kernel, run the following commands: [root@host~]# make UNAME_R=2.6.32-431.el6 [root@host~]# make UNAME_R=2.6.32-431.el6 install To view the list of the supported kernel versions, run the following command: [root@host~]# make list_kernels Reboot your machine for changes to take effect. iWARP driver installation on Cluster nodes ------------------------------------------ IMPORTANT: Please make sure that you have enabled password less authentication with ssh on the peer nodes for this feature to work. Chelsio's Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a single command. Follow the procedure mentioned below: i. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed. ii. Now, execute the following command: [root@host~]# ./install.py -C -m iii. Select the required T5/T4 configuration tuning option. The tuning options may vary depending on the Linux distribution. iv. Select the required Cluster Configuration. v. If you already have the required version of OFED software installed, you can skip this step. To install OFED-3.18-1 choose the "Install-OFED option". To skip this step, "select Skip-OFED". vi. The selected components will now be installed. The above command will install iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes listed in the file 4.1.2. Installing individual drivers ==================================== You can also choose to install drivers individually. Provided here are steps to build and install NIC, TOE, iWARP, Bonding, Bypass, UDP Segmentation Offload, DCB drivers and benchmarking tools. To know about other drivers, view help by running "make help". i. To build and install NIC driver without offload support: [root@host~]# make nic [root@host~]# make nic_install ii. To build and install NIC driver with offload support and Offload drivers: [root@host~]# make toe [root@host~]# make toe_install iii. To build and install Offload drivers without IPv6 support: [root@host~]# make toe_ipv4 [root@host~]# make toe_ipv4_install iv. To build and install iWARP driver against outbox OFED: [root@host~]# make iwarp [root@host~]# make iwarp_install v. To build and install bonding driver and Offload drivers: [root@host~]# make bonding [root@host~]# make bonding_install vi. To build and install all drivers without IPv6 support: [root@host~]# make ipv6_disable=1 [root@host~]# make ipv6_disable=1 install vii. The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= viii. To build and install drivers along with benchmarks: [root@host~]# make BENCHMARKS=1 [root@host~]# make BENCHMARKS=1 install ix. Unified Wire Manager will be installed by default. To skip the installation: [root@host~]# make INSTALL_UM=0 install x. The drivers will be installed as RPMs or Debian packages (for ubuntu). To skip this and install drivers: [root@host~]# make SKIP_RPM=1 install NOTE:To view the different configuration tuning options, view the help by typing [root@host~]# make help NOTE:If IPv6 is disabled in the machine, the drivers will be built and installed without IPv6 Offload support by default. 4.1.3. Firmware Update ====================== The firmware (v1.16.9.0) is installed on the system, typically in /lib/firmware/cxgb4, and the driver will auto-load the firmware if an update is required. The kernel must be configured to enable userspace firmware loading support: Device Drivers -> Generic Driver Options -> Userspace firmware loading support The firmware version can be verified using ethtool: [root@host~]# ethtool -i 4.1.4. Uninstalling Chelsio Unified Wire ======================================== Follow the steps mentioned below for uninstallation using CLI. a. From source -------------- Navigate to the ChelsioUwire-x.xx.x.x directory. Uninstall the source using the following command: [root@host~]# make uninstall NOTE:Uninstalling Unified Wire package will not uninstall Unified Wire Manager. Refer the section, "Uninstalling individual drivers/software" to remove the software manually. iWARP driver uninstallation on Cluster nodes ---------------------------------------------- To uninstall iWARP drivers on multiple Cluster nodes with a single command, run the following command: [root@host~]# ./install.py -C -m -u all The above command will remove Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. 4.1.5. Uninstalling individual drivers/software ================================================ You can also choose to uninstall drivers/software individually. Provided here are steps to uninstall NIC, TOE, iWARP, Bypass, UDP Segmentation Offload drivers and Unified Wire Manager(UM). To know about other drivers, view help by running "make help". i. To uninstall NIC driver: [root@host~]# make nic_uninstall ii. To uninstall drivers with offload support: [root@host~]# make toe_uninstall iii. To uninstall iWARP driver: [root@host~]# make iwarp_uninstall iv. To uninstall Unified Wire Manager (UM) [root@host~]# make uninstall UM_UNINST=1 OR [root@host~]# make tools_uninstall UM_UNINST=1 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.2. Network (NIC/TOE) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.2.1. Loading/Unloading the driver =================================== a. Loading the driver --------------------- IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. i. To load the driver in NIC mode(without offload support) [root@host~]# modprobe cxgb4 ii. To load driver in TOE mode(with offload support) [root@host~]# modprobe t4_tom NOTE:Offload support needs to be enabled upon each reboot of the system. This can be done manually as shown above. In VMDirect Path environment, it is recommended to load the offload driver using the following command: [root@host~]# modprobe t4_tom vmdirectio=1 b. Unloading the driver ------------------------ - To unload the driver in NIC mode(without offload support): [root@host~]# rmmod cxgb4 - A reboot is required to unload the driver in TOE(with Offload support). To avoid rebooting, follow the steps mentioned below: i. Load t4_tom driver with unsupported_allow_unload parameter. [root@host~]# modprobe t4_tom unsupported_allow_unload=1 ii. Stop all the offloaded traffic, servers and connections. Check for the reference count. [root@host~]# cat /sys/module/t4_tom/refcnt If the reference count is 0, the driver can be directly unloaded. Skip to step (iii). If the count is non-zero, load a COP policy which disables offload using the following procedure: a. Create a policy file which will disable offload [root@host~]# cat policy_file all => !offload b. Compile and apply the output policy file [root@host~]# cop –o no-offload.cop policy_file [root@host~]# cxgbtool ethX policy no-offload.cop iii. Unload the driver: [root@host~]# rmmod t4_tom [root@host~]# rmmod toecore [root@host~]# rmmod cxgb4 4.2.2. Instantiate Virtual Functions ===================================== To instantiate the Virtual functions, load the cxgb4 driver with 'num_vf' parameter with a non-zero value. For example: [root@host~]# modprobe cxgb4 num_vf=1,0,0,0 Each number instantiates the specified number of Virtual Functions for the corresponding Physical Function. The Virtual Functions can be assigned to Virtual Machines(Guest OS). A maximum of 64 Virtual Functions can be instantiated with 16 Virtual Functions per Physical Function. Loading the cxgb4 driver with "num_vf" parameter loads the cxgb4vf driver by default. Hence unload the cxgb4vf driver (on the host) before assigning Virtual Functions to the Virtual Machines(Guest OS), using the following command: [root@host~]# rmmod cxgb4vf 4.2.3. Enabling Busy waiting ============================ Busy waiting/polling is a technique where a process repeatedly checks to see if an event has occurred, by spinning in a tight loop. By making use of similar technique, Linux kernel provides the ability for the socket layer code to poll directly on an Ethernet device's Rx queue. This eliminates the cost of interrupts and context switching, and with proper tuning allows to achieve latency performance similar to that of hardware. Chelsio's NIC and TOE drivers support this feature and can be enabled on Chelsio supported devices to attain improved latency. To make use of BUSY_POLL feature, follow the steps mentioned below: i. Enable BUSY_POLL support in kernel config file by setting "CONFIG_NET_RX_BUSY_POLL=y" ii. Enable BUSY_POLL globally in the system by setting the values of following sysctl parameters depending on the number of connections: sysctl -w net.core.busy_read= sysctl -w net.core.busy_poll= Set the values of the above parameters to 50 for 100 or less connections; and 100 for more than 100 connections. NOTE: BUSY_POLL can also be enabled on a per-connection basis by making use of SO_BUSY_POLL socket option in the socket application code.Refer socket man-page for further details. 4.2.4. Performance Tuning ========================= To tune your system for better network performance, refer the "Performance Tuning" section of the Network (NIC/TOE) chapter in the User's Guide. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.3. iWARP (RDMA) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Loading/Unloading the Driver ============================ a. Loading the driver --------------------- IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the iWARP driver we need to load the NIC driver & core RDMA drivers first: [root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm b. Unloading the driver ----------------------- To unload the iWARP driver, run the following command: [root@host~]# rmmod iw_cxgb4 IMPORTANT:openmpi-1.4.3 can cause IMB benchmark stalls due to a shared memory BTL issue. This issue is fixed in openmpi-1.4.5 and later releases. Hence, it is recommended that you download and install the latest stable release from Open MPI's official website, http://www.open-mpi.org NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.4. iSCSI PDU Offload Target ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Loading/Unloading the Driver ============================ a. Loading the driver --------------------- IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. To load the module for RHEL distributions, run modprobe as follows: [root@host~]# modprobe chiscsi_t4 completion_on=1 b. Unloading the driver ----------------------- Use the following command to unload the module: [root@host~]# rmmod chiscsi_t4 NOTE:i. While using rpm-tar-ball for installation a. Uninstallation will result into chiscsi.conf file renamed into chiscsi.conf.rpmsave. b. It is advised to take a backup of chiscsi.conf file before you do an uninstallation and installation of new/same unified wire package. As re-installing/upgrading unified-wire package may lead to loss of chiscsi.conf file. ii. Installation/uninstallation using source-tar-ball will neither remove the conf file nor rename it. It will always be intact. However it is recommended to always take a backup of your configuration file for both methods of installation. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.5. iSCSI PDU Offload Initiator ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Loading/Unloading the Driver ============================ a. Loading the driver ---------------------- IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. The driver must be loaded by the root user. Any attempt to loading the driver as a regular user will fail. Load cxgb4i driver using the following command: [root@host~]# modprobe cxgb4i The cxgb4i module registers a new transport class "cxgb4i". On SLES distributions, load the driver with the '--allow' option: [root@host~]# modprobe cxgb4i --allow If loading of cxgb4i displays "unkown symbols found" error in dmesg, follow the steps mentioned below: i. Kill iSCSI daemon "iscsid" ii. View all the loaded iSCSI modules [root@host~]# lsmod | grep iscsi iii. Now, unload them using the following command: [root@host~]# rmmod iv. Finally reload the cxgb4i driver b. Unloading the driver -------------------------- [root@host~]# rmmod cxgb4i [root@host~]# rmmod libcxgbi NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.6. Offload Bonding driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.6.1. Loading/Unloading the Driver ==================================== a. Loading the driver --------------------- IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. The driver must be loaded by the root user. Any attempt to loading the driver as a regular user will fail. To load the Bonding driver (with offload support), run the following command: [root@host~]# modprobe bonding b. Unloading the driver ----------------------- To unload the bonding driver. [root@host~]# rmmod bonding 4.6.2. Offloading TCP traffic over a bonded interface ====================================================== The Chelsio Offload Bonding driver supports all the bonding modes in NIC Mode. In offload mode (t4_tom loaded) however, only the balance-rr (mode=0), active-backup (mode=1),balance-xor (mode=2) and 802.3ad (mode=4) modes are supported. To offload TCP traffic over a bonded interface, use the following method: i. Load the network driver with TOE support. [root@host~]# modprobe t4_tom ii. Create a bonded interface [root@host~]# modprobe bonding mode=1 miimon=100 iii. Bring up the bonded interface and enslave the interfaces to the bond [root@host~]# ifconfig bond0 up [root@host~]# ifenslave bond0 ethX ethY NOTE: "ethX" and "ethY" are interfaces of the same adapter. iv. Assign IPv4/IPv6 address to the bonded interface [root@host~]# ifconfig bond0 X.X.X.X/Y [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up v. Disable FRTO on the PEER: [root@host~]# sysctl -w net.ipv4.tcp_frto=0 All TCP traffic will be offloaded over the bonded interface now. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.7. Offload Multi-Adapter Failover (MAFO) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Multi-Adapter fail-over feature will work for link down events caused by: - Cable unplug on bonded interface - Bringing corresponding switch port down NOTE: The feature will not work if the bonded interfaces are administratively taken down. IMPORTANT: - Portions of this software are covered under US Patent "Failover and migration for full-offload network interface devices : US 8346919 B1" - Use of the covered technology is strictly limited to Chelsio ASIC-based solutions. 4.7.1. Loading/Unloading the Driver ==================================== a. Loading the driver --------------------- IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host~]# modprobe bonding b. Unloading the driver ----------------------- To unload the driver, run the following command: [root@host~]# rmmod bonding 4.7.2. Offloading TCP traffic over a bonded interface ====================================================== The Chelsio MAFO driver supports only the active-backup (mode=1) mode. To offload TCP traffic over a bonded interface, use the following method: i. Load the network driver with TOE support. [root@host~]# modprobe t4_tom ii. Create a bonded interface [root@host~]# modprobe bonding mode=1 miimon=100 iii. Bring up the bonded interface and enslave the interfaces to the bond [root@host~]# ifconfig bond0 up [root@host~]# ifenslave bond0 ethX ethY NOTE: "ethX" and "ethY" are interfaces of different adapters. iv. Assign IPv4/IPv6 address to the bonded interface [root@host~]# ifconfig bond0 X.X.X.X/Y [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up v. Disable FRTO on the PEER: [root@host~]# sysctl -w net.ipv4.tcp_frto=0 All TCP traffic will be offloaded over the bonded interface now and fail-over will happen in case of link-down event. NOTE: For more information on additional configuration options, please refer User's Guide. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.8. Offload IPv6 driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Loading/Unloading the Driver ============================ IPv6 must be enabled in your system (enabled by default) to use the Offload IPv6 feature.Also, Unified Wire package must be installed with IPv6 support (see section 4.1 for more information). a. Loading NIC & TOE drivers ------------------------------ IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. After installing Unified Wire package and rebooting the host, load the NIC (cxgb4) and TOE (t4_tom) drivers. The drivers must be loaded by root user. Any attempt to load the drivers as a regular user will fail. [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom All the IPv6 traffic will be offloaded now. b. Unloading NIC & TOE drivers ------------------------------ i. To unload the NIC driver: [root@host~]# rmmod cxgb4 ii. To unload the TOE driver: Please reboot the system to unload the TOE driver. To unload without rebooting, refer "Unloading the driver" in Network (NIC/TOE) section. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.9. Classification and Filtering ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Classification and Filtering feature enhances network security by controlling incoming traffic as they pass through network interface based on source and destination addresses, protocol, source and receiving ports, or the value of some status bits in the packet. 4.9.1. Usage ============= a. Creating Filter Rules ------------------------- Network driver (cxgb4) must be installed before setting the filter rule. i. If you haven't done already, run the Unified Wire Installer with the appropriate T5/T4 configuration tuning option to install the Network Driver. ii. Next, run the following command to load the network driver: [root@host~]# modprobe cxgb4 iii. Now, create filter rules using cxgbtool: [root@host~]# cxgbtool ethx filter action [pass/drop/switch] Where, ethX : Chelsio interface index : positive integer set as filter id action : Ingress packet disposition pass : Ingress packets will be passed through set ingress queues switch : Ingress packets will be routed to an output port with optional header rewrite. drop : Ingress packets will be dropped. NOTE: In case of multiple filter rules, the rule with the lowest filter index takes higher priority. b. Listing Filter Rules ----------------------- To list previously set filters, run the following command: [root@host~]# cxgbtool ethX filter show c. Removing Filter Rules ------------------------ To remove a filter, run the following command with the corresponding filter rule index [root@host~]# cxgbtool ethX filter index NOTE:For more information on additional parameters, refer to cxgbtool manual by running the "man cxgbtool" command. 4.9.2. Hash/DDR Filters ======================== The default (Unified Wire) configuration tuning option allows you to create LE-TCAM filters, which has a limit of 496 filter rules. If you wish to create more, select "T5 Hash Filter" configuration tuning option during installation which allows you to create HASH/DDR filters with a capacity of ~0.5 million filter rules. NOTE: Creating Hash/DDR Filters is currently supported only on T5 adapters. a. Creating Filter Rules ------------------------ Network driver (cxgb4) must be installed and loaded before setting the filter rule. i. If you haven’t done already, run the Unified Wire Installer with the "T5 Hash Filter" configuration tuning option to install the Network Driver. ii. Load the network driver with DDR filters support : [root@host~]# modprobe cxgb4 use_ddr_filters=1 iii. Now, create filter rules using cxgbtool: [root@host~]# cxgbtool ethX filter action [pass/drop/switch] fip lip fport lport hitcnts 1 cap maskless Where, ethX : Chelsio interface. index : Filter index. The user must provide a positive integer, which will be replaced by an automatically computed index, based on the hash (4-tuple). The index will be displayed after the filter rule is created successfully. action : Ingress packet disposition. pass : Ingress packets will be passed through set ingress queues. switch : Ingress packets will be routed to an output port with. optional header rewrite. drop : Ingress packets will be dropped. source IP/port : Source IP/port of incoming packet. destination IP/port: Destination IP/port of incoming packet. NOTE: "source IP","destination IP","source port" and "destination port" are mandatory parameters since Hash filters don't support masks and hence, 4-tuple must be supplied always for Hash filter. "cap maskless" parameter should be appended in order to create Hash/DDR filter rules. Otherwise the above command will create LE-TCAM filter rules. b. Listing Filter Rules ----------------------- - To list the Hash/DDR filters set, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4//hash_filters - To list both LE-TCAM and Hash/DDR filters set, run the following command: [root@host~]# cxgbtool ethX filter show c. Removing Filter Rules ------------------------ To remove a filter, run the following command with cap maskless parameter and corresponding filter rule index: [root@host~]# cxgbtool ethX filter index cap maskless NOTE: Filter rule index can be determined by referring the "hash_filters" file located in /sys/kernel/debug/cxgb4// NOTE: For more information on additional parameters, refer cxgbtool manual by running the man cxgbtool command. d. Hit Counters --------------- For LE-TCAM filters, hit counters will work simply by adding hitcnts 1 parameter to the filter rule. However, for Hash/DDR filters, you will have to make use of tracing feature and RSS queues. Here’s a step-by-step guide to enable hit counters for Hash/DDR filter rules: i. Enable tracing on T5 adapter. [root@host~]# cxgbtool ethX reg 0x09800=0x13 ii. Setup a trace filter [root@host~]# echo tx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace0 NOTE: Use "snaplen=60" in case of IPv6. iii. Configure the RSS Queue to receive traced packets. Determine the "RspQ ID" of the queue by looking at "Trace" QType in /sys/kernel/debug/cxgb4//sge_qinfo file [root@host~]# cxgbtool ethX reg 0x0a00c= The above step will trace all the packets transmitting from port1(tx1) to trace filter 0. Multi-tracing --------------- To enable hit counters for multiple Chelsio ports in Tx/Rx direction enable Multi-tracing. Using this we can configure 4 different RSS Queues separately corresponding to 4 trace-filters. i. Enable Tracing as well as MultiRSSFilter [root@host~]# cxgbtool ethX reg 0x09800=0x33 ii. Setup a trace filter [root@host~]# echo tx0 snaplen=40 > /sys/kernel/debug/cxgb4//trace0 iii. Configure the RSS Queue corresponding to trace0 filter configured above. Determine the "RspQ ID" of the queues by looking at "Trace" QType in /sys/kernel/debug/cxgb4//sge_qinfo file. [root@host~]# cxgbtool ethX reg 0x09808= iv. Similarly for other direction and for multiple ports run the follow commands: [root@host~]# echo rx0 snaplen=40 > /sys/kernel/debug/cxgb4//trace1 [root@host~]# echo tx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace2 [root@host~]# echo rx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace3 [root@host~]# cxgbtool ethX reg 0x09ff4= [root@host~]# cxgbtool ethX reg 0x09ffc= [root@host~]# cxgbtool ethX reg 0x0a004= NOTE: Use "snaplen=60" in case of IPv6. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.10. Traffic Management ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Traffic Management capabilities built-in to Chelsio T5/T4 CNAs can shape transmit data traffic through the use of sophisticated queuing and scheduling algorithms built-in to the T5/T4 ASIC hardware which provides fine-grained software control over latency and bandwidth parameters such as packet rate and byte rate. 4.10.1. Loading/Unloading the Driver ==================================== a. Loading the driver --------------------- IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. Traffic Management can be performed on non-offloaded connections as well as on offloaded connections. The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail.Run the following commands to load the TOE driver: [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom b. Unloading the driver ----------------------- Reboot the system to unload the driver. To unload without rebooting, refer "Unloading the driver" in Network (NIC/TOE) section. 4.10.2. Usage ============= a. Traffic Management of Non-Offloaded Connections -------------------------------------------------- The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for Non-offload connections: i. Load the network driver and bring up the interface [root@host~]# modprobe cxgb4 [root@host~]# ifconfig eth0 up ii. Bind connections with destination IP address 192.168.5.3 to NIC TX queue 3 [root@host~]# tc qdisc add dev eth0 root handle 1: multiq [root@host~]# tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dst 192.168.5.3 action skbedit queue_mapping 3 iii. Bind the NIC TX queue to class 0 [root@host~]# cxgbtool eth0 sched-queue 3 0 iv. Set the appropriate rule for class 0 [root@host~]# cxgbtool eth0 sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 b. Traffic Management of Offloaded Connections ---------------------------------------------- The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for offloaded connections: i. Load the TOE driver and bring up the interface: [root@host~]# modprobe t4_tom [root@host~]# ifconfig eth0 up ii. Create a new policy file (say new_policy_file) and add the following line to associate connections with the given scheduling class: src host 102.1.1.1 => offload class 0 iii. Compile the policy file using COP [root@host~]# cop -d -o iv. Apply the COP policy: [root@host~]# cxgbtool eth0 policy v. Set the appropriate rule for class 0 [root@host~]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 c. Traffic Management of Offloaded Connections with Modified Application ------------------------------------------------------------------------ The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for for offloaded connections with modified application. i. Load the TOE driver and bring up the interface [root@host~]# modprobe t4_tom [root@host~]# ifconfig eth0 up ii. Modify the application as mentioned in the Configuring Traffic Management section in the User's Guide. iii. Set the appropriate rule for class 0 [root@host~]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 NOTE:For more information on additional parameters, refer cop manual by running the "man cxgbtool" command. NOTE: For more information on additional configuration options, please refer User's Guide. 5. Support Documentation ================================================================================ The documentation for this release can be found inside the ChelsioUwire-x.xx.x.x/docs folder. It contains: - README - Release Notes 6. Customer Support ================================================================================ Please contact Chelsio support at support@chelsio.com for any issues regarding the product. ******************************************************************************** Copyright (C) 2016 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system,or transmitted in any form or by any means without the express written consent of Chelsio Communications.