**************************************** README **************************************** Chelsio Unified Wire for Linux Version : 3.5.0.4 (Alpha) Date : 08/14/2017 Overview ================================================================================ Chelsio Unified Wire software for Linux is an easy to use utility developed to provide installation of 64-bit Linux based drivers and tools for Chelsio's Unified Wire adapters. The Chelsio Unified Wire package provides an interactive installer to install various drivers and utilities. It consists of the following components: - Network (NIC/TOE) - Virtual Function Network (vNIC) - iWARP (RDMA) - iSER - WD-UDP - WD-TOE - NVMe-oF - LIO iSCSI Target Offload - iSCSI PDU Offload Target - iSCSI PDU Offload Initiator - Crypto Offload - Data Center Bridging (DCB) - FCoE Full Offload Initiator - Offload Bonding - Offload Multi-Adapter Failover(MAFO) - UDP Segmentation Offload and Pacing - Offload IPv6 - Classification and Filtering feature - Traffic Management feature (TM) - Unified Wire Manager (UM) - Unified Boot Software - DPDK - OVS Kernel Datapath Offload - Utility Tools (cop,cxgbtool,t4_perftune,benchmark tools,sniffer & tracer) - libs (iWARP, WD-UDP and WD-TOE libraries) ================================================================================ CONTENTS ================================================================================ - 1. Requirements - 2. Supported Operating Systems - 3. Supported Cards - 4. How To Use - 5. Support Documentation - 6. Customer Support 1. Requirements ================================================================================ Unified Wire Installer ---------------------- ncurses-devel package should be installed for installation using GUI mode (with Dialog utility) Virtual Function Network (vNIC) ------------------------------- - Chelsio adapter and an SR-IOV enabled system with supported platforms mentioned in section 2. - Intel Virtualization Technology for Directed I/O (VT-d) should be enabled in the BIOS. - PCI Express Slot should be ARI capable. iWARP (RDMA) ------------ - libnl-devel package should be installed for libiwpm installation. - If you are planning to upgrade OFED on one member of the cluster, the upgrade needs to be installed on all the members. - If you want to install OFED with NFS-RDMA support, please see "Setting up NFS-RDMA" in iWARP (RDMA) chapter in the User's Guide. iSER ---- - libnl-devel package should be installed for libiwpm installation. - Python v2.7 or above is required for targetcli installation. If Python v2.7 is not already present in the system, or if an older version exists, v2.7.10 provided in the package will be installed. NVMe-oF ------- - bzip2-devel,zlib-devel,ncurses-devel,sqlite-devel,libudev-devel packages for nvmetcli utility should be installed. - Python v2.7 or above. If Python v2.7 is not already present in the system, or if an older version exists, v2.7.10 provided in the package will be installed. LIO iSCSI Target Offload Driver ------------------------------- - The driver requires the following components to be installed to function. - Python v2.7 or above* (v2.7.10 provided in the package) - TargetCLI* (v2.1 provided in the package) - OpenSSL (Download from https://www.openssl.org/source/) * If not already present in the system, the components provided in the package will be installed. iSCSI PDU Offload Initiator -------------------------- - The iSCSI PDU Offload Initiator driver (cxgb4i) runs on top of NIC module (cxgb4) and open-iscsi-2.0-872/873/874 only, on a Chelsio card. - If you're planning to install iSCSI PDU Offload Initiator, please install openssl-devel package. Crypto Offload -------------- - Disable selinux and firewall. Unified Wire Manager -------------------- - Chelsio adapter and system with supported platforms mentioned in section 2. Unified Boot Software --------------------- - Chelsio adapter and system with supported platforms mentioned in section 2. - DOS bootable USB flash drive or Floppy Disk. DPDK ----- - flex, bison, byacc, patch, patchutils, autoconf, automake and rpm-build packages must be present in the machine (required for libpcap installation). OVS --- Before installing OVS, please ensure that the following dependencies are installed: - GCC 4.6+ - Python 2.7+ - Python-six - Autoconf 2.63+ - Automake 1.10+ - libtool 2.4+ For the complete list of dependencies please visit http://docs.openvswitch.org/en/latest/intro/install/general/ 2. Supported Operating Systems ================================================================================ The Chelsio Unified Wire software has been developed to run on 64-bit Linux based platforms. Following is the list of Drivers/Software and supported Linux distributions. 1. x86_64 Architecture ====================== |########################|#####################################################| | Linux Distribution | Driver/Software | |########################|#####################################################| |RHEL 7.3, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target, | |3.10.0-514.el7 |iSCSI Initiator,Crypto(Co-processor),DCB, | | |FCoE Initiator,Bonding,MAFO,UDP-SO,IPv6, | | |Sniffer & Tracer,Filtering,TM, | | |uBoot(PXE,FCoE,iSCSI),DPDK(UIO,VFIO),OVS | |------------------------|-----------------------------------------------------| |RHEL 7.2, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target, | |3.10.0-327.el7 |iSCSI Initiator,DCB,Bonding,MAFO,UDP-SO,IPv6, | | |Sniffer & Tracer,Filtering,TM,uBoot(PXE,iSCSI), | | |DPDK(UIO,VFIO) | |------------------------|-----------------------------------------------------| |RHEL 6.9, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target, | |2.6.32-696.el6 |iSCSI Initiator,DCB,Bonding,MAFO,UDP-SO,IPv6, | | |Sniffer & Tracer,Filtering,TM | |------------------------|-----------------------------------------------------| |RHEL 6.8, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target, | |2.6.32-642.el6 |iSCSI Initiator,DCB,Bonding,MAFO,UDP-SO,IPv6, | | |Sniffer & Tracer,Filtering,TM,uBoot(PXE,iSCSI) | |------------------------|-----------------------------------------------------| |SLES 12 SP2, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target, | |4.4.21-69-default |iSCSI Initiator,DCB,Bonding,MAFO,UDP-SO,IPv6, | | |Sniffer & Tracer,Filtering,TM,uBoot(PXE,iSCSI) | |------------------------|-----------------------------------------------------| |SLES 12 SP1, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target, | |3.12.49-11-default |iSCSI Initiator,DCB,Bonding,MAFO,UDP-SO,IPv6, | | |Sniffer & Tracer,Filtering,TM,uBoot(PXE,iSCSI) | |------------------------|-----------------------------------------------------| |SLES 11 SP4, |NIC/TOE,vNIC,iWARP,WD-UDP,WD-TOE,iSCSI Target, | |3.0.101-63-default |iSCSI Initiator,DCB,Bonding,MAFO,UDP-SO,IPv6, | | |Sniffer & Tracer,Filtering,TM,uBoot(PXE,iSCSI) | |------------------------|-----------------------------------------------------| |Ubuntu 16.04.1, |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |4.4.0-31-generic |iSCSI Initiator,Crypto(Inline-TLS),DCB,Bonding,MAFO, | | |UDP-SO,IPv6,Sniffer & Tracer,Filtering,TM | |------------------------|-----------------------------------------------------| |Ubuntu-14.04.4, |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |4.2.0-27-generic |iSCSI Initiator,DCB,Bonding,MAFO,UDP-SO,IPv6, | | |Sniffer & Tracer,Filtering,TM | |------------------------|-----------------------------------------------------| |Kernel.org linux-4.9.13*|NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |LIO iSCSI Target,iSCSI Target,iSCSI Initiator, | | |Crypto(Co-processor,Inline-TLS)^,DCB,Bonding,MAFO, | | |UDP-SO,IPv6,Sniffer & Tracer,Filtering,TM,OVS | |------------------------|-----------------------------------------------------| |Kernel.org linux-4.8 |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |LIO iSCSI Target,iSCSI Target,iSCSI Initiator, | | |DCB,Bonding,MAFO,UDP-SO,IPv6,Sniffer & Tracer, | | |Filtering,TM | |------------------------------------------------------------------------------| * Minimum 4.9 kernel version supported is 4.9.13 ^ 4.9.13 compiled on RHEL 7.3 NOTE: Other kernel versions have not been tested and are not guaranteed to work. 2. POWERPC64 Architecture ========================= |########################|#####################################################| | Linux Distribution | Driver/Software | |########################|#####################################################| |RHEL 7.3 (POWER8), |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator,Bonding, | |3.10.0-514.el7.ppc64le |MAFO,IPv6,Filtering,TM,DPDK(VFIO) | |------------------------------------------------------------------------------| NOTE: Other kernel versions have not been tested and are not guaranteed to work. 3. ARM64 Architecture ===================== |########################|#####################################################| | Linux Distribution | Driver/Software | |########################|#####################################################| |RHEL 7.3 (ARM64), |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator,Bonding, | |4.5.0-15.el7.aarch64 |MAFO,IPv6,Filtering,TM | |------------------------------------------------------------------------------| NOTE: Other kernel versions have not been tested and are not guaranteed to work. 3. Supported Hardware ================================================================================ 1.Chelsio Drivers/Software and supported adapters ================================================= |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |------------------------|-----------------------------------------------------| |T62100-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |iSCSI Target,LIO iSCSI Target,iSCSI Initiator,Crypto,| | |DCB,Bonding,MAFO,IPv6,UDP-SO,Filtering,TM,uBoot*, | | |DPDK,OVS | |------------------------|-----------------------------------------------------| |T62100-LP-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |iSCSI Target,LIO iSCSI Target,iSCSI Initiator,Crypto,| | |DCB,Bonding,MAFO,IPv6,UDP-SO,Filtering,TM,uBoot*, | | |DPDK,OVS | |------------------------|-----------------------------------------------------| |T62100-SO-CR |NIC/TOE,DPDK | |------------------------|-----------------------------------------------------| |T6425-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |iSCSI Target,LIO iSCSI Target,iSCSI Initiator, | | |FCoE Initiator,Crypto,DCB,Bonding,MAFO,IPv6,UDP-SO, | | |Filtering,TM,DPDK,OVS | |------------------------|-----------------------------------------------------| |T6225-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |iSCSI Target,LIO iSCSI Target,iSCSI Initiator, | | |FCoE Initiator,Crypto,DCB,Bonding,MAFO,IPv6,UDP-SO, | | |Filtering,TM,uBoot*,DPDK,OVS | |------------------------|-----------------------------------------------------| |T6225-LL-CR |DPDK,OVS,FCoE Initiator | |------------------------|-----------------------------------------------------| |T6225-SO-CR |NIC/TOE,iSER^,NVMe-oF^ | |------------------------|-----------------------------------------------------| |T580-OCP-SO |NIC,uBoot(PXE) | |------------------------|-----------------------------------------------------| |T520-OCP-SO |NIC,uBoot(PXE) | |------------------------|-----------------------------------------------------| |T520-BT |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |LIO iSCSI Target,iSCSI Target,iSCSI Initiator, | | |FCoE Initiator,Bonding,MAFO,IPv6,Sniffer & Tracer, | | |UDP-SO,Filtering,TM,UM,uBoot*,DPDK,OVS | |------------------------|-----------------------------------------------------| |T580-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |LIO iSCSI Target,iSCSI Target,iSCSI Initiator,DCB, | | |Bonding,MAFO,IPv6,Sniffer & Tracer,UDP-SO,Filtering, | | |TM,UM,uBoot*,DPDK,OVS | |------------------------|-----------------------------------------------------| |T580-SO-CR |NIC,Filtering,uBoot(PXE),DPDK | |------------------------|-----------------------------------------------------| |T580-LP-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |LIO iSCSI Target,iSCSI Target,iSCSI Initiator,DCB, | | |Bonding,MAFO,IPv6,Sniffer & Tracer,UDP-SO,Filtering, | | |TM,UM,uBoot*,DPDK,OVS | |------------------------|-----------------------------------------------------| |T520-LL-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |LIO iSCSI Target,iSCSI Target,iSCSI Initiator,DCB, | | |FCoE Initiator,Bonding,MAFO,IPv6,Sniffer & Tracer, | | |UDP-SO,Filtering,TM,UM,uBoot*,DPDK,OVS | |------------------------|-----------------------------------------------------| |T520-SO-CR |NIC,uBoot(PXE),DPDK | |------------------------|-----------------------------------------------------| |T520-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |LIO iSCSI Target, iSCSI Target,iSCSI Initiator,DCB, | | |FCoE Initiator,Bonding,MAFO,IPv6,Sniffer & Tracer, | | |UDP-SO,Filtering,TM,UM,uBoot*,DPDK,OVS | |------------------------|-----------------------------------------------------| |T540-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,NVMe-oF, | | |LIO iSCSI Target,iSCSI Target,iSCSI Initiator,UDP-SO,| | |Bonding,MAFO,IPv6,Sniffer & Tracer,Filtering,uBoot*, | | |DPDK,OVS | |------------------------|-----------------------------------------------------| |T420-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | | |iSCSI Initiator,DCB,Bonding,MAFO,IPv6, | | |Sniffer & Tracer,UDP-SO,Filtering,TM,UM | |------------------------|-----------------------------------------------------| |T440-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | | |iSCSI Initiator,Bonding,MAFO,IPv6,Sniffer & Tracer, | | |UDP-SO,Filtering,TM,UM | |------------------------|-----------------------------------------------------| |T422-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | | |iSCSI Initiator,Bonding,MAFO,IPv6,Sniffer & Tracer, | | |UDP-SO,Filtering,TM,UM | |------------------------|-----------------------------------------------------| |T420-SO-CR |NIC,vNIC,Bonding,MAFO,IPv6,Filtering,UM | |------------------------|-----------------------------------------------------| |T404-BT |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | | |iSCSI Initiator,Bonding,MAFO,IPv6,Sniffer & Tracer, | | |UDP-SO,Filtering,TM,UM | |------------------------|-----------------------------------------------------| |T420-BCH |NIC/TOE,iSCSI Target,iSCSI Initiator,Bonding,MAFO, | | |IPv6,Sniffer & Tracer,UDP-SO,Filtering,TM,UM | |------------------------|-----------------------------------------------------| |T440-LP-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | | |iSCSI Initiator,Bonding,MAFO,IPv6,Sniffer & Tracer, | | |UDP-SO,Filtering,TM,UM | |------------------------|-----------------------------------------------------| |T420-BT |NIC/TOE,vNIC,iSCSI Target,iSCSI Initiator,Bonding, | | |MAFO,IPv6,Sniffer & Tracer,UDP-SO,Filtering,TM,UM | |------------------------|-----------------------------------------------------| |T420-LL-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | | |iSCSI Initiator,DCB,Bonding,MAFO,IPv6, | | |Sniffer & Tracer,UDP-SO,Filtering,TM,UM | |------------------------|-----------------------------------------------------| |T420-CX |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | | |iSCSI Initiator,Bonding,MAFO,IPv6,Sniffer & Tracer, | | |UDP-SO,Filtering,TM,UM | |------------------------------------------------------------------------------| * PXE,FCoE,iSCSI ^ Limited number of offload connections supported. 2. Unified Wire Manager (UM) ============================ Supported T3 adapters --------------------- - S302E - S302E-C - S310E-CR - S310E-CR-C - S310E-CXA - S310E-SR+ - S310E-SR - S310E-BT - S320E-CR - S320E-LP-CR - S320E-CXA - S320EM-BS - S320EM-BCH - N320E-G2-CR - N320E - N320E-CXA - N320E-BT - N310E - N310E-CXA 3. Unified Boot Software ======================== Supported hardware platforms ---------------------------- - DELL PowerEdge T710 - DELL PowerEdge 2950 - DELL PowerEdge T110 - Dell T5600 - IBM X3650 M2 - IBM X3650 M4* - HP ProLiant DL385G2 - Supermicro X7DWE - Supermicro X8DTE-F - Supermicro X8STE - Supermicro X8DT6 - Supermicro X9SRL-F - Supermicro X9SRE-3F - ASUS P5KPL - ASUS P8Z68 * If system BIOS version is lower than 1.5 and both Legacy and uEFI are enabled, please upgrade to 1.5 or higher. Otherwise the system will hang during POST. Supported Switches ------------------- - Cisco Nexus 5010 with 5.1(3)N1(1a) firmware - Arista DCS-7124S-F - Mellanox SX_PPC_M460EX NOTE:Other platforms/switches have not been tested and are not guaranteed to work. 4. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chelsio Unified Wire ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Installing Chelsio Unified Wire =============================== There are two main methods to install the Chelsio Unified Wire package: from source and from RPM. If you decide to use source, you can install the package using CLI or GUI mode. If you decide to use RPM, you can install the package using Menu or CLI mode. The following table describes the various "configuration tuning options" available during installation and drivers/software installed with each option by default: |#######################################|######################################| | Configuration Tuning Option | Driver/Software installed | |#######################################|######################################| |Unified Wire(Default) |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,WD-TOE,| | |NVMe-oF,LIO iSCSI Target,iSCSI Target,| | |iSCSI Initiator,Crypto,DCB, | | |FCoE Initiator,Bonding,MAFO,IPv6, | | |Sniffer & Tracer,UM,Filtering,TM,DPDK,| | |OVS | |---------------------------------------|--------------------------------------| |Low latency Networking |TOE,iWARP,WD-UDP,WD-TOE,IPv6,Bonding, | | |MAFO,UM | |---------------------------------------|--------------------------------------| |High capacity RDMA |iWARP,UM | |---------------------------------------|--------------------------------------| |RDMA Performance |iWARP,iSER,NVMe-oF,UM | |---------------------------------------|--------------------------------------| |High capacity TOE |TOE,Bonding,MAFO,IPv6,UM | |---------------------------------------|--------------------------------------| |iSCSI Performance (T5/T6) |LIO iSCSI Target,iSCSI Target, | | |iSCSI Initiator,Bonding,DCB,UM | |---------------------------------------|--------------------------------------| |UDP Seg.Offload & Pacing (T4/T5) |UDP-SO,Bonding,UM | |---------------------------------------|--------------------------------------| |Wire Direct Latency (T5/T6) |TOE,iWARP,WD-UDP,WD-TOE,UM | |---------------------------------------|--------------------------------------| |High Capacity WD |WD-UDP,WD-TOE,UM | |---------------------------------------|--------------------------------------| |Hash Filter (T5/T6) |Filtering,UM | |------------------------------------------------------------------------------| Enabling RDMA on ARM platforms: ------------------------------- RDMA is disabled by default in RHEL 7.X build of ARM architecture. To enable this feature, follow the steps mentioned below: a) Download the kernel source package and extract it. b) Create a kernel configuration file. [root@host~]# make oldconfig c) The above command will create a configuration file ".config" in the same location. Edit the file and enable the following parameters as follows: CONFIG_NET_VENDOR_CHELSIO=y CONFIG_INFINIBAND=y d) Compile the kernel. e) During kernel compilation, please ensure that the following parameters are set as follows: CONFIG_CHELSIO_T1=m CONFIG_CHELSIO_T1_1G=y CONFIG_CHELSIO_T3=m CONFIG_CHELSIO_T4=m CONFIG_CHELSIO_T4VF=m CONFIG_INFINIBAND_USER_MAD=m CONFIG_INFINIBAND_USER_ACCESS=m CONFIG_INFINIBAND_USER_MEM=y CONFIG_INFINIBAND_CXGB3=m CONFIG_INFINIBAND_CXGB3_DEBUG=y CONFIG_INFINIBAND_CXGB4=m CONFIG_SCSI_CXGB3_ISCSI=m CONFIG_SCSI_CXGB4_ISCSI=m f) Install the kernel. g) Reboot into the newly installed kernel. Mounting debugfs ---------------- All driver debug data is stored in debugfs, which will be mounted in most cases. If not, mount it manually using: [root@host~]# mount -t debugfs none /sys/kernel/debug Allowing unsupported modules on SLES ------------------------------------ On SLES11 SPx platforms, edit the "/etc/modprobe.d/unsupported-modules" file and change "allow_unsupported_modules" to 1. On SLES12 SPx platforms, edit the "/etc/modprobe.d/10-unsupported-modules.conf" file and change "allow_unsupported_modules" to 1. Installation ------------ Follow the steps mentioned below for installation using CLI. For GUI or Menu based installation, refer the User's Guide. 1.1. From source ---------------- a) Download Chelsio Unified Wire driver package. b) Untar the tarball [root@host~]# tar zxvf .tar.gz c) Change your current working directory to Chelsio Unified Wire package directory. Build the source: [root@host~]# make d) Install the drivers, tools and libraries: [root@host~]# make install e) The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= install NOTE: To view the different configuration tuning options, view help by typing [root@host~]# make help f) Reboot your machine for changes to take effect. IMPORTANT:Steps (d) and (e) mentioned above will NOT install Crypto, DCB, WD-TOE drivers, and benchmark tools. They will have to be installed manually. Please refer to section "Installing individual drivers" for instructions on installing them. iWARP driver installation on Cluster nodes ------------------------------------------ IMPORTANT: Please make sure that you have enabled password less authentication with ssh on the peer nodes for this feature to work. Chelsio's Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a single command. Follow the procedure mentioned below: a) Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed. b) Now, execute the following command: [root@host~]# ./install.py -C -m c) Select the required configuration tuning option. The tuning options may vary depending on the Linux distribution. d) Select the required Cluster Configuration. e) If you already have the required version of OFED software installed, you can skip this step. To install OFED-3.18-3 choose the "Install-OFED option". To skip this step, "select Skip-OFED". f) The selected components will now be installed. The above command will install iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes listed in the file 1.2. From RPM (tarball) ----------------------- NOTE: - IPv6 should be enabled in the machine to use the RPM Packages. - Drivers installed from RPM Packages do not have DCB support. a) Download the tarball specific to your operating system and architecture. b) Untar the tarball E.g. For RHEL 6.9, untar using the following command: [root@host~]# tar zxvf -RHEL6.9_x86_64.tar.gz c) Change your current working directory to Chelsio Unified Wire package directory. Run the following command: [root@host~]# ./install.py -i nic_toe :NIC and TOE drivers only all :all Chelsio drivers built against inbox OFED udpso :UDP segmentation offload capable NIC and TOE drivers only wd :Wire Direct drivers and libraries only NOTE: The Installation options may vary depending on the Linux distribution. d) The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following command: [root@host~]# ./install.py -i -c NOTE: To view the different configuration tuning options, view the help by typing [root@host~]# ./install.py -h e) To install OFED and Chelsio Drivers built against OFED, run the above command with -o option. [root@host~]# ./install.py -i -c -o f) Reboot your machine for changes to take effect. NOTE:If the installation aborts with the message "Resolve the errors/dependencies manually and restart the installation", please go through the install.log to resolve errors/dependencies and then start the installation again. iWARP driver installation on cluster nodes ------------------------------------------- IMPORTANT:Please make sure that you have enabled password less authentication with ssh on the peer nodes for this feature to work. a) Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed. b) Navigate to driver package directory and execute the following command: [root@host~]# ./install.py -C -m -i -c -o Here, -o parameter will install OFED and Chelsio drivers built against OFED The above command will install iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes listed in the file c) Reboot your machine for changes to take effect. Installing individual drivers ============================= You can also choose to install drivers individually. Provided here are steps to build and install NIC,TOE,iWARP,Bonding,UDP Segmentation Offload,Crypto Offload, DCB drivers and benchmarking tools. To know about other drivers, view help by running "make help". - To build and install NIC driver without offload support: [root@host~]# make nic [root@host~]# make nic_install - To build and install NIC driver with offload support and Offload drivers: [root@host~]# make toe [root@host~]# make toe_install - To build and install Offload drivers without IPv6 support: [root@host~]# make toe_ipv4 [root@host~]# make toe_ipv4_install - To build and install iWARP driver against outbox OFED: [root@host~]# make iwarp [root@host~]# make iwarp_install - To build and install bonding driver and Offload drivers: [root@host~]# make bonding [root@host~]# make bonding_install - To build and install all drivers without IPv6 support: [root@host~]# make ipv6_disable=1 [root@host~]# make ipv6_disable=1 install - To build and install Crypto Offload driver: [root@host~]# make crypto [root@host~]# make crypto_install - To build and install all drivers with DCB support: [root@host~]# make dcbx=1 [root@host~]# make dcbx=1 install - The offload drivers support UDP Segmentation Offload with limited number of connections (1024 connections).To build and install UDP Offload drivers which support large number of offload connections (approx 10K), [root@host~]# make udp_offload [root@host~]# make udp_offload_install - The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= - To build and install drivers along with benchmarks: [root@host~]# make BENCHMARKS=1 [root@host~]# make BENCHMARKS=1 install - Unified Wire Manager will be installed by default. To skip the installation: [root@host~]# make INSTALL_UM=0 install - The drivers will be installed as RPMs or Debian packages (for ubuntu). To skip this and install drivers: [root@host~]# make SKIP_RPM=1 install NOTE:To view the different configuration tuning options, view the help by typing [root@host~]# make help NOTE:If IPv6 is disabled in the machine, the drivers will be built and installed without IPv6 Offload support by default. Firmware Update =============== The firmware (v1.16.59.0) is installed on the system, typically in /lib/firmware/cxgb4, and the driver will auto-load the firmware if an update is required. The kernel must be configured to enable userspace firmware loading support: Device Drivers -> Generic Driver Options -> Userspace firmware loading support The firmware version can be verified using ethtool: [root@host~]# ethtool -i Uninstalling Chelsio Unified Wire ================================= There are two methods to uninstall the Chelsio Unified Wire package: from source and from RPM. If you decide to use source, you can uninstall the package using CLI or GUI mode. Follow the steps mentioned below for uninstallation using CLI. For GUI based uninstallation, refer the User's Guide. - From source ------------- Navigate to the driver package directory. Uninstall the source using the following command: [root@host~]# make uninstall NOTE:Uninstalling Unified Wire package will not uninstall Unified Wire Manager. Refer the section, "Uninstalling individual drivers/software" to remove the software manually. iWARP driver uninstallation on Cluster nodes ---------------------------------------------- To uninstall iWARP drivers on multiple Cluster nodes with a single command, run the following command: [root@host~]# ./install.py -C -m -u all The above command will remove Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. - From RPM (tar-ball) --------------------- Go to the driver package directory. Run the following command: [root@host~]# ./uninstall.py Here, inbox : for removing all Chelsio drivers. ofed : for removing OFED and Chelsio drivers. NOTE:The uninstallation options may vary depending on Linux distribution. View help by running the following command for more information: [root@host~]# ./uninstall.py -h NOTE:Uninstalling Unified Wire package will not uninstall Unified Wire Manager. Refer the "Unified Wire Manager (UM)" section to remove the software manually. iWARP driver uninstallation on Cluster nodes ----------------------------------------------- To uninstall iWARP drivers on multiple Cluster nodes with a single command, run the following: [root@host~]# ./install.py -C -m -u The above command will remove Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. Uninstalling individual drivers/software ======================================== You can also choose to uninstall drivers/software individually. Provided here are steps to uninstall NIC, TOE,iWARP,UDP Segmentation Offload drivers and Unified Wire Manager(UM). To know about other drivers, view help by running "make help". - To uninstall NIC driver: [root@host~]# make nic_uninstall - To uninstall drivers with offload support: [root@host~]# make toe_uninstall - To uninstall iWARP driver: [root@host~]# make iwarp_uninstall - To uninstall UDP Segmentation Offload driver: [root@host~]# make udp_offload_uninstall - To uninstall Unified Wire Manager (UM) [root@host~]# make uninstall UM_UNINST=1 OR [root@host~]# make tools_uninstall UM_UNINST=1 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Network (NIC/TOE) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== - To build and install NIC only driver (without offload support): [root@host~]# make nic_install - To build and install NIC driver with offload support and Offload drivers: [root@host~]# make install NOTE: For more installation options, please refer Unifier Wire section. Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. - To load the driver in NIC mode(without offload support) [root@host~]# modprobe cxgb4 - To load driver in TOE mode(with offload support) [root@host~]# modprobe t4_tom NOTE:Offload support needs to be enabled upon each reboot of the system. This can be done manually as shown above. In VMDirect Path environment, it is recommended to load the offload driver using the following command: [root@host~]# modprobe t4_tom vmdirectio=1 Disabling Auto-negotiation ========================== Auto-negotiation is enabled by default for T6 adapters. Few 100G switches still have open bugs around this functionality. If you face any link issue with such switches, disable this feature using the command below: [root@host~]# ethtool -s ethX autoneg off Enabling Busy waiting ===================== Busy waiting/polling is a technique where a process repeatedly checks to see if an event has occurred, by spinning in a tight loop. By making use of similar technique, Linux kernel provides the ability for the socket layer code to poll directly on an Ethernet device's Rx queue. This eliminates the cost of interrupts and context switching, and with proper tuning allows to achieve latency performance similar to that of hardware. Chelsio's NIC and TOE drivers support this feature and can be enabled on Chelsio supported devices to attain improved latency. To make use of BUSY_POLL feature, follow the steps mentioned below: a) Enable BUSY_POLL support in kernel config file by setting "CONFIG_NET_RX_BUSY_POLL=y" b) Enable BUSY_POLL globally in the system by setting the values of following sysctl parameters depending on the number of connections: sysctl -w net.core.busy_read= sysctl -w net.core.busy_poll= Set the values of the above parameters to 50 for 100 or less connections; and 100 for more than 100 connections. NOTE: BUSY_POLL can also be enabled on a per-connection basis by making use of SO_BUSY_POLL socket option in the socket application code.Refer socket man-page for further details. Precision Time Protocol (PTP) ============================= ptp4l tool (installed during Unified Wire installation) is used to enable timestamping on slave clocks: a) To use ptp4l with hardware timestamping capable drivers and NIC, start the ptp4l tool on master using the "-i" option: [root@host~]# ptp4l -i -H -m b) Start the tool on slave only mode: [root@host~]# ptp4l -i -H -m -s c) Synchronize the system clock to a PTP hardware clock (PHC). [root@host~]# phc2sys -s -c CLOCK_REALTIME -w -m Performance Tuning ================== To tune your system for better network performance, refer the "Performance Tuning" section of the Network (NIC/TOE) chapter in the User's Guide. NOTE: For more information on additional configuration options, please refer User's Guide. Driver Unloading ================ - To unload the driver in NIC mode(without offload support): [root@host~]# rmmod cxgb4 - A reboot is required to unload the driver in TOE(with Offload support). To avoid rebooting, follow the steps mentioned below: a) Load t4_tom driver with unsupported_allow_unload parameter. [root@host~]# modprobe t4_tom unsupported_allow_unload=1 b) Stop all the offloaded traffic, servers and connections. Check for the reference count. [root@host~]# cat /sys/module/t4_tom/refcnt If the reference count is 0, the driver can be directly unloaded. Skip to step (c). If the count is non-zero, load a COP policy which disables offload using the following procedure: i. Create a policy file which will disable offload [root@host~]# cat policy_file all => !offload ii. Compile and apply the output policy file [root@host~]# cop –o no-offload.cop policy_file [root@host~]# cxgbtool ethX policy no-offload.cop c) Unload the driver: [root@host~]# rmmod t4_tom [root@host~]# rmmod toecore [root@host~]# rmmod cxgb4 NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Virtual Function Network (vNIC) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== To install vNIC driver and firmware: [root@host~]# make vnic_install Instantiate Virtual Functions ============================= To instantiate virtual functions on the host, run the following commands: [root@host~]# modprobe cxgb4 [root@host~]# echo n > /sys/class/net/ethX/device/driver//sriov_numvfs Here, "ethX" is the interface and "n" specifies the number of virtual functions to be instantiated per physical function (bus_id). The virtual functions can be assigned to virtual machines (guests). A maximum of 64 virtual functions can be instantiated with 16 virtual functions per physical function. NOTE: To get familiar with physical and virtual function terminologies, please refer the PCI Express specification. Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The vNIC (cxgb4vf) driver must be loaded on the Guest OS by the root user. Any attempt to load the driver as a regular user will fail. To load the driver execute the following command: [root@host~]# modprobe cxgb4vf Configuration ============= This section describes the method to rate-limit traffic passing through virtual functions (VFs). a) The VF rate limit needs to be set on the Host (hypervisor). Apply rate-limiting using: [root@host~]# ip link set dev mgmtpfXX vf rate Here, - mgmtpfXX is the management interface to be used. For each PF on which VFs are instantiated, 1 interface will be created (in "ifconfig -a"). - vf_number is VF on which rate-limiting is applied. Value 0-15. b) Run traffic over the VF and the throughput should be rate-limited as per the values set in the previous step. Driver Unloading ================ To unload the driver execute the following command: [root@host~]# rmmod cxgb4vf NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iWARP (RDMA) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== To install iWARP driver, WD-UDP Libraries and firmware. [root@host~]# make iwarp_install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the iWARP driver we need to load the NIC driver & core RDMA drivers first: [root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm Optionally, you can start iWARP Port Mapper daemon to enable port mapping: [root@host~]# iwpmd Driver Unloading ================ To unload the iWARP driver, run the following command: [root@host~]# rmmod iw_cxgb4 IMPORTANT:openmpi-1.4.3 can cause IMB benchmark stalls due to a shared memory BTL issue. This issue is fixed in openmpi-1.4.5 and later releases. Hence, it is recommended that you download and install the latest stable release from Open MPI's official website, http://www.open-mpi.org NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSER ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NOTE: This driver will be fully functional in the next release. Kernel Installation =================== a) To install 4.9.13 kernel with iSER components enabled, use the following command: [root@host~]# make kernel_install NOTE: If you wish to use custom 4.8.X/4.9.X kernel, enable the following iSER parameters in the kernel configuration file and then proceed with kernel installation: CONFIG_ISCSI_TARGET=m CONFIG_INFINIBAND_ISER=m CONFIG_INFINIBAND_ISERT=m b) Boot into the new kernel and install Chelsio Unified Wire. Driver Installation =================== To install Chelsio iWARP driver, iSER libraries and targetcli utils: [root@host~]# make iser_install Driver Loading ============== Follow the steps mentioned below on both target and initiator machines: a) Unload Chelsio iWARP driver if previously loaded: [root@host~]# rmmod iw_cxgb4 b) Load the following modules: [root@host~]# modprobe iw_cxgb4 mpa_rev=2 [root@host~]# modprobe rdma_ucm c) Start the iWARP Port Mapper Daemon: [root@host~]# iwpmd d) Bring up the Chelsio interface(s): [root@host~]# ifconfig ethX x.x.x.x up e) Run the following script for IRQ mapping: [root@host~]# t4_perftune.sh f) On target, run the following command: [root@host~]# modprobe ib_isert On initiator, run the following command: [root@host~]# modprobe ib_iser Configuration ============= a) Configure LIO target with iSER support, using ramdisk as LUN: [root@host~]# targetcli /backstores/ramdisk create name=ram0 size=1GB [root@host~]# targetcli /iscsi create wwn=iqn.2003-01.org.lun0.target [root@host~]# targetcli /iscsi/iqn.2003-01.org.lun0.target/tpg1/luns create /backstores/ramdisk/ram0 [root@host~]# targetcli /iscsi/iqn.2003-01.org.lun0.target/tpg1 set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 [root@host~]# targetcli saveconfig b) Discover the LIO target using OpeniSCSI initiator: [root@host~]# iscsiadm -m discovery -t st -p 102.10.10.4 c) Enable iSER support in LIO target: [root@host~]# targetcli /iscsi/iqn.2003-01.org.lun0.target/tpg1/portals/0.0.0.0:3260 enable_iser boolean=True d) Login from the initiator with iSER as transport: [root@host~]# iscsiadm -m node -p 102.10.10.4 -T iqn.2003-01.org.lun0.target --op update -n node.transport_name -v iser [root@host~]# iscsiadm -m node -p 102.10.10.4 -T iqn.2003-01.org.lun0.target --login HMA ==== To use HMA, please ensure that Unified Wire is installed using the "Unified Wire(Default)" configuration tuning option. Currently 256 IPv4/128 IPv6 iSER connections are supported on T6225-SO-CR adapter. Driver Unloading ================ - On target, run the following commands: [root@host~]# rmmod ib_isert [root@host~]# rmmod iw_cxgb4 - On initiator, run the following commands: [root@host~]# rmmod ib_iser [root@host~]# rmmod iw_cxgb4 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ WD-UDP ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Instalallation ===================== To install WD-TOE driver (offload), WD-UDP Libraries and iWARP driver: [root@host~]# wdtoe_wdudp_install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb InfiniBand modules from the OFED package should be loaded before proceeding. Load the cxgb4, iw_cxgb4 and rdma_ucm drivers: [root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm Configuring WD-UDP ================== Preload "libcxgb4_sock" using one of the methods mentioned below when starting your application: Preloading using wdload script ------------------------------ [root@host~]# PROT=UDP wdload /your_application The above command will generate an end point file, libcxgb4_sock.conf at /etc/. Parameters like interface name and port number can be changed in this file. NOTE: If you encounter error while using wdload on kernels built on RHEL 5.x distribution, then run the above command as: [root@host~]# NUMA=0 PROT=UDP wdload /your_application The following example shows how to run Netperf with WD-UDP: server: [root@host~]# PROT=UDP wdload netserver -f -p client: [root@host~]# PROT=UDP wdload netperf -H -p -t UDP_RR Preloading manually ------------------- Create a configuration file that defines which UDP endpoints should be accelerated, their vlan and priority if any, as well as which interface/port should be used. The file /etc/libcxgb4_sock.conf contains these endpoint entries. Create this file on all systems using libcxgb4_sock. Here is the syntax: Syntax: endpoint { attributes } ... where attributes include: interface = interface-name port = udp-port-number E.g: endpoint {interface=eth2 port=8888} endpoint {interface=eth3 port=9999} The above file defines 2 accelerated endpoints, port 8888 which will use eth2, and port 9999 which will use eth3. Now, preload libcxgb4_sock using the following command: [root@host~]# CXGB4_SOCK_CFG= LD_PRELOAD=libcxgb4_sock.so /your_application NOTE: i. In WD-UDP only one application can be run per Terminator device per UDP port number. For running 2 concurrent netperf UDP_RR tests, each must use a unique UDP port number or Terminator device. E.g.: endpoint {interface=eth2 port=8888} endpoint {interface=eth2 port=9000} The above file defines 2 accelerated endpoints, port 8888 and port 9000 both will be used by eth2 only. ii. In order to offload IPv6 UDP sockets, please select "low latency networking" as configuration tuning option during installation. iii. Jumbo frames of 9000B are supported only on kernel 2.6.32 and above. Driver Unloading ================ To unload the iWARP driver, run the following command: [root@host~]# rmmod iw_cxgb4 NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ WD-TOE ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NOTE: This is an alpha release. Driver Installation =================== Before proceeding, please ensure that the offload drivers are installed with WD-TOE support. They are installed by default with Low Latency Networking, T5 Wire Direct Latency or High Capacity WD configuration tuning Options. With any other configuration tuning option, the installation needs to be customized. Alternatively, you can use the following command to install WD-TOE driver (offload) and firmware: [root@host~]# make wdtoe_install Driver Loading ============== The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail. Run the following commands to load the drivers: [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom Driver Unloading ================ Reboot the system to unload the driver. Example ======= To run Netperf application with WD-TOE, a) Start netserver at the PEER, using the following command: [root@host~]# PROT=TCP wdload netserver -D -4 b) On the Test Machine, run the following command to run netperf application. [root@host~]# PROT=TCP wdload netperf -H -t TCP_RR -l 10 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NVMe-oF ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Kernel Installation =================== a) To install 4.9.13 kernel with NVMe-oF components enabled, use the following command: [root@host~]# make kernel_install NOTE: If you wish to use custom 4.8.X/4.9.X kernel, enable the following parameters in the kernel configuration file and then proceed with kernel installation: CONFIG_BLK_DEV_NVME=m CONFIG_NVME_RDMA=m CONFIG_NVME_TARGET=m CONFIG_NVME_TARGET_RDMA=m CONFIG_NVME_RDMA=m CONFIG_BLK_DEV_NULL_BLK=m CONFIG_CONFIGFS_FS=y b) Boot into the new kernel and install Chelsio Unified Wire. Driver Installation =================== To install iWARP driver and NVMe utilities: [root@host~]# make nvme_install Driver Loading ============== Follow the steps mentioned below on both target and initiator machines: a) Load the following modules: [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm b) Bring up the Chelsio interface(s): [root@host~]# ifconfig ethX x.x.x.x up c) Mount configfs by running the below command: [root@host~]# mount -t configfs none /sys/kernel/config d) On target, run the following commands: [root@host~]# modprobe null_blk [root@host~]# modprobe nvmet [root@host~]# modprobe nvmet-rdma On initiator, run the following commands: [root@host~]# modprobe nvme [root@host~]# modprobe nvme-rdma Configuration ============= Target ------ a) The following commands will configure target using nvmetcli with a LUN: [root@host~]# nvmetcli /> cd subsystems /subsystems> create nvme-ram0 /subsystems> cd nvme-ram0/namespaces /subsystems/n...m0/namespaces> create nsid=1 /subsystems/n...m0/namespaces> cd 1 /subsystems/n.../namespaces/1> set device path=/dev/ram1 /subsystems/n.../namespaces/1> cd ../.. /subsystems/nvme-ram0> set attr allow_any_host=1 /subsystems/nvme-ram0> cd namespaces/1 /subsystems/n.../namespaces/1> enable /subsystems/n.../namespaces/1> cd ../../../.. /> cd ports /ports> create 1 /ports> cd 1/ /ports/1> set addr adrfam=ipv4. /ports/1> set addr trtype=rdma /ports/1> set addr trsvcid=4420 /ports/1> set addr traddr=102.1.1.102 /ports/1> cd subsystems /ports/1/subsystems> create nvme-ram0 b) Save the target configuration to a file: /ports/1/subsystems> saveconfig /root/nvme-target_setup /ports/1/subsystems> exit c) Clear the targets: [root@host~]# nvmetcli clear Initiator --------- a) Discover the target [root@host~]# nvme discover -t rdma -a -s 4420 b) Connect to target Connecting to a specific target: [root@host~]# nvme connect -t rdma -a -s 4420 -n Connecting to all targets configured on a portal [root@host~]# nvme connect-all -t rdma -a -s 4420 c) List the connected targets [root@host~]# nvme list d) Format and mount the NVMe disks shown with the above command. e) Disconnect from the target and unmount the disk: [root@host~]# nvme disconnect -d NOTE: nvme_disk_name is the name of the device (Ex:nvme0n1) and not the device path. HMA ---- To use HMA, please ensure that Unified Wire is installed using the "Unified Wire(Default)" configuration tuning option. Currently 128 IPv4 connections are supported on T6225-SO-CR adapter, provided I/O queues is set to 1 while logging into the target. Example: Connecting to a specific target setting I/O queue=1, which takes 2 queues per target. [root@host~]# nvme connect -i 1 -t rdma -a -s 4420 -n Driver Unloading ================ Follow the steps mentioned below to unload the drivers: On target, run the following commands: [root@host~]# rmmod nvmet-rdma [root@host~]# rmmod nvmet [root@host~]# rmmod iw_cxgb4 On initiator, run the following commands: [root@host~]# rmmod nvme-rdma [root@host~]# rmmod nvme [root@host~]# rmmod iw_cxgb4 NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ LIO iSCSI Target Offload ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Kernel Installation =================== To install 4.9.13 kernel with LIO iSCSI Target Offload enabled, use the following command: [root@host~]# make kernel_install Boot into the new kernel and install Chelsio Unified Wire. In case of 3.14.57 kernel, follow the below steps: a) Download the kernel from kernel.org. b) Untar the tar-ball. c) Change your working directory to kernel package directory and run the following command to invoke the installation menu. [root@host~]# make menuconfig d) Select "Device Drivers" > "Generic Target Core Mod (TCM) and ConfigFS Infrastructure". e) Enable "Linux-iSCSI.org iSCSI Target Mode Stack". f) Select Save. g) Exit from the installation menu. h) Untar the patch file: [root@host~]# cp /root//src/cxgbit/patch/linux_3-14.a . [root@host~]# ar xvf linux_3-14.a i) Apply all the patches to kernel source one by one: [root@host~]# patch -p1 < .patch j) Continue with kernel installation as usual. k) Reboot to the newly installed kernel. Verify by running "uname -a" command. l) Install the Unified Wire package as mentioned in the "Chelsio Unified Wire" section. Driver Installation =================== To install LIO driver and targetcli utils: [root@host~]# make lio_install Driver Loading ============== IMPORTANT: Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Loading Network Driver ---------------------- The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver in NIC mode (without offload support) [root@host~]# modprobe cxgb4 Bring up the interface. [root@host~]# ifconfig ethX up Loading LIO iSCSI Target Offload driver --------------------------------------- Load the LIO iSCSI Target Offload driver (cxgbit) using: [root@host~]# modprobe cxgbit Driver Configuration ==================== Configuring LIO iSCSI Target ----------------------------- The LIO iSCSI Target needs to be configured before it can become useful. Please refer the user manual at http://www.linux-iscsi.org/Doc/LIO Admin Manual.pdf to do so. Offloading LIO iSCSI Connection -------------------------------- To offload the LIO iSCSI Target, use the following command: [root@host~]# echo 1 > /sys/kernel/config/target/iscsi//tpgt_1/np/\:3260/cxgbit Execute the above command for every portal address listening on Chelsio interface. Running LIO iSCSI and Network Traffic Concurrently -------------------------------------------------- If you wish to run network traffic with offload support (TOE) and LIO iSCSI traffic together, follow the steps mentioned below: a) If not done already, load network driver with offload support (TOE): [root@host~]# modprobe t4_tom b) Create a new policy file: [root@host~]# cat c) Add the following lines to offload all traffic except LIO iSCSI: listen && src port && src host => !offload all => offload d) Compile the policy: [root@host~]# cop -d -o e) Apply the policy: [root@host~]# cxgbtool ethX policy Performance Tuning ------------------ The driver package provides tools to auto tune the system for best performance. Please refer the User's Guide for instructions. Driver Unloading ================ Unloading LIO iSCSI Target Offload driver ----------------------------------------- To unload the LIO iSCSI Target Offload driver, follow the steps mentioned below: a) Log out from the initiator b) Run the following command: [root@host~]# echo 0 > /sys/kernel/config/target/iscsi//tpgt_1/np/\:3260/cxgbit Execute the above command for every portal address listening on Chelsio interface. c) Unload the driver: [root@host~]# rmmod cxgbit Unloading Network driver ------------------------ - To unload the driver in NIC mode (without offload support): [root@host~]# rmmod cxgb4 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSCSI PDU Offload Target ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== To install iSCSI-target driver,firmware and utilities: [root@host~]# make iscsi_pdu_target_install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb To load the module, run modprobe as follows: [root@host~]# modprobe chiscsi_t4 Driver Unloading ================ Use the following command to unload the module: [root@host~]# rmmod chiscsi_t4 NOTE:i. While using rpm-tar-ball for installation a. Uninstallation will result into chiscsi.conf file renamed into chiscsi.conf.rpmsave. b. It is advised to take a backup of chiscsi.conf file before you do an uninstallation and installation of new/same unified wire package. As re-installing/upgrading unified-wire package may lead to loss of chiscsi.conf file. ii. Installation/uninstallation using source-tar-ball will neither remove the conf file nor rename it. It will always be intact. However it is recommended to always take a backup of your configuration file for both methods of installation. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSCSI PDU Offload Initiator ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== To install open-iSCSI,iSCSI-initiator,firmware and utilities: [root@host~]# make iscsi_pdu_initiator_install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to loading the driver as a regular user will fail. Load cxgb4i driver using the following command: [root@host~]# modprobe cxgb4i The cxgb4i module registers a new transport class "cxgb4i". If loading of cxgb4i displays "unkown symbols found" error in dmesg, follow the steps mentioned below: a) Kill iSCSI daemon "iscsid" b) View all the loaded iSCSI modules [root@host~]# lsmod | grep iscsi c) Now, unload them using the following command: [root@host~]# rmmod d) Finally reload the cxgb4i driver Driver Unloading ================ [root@host~]# rmmod cxgb4i [root@host~]# rmmod libcxgbi NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Crypto Offload ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Kernel Installation =================== a) Incase of: - 4.9.13 kernel, use the following command to build and install the kernel with Crypto components enabled by default: [root@host~]# make kernel_install - RHEL 7.3, 3.10.0-514.el7 kernel, the kernel needs to be patched with https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4afa5f9617927453ac04b24b584f6c718dfb4f45 to use digests. - Ubuntu 16.04.1, 4.4.0-31-generic kernel, no extra kernel configuration is needed to use inline-TLS. b) Boot into the new kernel and install Chelsio Crypto drivers using the following command: [root@host~]# make crypto_install c) Reboot machine for changes to take effect. Driver Installation =================== To install Crypto driver and Chelsio Openssl modules: [root@host~]# make crypto_install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Co-processor ------------ a) To load Crypto Offload driver in Co-processor mode(chcr), run the following command: [root@host~]# modprobe cxgb4 [root@host~]# modprobe chcr b) Bring up the Chelsio network interface: [root@host~]# ifconfig ethX up Where ethX is the Chelsio interface. Inline ------ To load Crypto Offload driver in Inline mode, load the network driver in TOE mode: [root@host~]# modprobe t4_tom Configuration ============= Co-processor ------------ To view the complete list of supported cryptographic algorithms, use the following command: [root@host~]# cat /proc/crypto|grep -i chcr The following applications can be offloaded by Chelsio Co-processor: - Apache - Dmcrypt - SFTP - OpenVPN - Strongswan To use digests, uncomment the "DIGESTS" line in "af_alg_engine" section of openSSL configuration file, /etc/pki/tls/openssl.cnf: [af_alg_engine] CIPHERS=aes-128-cbc aes-192-cbc aes-256-cbc aes-128-gcm aes-192-gcm aes-256-gcm aes-128-ctr aes-192-ctr aes-256-ctr DIGESTS=sha1 sha224 sha256 sha384 sha512 To verify if Chelsio Co-processor is used by the above applications, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4//crypto Chelsio Crypto Co-processor Stats aes_ops: 1016 digest_ops: 323 aead_ops: 2739611 comp: 2740950 error: 0 Fallback: 9 Inline ------ - Configuring TLS Offload ports To configure TLS offload ports, COP rule for each TCP port must be defined. Follow the steps mentioned below: a) Create a new policy file and add the following line for each TCP port: src or dst port => offload tls mss 32 bind random b) Compile the policy: [root@host~]# cop -d -o c) Apply the policy: [root@host~]# cxgbtool policy - Configuring Chelsio OpenSSL OpenSSL which supports Inline offload is installed as part of Unified Wire package. It is installed in /usr/chssl/bin a) Start TLS offload Server: [root@host~]# cd /usr/chssl/bin [root@host~]# ./openssl s_server -key -cert -accept 443 -cipher AES128-GCM-SHA256 -WWW b) Start TLS offload Client: [root@host~]# cd /usr/chssl/bin [root@host~]# ./openssl s_time -connect : -www / To compile applications using Chelsio OpenSSL library: [root@host~]# export LD_LIBRARY_PATH=/usr/chssl/lib/ [root@host~]# gcc -g -o -lcrypto -lssl -L/usr/chssl/lib/ Driver Unloading ================ To unload Crypto Offload driver in Co-processor mode, run the following command: [root@host~]# rmmod chcr To unload Crypto Offload driver in Inline mode, unload the network driver in TOE mode, as mentioned under "Driver Unloading" in "Network (NIC/TOE)" section. NOTE: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ DCB ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== To build and install all drivers with DCB support: [root@host~]# make dcbx=1 install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Before proceeding, please ensure that Unified Wire is installed with DCB support as mentioned in CLI mode (individual drivers) section of Unified Wire Installer section. The switch ports need to be enabled with DCBX configuration(Class mapping, ETS and PFC). Upon loading the network/storage driver and interface bringup, firmware completes DCBX negotiation with the switch. [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom [root@host~]# ifconfig ethX up [root@host~]# modprobe csiostor The negotiated DCBX parameters can be reviewed at /sys/kernel/debug/cxgb4//dcb_info The storage driver(FCoE Full Offload Initiator) uses the DCBX negotiated parameters (ETS, PFC etc.) Without any further configuration. The network drivers (cxgb4, t4_tom) and iSCSI drivers (cxgb4i, chiscsi) needs further VLAN configuration to be setup, which is explained in the next section "Running NIC & iSCSI Traffic together with DCBx". Running NIC & iSCSI Traffic together with DCBx ============================================== NOTE: Please refer to "iSCSI PDU Offload Initiator" section to configure iSCSI Initiator. Use the following procedure to run NIC and iSCSI traffic together with DCBx enabled. a) Identify the VLAN priority configured for NIC and iSCSI class of traffic on the switch. b) Create VLAN interfaces for running NIC and iSCSI traffic, and configure corresponding VLAN priority. Example: Switch is configured with a VLAN priority of 2 and 5 for NIC and iSCSI class of traffic respectively. NIC traffic is run on VLAN10 and iSCSI traffic is run on VLAN20. Assign proper VLAN priorities on the interface (here eth5), using the following commands on the host machine: [root@host~]# vconfig set_egress_map eth5.10 0 2 [root@host~]# vconfig set_egress_map eth5.20 5 5 NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ FCoE Full Offload Initiator ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== To install FCoE PDU offload target driver: [root@host~]# fcoe_pdu_offload_target_install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver, execute the following command: [root@host~]# modprobe csiostor Configuring the switch and Troubleshooting ========================================== Please refer "Software Configuration and Fine-tuning" section in User's Guide Driver Unloading ================ To unload the driver, execute the following command: [root@host~]# modprobe -r csiostor NOTE:If multipath services are running, unload of FCoE driver is not possible. Stop the multipath service and then unload the driver. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Offload Bonding driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== To install bonding driver and firmware (offload). [root@host~]# make bonding_install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to loading the driver as a regular user will fail. To load the Bonding driver (with offload support), run the following command: [root@host~]# modprobe bonding Offloading TCP traffic over a bonded interface ============================================== The Chelsio Offload Bonding driver supports all the bonding modes in NIC Mode. In offload mode (t4_tom loaded) however, only the balance-rr (mode=0), active-backup (mode=1),balance-xor (mode=2) and 802.3ad (mode=4) modes are supported. To offload TCP traffic over a bonded interface, use the following method: a) Load the network driver with TOE support. [root@host~]# modprobe t4_tom b) Create a bonded interface [root@host~]# modprobe bonding mode=1 miimon=100 c) Bring up the bonded interface and enslave the interfaces to the bond [root@host~]# ifconfig bond0 up [root@host~]# ifenslave bond0 ethX ethY NOTE: "ethX" and "ethY" are interfaces of the same adapter. d) Assign IPv4/IPv6 address to the bonded interface [root@host~]# ifconfig bond0 X.X.X.X/Y [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up e) Disable FRTO on the PEER: [root@host~]# sysctl -w net.ipv4.tcp_frto=0 All TCP traffic will be offloaded over the bonded interface now. Driver Unloading ================ To unload the bonding driver. [root@host~]# rmmod bonding NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Offload Multi-Adapter Failover (MAFO) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Installation ============ To install the MAFO feature: [root@host~]# make bonding_install Multi-Adapter fail-over feature will work for link down events caused by: - Cable unplug on bonded interface - Bringing corresponding switch port down NOTE: The feature will not work if the bonded interfaces are administratively taken down. IMPORTANT: - Portions of this software are covered under US Patent "Failover and migration for full-offload network interface devices : US 8346919 B1" - Use of the covered technology is strictly limited to Chelsio ASIC-based solutions. Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver (with offload support), run the following command: [root@host~]# modprobe bonding Offloading TCP traffic over a bonded interface ============================================== The Chelsio MAFO driver supports only the active-backup (mode=1) mode. To offload TCP traffic over a bonded interface, use the following method: a) Load the network driver with TOE support. [root@host~]# modprobe t4_tom b) Create a bonded interface [root@host~]# modprobe bonding mode=1 miimon=100 c) Bring up the bonded interface and enslave the interfaces to the bond [root@host~]# ifconfig bond0 up [root@host~]# ifenslave bond0 ethX ethY NOTE: "ethX" and "ethY" are interfaces of different adapters. d) Assign IPv4/IPv6 address to the bonded interface [root@host~]# ifconfig bond0 X.X.X.X/Y [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up e) Disable FRTO on the PEER: [root@host~]# sysctl -w net.ipv4.tcp_frto=0 f) Disable TCP timestamps: [root@host~]# sysctl -w net.ipv4.tcp_timestamps=0 All TCP traffic will be offloaded over the bonded interface now and fail-over will happen in case of link-down event. Driver Unloading ================ To unload the driver, run the following command: [root@host~]# rmmod bonding NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ UDP Segmentation Offload and Pacing ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== The offload drivers support UDP Segmentation Offload with limited number of connections (1024 connections).To build and install UDP Offload drivers which support large number of offload connections (approx 10K), [root@host~]# make udp_offload_install NOTE: For more installation options, please refer Unifier Wire section. Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. Run the following commands to load the driver: [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom Though normally associated with the Chelsio TCP Offload engine, the t4_tom module is required in order to allow for the proper redirection of UDP socket calls. Driver Unloading ================ Reboot the system to unload the driver. To unload without rebooting, refer "Unloading the driver" in Network (NIC/TOE) section. NOTE: For more information on additional configuration options, please refer User's Guide. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Offload IPv6 driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ IPv6 must be enabled in your system (enabled by default) to use the Offload IPv6 feature.Also, Unified Wire package must be installed with IPv6 support (see section 4.1 for more information). Installation ============ To build and install all drivers without IPv6 support: [root@host~]# make ipv6_disable=1 install NOTE: For more installation options, please refer Unifier Wire section. NIC & TOE Driver Loading ======================== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb After installing Unified Wire package and rebooting the host, load the NIC (cxgb4) and TOE (t4_tom) drivers. The drivers must be loaded by root user. Any attempt to load the drivers as a regular user will fail. [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom All the IPv6 traffic will be offloaded now. NIC & TOE Driver Unloading ========================== - To unload the NIC driver: [root@host~]# rmmod cxgb4 - To unload the TOE driver: Please reboot the system to unload the TOE driver. To unload without rebooting, refer "Unloading the driver" in Network (NIC/TOE) section. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ WD Sniffing and Tracing ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The objective of these utilities (wd_sniffer and wd_tcpdump_trace) is to provide sniffing and tracing capabilities by making use of Terminator's hardware features. Sniffer- Involves targeting specific multicast traffic and sending it directly to user space. Tracer - All tapped traffic is forwarded to user space and also pushed back on the wire via the internal loop back mechanism In either mode the targeted traffic bypasses the kernel TCP/IP stack and is delivered directly to user space by means of a RX queue which is defined by the register MPS_TRC_RSS_CONTROL. Installation ============ To install Sniffer tracing & filtering tcpdump and iWARP driver: [root@host~]#make sniffer_install NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Classification and Filtering ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1. LE-TCAM Filters ================= Creating Filter Rules --------------------- Network driver (cxgb4) must be installed before setting the filter rule. a) If you haven't done already, run the Unified Wire Installer with the appropriate configuration tuning option to install the Network Driver. b) Load the network driver and bring up the Chelsio interface: [root@host~]# modprobe cxgb4 [root@host~]# ifconfig ethX up c) Now, create filter rules using cxgbtool: [root@host~]# cxgbtool ethx filter action [pass/drop/switch] Where, ethX : Chelsio interface. index : positive integer set as filter id. 0-495 for T5 adapters; 0-559 for T6 adapters. action : Ingress packet disposition. pass : Ingress packets will be passed through set ingress queues. switch : Ingress packets will be routed to an output port with optional header rewrite. drop : Ingress packets will be dropped. prio 1 : Optional for T5. Mandatory for T6 indices 0-63; Should not be added for T6 indices 64-559. hitcnts 1: To enable hit counts in cxgbtool filter show output. NOTE: In case of multiple filter rules, the rule with the lowest filter index takes higher priority. Listing Filter Rules -------------------- To list previously set filters, run the following command: [root@host~]# cxgbtool ethX filter show OR [root@host~]# cat /sys/kernel/debug/cxgb4//filters Removing Filter Rules --------------------- To remove a filter, run the following command with the corresponding filter rule index [root@host~]# cxgbtool ethX filter NOTE:For more information on additional parameters, refer to cxgbtool manual by running the "man cxgbtool" command. 2. Hash/DDR Filters =================== Creating Filter Rules --------------------- Network driver (cxgb4) must be installed and loaded before setting the filter rule. a) If you haven’t done already, run the Unified Wire Installer with the "T5/T6 Hash Filter" configuration tuning option to install the Network Driver. b) Load the network driver with DDR filters support and bring up the Chelsio interface: [root@host~]# modprobe cxgb4 use_ddr_filters=1 [root@host~]# ifconfig ethX up c) Now, create filter rules using cxgbtool: [root@host~]# cxgbtool ethX filter action [pass/drop/switch] fip lip fport lport proto Where, ethX : Chelsio interface. index : Filter index. For LE-TCAM filters, filter index should be 0-495 for T5 adapters and 0-559 for T6 adapters. In case of Hash/DDR filter, the index will be ignored and replaced by an automatically computed value, based on the hash (4-tuple). The index will be displayed after the filter rule is created successfully. action : Ingress packet disposition. pass : Ingress packets will be passed through set ingress queues. switch : Ingress packets will be routed to an output port with. optional header rewrite. drop : Ingress packets will be dropped. source_ip/port : Source IP/port of incoming packet. destination_ip/port : Destination IP/port of incoming packet. protocol : TCP by default. To change, specify the corresponding internet protcol number. E.g. for UDP, use 17. hitcnts 1 : To enable hit counts in cxgbtool filter show output. cap maskless : This is mandatory for hash filter. If not provided, LE-TCAM filter will be created at the specified index. NOTE: In case of Hash/DDR filters, source_ip, destination_ip, source_port and destination_port are mandatory, since the filters don't support masks and hence, 4-tuple must always be supplied. "Proto" is also a mandatory parameter. Listing Filter Rules -------------------- - To list the Hash/DDR filters set, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4//hash_filters - To list both LE-TCAM and Hash/DDR filters set, run the following command: [root@host~]# cxgbtool ethX filter show Removing Filter Rules ---------------------- To remove a filter, run the following command with cap maskless parameter and corresponding filter rule index: [root@host~]# cxgbtool ethX filter cap maskless NOTE: Filter rule index can be determined by referring the "hash_filters" file located in /sys/kernel/debug/cxgb4// NOTE: For more information on additional parameters, refer cxgbtool manual by running the man cxgbtool command. Filter Priority --------------- By default, Hash/DDR filter has priority over LE-TCAM filter. To override this, the LE-TCAM filter should be created with prio option. For example: [root@host~]# cxgbtool ethx filter action prio 1 Where index is a positive integer set as filter id. 0-495 for T5 adapters and 0-63 for T6 adapters. Swap MAC Feature ----------------- Chelsio’s T6/T5 Swap MAC feature swaps packet source MAC and destination MAC addresses. This is applicable only for switch filter rules. Here’s an example: [root@host~]# cxgbtool eth2 filter 100 action switch lip 102.2.2.1 fip 102.2.2.2 lport 5001 fport 14000 hitcnts 1 iport 1 eport 0 swapmac 1 proto 17 cap maskless Hash-Filter Index = 21936 The above example will swap source and destination MAC addresses of UDP packets (matching above 4 tuple) received on adapter port 1 and then switch them to port 0. Traffic Mirroring ----------------- Enabling Mirroring ------------------ To enable traffic mirroring, follow the steps mentioned below: a) If not done already, install Unified Wire with Hash Filter configuration tuning option as mentioned in the Unfied Wire section. b) Enable vnic_id match for filterMode in Hash filter config file, t5-config.txt, located in /lib/firmware/cxgb4/ filterMode = fragmentation, mpshittype, protocol, vnic_id, port, fcoe filterMask = port, protocol, vnic_id c) Unload network driver (cxgb4) and reload it with mirroring enabled. [root@host~]# rmmod cxgb4 [root@host~]# modprobe cxgb4 enable_mirror=1 use_ddr_filters=1 d) The traffic will now be mirrored and received via mirror PF/VF corresponding to each port. Switch Filter with Mirroring ---------------------------- The following example explains the method to switch and mirror traffic simultaneously: a) Obtain the PF and VF values of the incoming port from /sys/kernel/debug/cxgb4//mps_tcam b) Create the desired switch filter rule: [root@host~]# cxgbtool ethX filter 0 fip 102.8.8.2 lip 102.8.8.1 fport 20000 lport 12865 pf 4 vf 64 action switch iport 0 eport 1 cap maskless The above hash filter rule is created to switch traffic from 102.8.8.2,20000 to 102.8.8.1,12865 received on port 0 to be switched out from port 1. The traffic will be switched and simultaneously received on mirror queues and network stack of host as mirroring is enabled. Filtered Traffic Mirroring -------------------------- Once mirroring is enabled, all the traffic received on a physical port will be duplicated. The following example explains the method to filter out the redundant traffic and receive only specific traffic on mirror queues: a) Obtain the mirror PF and VF values from dmesg [root@host~]# dmesg .... ... cxgb4 0000:02:00.4: Port 0 Traffic Mirror PF = 4; VF = 66 cxgb4 0000:02:00.4: Port 1 Traffic Mirror PF = 4; VF = 67 b) Create a DROP-ALL rule as below: [root@host~]# cxgbtool ethX filter 255 pf 4 vf 66 action drop Where, 255 is the last index of available TCAM filters. This will create a catch-all DROP filter for Mirror PF/VF of port 0. Similarly, create DROP filters for rest of Mirror PF/VF. c) Create specific filter rules to allow specific traffic to be received on mirror queues as below: [root@host~]# cxgbtool ethX filter 1 lip 102.8.8.1 fip 102.8.8.2 lport 12865 fport 20000 pf 4 vf 66 action pass Now, the above specific traffic (from 102.8.8.2,20000 to 102.8.8.1,12865) will be received in Mirror receive queues and network stack of host. Packet Tracing and Hit Counters ------------------------------- For T5/T6 LE-TCAM and T6 Hash/DDR filters, hit counters will work simply by adding hitcnts 1 parameter to the filter rule. However, for T5 Hash/DDR filters, you will have to make use of tracing feature and RSS queues. Here’s a step-by-step guide to enable packet tracing and hit counters for Hash/DDR filter rules: a) Load nerwork driver with the following parameters: [root@host~]# modprobe cxgb4 use_ddr_filters=1 enable_traceq=1 b) Configure the required filter rules. c) Enable tracing on adapter. [root@host~]# cxgbtool ethX reg 0x09800=0x13 d) Setup a trace filter [root@host~]# echo tx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace0 Here, "snaplen" is the length in bytes to be captured. NOTE: Use "snaplen=60" in case of IPv6. The above step will trace all the packets transmitting from port1(tx1) to trace filter 0. e) Configure the RSS Queue to receive traced packets. Determine the "RspQ ID" of the queue by looking at "Trace" QType in /sys/kernel/debug/cxgb4//sge_qinfo file [root@host~]# cxgbtool ethX reg 0x0a00c= Now the traced packets can be seen in tcpdump and the hit counters will also increment. Multi-tracing --------------- To enable packet capture or hit counters for multiple Chelsio ports in Tx/Rx direction enable Multi-tracing. Using this we can configure 4 different RSS Queues separately corresponding to 4 trace-filters. a) Enable Tracing as well as MultiRSSFilter [root@host~]# cxgbtool ethX reg 0x09800=0x33 b) Setup a trace filter [root@host~]# echo tx0 snaplen=40 > /sys/kernel/debug/cxgb4//trace0 c) Configure the RSS Queue corresponding to trace0 filter configured above. Determine the "RspQ ID" of the queues by looking at "Trace" QType in /sys/kernel/debug/cxgb4//sge_qinfo file. [root@host~]# cxgbtool ethX reg 0x09808= d) Similarly for other direction and for multiple ports run the follow commands: [root@host~]# echo rx0 snaplen=40 > /sys/kernel/debug/cxgb4//trace1 [root@host~]# echo tx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace2 [root@host~]# echo rx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace3 [root@host~]# cxgbtool ethX reg 0x09ff4= [root@host~]# cxgbtool ethX reg 0x09ffc= [root@host~]# cxgbtool ethX reg 0x0a004= NOTE: Use "snaplen=60" in case of IPv6. NOTE: For more information on additional configuration options, please refer User's Guide. 3. NAT Filtering ================ T6/T5 adapters can change source & destination IP addresses and source & destination port numbers achieving NAT functionality. This feature is supported with both LE-TCAM and Hash filters. NOTE: This feature is only supported with filter action switch. Syntax: [root@host~]# cxgbtool ethX filter action switch fip lip fport lport nat nat_fip nat_lip nat_fport nat_lport Where, ethX : Chelsio interface. source_ip/port : Source IP/port of incoming packet. destination_ip/port : Destination IP/port of incoming packet. new_source_ip/port : Source IP/port to be translated to. new_destination_ip/port: Destination IP/port to be translated to. mode :Combination of IP/port to be translated. "all" will translate all 4-tuple fields. To see other modes, refer cxgbtool manual page. For more information and examples, refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Traffic Management ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1. Driver Loading ================= IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers: [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Traffic Management can be performed on non-offloaded connections as well as on offloaded connections. The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user will fail.Run the following commands to load the TOE driver: [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom 2. Driver Unloading =================== Reboot the system to unload the driver. To unload without rebooting, refer "Unloading the driver" in Network (NIC/TOE) section. 3. Usage ======== Traffic Management of Non-Offloaded Connections ----------------------------------------------- The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for Non-offload connections: a) Load the network driver and bring up the interface [root@host~]# modprobe cxgb4 [root@host~]# ifconfig eth0 up b) Bind connections with destination IP address 192.168.5.3 to NIC TX queue 3 [root@host~]# tc qdisc add dev eth0 root handle 1: multiq [root@host~]# tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dst 192.168.5.3 action skbedit queue_mapping 3 c) Bind the NIC TX queue to class 0 [root@host~]# cxgbtool eth0 sched-queue 3 0 d) Set the appropriate rule for class 0 [root@host~]# cxgbtool eth0 sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 Traffic Management of Offloaded Connections ------------------------------------------- The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for offloaded connections: a) Load the TOE driver and bring up the interface: [root@host~]# modprobe t4_tom [root@host~]# ifconfig eth0 up b) Create a new policy file (say new_policy_file) and add the following line to associate connections with the given scheduling class: src host 102.1.1.1 => offload class 0 NOTE: If no specified rule matches a connection, a default setting will be used which disables offload for that connection. That is, there will always be a final implicit rule following all the rules in the input rule set of: all => !offload c) Compile the policy file using COP [root@host~]# cop -d -o d) Apply the COP policy: [root@host~]# cxgbtool eth0 policy e) Set the appropriate rule for class 0 [root@host~]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 Traffic Management of Offloaded Connections with Modified Application --------------------------------------------------------------------- The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for for offloaded connections with modified application. a) Load the TOE driver and bring up the interface [root@host~]# modprobe t4_tom [root@host~]# ifconfig eth0 up b) Modify the application as mentioned in the Configuring Traffic Management section in the User's Guide. c) Set the appropriate rule for class 0 [root@host~]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 NOTE:For more information on additional parameters, refer cop manual by running the "man cxgbtool" command. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Unified Wire Manager ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1. Installation =============== Chelsio Unified Wire has been designed to install Unified Wire Manager (UM) by default. All the three UM components, i.e. Management Agent, Client and Station, will be installed on selecting any of the Terminator 4/Terminator 5 configuration tuning options during installation. Hence, no separate installation is required. 2. Management Station Configuration =================================== Running Management Station on RHEL 6.x -------------------------------------- a) Start/Restart Apache httpd daemon: [root@host~]# service httpd start/restart b) Start/Restart the Management Station: [root@host~]# /etc/init.d/chelsio-mgmtstd start/restart Running Management Station on SLES11SP3 --------------------------------------- a) On SLES11SP3, Management Station needs to be configured before running. Hence, execute the following commands and provide valid inputs. [root@host~]# cd /etc/apache2 [root@host~]# openssl genrsa -des3 -out server.key 1024 [root@host~]# openssl req -new -key server.key -out server.csr [root@host~]# cp server.key server.key.org [root@host~]# openssl rsa -in server.key.org -out server.key [root@host~]# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt [root@host~]# cp server.crt ./ssl.crt [root@host~]# cp server.key ./ssl.key b) Start/Restart Apache services [root@host~]# rcapache2 start/restart c) Start/Restart the Management Station: [root@host~]# /etc/init.d/chelsio-mgmtstd start/restart 3. Un-Installation =================== Use the following query command to determine the name of the agent/client RPM [root@host~]# rpm -qa | grep uwire Now,Uninstall the RPM using the following syntax: [root@host~]# rpm -e 4. Verifying Agent/Client status ================================ a) Use following command to check whether Agent is running. [root@host~]# /etc/init.d/Chelsio-uwire_mgmtd status b) Use the following query command to determine if Management Client is installed: [root@host~]# chelsio_uwcli -V The above query should confirm that Management Client is installed by displaying a similar result: Unified Manager client CLI version : 2.x.yy 5. Agent/Station Start/Stop/restart =================================== - Use following command to start, stop or restart Agent: [root@host~]# /etc/init.d/Chelsio-uwire_mgmtd - Use following command to start, stop or restart Management station: [root@host~]# /etc/init.d/chelsio-mgmtstd 6. Client Usage =============== Use the following commands to view Management client Help file. [root@host~]# chelsio_uwcli /? NOTE: For a detailed explanation on usage and configuration of various UM components, please refer the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Unified Boot Software ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1. Flashing firmware & option ROM ================================= Depending on the boot mode selected, Chelsio Unified Boot provides two methods to flash firmware and option ROM onto Chelsio adapters: Flash utility "cfut4" for Legacy mode and "HII" for uEFI mode. Both methods also provide the functionality to update/erase Hardware configuration and Phy Firmware files. Preparing USB flash drive ------------------------- This document assumes that you are using an USB flash drive as a storage media for the necessary files. Follow the steps below to prepare the drive: a) Create a DOS bootable USB flash drive. b) Create a directory "CHELSIO" on USB flash drive. c) If you haven't done already, download Chelsio Unified Wire driver package from Chelsio Download Center, service.chelsio.com d) Untar the downloaded package and change your working directory to "OptionROM" directory. [root@host~]# tar zxvf .tar.gz [root@host~]# cd /Uboot/OptionROM e) Copy all the files and place them in the CHELSIO directory created on the USB flash drive. f) Plug-in the USB flash drive in the system on which the Chelsio CNA is installed. g) Reboot the system and enter the system BIOS setup. h) Make the USB flash drive as the primary boot device. i) Save the changes and exit. Legacy ------ a) Configure the system having Chelsio CNA to boot in Legacy mode. b) Once the system boots from the USB flash drive, change your working directory to CHELSIO directory: C:\>cd CHELSIO c) Run the following command to list all Chelsio CNAs present in the system. The list displays a unique index for each CNA found. C:\CHELSIO>cfut4 -l d) Delete any previous version of Option ROM flashed onto the CNA: C:\CHELSIO>cfut4 -d -xb Here, idx is the CNA index found in step (c) e) Delete any previous firmware using the following command: C:\CHELSIO>cfut4 -d -xh -xf f) Delete any previous Option ROM settings: C:\CHELSIO>cfut4 -d -xc g) Run the following command to flash the appropriate firmware. C:\CHELSIO>cfut4 -d -uf .bin h) Flash the Option ROM onto the Chelsio CNA using the following command: C:\CHELSIO>cfut4 -d -ub cubt4.bin Here, "cubt4.bin" is the unified option ROM image file present in the CHELSIO directory. i) Reboot the system for changes to take effect. j) To configure the base MAC address (optional), use the below command: C:\CHELSIO>cfut4 -d -um Here, idx is the CNA index found in step (c) For example, C:\CHELSIO>cfut4 -d 0 -um 000743000123 uEFI ----- To configure Chelsio CNA using HII in uEFI mode, please refer User's Guide. Default Option ROM Settings -------------------------------- If you wish to restore option ROM settings to their default values, i.e., PXE enabled, iSCSI and FCoE disabled, use any of the methods mentioned below: - Using Option ROM (boot level) For Legacy PXE, boot system into Chelsio’s Unified Boot Setup utility and press F8. For uEFI PXE, boot system into uEFI mode and press F3. - Using cxgbtool (OS level) Change your working directory to OptionROM directory and use cxgbtool to flash the default boot configuration onto the adapter: [root@host~]# cd /Uboot/OptionROM/ [root@host~]# cxgbtool loadboot-cfg bootcfg 2. Driver Update Disk (DUD) ============================ The following section describes the procedure to create Driver Update Disks for RHEL and SLES distributions, for Chelsio adapters. In case of T4 adapters, you can skip this step and use inbox drivers to install the operating system. Creating Driver Disk for RedHat Enterprise Linux ------------------------------------------------ a) If you haven't done already, download Chelsio Unified Wire driver package from Chelsio Download Center, service.chelsio.com b) Untar the package. [root@host~]# tar zxvf .tar.gz c) Change your working directory to "LinuxDUD" directory. [root@host~]# cd /Uboot/LinuxDUD d) Insert a blank, formatted USB flash drive. e) Depending on the distribution to be installed, copy the corresponding image file to the USB drive.For example, execute the following command for RHEL 6.6 [root@host~]# cp Chelsio-DriverUpdateDisk-RHEL6.6-x86_64-x.xx.x.x.img NOTE: For RHEL 7.X, use Chelsio-DriverUpdateDisk-RHEL7.X-x86_64-x.xx.x.x.iso Creating Driver Disk for Suse Enterprise Linux ---------------------------------------------- a) If you haven't done already, download Chelsio Unified Wire driver package from Chelsio Download Center, service.chelsio.com b) Untar the package [root@host~]# tar zxvf .tar.gz c) Insert a blank USB drive. d) Format the USB drive [root@host~]# mkfs.vfat /dev/sda1 e) Depending on the distribution to be installed, copy the corresponding image file to the USB drive.For example, execute the following command for SLES11sp4: [root@host~]# dd if=/root//Uboot/LinuxDUD/Chelsio-DriverUpdateDisk-SLES11sp4-x86_64-x.xx.x.x.img of=/dev/sda1 IMPORTANT:Please make sure the switch is upgraded to the latest available firmware, before proceeding with the operating system installation. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ DPDK ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ IMPORTANT: Please ensure that any existing version of DPDK driver is uninstalled before proceeding. Driver Installation =================== - To install DPDK source, testpmd,tools and Pktgen: [root@host~]# make dpdk_install NOTE: - The above command will install the driver with the following build configuration: arch: x86_64/ppc_64 machine: native/power8 execenv: linuxapp toolchain: gcc - The above commands will create a target environment directory "x86_64-native-linuxapp-gcc/ppc_64-power8-linuxapp-gcc" in the driver package directory. - The pktgen application has been modified to disable packet classification, to reflect correct Rx rate as packet classification is an expensive operation. Flashing Firmware Configuration File ==================================== a) If not done already, load the kernel mode NIC driver (cxgb4): [root@host~]# modprobe cxgb4 b) Flash the firmware configuration file (for T6 adapters): [root@host~]# cxgbtool loadcfg /lib/firmware/cxgb4/t6-config.txt NOTE: For T5 adapters, use t5-config.txt. Driver Loading ============== NOTE: Before proceeding, unload all Chelsio inbox and outbox kernel mode drivers using the following commands: [root@host~]# rmmod cxgb4 [root@host~]# rmmod csiostor The driver must be loaded by the root user. Any attempt to load as a regular user will fail. UIO support ----------- Follow the steps mentioned below to load DPDK driver with UIO support: a) Disable Intel VT-d in system BIOS. b) Turn off Intel iommu by adding below entry in Kernel grub/grub2 menu: intel_iommu=off c) Reboot the system for changes to take effect. d) Load the UIO module: [root@host~]# modprobe uio [root@host~]# insmod Chelsio-DPDK-x.x.x.x/src/DPDK/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko VFIO support ------------- Follow the steps mentioned below to load DPDK driver with VFIO support: a) Enable Intel VT-d in system BIOS. b) Add the following entry to grub/grub2 menu: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 E.g.: kernel /vmlinuz-3.13.0-32-generic ro root=UUID=5149fae1-c52b-42a9-a48c-b0a70937e8fb intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet c) Reboot the system for changes to take effect. d) Load the VFIO module: [root@host~]# modprobe vfio-pci Configuration ============= Huge Pages ----------- Using script ------------ Run the "res_hugepages.sh" shell script (copied to /sbin during installation): [root@host~]# res_hugepages.sh Manual ------ a) Mount hugetlbfs: [root@host~]# mkdir -p /mnt/huge [root@host~]# mount -t hugetlbfs nodev /mnt/huge b) Reserve Huge Page memory manually: - x86_64: [root@host~]# echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages - POWERPC64: [root@host~]# echo 512 > /sys/devices/system/node/node0/hugepages/hugepages-16384kB/nr_hugepages In case of dual socket machines, run the above command for other CPU nodes. Binding network ports --------------------- NOTE: Please make sure that adapter's physical function 4 is used, since it is assigned for all NIC functions of the adapter. - Run the following command to bind network ports to DPDK environment with UIO support: [root@host~]# dpdk_nic_bind.py --bind=igb_uio Now, Verify using: [root@host~]# dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ 0000:03:00.4 'T6225-CR Unified Wire Ethernet Controller' drv=igb_uio unused= - Run the following command to bind network ports to DPDK environment with VFIO support: [root@host~]# dpdk_nic_bind.py --bind=vfio-pci Now, Verify using: [root@host~]# dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ 0000:03:00.4 'T6225-CR Unified Wire Ethernet Controller' drv=vfio-pci unused= Unbinding network ports ----------------------- a) Run the following command to unbind network ports from DPDK environment: [root@host~]# dpdk_nic_bind.py -u b) Run the following command to verify: [root@host~]# dpdk_nic_bind.py --status Running DPDK Test Applications ============================== For information(syntax,examples) regarding test applications Testpmd and Pktgen, and configuring RSS, Flow Control and Jumbo frames, please refer user's guide. Driver Unloading ================ UIO support ------------ Run the following commands to unload DPDK driver with UIO support: [root@host~]# rmmod igb_uio [root@host~]# rmmod uio If unloading "uio" module reports an error, unload the following dependent modules and try again: [root@host~]# rmmod bnx2fc [root@host~]# rmmod bnx2i [root@host~]# rmmod cnic VFIO support ------------ Run the following command to unload DPDK driver with VFIO support: [root@host~]# rmmod vfio-pci Driver Uninstallation ===================== Uninstall the DPDK driver,testpmd and Pktgen tools, and all the tools in the DPDK depository: [root@host~]# make dpdk_uninstall ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ OVS Kernel Datapath Offload ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== To install OVS modules and NIC driver with offload support: [root@host~]# make ovs_install Configuration ============= Before proceeding: a) Please ensure that Unified Wire is installed with "Hash Filter (T5/T6)" configuration tuning option before proceeding. b) Edit the hash config file in /lib/firmware/cxgb4/. Use "t6-config.txt" for T6 adapters and "t5-config.txt" for T5 adapters: filterMode = fragmentation,mpshittype,macmatch,ethertype,protocol,port filterMask = ethertype,protocol,port Configuring OVS Machine ----------------------- The following example explains the procedure to configure an OVS machine: ------------ -------- | OVS Switch | -------- | HOST A | ------->| eth2 eth3 |<------- | HOST B | -------- ------------ -------- "eth2" and "eth3" are Chelsio interfaces. a) Load NIC (cxgb4) driver with hash-filter support: [root@host~]# modprobe cxgb4 use_ddr_filters=1 b) Bring up the Chelsio interfaces in promiscous mode: [root@host~]# ifconfig eth2 promisc up [root@host~]# ifconfig eth3 promisc up c) Load Open vSwitch module: [root@host~]# modprobe openvswitch d) Configure OVS [root@host~]# ovs-appctl exit [root@host~]# pkill -9 ovs [root@host~]# rm -rf /usr/local/etc/ovs-vswitchd.conf [root@host~]# rm -rf /usr/local/var/run/openvswitch/db.sock [root@host~]# rm -rf /usr/local/etc/openvswitch/conf.db [root@host~]# touch /usr/local/etc/ovs-vswitchd.conf [root@host~]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db /src/openvswitch-x.x.x/vswitchd/vswitch.ovsschema [root@host~]# ovsdb-server /usr/local/etc/openvswitch/conf.db --remote=punix:/usr/local/var/run/openvswitch/db.sock \ --remote=db:Open_vSwitch,Open_vSwitch,manager_options --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --log-file [root@host~]# ovs-vsctl --no-wait init [root@host~]# export DB_SOCK=/usr/local/var/run/openvswitch/db.sock [root@host~]# ovs-vswitchd --pidfile --detach NOTE: Ports on OVS bridge must be added in the same order as the adapter, since there's no mapping between OVS and physical ports. e) Create an OVS brige and add Chelsio interfaces to it: [root@host~]# ovs-vsctl add-br br0 [root@host~]# sleep 2 [root@host~]# ifconfig br0 up [root@host~]# ovs-vsctl add-port br0 eth2 [root@host~]# sleep 5 [root@host~]# ovs-vsctl add-port br0 eth3 [root@host~]# sleep 5 [root@host~]# ovs-vsctl show f) Now ping from Host A to Host B to verify that OVS is configured successfully. g) Stop the ping traffic and delete all the flows on switch: [root@host~]# ovs-ofctl del-flows br0 h) Create a new flow based on your requirement. In this case, we have configured to drop IPv4 TCP packets from host A (source IP 10.1.1.56) to host B (destination IP 10.1.1.62) [root@host~]# ovs-ofctl add-flow br0 in_port=1,dl_type=0x800,nw_proto=6,nw_src=10.1.1.56,nw_dst=10.1.1.66,action=drop i) Verify the flow dump: [root@host~]# ovs-ofctl dump-flows br0 Start traffic from Host A to Host B and observe that the packets which match the flow will be dropped. Setting Up ODL with OVS ----------------------- The following example explains the method to set up OpenDaylight(ODL) using OVS: ------------ | ODL | | Controller | ------------ ^ | |(private network) ------------ -------- | OVS_Switch | -------- | HOST A | ------->| eth2 eth3 |<------- | HOST B | -------- ------------ -------- On the ODL controller setup, a) Download latest Java Development Kit. b) Untar the tar file. c) Create an entry in .bashrc which points to the extracted folder: export JAVA_HOME=/jdk1.8.0_92 export PATH=$PATH:$JAVA_HOME d) Logout & log back in. e) Download ODL controller pre-built zip package. f) Unzip the package and change your working directory to "opendaylight" g) Run the script "run.sh" and wait for ~3 minutes for the controller to be setup. h) Open a web browser and enter the address http://localhost:8080 i) Login with admin keyword for both username and password. j) On the OVS machine, add the bridge to the controller and disable in-band: [root@host ~]# ovs-vsctl set-controller br0 tcp::6633 [root@host ~]# ovs-vsctl set bridge br0 other-config:disable-in-band=true k) Refresh the webpage on the ODL controller and you should see the OVS details. l) Goto "Flows" tab, add and install a flow. m) Verify the flow dump on the OVS machine: [root@host ~]# ovs-ofctl dump-flows br0 Run traffic between hosts which matches the flow and verify if the "n_packet" counter is incrementing. Configuring VXLAN using OVS ---------------------------- The following steps describe the method to configure VXLAN using OVS with single port on Server and Client machines. Server ------ a) Edit the hash-filter config file as: filterMode = fragmentation, mpshittype, ethertype, vnic_id, port filterMask = ethertype, vnic_id, port vnicMode = encapsulation b) Load NIC (cxgb4) driver with hash-filter support: [root@host~]# modprobe cxgb4 use_ddr_filters=1 c) Bring up the Chelsio interfaces in promiscous mode: [root@host~]# ifconfig ethX promisc up [root@host~]# ifconfig ethY promisc up d) Load Open vSwitch module: [root@host~]# modprobe openvswitch e) Configure OVS: [root@host~]# ovs-appctl exit [root@host~]# pkill -9 ovs [root@host~]# rm -rf /usr/local/etc/ovs-vswitchd.conf [root@host~]# rm -rf /usr/local/var/run/openvswitch/db.sock [root@host~]# rm -rf /usr/local/etc/openvswitch/conf.db [root@host~]# touch /usr/local/etc/ovs-vswitchd.conf [root@host~]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db /src/openvswitch-x.x.x/vswitchd/vswitch.ovsschema [root@host~]# ovsdb-server /usr/local/etc/openvswitch/conf.db --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --log-file [root@host~]# ovs-vsctl --no-wait init [root@host~]# export DB_SOCK=/usr/local/var/run/openvswitch/db.sock [root@host~]# ovs-vswitchd --pidfile --detach [root@host~]# ovs-vsctl add-br br0 [root@host~]# ifconfig br0 /24 up [root@host~]# ovs-vsctl add-port br0 -- set interface type=vxlan options:remote_ip= options:local_ip= options:key=flow [root@host~]# ovs-vsctl add-port br0 f) Disable GRO on the chelsio adapter,bridge and VXLAN interfaces: [root@host~]# ethtool -K gro off [root@host~]# ethtool -K gro off [root@host~]# ethtool -K br0 gro off [root@host~]# ethtool -K vxlan_sys_* gro off g) Set a rule to match packets with VNI=42 and drop: [root@host~]# ovs-ofctl add-flow br0 in_port=2,tun_id=0x2a,action=drop Client ------ a) Follow steps (a)-(f) described in the the previous section ("Server"). b) Set a rule to set VNI to 42 and send traffic: [root@host~]# ovs-ofctl add-flow br0 in_port=LOCAL,action=set_tunnel:0x2a,output:2 Driver Uninstallation ===================== [root@host~]# make ovs_uninstall 5. Support Documentation ================================================================================ The documentation for this release can be found inside the /docs folder. It contains: - README - Release Notes - User's Guide 6. Customer Support ================================================================================ Please contact Chelsio support at support@chelsio.com for any issues regarding the product. ******************************************************************************** Copyright (C) 2017 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system,or transmitted in any form or by any means without the express written consent of Chelsio Communications.