**************************************** README **************************************** Chelsio Unified Wire for Linux Version : 3.19.0.3 Date : 11/22/2024 Overview ================================================================================ Chelsio Unified Wire software for Linux is an easy to use utility developed to provide installation of 64-bit Linux based drivers and tools for Chelsio's Unified Wire adapters. The Chelsio Unified Wire package provides an interactive installer to install various drivers and utilities. Chelsio Unified Wire consists of the following components: - Network (NIC/TOE) - Virtual Function Network (vNIC) - iWARP RDMA Offload - iSER (Target and Initiator) - WD-UDP - NVMe-oF iWARP (Target and Initiator) - SPDK NVMe-oF iWARP (Target and Initiator) - NVMe-oF TOE (Target and Initiator) - SPDK NVMe-oF TOE Target - SoftiWARP Initiator - LIO iSCSI Target Offload - iSCSI PDU Offload Target - iSCSI PDU Offload Initiator - Crypto Offload - Data Center Bridging (DCB) - FCoE Full Offload Initiator - Offload Bonding - Offload Multi-Adapter Failover (MAFO) - UDP Segmentation Offload and Pacing - Offload IPv6 - WD Sniffing and Tracing - Classification and Filtering - OVS Kernel Datapath Offload - Mesh Topology - Traffic Management feature (TM) - Unified Boot Software - Utility Tools (cop,cxgbtool,t4_perftune,benchmark tools) ================================================================================ CONTENTS ================================================================================ - 1. Supported Operating Systems - 2. Supported Hardware - 3. How To Use - 4. Support Documentation - 5. Customer Support 1. Supported Operating Systems ================================================================================ The Chelsio Unified Wire software has been developed to run on 64-bit Linux based platforms. Following is the list of Drivers/Software and supported Linux distributions. x86_64 Architecture =================== |############################|#################################################| | Linux Distribution | Driver/Software | |############################|#################################################| |RHEL/Rocky/AlmaLinux 9.3, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | |5.14.0-362.8.1.el9_3.x86_64 |SPDK NVMe-oF iWARP*,NVMe-oF TOE,SPDK NVMe-oF TOE*| | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,IPv6 | | |Crypto,DCB,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |Sniffer & Tracer,Filtering,Mesh,TM,uBoot** | |----------------------------|-------------------------------------------------| |RHEL/Rocky/AlmaLinux 9.2, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | |5.14.0-284.11.1.el9_2.x86_64|SPDK NVMe-oF iWARP*,NVMe-oF TOE,SPDK NVMe-oF TOE*| | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,IPv6 | | |Crypto,DCB,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |Sniffer & Tracer,Filtering,Mesh,TM,uBoot** | |----------------------------|-------------------------------------------------| |RHEL/Rocky/AlmaLinux 8.9, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | |4.18.0-513.5.1.el8_9.x86_64 |SPDK NVMe-oF iWARP^,NVMe-oF TOE,SPDK NVMe-oF TOE^| | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,IPv6 | | |Crypto,DCB,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |Sniffer & Tracer,Filtering,Mesh,TM,uBoot** | |----------------------------|-------------------------------------------------| |RHEL/Rocky/AlmaLinux 8.8, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | |4.18.0-477.10.1.el8_8.x86_64|SPDK NVMe-oF iWARP^,NVMe-oF TOE,SPDK NVMe-oF TOE^| | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,IPv6, | | |Crypto,DCB,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |Sniffer & Tracer,Filtering,Mesh,TM,uBoot** | |----------------------------|-------------------------------------------------| |RHEL 7.9, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP,DCB,| |3.10.0-1160.el7.x86_64 |SPDK NVMe-oF iWARP,SPDK NVMe-oF TOE,UDP-SO,IPv6, | | |LIO iSCSI Target,iSCSI Initiator,FCoE Initiator, | | |Crypto (Co-processor IPsec),Bonding,MAFO,TM,OVS, | | |Filtering,Sniffer & Tracer,Mesh,uBoot** | |----------------------------|-------------------------------------------------| |SLES 15 SP4, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | |5.14.21-150400.22-default |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,IPv6, | | |Crypto,DCB,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |Sniffer & Tracer,Filtering,Mesh,TM,uBoot** | |----------------------------|-------------------------------------------------| |Ubuntu 22.04.5, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | |5.15.0-125-generic |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,IPv6, | | |Crypto,DCB,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |Sniffer & Tracer,Filtering,Mesh,TM | |----------------------------|-------------------------------------------------| |Ubuntu 20.04.6, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | |5.4.0-146-generic |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,IPv6, | | |Crypto,DCB,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |Sniffer & Tracer,Filtering,Mesh,TM | |----------------------------|-------------------------------------------------| |Debian 12.7, |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP,TM, | |6.1.0-25-amd64 |SPDK NVMe-oF iWARP,NVMe-oF TOE,Bonding,MAFO,DCB, | | |LIO iSCSI Target,iSCSI Initiator,Crypto,UDP-SO, | | |FCoE Initiator,Sniffer & Tracer,Filtering,IPv6, | | |Mesh | |----------------------------|-------------------------------------------------| |Kernel.org linux-6.6.60 |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,DCB, | | |Crypto^^,FCoE Initiator,Bonding,MAFO,UDP-SO,IPv6,| | |Sniffer & Tracer,Filtering,Mesh,TM | |----------------------------|-------------------------------------------------| |Kernel.org linux-6.1.116 |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Initiator,DCB, | | |Crypto^^,FCoE Initiator,Bonding,MAFO,UDP-SO,IPv6,| | |Sniffer & Tracer,Filtering,Mesh,TM | |------------------------------------------------------------------------------| * Not supported on Rocky/AlmaLinux ^ Not supported on AlmaLinux ^^ Kernel compiled on RHEL 9.X/8.X ** PXE,FCoE,iSCSI NOTE: Other kernel versions have not been tested and are not guaranteed to work. POWERPC64 Architecture ====================== |#########################|####################################################| | Linux Distribution | Driver/Software | |#########################|####################################################| |RHEL 7.6 (POWER8), |NIC/TOE,iWARP,LIO iSCSI Target,iSCSI Initiator, | |3.10.0-957.el7.ppc64le |Bonding,MAFO,IPv6,Filtering,TM | |-------------------------|----------------------------------------------------| |RHEL 7.5 (POWER8), |NIC/TOE,iWARP,LIO iSCSI Target,iSCSI Initiator, | |3.10.0-862.el7.ppc64le |Bonding,MAFO,IPv6,Filtering,TM | |------------------------------------------------------------------------------| NOTE: Other kernel versions have not been tested and are not guaranteed to work. ARM64 Architecture ================== |#########################|####################################################| | Linux Distribution | Driver/Software | |#########################|####################################################| |RHEL 7.6 (ARM64), |NIC/TOE,iSER,NVMe-oF iWARP,LIO iSCSI Target,Bonding,| |4.14.0-115.el7a.aarch64 |MAFO,iSCSI Initiator,Crypto (Co-processor IPsec), | | |IPv6,Filtering,TM | |-------------------------|----------------------------------------------------| |RHEL 7.5 (ARM64), |NIC/TOE,iWARP,iSER,NVMe-oF iWARP,LIO iSCSI Target, | |4.14.0-49.el7a.aarch64 |iSCSI Initiator,Crypto (Co-processor IPsec),Bonding,| | |MAFO,IPv6,Filtering,TM | |------------------------------------------------------------------------------| NOTE: Other kernel versions have not been tested and are not guaranteed to work. 2. Supported Hardware ================================================================================ Chelsio Drivers/Software and supported adapters =============================================== |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T62100-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,Crypto,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T62100-LP-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,Crypto,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T62100-SO-CR |NIC,vNIC,SoftiWARP,Crypto(Co-processor),DCB, | | |Filtering^,OVS^,Mesh(NIC),TM(NIC),uBoot(PXE) | |------------------------|-----------------------------------------------------| |T62100-SO-OCP3 |NIC,vNIC,SoftiWARP,Crypto(Co-processor),DCB, | | |Filtering^,OVS^,Mesh(NIC),TM(NIC),uBoot(PXE) | |------------------------|-----------------------------------------------------| |T6425-CR** |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,Crypto,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T6225-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,Crypto,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T6225-LL-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,Crypto,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T6225-OCP3 |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP,TM | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,Crypto,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,uBoot(PXE) | |------------------------|-----------------------------------------------------| |T580-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T580-LP-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T580-SO-CR |NIC,vNIC,SoftiWARP,DCB,Filtering^,OVS^,Mesh(NIC), | | |TM(NIC),uBoot(PXE) | |------------------------|-----------------------------------------------------| |T580-OCP-SO |NIC,vNIC,SoftiWARP,DCB,Filtering^,OVS^,Mesh(NIC), | | |TM(NIC),uBoot(PXE) | |------------------------|-----------------------------------------------------| |T540-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T540-LP-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T540-SO-CR |NIC,vNIC,SoftiWARP,DCB,Filtering^,OVS^,Mesh(NIC), | | |TM(NIC) | |------------------------|-----------------------------------------------------| |T540-BT |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T520-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T520-LL-CR |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------|-----------------------------------------------------| |T520-SO-CR |NIC,vNIC,SoftiWARP,DCB,Filtering^,OVS^,Mesh(NIC), | | |TM(NIC),uBoot(PXE) | |------------------------|-----------------------------------------------------| |T520-OCP-SO |NIC,vNIC,SoftiWARP,DCB,Filtering^,OVS^,Mesh(NIC), | | |TM(NIC),uBoot(PXE) | |------------------------|-----------------------------------------------------| |T520-BT |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE, | | |SoftiWARP,LIO iSCSI Target,iSCSI Target,DCB, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO,UDP-SO, | | |IPv6,Sniffer & Tracer,Filtering,OVS,Mesh,TM,uBoot* | |------------------------------------------------------------------------------| * PXE,FCoE,iSCSI ** All 4-ports of T6425-CR adapter will be functional only if PCIe x8 -> 2x PCIe x4 slot bifurcation is supported by the system and enabled in BIOS. Otherwise, only 2-ports will be functional. ^ Hash filter not supported. Memory-free Adapters ==================== |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T6225-SO-CR |NIC/TOE,vNIC,iWARP*,iSER Initiator*,iSCSI Initiator*,| | |NVMe-oF iWARP Initiator*,SoftiWARP Initiator,DCB, | | |NVMe-oF TOE Initiator*,Crypto(Co-processor),Mesh*, | | |Filtering^,OVS^,TM(NIC),WD-UDP,uBoot(PXE) | |------------------------|-----------------------------------------------------| |T6225-OCP3 |NIC/TOE,vNIC,iWARP*,iSER Initiator*,iSCSI Initiator*,| | |NVMe-oF iWARP Initiator*,SoftiWARP Initiator,DCB, | | |NVMe-oF TOE Initiator*,Crypto(Co-processor),Mesh*, | | |Filtering^,OVS^,TM(NIC),WD-UDP,uBoot(PXE) | |------------------------------------------------------------------------------| * 256 IPv4/128 IPv6 offload connections supported. ^ Hash filter not supported. Unified Boot Software ===================== Supported hardware platforms ---------------------------- - DELL PowerEdge R610 - DELL PowerEdge R720 - IBM X3650 M4* - Lenovo X3650 M5 - Lenovo ThinkSystem SR650 - HP ProLiant DL180 gen9 - HP ProLiant DL385G2 - Supermicro X10DRi - Supermicro X11SSL-CF - ASUS z390 - QuantaGrid D51B-1U - AMD EPYC 7551 * If the system BIOS version is lower than 1.5 and both Legacy and uEFI are enabled, upgrade to version 1.5 or higher to avoid system hang during POST. Supported Switches ------------------- - Cisco Nexus 5010 with 5.1(3)N1(1a) firmware - Arista DCS-7124S-F - Mellanox SX_PPC_M460EX NOTE: Other platforms/switches have not been tested and are not guaranteed to work. 3. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chelsio Unified Wire ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== - Package Manager yum/apt/zypper should be configured using any of the OS recommended ways to resolve and install missing packages. - make, gcc, kernel-devel and kernel-headers packages should be installed for the compilation of drivers and utilities. - python3 should be installed for Chelsio Unified Wire package scripts to run. In case of older distributions which do not support python3, source package CLI (make) should be used for the installation. Installing Chelsio Unified Wire =============================== There are two main methods to install the Chelsio Unified Wire package: From source and from RPM. If you decide to use source, you can install the package using CLI or GUI mode. If you decide to use RPM, you can install the package using Menu or CLI mode. RPM packages support only distro base kernels. In case of updated/custom kernels, use source package. The following table describes the various "configuration tuning options" available during installation and drivers/software installed with each option by default: |#############################|################################################| |Configuration Tuning Option | Driver/Software installed | |#############################|################################################| |Unified Wire (Default) |NIC/TOE,vNIC,iWARP,iSER,WD-UDP,NVMe-oF iWARP, | | |SPDK NVMe-oF iWARP,NVMe-oF TOE,SPDK NVMe-oF TOE,| | |SoftiWARP,LIO iSCSI Target,iSCSI Target, | | |iSCSI Initiator,FCoE Initiator,Bonding,MAFO, | | |UDP-SO,IPv6,Sniffer & Tracer,Filtering,Mesh,TM | |-----------------------------|------------------------------------------------| |Low latency Networking |TOE,iWARP,WD-UDP,IPv6,Bonding,MAFO | |-----------------------------|------------------------------------------------| |High capacity RDMA |iWARP | |-----------------------------|------------------------------------------------| |RDMA Performance |iWARP,iSER,NVMe-oF iWARP | |-----------------------------|------------------------------------------------| |High capacity TOE |TOE,Bonding,MAFO,IPv6 | |-----------------------------|------------------------------------------------| |iSCSI Performance (T5) |LIO iSCSI Target,iSCSI Target,iSCSI Initiator, | | |Bonding,DCB | |-----------------------------|------------------------------------------------| |UDP Seg.Offload & Pacing (T5)|UDP-SO,Bonding | |-----------------------------|------------------------------------------------| |Wire Direct Latency |TOE,iWARP,WD-UDP | |-----------------------------|------------------------------------------------| |High Capacity WD |WD-UDP | |-----------------------------|------------------------------------------------| |High Capacity Hash Filter |Filtering | |-----------------------------|------------------------------------------------| |NVMe Performance (T6) |iWARP,NVMe-oF iWARP,SPDK NVMe-oF iWARP | |-----------------------------|------------------------------------------------| |High Capacity VF |NIC,vNIC | |------------------------------------------------------------------------------| IMPORTANT: Crypto, DCB and OVS drivers will not be installed by default. Refer to the respective sections for instructions on installing them. Mounting debugfs ---------------- All the driver debug data is stored in debugfs, which will be mounted in most cases. If not, mount it manually. [root@host~]# mount -t debugfs none /sys/kernel/debug Configuring IPv6 ---------------- The interfaces should come up with a link-local IPv6 address for complete and fully functional IPv6 configuration. Update the Interface network-script with ONBOOT="yes". Installation ------------ Follow the steps mentioned below for installation using CLI. For GUI or Menu based installation, refer the User's Guide. 1.1. From source ---------------- 1. Download Chelsio Unified Wire driver package. 2. Extract the downloaded package. [root@host~]# tar zxvfm ChelsioUwire-x.x.x.x.tar.gz 3. Change your current working directory to the Chelsio Unified Wire package directory and build the source. [root@host~]# cd ChelsioUwire-x.x.x.x [root@host~]# make 4. Install the drivers, tools and libraries. [root@host~]# make install 5. The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= install NOTE: To view the different configuration tuning options, view help by typing [root@host~]# make help 6. Reboot your machine for changes to take effect. IMPORTANT: Steps 3 and 4 mentioned above does NOT install Crypto, DCB, OVS drivers and benchmark tools. They have to be installed manually. Refer to their respective sections for instructions on installing them. Installation using additional flags ----------------------------------- Provided here are steps to build and install drivers using additional flags. For the complete list, view help by running "make help". Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x - To build and install all drivers without IPv6 support. [root@host~]# make ipv6_disable=1 [root@host~]# make ipv6_disable=1 install - The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= - To build and install drivers along with benchmarks. [root@host~]# make BENCHMARKS=1 [root@host~]# make BENCHMARKS=1 install - The drivers will be installed as RPMs or Debian packages (for Ubuntu/Debian). To skip this and install drivers, [root@host~]# make SKIP_RPM=1 install - The installer removes the Chelsio specific drivers (inbox/outbox) from initramfs. To skip this and install drivers, [root@host~]# make SKIP_INIT=1 install - The installer checks for the required dependency packages and will install them if they are missing from the machine. To skip this and install drivers, [root@host~]# make SKIP_DEPS=1 install NOTE: - To view the different configuration tuning options, view the help by typing [root@host~]# make help - If IPv6 is disabled in the machine, the drivers will be built and installed without IPv6 Offload support by default. iWARP Driver Installation on Cluster Nodes ------------------------------------------ IMPORTANT: Ensure that you have enabled password less authentication with ssh on the peer nodes for this feature to work. Chelsio's Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a single command. Follow the procedure mentioned below: 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed. 3. Now, execute the following command: [root@host~]# ./install.py -C -m 4. Select the required configuration tuning option. The tuning options may vary depending on the Linux distribution. 5. Select the required Cluster Configuration. 6. The selected components will now be installed. The above command installs iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes listed in the file. 1.2. From RPM (tarball) ----------------------- NOTE: - IPv6 should be enabled in the machine to use the RPM Packages. - Drivers installed from RPM Packages do not have DCB support. 1. Download the tarball specific to your operating system and architecture. 2. Extract the downloaded package. Example: For RHEL/Rocky/AlmaLinux 8.9, extract using the following command: [root@host~]# tar zxvfm ChelsioUwire-x.x.x.x-RHEL8.9-x86_64.tar.gz 3. Change your current working directory to the Chelsio Unified Wire package directory and install the driver. [root@host~]# cd ChelsioUwire-x.x.x.x-- [root@host~]# ./install.py -i nic_toe : NIC and TOE drivers only. all : all Chelsio drivers. udpso : UDP segmentation offload capable NIC and TOE drivers only. wd : Wire Direct drivers and libraries only. crypto : Crypto drivers and OpenSSL modules. ovs : OVS modules and NIC driver. NOTE: The Installation options may vary depending on the Linux distribution. 4. The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following command: [root@host~]# ./install.py -i -c NOTE: To view the different configuration tuning options, view the help by typing. [root@host~]# ./install.py -h 5. Reboot your machine for changes to take effect. NOTE: If the installation aborts with the message "Resolve the errors/dependencies manually and restart the installation", go through the install.log to resolve errors/dependencies and then start the installation again. iWARP Driver Installation on Cluster Nodes ------------------------------------------- IMPORTANT: Ensure that you have enabled password less authentication with ssh on the peer nodes for this feature to work. 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x-- 2. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed. 3. Install iWARP and TOE drivers. [root@host~]# ./install.py -C -m -i -c The above command installs iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes listed in the file 4. Reboot your machine for changes to take effect. Firmware Update =============== The firmware (v1.27.5.0) is installed on the system, typically in /lib/firmware/cxgb4, and the driver auto-loads the firmware if an update is required. The kernel must be configured to enable userspace firmware loading support. Device Drivers > Generic Driver Options > Userspace firmware loading support The firmware version can be verified using ethtool. [root@host~]# ethtool -i Refer to the User's Guide for detailed steps of firmware signature verification. Uninstalling Chelsio Unified Wire ================================= There are two methods to uninstall the Chelsio Unified Wire package: From source and from RPM. If you decide to use source, you can uninstall the package using CLI or GUI mode. Follow the steps mentioned below for uninstallation using CLI. For GUI based uninstallation, refer to the User's Guide. From source ----------- 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Uninstall the source. [root@host~]# make uninstall Uninstalling Individual Drivers/Software ---------------------------------------- You can also choose to uninstall drivers/software individually. Provided here are steps to uninstall few of them. For the complete list, view help by running "make help". Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x - To uninstall NIC driver, [root@host~]# make nic_uninstall - To uninstall drivers with offload support, [root@host~]# make toe_uninstall - To uninstall iWARP driver, [root@host~]# make iwarp_uninstall - To uninstall UDP Segmentation Offload driver, [root@host~]# make udp_offload_uninstall iWARP Driver Uninstallation on Cluster Nodes -------------------------------------------- 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Uninstall iWARP drivers on multiple Cluster nodes. [root@host~]# ./install.py -C -m -u all The above command removes the Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. From RPM (tar-ball) ------------------- 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x-- 2. Run the following command: [root@host~]# ./uninstall.py iWARP Driver Uninstallation on Cluster Nodes -------------------------------------------- To uninstall iWARP drivers on multiple Cluster nodes, 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x-- 2. Run the following command: [root@host~]# ./install.py -C -m -u The above command removes the Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Network (NIC/TOE) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x - To build and install NIC only driver (without offload support), [root@host~]# make nic_install - To build and install drivers with offload support, [root@host~]# make toe_install NOTE: For more installation options, run "make help" or "install.py -h". 2. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user may fail. - To load the driver in NIC mode (without offload support), [root@host~]# modprobe cxgb4 - To load driver in TOE mode (with offload support), [root@host~]# modprobe t4_tom NOTE: Offload support needs to be enabled upon each reboot of the system. This can be done manually as shown above. In VMDirect Path environment, it is recommended to load the offload driver using the following command: [root@host~]# modprobe t4_tom vmdirectio=1 Enabling TCP Offload ==================== Load the offload drivers and bring up the Chelsio interface. [root@host~]# modprobe t4_tom [root@host~]# ifconfig ethX up All TCP traffic will be offloaded over the Chelsio interface now. To see the number of connections offloaded, [root@host~]# cat /sys/kernel/debug/cxgb4//tids Enabling Busy Waiting ===================== Busy waiting/polling is a technique where a process repeatedly checks to see if an event has occurred, by spinning in a tight loop. By making use of similar technique, Linux kernel provides the ability for the socket layer code to poll directly on an Ethernet device's Rx queue. This eliminates the cost of interrupts and context switching, and with proper tuning allows to achieve latency performance similar to that of hardware. Chelsio's NIC and TOE drivers support this feature and can be enabled on Chelsio supported devices to attain improved latency. To make use of BUSY_POLL feature, follow the steps mentioned below: 1. Enable BUSY_POLL support in kernel config file by setting "CONFIG_NET_RX_BUSY_POLL=y". 2. Enable BUSY_POLL globally in the system by setting the values of following sysctl parameters depending on the number of connections: sysctl -w net.core.busy_read= sysctl -w net.core.busy_poll= Set the values of the above parameters to 50 for 100 or less connections; and 100 for more than 100 connections. NOTE: BUSY_POLL can also be enabled on a per-connection basis by making use of SO_BUSY_POLL socket option in the socket application code. Refer to the socket man-page for further details. Precision Time Protocol (PTP) ============================= IMPORTANT: This feature is not supported on RHEL 6.X platforms. ptp4l tool (installed during Unified Wire installation) is used to synchronise clocks. 1. Load the network driver on all master and slave nodes. [root@host~]# modprobe cxgb4 2. Assign IP addresses and ensure that master and slave nodes are connected. 3. Start the ptp4l tool on master using the Chelsio interface. [root@host~]# ptp4l -i -H -m 4. Start the tool on slave nodes. [root@host~]# ptp4l -i -H -m -s NOTE: To view the complete list of available options, refer to the ptp4l help manual. 5. Synchronize the system clock to a PTP hardware clock (PHC) on slave nodes. [root@host~]# phc2sys -s -c CLOCK_REALTIME -w -m VXLAN Offload ============= The Chelsio adapters are uniquely capable of offloading the processing of VXLAN encapsulated frames such that all stateless offloads (checksums and TSO) are preserved, resulting in significant performance benefits. This is enabled by default on loading the driver. For information regarding the configuration, refer to the User's Guide. HMA === To use HMA, ensure that the Unified Wire is installed using the "Unified Wire (Default)" configuration tuning option. Currently, 256 IPv4/128 IPv6 TOE Offload connections are supported on T6 25G SO adapters. To see the number of connections offloaded, [root@host~]# cat /sys/kernel/debug/cxgb4//tids Performance Tuning ================== To tune your system for better network performance, refer to the "Performance Tuning" section of the Network (NIC/TOE) chapter in the User's Guide. NOTE: For more information on additional configuration options, refer to the User's Guide. Driver Unloading ================ - To unload the driver in NIC mode (without offload support), [root@host~]# rmmod cxgb4 - A reboot is required to unload the driver in TOE (with Offload support). To avoid rebooting, follow the steps mentioned below: 1. Load t4_tom driver with unsupported_allow_unload parameter. [root@host~]# modprobe t4_tom unsupported_allow_unload=1 2. Stop all the offloaded traffic, servers and connections. Check for the reference count. [root@host~]# cat /sys/module/t4_tom/refcnt If the reference count is 0, the driver can be directly unloaded. Skip to step 3. If the count is non-zero, load a COP policy which disables offload using the following procedure: i. Create a policy file which disables offload. [root@host~]# cat policy_file all => !offload ii. Compile and apply the output policy file. [root@host~]# cop –o no-offload.cop policy_file [root@host~]# cxgbtool ethX policy no-offload.cop 3. Unload the driver. [root@host~]# rmmod t4_tom [root@host~]# rmmod toecore [root@host~]# rmmod cxgb4 NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Virtual Function Network (vNIC) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The Virtual Function implementation for Chelsio adapters comprises of two modules: - Standard NIC driver module, cxgb4, which runs on base Hypervisor and is responsible for instantiation and management of the PCIe Virtual Functions (VFs) on the adapter. - VF NIC driver module, cxgb4vf, which runs on Virtual Machine (VM) guest OS using VFs "attached" through Hypervisor VM initiation commands. Pre-requisites ============== Ensure that following requirements are met before driver installation: - PCI Express Slot should be ARI capable. - SR-IOV should be enabled in the machine. - Intel Virtualization Technology for Directed I/O (VT-d) should be enabled in the BIOS. - Add intel_iommu=on to the kernel command line in grub/grub2 menu, to use VFs in VMs. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. On the host, install network driver. [root@host~]# make nic_install 3. On the guest (VM), install vNIC driver. [root@host~]# make vnic_install NOTE: For more installation options, run "make help" or "install.py -h". 4. Reboot your machine for changes to take effect. Instantiate Virtual Functions ============================= To instantiate Virtual Functions (VFs) on the host, run the following commands: [root@host~]# modprobe cxgb4 [root@host~]# echo n > /sys/class/net/ethX/device/driver//sriov_numvfs Here, "ethX" is the interface and "n" specifies the number of VFs to be instantiated per physical function (bus_id). VFs can be instantiated only from PFs 0 - 3 of the Chelsio adapter. A maximum of 64 virtual functions can be instantiated with 16 virtual functions per physical function. NOTE: To get familiar with physical and virtual function terminologies, refer to the PCI Express specification. Unload the vNIC driver on the host (if loaded). [root@host~]# rmmod cxgb4vf The virtual functions can now be assigned to virtual machines (guests). Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The vNIC (cxgb4vf) driver must be loaded on the Guest OS by the root user. Any attempt to load the driver as a regular user may fail. To load the driver execute the following command: [root@host~]# modprobe cxgb4vf Configuration ============= VF Communication ---------------- Once the VF driver (cxgb4vf) is loaded in the VM and the VF interface is up with an IP address, it will be able to communicate (send/receive network traffic). [root@host~]# modprobe cxgb4vf [root@host~]# ifconfig ethX up 2-port card: VFs of PF0 and PF2 can communicate with each other and with hosts connected to Port 0. VFs of PF1 and PF3 can communicate with each other and with hosts connected to Port 1. 4-port card: VFs of PF0 can communicate with each other and with hosts connected to Port 0. VFs of PF1 can communicate with each other and with hosts connected to Port 1. VFs of PF2 can communicate with each other and with hosts connected to Port 2. VFs of PF3 can communicate with each other and with hosts connected to Port 3. By default, the VFs (in VM) cannot communicate with PFs (on Host). To enable this communication, set ethtool private flag port_tx_vm_wr for PF interface (on Host). [root@host~]# ethtool --set-priv-flags ethX port_tx_vm_wr on VF Link State ------------- VF link state depends on the physical port link status to which the VF is mapped to. Refer to the above section for VF to physical port mappings. To override this and always enable the VF link, follow the below procedure. This enables VF to VF communication irrespective of the physical port link status. 1. After instantiating the VFs, check the current VF link state using the below command on Host (hypervisor). By default, it is auto. [root@host~]# ip link show mgmtpfX,Y 2. Enable the VF link state for the required VFs. [root@host~]# ip link set dev mgmtpfX,Y vf Z state enable 3. The VFs can then be assigned to Virtual Machines. On loading cxgb4vf driver in the VM and bringing up the VF interface, the VF will be enabled. It can then communicate with other VFs (which are enabled) irrespective of physical link. To revert to default behaviour, set the VF link state to auto. [root@host~]# ip link set dev mgmtpfX,Y vf Z state auto VF Rate Limiting ---------------- This section describes the method to rate-limit traffic passing through virtual functions (VFs). 1. The VF rate limit needs to be set on the Host (hypervisor). Apply rate-limiting. [root@host~]# ip link set dev mgmtpfXX vf rate Here, - mgmtpfXX is the management interface to be used. For each PF on which VFs are instantiated, 1 management interface will be created (in "ifconfig -a"). - vf_number is VF on which rate-limiting is applied. Value 0-15. 2. Run traffic over the VF and the throughput should be rate-limited as per the values set in the previous step. Bonding ------- The VF network interfaces (assigned to a VM) can be aggregated into a single logical bond interface effectively combining the bandwidth into a single connection. It also provides redundancy in case one of the link fails. Execute the following steps in the VM (attached with more than 1 VF interface): 1. Load the Virtual Function network driver. [root@host~]# modprobe cxgb4vf 2. Create a bond interface. [root@host~]# modprobe bonding mode= 3. Bring up the bond interface and enslave the VF interfaces to the bond. [root@host~]# ifconfig bond0 up [root@host~]# ifenslave bond0 ethX ethY NOTE: ethX and ethY are the VF interfaces attached to the same VM. It is recommended to use VFs of different Ports to achieve redundancy in case of link failures. 4. Assign IPv4/IPv6 address to the bond interface. [root@host~]# ifconfig bond0 X.X.X.X/Y [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up High Capacity VF Configuration ------------------------------ Chelsio adapters by default support 16 VFs per PF. To use more VFs per PF, refer to the User's Guide. IMPORTANT: This feature is currently supported only on T6225-SO-CR adapter. Driver Unloading ================ To unload the driver execute the following command: [root@host~]# rmmod cxgb4vf NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iWARP RDMA Offload ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Ensure that following requirements are met before driver installation: - Uninstall any OFED present in the machine. - rdma-core-devel package should be installed on RHEL/Rocky/AlmaLinux 9.X/8.X/7.X and SLES 15 SP4 systems. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install iWARP drivers and libraries. [root@host~]# make iwarp_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user may fail. To load the iWARP driver, load the NIC and core RDMA drivers first. Run the following commands: [root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm Optionally, you can start iWARP Port Mapper daemon to enable port mapping. [root@host~]# iwpmd HMA ==== To use HMA, ensure that the Unified Wire is installed using the "Unified Wire (Default)" configuration tuning option. Currently, 256 IPv4/128 IPv6 iWARP Offload connections are supported on T6 25G SO adapters. To see the number of connections offloaded, [root@host~]# cat /sys/kernel/debug/cxgb4//tids Performance Tuning ================== To tune your system for better performance, refer to the "Performance Tuning" section of the iWARP (RDMA) chapter in the User's Guide. Driver Unloading ================ To unload the iWARP driver, run the following command: [root@host~]# rmmod iw_cxgb4 NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSER ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Ensure that the following requirements are met before driver installation: - Uninstall any OFED present in the machine. - The rdma-core-devel package should be installed on RHEL/Rocky/AlmaLinux 9.X/8.X/7.X and SLES 15 SP4 systems. - targetcli is automatically installed by the Chelsio Unified Wire installer using the package manager yum/apt/zypper, if missing from the system. In case you wish to use a different version of targetcli, it is highly recommended to install v2.1.fb29 or higher version. Kernel Configuration ==================== Kernel.org linux-6.6.X/6.1.X ----------------------------- 1. Ensure that the following iSER components are enabled in the kernel configuration file: CONFIG_ISCSI_TARGET CONFIG_INFINIBAND_ISER CONFIG_INFINIBAND_ISERT 2. If the iSER components are not enabled, enable them as follows: CONFIG_ISCSI_TARGET=m CONFIG_INFINIBAND_ISER=m CONFIG_INFINIBAND_ISERT=m 3. Compile and install the kernel, then boot into the new kernel and install the Chelsio Unified Wire. RHEL/Rocky/AlmaLinux 9.X/8.X/7.X, Ubuntu 22.04.X/20.04.X, Debian 12.X/11.X, SLES 15 SP4 --------------------------------------------------------------------------- No additional kernel configuration is required. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install the Chelsio iSER driver, libraries and targetcli utilities. [root@host~]# make iser_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. [root@host~]# reboot Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with Unified Wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Follow the steps mentioned below on both target and initiator machines: 1. Unload Chelsio iWARP driver if previously loaded. [root@host~]# rmmod iw_cxgb4 2. Load the following modules. [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm 3. Start the iWARP Port Mapper Daemon. [root@host~]# iwpmd 4. Bring up the Chelsio interface(s). [root@host~]# ifconfig ethX x.x.x.x up 5. On target, run the following command: [root@host~]# modprobe ib_isert On initiator, run the following command: [root@host~]# modprobe ib_iser Configuration ============= 1. Configure LIO target with iSER support, using ramdisk as LUN. [root@host~]# targetcli /backstores/ramdisk create name=ram0 size=1GB [root@host~]# targetcli /iscsi create wwn=iqn.2003-01.org.lun0.target [root@host~]# targetcli /iscsi/iqn.2003-01.org.lun0.target/tpg1/luns create /backstores/ramdisk/ram0 [root@host~]# targetcli /iscsi/iqn.2003-01.org.lun0.target/tpg1 set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 [root@host~]# targetcli saveconfig 2. Discover the LIO target using OpeniSCSI initiator. [root@host~]# iscsiadm -m discovery -t st -p 102.10.10.4 3. Enable iSER support in LIO target. [root@host~]# targetcli /iscsi/iqn.2003-01.org.lun0.target/tpg1/portals/0.0.0.0:3260 enable_iser boolean=True 4. Login from the initiator with iSER as transport. [root@host~]# iscsiadm -m node -p 102.10.10.4 -T iqn.2003-01.org.lun0.target --op update -n node.transport_name -v iser [root@host~]# iscsiadm -m node -p 102.10.10.4 -T iqn.2003-01.org.lun0.target --login HMA ==== To use HMA, ensure that the Unified Wire is installed using the "Unified Wire (Default)" configuration tuning option. Currently, 256 IPv4/128 IPv6 iSER Initiator Offload connections are supported on the T6 25G SO adapters. To see the number of connections offloaded, [root@host~]# cat /sys/kernel/debug/cxgb4//tids Performance Tuning ================== To tune your system for better performance, refer to the "Performance Tuning" section of the iSER chapter in the User's Guide. Driver Unloading ================ - On target, run the following commands: [root@host~]# rmmod ib_isert [root@host~]# rmmod iw_cxgb4 - On initiator, run the following commands: [root@host~]# rmmod ib_iser [root@host~]# rmmod iw_cxgb4 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ WD-UDP ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Ensure that following requirements are met before driver installation: - Uninstall any OFED present in the machine. - The rdma-core-devel package should be installed on RHEL/Rocky/AlmaLinux 9.X/8.X/7.X and SLES 15 SP4 systems. - IOMMU should be disabled by adding intel_iommu/amd_iommu=off to the grub/grub2 kernel command line. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install iWARP drivers and WD-UDP Libraries. [root@host~]# make iwarp_install NOTE: For more installation options, run "make help" or "install.py -h". Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Load the cxgb4, iw_cxgb4 and rdma_ucm drivers. [root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm Configuring WD-UDP ================== Preload "libcxgb4_sock" using one of the methods mentioned below when starting your application: Preloading using wdload script ------------------------------ [root@host~]# PROT=UDP wdload /your_application The above command generates an end point file, libcxgb4_sock.conf at /etc/. Parameters like interface name and port number can be changed in this file. The following example shows how to run Netperf with WD-UDP: Server: [root@host~]# PROT=UDP wdload netserver -f -p -D -L Client: [root@host~]# PROT=UDP wdload netperf -H -p -t UDP_RR Preloading manually ------------------- Create a configuration file that defines which UDP endpoints should be accelerated, their vlan and priority if any, as well as which interface/port should be used. The file /etc/libcxgb4_sock.conf contains these endpoint entries. Create this file on all systems using libcxgb4_sock. Here is the syntax: Syntax: endpoint { attributes } ... where attributes include: interface = interface-name port = udp-port-number Example: To accelerate all applications that preload libcxgb4_sock using eth2, you only need one entry in /etc/libcxgb4_sock.conf. endpoint {interface=eth2 port=0} For VLAN support, create your VLANs using the normal OS service (like vconfig, for example), then add entries to define the VLAN and priority for each endpoint to be accelerated. endpoint {interface = eth2.5 port=10000} endpoint {interface = eth2.7 priority=3 port=9000} Now, preload libcxgb4_sock using the following command: [root@host~]# CXGB4_SOCK_CFG= LD_PRELOAD=libcxgb4_sock.so /your_application NOTE: i. In WD-UDP, only one application can be run per Terminator device per UDP port number. For running 2 concurrent netperf UDP_RR tests, each must use a unique UDP port number. Example: endpoint {interface=eth2 port=8888} endpoint {interface=eth2 port=9000} ii. In order to offload IPv6 UDP sockets, select "low latency networking" as configuration tuning option during installation. Multiple interfaces ------------------- To run on multiple interfaces, it is recommended to create a configuration file for each interface with the corresponding ports to offload. The applications can be started as below: [root@host~]# CXGB4_SOCK_CFG= PROT=UDP wdload [root@host~]# CXGB4_SOCK_CFG= PROT=UDP wdload Driver Unloading ================ To unload the iWARP driver, run the following command: [root@host~]# rmmod iw_cxgb4 NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NVMe-oF iWARP ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Ensure that following requirements are met before driver installation: - Uninstall any OFED present in the machine. - The rdma-core-devel package should be installed on RHEL/Rocky/AlmaLinux 9.X/8.X/7.X and SLES 15 SP4 systems. Kernel Configuration ==================== Kernel.org linux-6.6.X/6.1.X ----------------------------- 1. Ensure that the following NVMe-oF iWARP components are enabled in the kernel configuration file: CONFIG_BLK_DEV_NVME CONFIG_NVME_RDMA CONFIG_NVME_TARGET CONFIG_NVME_TARGET_RDMA CONFIG_BLK_DEV_NULL_BLK CONFIG_CONFIGFS_FS 2. If the NVMe-oF iWARP components are not enabled, enable them as follows: CONFIG_BLK_DEV_NVME=m CONFIG_NVME_RDMA=m CONFIG_NVME_TARGET=m CONFIG_NVME_TARGET_RDMA=m CONFIG_BLK_DEV_NULL_BLK=m CONFIG_CONFIGFS_FS=y 3. Compile and install the kernel, then boot into the new kernel and install the Chelsio Unified Wire. RHEL/Rocky/AlmaLinux 9.X/8.X/7.X, Ubuntu 22.04.X/20.04.X, Debian 12.X/11.X, SLES 15 SP4 --------------------------------------------------------------------------- No additional kernel configuration is required. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install iWARP RDMA Offload driver and NVMe utilities. [root@host~]# make nvme_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Follow the steps mentioned below on both target and initiator machines: 1. Load the following drivers: [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm 2. Bring up the Chelsio interface(s). [root@host~]# ifconfig ethX x.x.x.x up 3. Mount configfs. [root@host~]# mount -t configfs none /sys/kernel/config 4. On target, load the following drivers: [root@host~]# modprobe null_blk [root@host~]# modprobe nvmet [root@host~]# modprobe nvmet-rdma On initiator, load the following drivers: [root@host~]# modprobe nvme [root@host~]# modprobe nvme-rdma Configuration ============= Target ------ 1. The following commands configure target using nvmetcli with a LUN: [root@host~]# nvmetcli /> cd subsystems /subsystems> create nvme-ram0 /subsystems> cd nvme-ram0/namespaces /subsystems/n...m0/namespaces> create nsid=1 /subsystems/n...m0/namespaces> cd 1 /subsystems/n.../namespaces/1> set device path=/dev/ram1 /subsystems/n.../namespaces/1> cd ../.. /subsystems/nvme-ram0> set attr allow_any_host=1 /subsystems/nvme-ram0> cd namespaces/1 /subsystems/n.../namespaces/1> enable /subsystems/n.../namespaces/1> cd ../../../.. /> cd ports /ports> create 1 /ports> cd 1/ /ports/1> set addr adrfam=ipv4. /ports/1> set addr trtype=rdma /ports/1> set addr trsvcid=4420 /ports/1> set addr traddr=102.1.1.102 /ports/1> cd subsystems /ports/1/subsystems> create nvme-ram0 2. Save the target configuration to a file. /ports/1/subsystems> saveconfig /root/nvme-target_setup /ports/1/subsystems> exit 3. To clear the targets, [root@host~]# nvmetcli clear Initiator --------- 1. Discover the target. [root@host~]# nvme discover -t rdma -a -s 4420 2. Connect to target. Connecting to a specific target. [root@host~]# nvme connect -t rdma -a -s 4420 -n Connecting to all targets configured on a portal. [root@host~]# nvme connect-all -t rdma -a -s 4420 3. List the connected targets. [root@host~]# nvme list 4. Format and mount the NVMe disks shown with the above command. 5. Disconnect from the target and unmount the disk. [root@host~]# nvme disconnect -d NOTE: nvme_disk_name is the name of the device (Ex:nvme0n1) and not the device path. HMA ---- To use HMA, ensure that Unified Wire is installed using the "Unified Wire (Default)" configuration tuning option. Currently, 256 IPv4/128 IPv6 NVMe-oF iWARP Initiator Offload connections are supported on T6 25G SO adapters. To see the number of connections offloaded, [root@host~]# cat /sys/kernel/debug/cxgb4//tids The total number of connections depends on the devices used and I/O queues. For example, if the Initiator connects to 2 target devices with 4 I/O queues per device (-i 4), a total of 10 NVMe-oF iWARP connections are used. Performance Tuning ------------------ To tune your system for better performance, refer to the "Performance Tuning" section of the NVMe-oF chapter in the User's Guide. Driver Unloading ================ Follow the steps mentioned below to unload the drivers: On target, run the following commands: [root@host~]# rmmod nvmet-rdma [root@host~]# rmmod nvmet [root@host~]# rmmod iw_cxgb4 On initiator, run the following commands: [root@host~]# rmmod nvme-rdma [root@host~]# rmmod nvme [root@host~]# rmmod iw_cxgb4 NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ SPDK NVMe-oF iWARP ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Ensure that the following requirements are met before driver installation: - Uninstall any OFED present in the machine. - The rdma-core-devel package should be installed on RHEL/Rocky Linux 9.X/8.X/7.X and SLES 15 SP4 systems. Kernel Configuration ==================== Kernel.org linux-6.6.X/6.1.X ----------------------------- 1. Ensure that the following SPDK NVMe-oF iWARP components are enabled in the kernel configuration file: CONFIG_BLK_DEV_NVME CONFIG_NVME_RDMA CONFIG_NVME_TARGET CONFIG_NVME_TARGET_RDMA CONFIG_BLK_DEV_NULL_BLK CONFIG_CONFIGFS_FS 2. If the SPDK NVMe-oF iWARP components are not enabled, enable them as follows: CONFIG_BLK_DEV_NVME=m CONFIG_NVME_RDMA=m CONFIG_NVME_TARGET=m CONFIG_NVME_TARGET_RDMA=m CONFIG_BLK_DEV_NULL_BLK=m CONFIG_CONFIGFS_FS=y 3. Compile and install the kernel, then boot into the new kernel and install the Chelsio Unified Wire. RHEL/Rocky Linux 9.X/8.X/7.X, Ubuntu 22.04.X/20.04.X, SLES 15 SP4 ----------------------------------------------------------------- No additional kernel configuration is required. Driver Installation =================== 1. rdma-core version higher than 23 is recommended for SPDK NVMe-oF iWARP. Below are the steps to install version 27: [root@host ~]# wget "https://github.com/linux-rdma/rdma-core/releases/download/v27.0/rdma-core-27.0.tar.gz" [root@host ~]# tar zxfv rdma-core-27.0.tar.gz [root@host ~]# tar cjf /root/rpmbuild/SOURCES/rdma-core-27.0.tgz rdma-core-27.0/ [root@host rdma-core-27.0]# rpmbuild -ba redhat/rdma-core.spec [root@host ~]# cd /root/rpmbuild/RPMS/x86_64/ [root@host x86_64]# rpm -ivh *27*.rpm NOTE: Skip this step if the system already has the recommended version. 2. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 3. Install iWARP RDMA Offload driver and NVMe utilities. [root@host~]# make nvme_install NOTE: For more installation options, run "make help" or "install.py -h". 4. Reboot your machine for changes to take effect. [root@host~]# reboot Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Follow the steps mentioned below on both target and initiator machines: 1. Load the iWARP RDMA Offload drivers: [root@host~]# modprobe iw_cxgb4 [root@host~]# modprobe rdma_ucm 2. Bring up the Chelsio interface(s). [root@host~]# ifconfig ethX x.x.x.x up Configuration ============= Target ------ 1. Download SPDK v23.01.1 LTS. [root@host~]# git clone https://github.com/spdk/spdk [root@host~]# cd spdk [root@host~]# git checkout v23.01.1 [root@host~]# git submodule update –init Change the below in CONFIG file. CONFIG_FIO_PLUGIN=y FIO_SOURCE_DIR= CONFIG_RDMA=y CONFIG_RDMA_SEND_WITH_INVAL=y 2. Run the below script to check that minimum SPDK dependencies are installed. [root@host~]# cd spdk [root@host~]# sh scripts/pkgdep.sh 3. Compile SPDK with RDMA and install it. [root@host~]# make clean ; ./configure --with-rdma; make; make install 4. Configure Huge Pages. [root@host~]# mkdir -p /mnt/huge [root@host~]# echo 8192 > /proc/sys/vm/nr_hugepages [root@host~]# echo 0 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages [root@host~]# echo 8192 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages [root@host~]# vim /etc/fstab nodev /dev/hugepages hugetlbfs pagesize=2MB 0 0 nodev /mnt/huge hugetlbfs pagesize=1GB 0 0 [root@host~]# mount -a [root@host~]# cd spdk [root@host~]# NRHUGE=8192 scripts/setup.sh 5. Start the SPDK NVMe-oF iWARP target. [root@host~]# spdk/build/bin/nvmf_tgt -m 0xFFF & 6. Below are the sample configuration steps to create a malloc LUN. [root@host~]# spdk/scripts/rpc.py nvmf_create_transport -t RDMA -c 8192 -u 131072 -n 8192 -b 256 [root@host~]# spdk/scripts/rpc.py bdev_malloc_create -b Malloc$i 256 512 [root@host~]# spdk/scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode0 -a -s SPDK00000000000000 -d SPDK_Controller0 [root@host~]# spdk/scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode0 Malloc0 [root@host~]# spdk/scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode0 -t rdma -a 10.1.1.163 -s 4420 Initiator --------- SPDK NVMe-oF iWARP target works seamlessly with SPDK NVMe-oF iWARP initiator or any standard Linux kernel initiators. refer to the NVMe-oF iWARP Initiator section for steps to use Linux kernel initiator. To use the SPDK NVMe-oF iWARP Initiator, 1. Follow steps 1 through 4 of the SPDK Target section above to configure and install SPDK. 2. Connect to the target using fio plugin. [root@host~]# LD_PRELOAD=/root/spdk/build/fio/spdk_nvme fio --rw=randread/randwrite --name=random --norandommap=1 --ioengine=/root/spdk/build/fio/spdk_nvme --thread=1 --size=400m --group_reporting --exitall --invalidate=1 --direct=1 --filename='trtype=RDMA adrfam=IPv4 traddr=10.1.1.163 trsvcid=4420 subnqn=nqn.2016-06.io.spdk\:cnode0 ns=1' --time_based --runtime=20 --iodepth=64 --numjobs=4 --unit_base=1 --bs= --kb_base=1000 --ramp_time=3 Performance Tuning ------------------ To tune your system for better performance, refer to the "Performance Tuning" section of the SPDK NVMe-oF chapter in the User's Guide. Driver Unloading ================ Follow the steps mentioned below to unload the drivers: [root@host~]# rmmod iw_cxgb4 NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NVMe-oF TOE ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Kernel Configuration ==================== Kernel.org linux-6.6.X/6.1.X ----------------------------- 1. Ensure that the following NVMe/TCP components are enabled in the kernel configuration file: CONFIG_NVME_CORE CONFIG_NVME_FABRICS CONFIG_NVME_TCP CONFIG_NVME_TARGET CONFIG_NVME_TARGET_TCP CONFIG_BLK_DEV_NVME CONFIG_BLK_DEV_NULL_BLK CONFIG_CONFIGFS_FS 2. If the NVMe/TCP components are not enabled, enable them as follows: CONFIG_NVME_CORE=m CONFIG_NVME_FABRICS=m CONFIG_NVME_TCP=m CONFIG_NVME_TARGET=m CONFIG_NVME_TARGET_TCP=m CONFIG_BLK_DEV_NVME=m CONFIG_BLK_DEV_NULL_BLK=m CONFIG_CONFIGFS_FS=y 3. Compile and install the kernel, then boot into the new kernel and install the Chelsio Unified Wire. RHEL/Rocky/AlmaLinux 9.X/8.X, Ubuntu 22.04.X/20.04.X, Debian 12.X/11.X, SLES 15 SP4 ----------------------------------------------------------------------- No additional kernel configuration is required. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install TOE driver and NVMe utilities. [root@host~]# make nvme_toe_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Follow the steps mentioned below on both target and initiator machines: 1. Load the TOE driver. [root@host~]# modprobe t4_tom 2. Bring up the Chelsio interface(s). [root@host~]# ifconfig ethX x.x.x.x up 3. Mount configfs. [root@host~]# mount -t configfs none /sys/kernel/config 4. Apply cop policy to disable DDP and Rx Coalesce. [root@host~]# cat all => offload !ddp !coalesce [root@host~]# cop -d -o [root@host~]# cxgbtool ethX policy NOTE: The policy applied using cxgbtool is not persistent and should be applied every time drivers are reloaded, or the machine is rebooted. The applied cop policies can be read using, [root@host~]# cat /proc/net/offload/toeX/read-cop 5. Load the nvme drivers. On target, run the following commands: [root@host~]# modprobe null_blk [root@host~]# modprobe nvmet [root@host~]# modprobe nvmet-tcp On initiator, run the following commands: [root@host~]# modprobe nvme [root@host~]# modprobe nvme-tcp Configuration ============= Target ------ 1. The following commands configure target using nvmetcli with a LUN. [root@host~]# nvmetcli /> cd subsystems /subsystems> create nvme-ram0 /subsystems> cd nvme-ram0/namespaces /subsystems/n...m0/namespaces> create nsid=1 /subsystems/n...m0/namespaces> cd 1 /subsystems/n.../namespaces/1> set device path=/dev/ram1 /subsystems/n.../namespaces/1> cd ../.. /subsystems/nvme-ram0> set attr allow_any_host=1 /subsystems/nvme-ram0> cd namespaces/1 /subsystems/n.../namespaces/1> enable /subsystems/n.../namespaces/1> cd ../../../.. /> cd ports /ports> create 1 /ports> cd 1/ /ports/1> set addr adrfam=ipv4 /ports/1> set addr trtype=tcp /ports/1> set addr trsvcid=4420 /ports/1> set addr traddr=102.1.1.102 /ports/1> cd subsystems /ports/1/subsystems> create nvme-ram0 2. Save the target configuration to a file. /ports/1/subsystems> saveconfig /root/nvme-target_setup /ports/1/subsystems> exit 3. Clear the targets. [root@host~]# nvmetcli clear Initiator --------- 1. Discover the target. [root@host~]# nvme discover -t tcp -a -s 4420 2. Connect to target. Connecting to a specific target. [root@host~]# nvme connect -t tcp -a -s 4420 -n Connecting to all targets configured on a portal. [root@host~]# nvme connect-all -t tcp -a -s 4420 3. List the connected targets. [root@host~]# nvme list 4. Format and mount the NVMe disks shown with the above command. 5. Disconnect from the target and unmount the disk. [root@host~]# nvme disconnect -d NOTE: nvme_disk_name is the name of the device (Ex:nvme0n1) and not the device path. HMA --- To use HMA, ensure that Unified Wire is installed using the "Unified Wire (Default)" configuration tuning option. Currently, 256 IPv4/128 IPv6 NVMe-oF TOE Initiator Offload connections are supported on the T6 25G SO adapters. To see the number of connections offloaded, [root@host~]# cat /sys/kernel/debug/cxgb4//tids The total number of connections depends on the devices used and I/O queues. For example, if the Initiator connects to 2 target devices with 4 I/O queues per device (-i 4), a total of 10 NVMe-oF TOE connections are used. Performance Tuning ------------------ To tune your system for better performance, refer ot the "Performance Tuning" section of the NVMe-TOE chapter in the User's Guide. Driver Unloading ================ Follow the steps mentioned below to unload the drivers: On target, run the following commands: [root@host~]# rmmod nvmet-rdma [root@host~]# rmmod nvmet On initiator, run the following commands: [root@host~]# rmmod nvme-rdma [root@host~]# rmmod nvme To unload TOE driver, see Software/Driver Unloading section in Network (NIC/TOE) section. NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ SPDK NVMe-oF TOE ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Kernel Configuration ==================== Kernel.org linux-6.6.X/6.1.X ----------------------------- 1. Ensure that the following NVMe/TCP components are enabled in the kernel configuration file: CONFIG_NVME_CORE CONFIG_NVME_FABRICS CONFIG_NVME_TCP CONFIG_NVME_TARGET CONFIG_NVME_TARGET_TCP CONFIG_BLK_DEV_NVME CONFIG_BLK_DEV_NULL_BLK CONFIG_CONFIGFS_FS 2. If the NVMe/TCP components are not enabled, then enable them as below: CONFIG_NVME_CORE=m CONFIG_NVME_FABRICS=m CONFIG_NVME_TCP=m CONFIG_NVME_TARGET=m CONFIG_NVME_TARGET_TCP=m CONFIG_BLK_DEV_NVME=m CONFIG_BLK_DEV_NULL_BLK=m CONFIG_CONFIGFS_FS=y 3. Compile and install the kernel, then boot into the new kernel and install the Chelsio Unified Wire. RHEL/Rocky 9.X/8.X/7.X, Ubuntu 22.04.X/20.04.X, SLES 15 SP4 ----------------------------------------------------------- No additional kernel configuration is required. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install SPDK NVMe-oF TOE driver and NVMe utilities. [root@host~]# make nvme_toe_spdk_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Follow the steps mentioned below on the target machine: 1. Load the SPDK NVMe-oF TOE driver. [root@host~]# modprobe chtcp 2. Bring up the Chelsio interface(s). [root@host~]# ifconfig ethX x.x.x.x up Configuration ============= Target ------ 1. SPDK v23.01.1 LTS, customized to support TCP/IP offload and kernel bypass for SPDK NVMe-oF TCP Target is part of Chelsio Unified Wire package. Change your current working directory to the Chelsio SPDK directory. [root@host~]# cd ChelsioUwire-x.x.x.x/build/src/chspdk/user/spdk/ 2. Configure Huge Pages. [root@host~]# mkdir -p /mnt/huge [root@host~]# echo 8192 > /proc/sys/vm/nr_hugepages [root@host~]# echo 0 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages [root@host~]# echo 8192 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages [root@host~]# vim /etc/fstab nodev /dev/hugepages hugetlbfs pagesize=2MB 0 0 nodev /mnt/huge hugetlbfs pagesize=1GB 0 0 [root@host~]# mount -a [root@host~]# NRHUGE=8192 scripts/setup.sh 3. Start the target. [root@host spdk]# ./build/bin/nvmf_tgt -m 4. Below are the sample configuration steps to create a LUN with null device. SPDK_PATH=$'ChelsioUwire-x.x.x.x/build/src/chspdk/user/spdk/' $SPDK_PATH/scripts/rpc.py nvmf_create_transport -t TCP $SPDK_PATH/scripts/rpc.py bdev_null_create Null0 1024 4096 $SPDK_PATH/scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode0 -a -s SPDK00000000000000 -d SPDK_Controller0 $SPDK_PATH/scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode0 Null0 $SPDK_PATH/scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode0 -t tcp -a 10.1.1.163 -s 4420 Initiator --------- SPDK NVMe-oF TOE target works seamlessly with SPDK NVMe-oF TCP initiator or any kernel mode initiators. Refer to the "NVMe-oF TOE Initiator" section for steps to connect to the target. Driver Unloading ================ Run the below commands on the target machine to unload the drivers: [root@host~]# rmmod chtcp [root@host~]# rmmod cxgb4 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ SoftiWARP ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Kernel Configuration ==================== Kernel.org linux-6.6.X/6.1.X ----------------------------- 1. Ensure that the following SoftiWARP component is enabled in the kernel configuration file: CONFIG_RDMA_SIW 2. If SoftiWARP component is not enabled, enable it as follows: CONFIG_RDMA_SIW=m 3. Compile and install the kernel, then boot into the new kernel and install the Chelsio Unified Wire. RHEL/Rocky/AlmaLinux 9.X/8.X, Ubuntu 22.04.X/20.04.X, SLES 15 SP4 ----------------------------------------------------------------- No additional kernel configuration is required. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install network driver and NVMe, iSER utilities. [root@host~]# make install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Follow the steps mentioned below the Initiator/Client machine: 1. Load the network driver (cxgb4). [root@host~]# modprobe cxgb4 2. Load the SoftiWARP driver (siw). [root@host~]# modprobe siw 3. Unload the iWARP RDMA offload driver (iw_cxgb4). [root@host~]# rmmod iw_cxgb4 Configuration ============= Initiator/Client ---------------- IMPORTANT: Disable iWARP Port Mapper (iwpmd) service on Target and Initiator. [root@host~]# systemctl stop iwpmd 1. RDMA tool (rdma) is used to configure the siw device. It is installed by default from RHEL/Rocky Linux 8.4, Ubuntu 20.04, and SLES 15 SP4 distributions. If not present in the machine, install it from the latest iproute2 package (https://git.kernel.org/pub/scm/network/iproute2/iproute2.git). 2. Configure the siw device. [root@host~]# rdma link add type siw netdev [root@host~]# ifconfig ethX up 3. Verify the configuration using ibv_devices. 4. The initiator/client can now connect to the target/server machines. Refer to the NVMe-oF iWARP initiator and iSER initiator sections for steps to connect to the respective targets. Driver Unloading ================ Follow the below steps to unload the SoftiWARP and network drivers: [root@host~]# rmmod siw [root@host~]# rmmod cxgb4 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ LIO iSCSI Target Offload ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Kernel Configuration ==================== RHEL/Rocky/AlmaLinux 9.X/8.X/7.X -------------------------------- 1. Download the kernel source RPM kernel-3.10.0-xxx.el7.src.rpm for your distribution. 2. Install the kernel source. [root@host~]# rpm -ivh kernel-3.10.0-xxx.el7.src.rpm 3. Prepare the kernel source. [root@host~]# cd /root/rpmbuild/SPECS/ [root@host~]# rpmbuild -bp kernel.spec --nodeps [root@host~]# cd /root/rpmbuild/BUILD/kernel-3.10.0-xxx.el7/linux-3.10.0-xxx.el7.x86_64/ [root@host~]# make prepare 4. Copy the source to /usr/src directory. [root@host~]# cp -r linux-3.10.0-xxx.el7 /usr/src 5. Proceed with driver installation as directed in the "Driver Installation" section. Kernel.org linux-6.6.X/6.1.X ----------------------------- Follow the below steps to use a 6.6.X/6.1.X kernel version, 1. Download the kernel from kernel.org. 2. Extract the tar-ball. 3. Change your working directory to the kernel directory and invoke the installation menu. [root@host~]# make menuconfig 4. Select Device Drivers > Generic Target Core Mod (TCM) and ConfigFS Infrastructure. 5. Enable Linux-iSCSI.org iSCSI Target Mode Stack as a Module (if not already enabled). 6. Select Save. 7. Exit from the installation menu. 8. Continue with kernel installation as usual. 9. Boot into the new kernel and proceed with driver installation as directed in the "Driver Installation" section. Kernel.org linux-4.9.X ---------------------- 1. Download the stable version of 4.9 from kernel.org. 2. Extract the tar-ball. 3. Change your working directory to the kernel package directory and run the following command to invoke the installation menu. [root@host~]# make menuconfig 4. Select "Device Drivers" > "Generic Target Core Mod (TCM) and ConfigFS Infrastructure". 5. Enable "Linux-iSCSI.org iSCSI Target Mode Stack". 6. Select Save. 7. Exit from the installation menu. 8. Apply the patch provided in the Unified Wire package. [root@host~]# patch -p1 < /root//src/cxgbit/patch/iscsi_target.patch 9. Continue with kernel installation as usual. 10. Boot into the new kernel and proceed with driver installation as directed in the "Driver Installation" section. Ubuntu 22.04.X/20.04.X ---------------------- 1. Clone Ubuntu Linux kernel source repository. Ubuntu 22.04.X: [root@host~]# git clone https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy Ubuntu 20.04.X: [root@host~]# git clone https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal Ubuntu 18.04.X: [root@host~]# git clone https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic 2. Check the booted kernel version using "uname -r" 3. Find the git tag which matches the kernel version. [root@host~]# cd jammy/ [root@host~]# git tag -l Ubuntu-* | grep -i 5.15.0-25 Ubuntu-5.15.0-25.25 4. Check out to the changeset. [root@host~]# git checkout Ubuntu-5.15.0-25.25 5. Proceed with driver installation as directed in the "Driver Installation" section. SLES 15 SP4 ----------- 1. Install the kernel source. [root@host~]# zypper install kernel-source Debian 12.X/11.X ---------------- 1. Install the kernel source. Debian 12.X: [root@host~]# apt install linux-source-6.1 Debian 11.X: [root@host~]# apt install linux-source-5.10 2. Extract the kernel source. [root@host~]# cd /usr/src [root@host~]# tar xf linux-source-x.x.tar.xz Pre-requisites ============== Ensure that following requirements are met before driver installation: - targetcli is automatically installed by the Chelsio Unified Wire installer using the package manager yum/apt/zypper, if missing from the system. If you wish to use a different version, it is highly recommended to install v2.1.fb44 or higher versions. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install LIO Target driver and targetcli utils. [root@host~]# make lio_install In case of RHEL/Rocky/AlmaLinux 9.X/8.X/7.X, Ubuntu 22.04.X/20.04.X and Debian 12.X/11.X, the kernel source (KSRC), and kernel object (KOBJ) should be speicifed. CLI mode -------- [root@host~]# make lio_install KSRC="" KOBJ="" Example: For Ubuntu 22.04, [root@host~]# make lio_install KSRC=/root/jammy/ KOBJ=/lib/modules/5.15.0-25-generic/build CLI mode (without Dialog utility) --------------------------------- [root@host~]# ./install.py --ksrc= --kobj= Example: For RHEL 7.9, [root@host~]# ./install.py --ksrc=/usr/src/linux-3.10.0-1160.el7 --kobj=/lib/modules/3.10.0-1160.el7.x86_64/build/ GUI mode -------- [root@host~]# ./install.py --set-kpath Provide the paths for kernel source and kernel object on the last screen of the installer. Select "OK". NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user may fail. 1. Load network driver (cxgb4). [root@host~]# modprobe cxgb4 2. Bring up the interface. [root@host~]# ifconfig ethX up 3. Load the LIO iSCSI Target Offload driver (cxgbit). [root@host~]# modprobe cxgbit Driver Configuration ==================== Configuring LIO iSCSI Target ----------------------------- The LIO iSCSI Target needs to be configured before it can become useful. Refer to the targetcli man page using the "man targetcli" command to do so. Offloading LIO iSCSI Connection -------------------------------- To offload the LIO iSCSI Target, use the following command: [root@host~]# targetcli /iscsi//tpg1/portals/\:3260 enable_offload boolean=True Execute the above command for every portal address listening on the Chelsio interface. Running LIO iSCSI and Network Traffic Concurrently -------------------------------------------------- If you wish to run network traffic with offload support (TOE) and LIO iSCSI traffic together, follow the steps mentioned below: 1. If not done already, load network driver with offload support (TOE). [root@host~]# modprobe t4_tom 2. Create a new policy file. [root@host~]# cat 3. Add the following lines to offload all traffic except LIO iSCSI. listen && src port && src host => !offload all => offload 4. Compile the policy. [root@host~]# cop -d -o 5. Apply the policy. [root@host~]# cxgbtool ethX policy NOTE: The policy applied using cxgbtool is not persistent and should be applied every time drivers are reloaded or the machine is rebooted. The applied cop policies can be read using, [root@host~]# cat /proc/net/offload/toeX/read-cop Performance Tuning ================== To tune your system for better performance, refer to the "Performance Tuning" section of the LIO iSCSI Target Offload chapter in the User's Guide. Driver Unloading ================ Unloading LIO iSCSI Target Offload driver ----------------------------------------- To unload the LIO iSCSI Target Offload driver, follow the steps mentioned below: 1. Log out from the initiator 2. Run the following command: [root@host~]# targetcli /iscsi//tpg1/portals/\:3260 enable_offload boolean=False Execute the above command for every portal address listening on the Chelsio interface. 3. Unload the driver. [root@host~]# rmmod cxgbit Unloading Network driver ------------------------ - To unload the driver in NIC mode (without offload support). [root@host~]# rmmod cxgb4 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSCSI PDU Offload Initiator ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Ensure that following requirements are met before driver installation: - The iSCSI PDU Offload Initiator driver (cxgb4i) runs on top of NIC driver (cxgb4) and open-iscsi version greater than 2.0-872 on a Chelsio card. - openssl-devel package should be installed. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install open-iSCSI, iSCSI-initiator, firmware and utilities. [root@host~]# make iscsi_pdu_initiator_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to loading the driver as a regular user may fail. Load cxgb4i driver using the following command: [root@host~]# modprobe cxgb4i The cxgb4i module registers a new transport class "cxgb4i". If loading of cxgb4i displays "unkown symbols found" error in dmesg, follow the steps mentioned below: 1. View all the loaded iSCSI modules [root@host~]# lsmod | grep iscsi 2. Now, unload them using the following command: [root@host~]# rmmod 3. Finally reload the cxgb4i driver Accelerating open-iSCSI Initiator ================================= The following steps need to be taken to accelerate the open-iSCSI initiator: I. Configuring interface (iface) file ------------------------------------- Create the file automatically by loading cxgb4i driver and then executing the following command: [root@host~]# iscsiadm -m iface Alternatively, you can create an interface file located under iface directory for the new transport class cxgb4i in the following format: iface.iscsi_ifacename = iface.hwaddress = iface.transport_name = cxgb4i iface.net_ifacename = iface.ipaddress = Here, iface.iscsi_ifacename : Interface file in /etc/iscsi/ifaces/ iface.hwaddress : MAC address of the Chelsio interface via which iSCSI traffic will be running. iface.transport_name : Transport name, which is cxgb4i. iface.net_ifacename : Chelsio interface via which iSCSI traffic will be running. iface.ipaddress : IP address which is assigned to the interface. II. Discovery and Login ----------------------- 1. Start Daemon from /sbin. [root@host~]# iscsid NOTE: If iscsid is already running, then kill the service and start it as shown above after installing the Chelsio Unified Wire package. 2. Discover iSCSI target. [root@host~]# iscsiadm -m discovery -t st -p : -I 3. Log into an iSCSI target. [root@host~]# iscsiadm -m node -T -p : -I -l If the login fails with an error message in the format of ERR! MaxRecvSegmentLength too big. Need to be <= . in dmesg, edit the iscsi/iscsid.conf file and change the setting for MaxRecvDataSegmentLength: node.conn[0].iscsi.MaxRecvDataSegmentLength = 8192 IMPORTANT: Always take a backup of iscsid.conf file before installing Chelsio Unified Wire package. Although the file is saved to iscsid.rpmsave after uninstalling the package using RPM, you are still advised to take a backup. 4. Log out from an iSCSI Target. [root@host~]# iscsiadm -m node -T -p : -I -u NOTE: Other options can be found by typing iscsiadm --help HMA ==== To use HMA, ensure that the Unified Wire is installed using the "Unified Wire (Default)" configuration tuning option. 1. Use LIO iSCSI Target in offload mode. 2. Configure MTU 9000 for the Chelsio Interfaces. 3. Load the driver using the following parameters. [root@host~]# modprobe cxgb4i cxgb4i_snd_win=131072 cxgb4i_rcv_win=262144 Currently, 256 IPv4/128 IPv6 iSCSI PDU Offload Initiator connections are supported on T6 25G SO adapters. The following command shows the number of offloaded connections. [root@host~]# cat /sys/kernel/debug/cxgb4//tids Performance Tuning ================== To tune your system for better performance, refer to the "Performance Tuning" section of the iSCSI PDU Offload Initiator chapter in the User's Guide. Driver Unloading ================ [root@host~]# rmmod cxgb4i [root@host~]# rmmod libcxgbi NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Crypto Offload ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Ensure that the SELinux and firewall are disabled. Kernel Configuration ==================== Kernel.org linux-6.6.X/6.1.X ----------------------------- 1. Enable the options listed in the User's Guide in the kernel configuration file. 2. Compile and install the kernel, then boot into the new kernel and install the Chelsio Unified Wire. RHEL/Rocky/AlmaLinux 9.X/8.X/7.X, Ubuntu 22.04.X/20.04.X, RHEL 7.5/7.6 ARM, Debian 12.X/11.X, SLES 15 SP4 ----------------------------------------------------------------------------- No additional kernel configuration is required. Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install Crypto driver. [root@host~]# make crypto_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot the machine for changes to take effect. [root@host~]# reboot   Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Inline ------ 1. To load Crypto Offload driver in Inline mode, [root@host~]# modprobe t4_tom 2. Bring up the Chelsio network interface. [root@host~]# ifconfig ethX up Where ethX is the Chelsio interface. Co-processor ------------ 1. To load Crypto Offload driver in Co-processor mode (chcr), [root@host~]# modprobe cxgb4 [root@host~]# modprobe chcr 2. Bring up the Chelsio network interface. [root@host~]# ifconfig ethX up Where ethX is the Chelsio interface. Configuration ============= Configuring OpenSSL ------------------- OpenSSL v3.0.12 with kTLS support is installed by Unified Wie installer at /usr/opensslv3/bin. Additionally, all the necessary openssl configuration files are updated automatically. Instead, if you wish to install the OpenSSL v3.0.12 with kTLS support and do the configuration, follow the below mentioned steps: 1. Download the OpenSSL v3.0.12, compile with kTLS support and install it. [root@host~]# wget https://www.openssl.org/source/openssl-3.0.12.tar.gz [root@host~]# tar xf openssl-3.0.12.tar.gz [root@host~]# cd openssl-3.0.12 [root@host~]# ./config shared enable-ktls --prefix=/root/sslv3-ktls --openssldir=/root/sslv3-ktls -Wl,-R/root/sslv3-ktls/lib64 [root@host~]# make && make install 2. Update openssl.cnf to enable kTLS during runtime. [root@host~]# vim /root/sslv3-ktls/openssl.cnf … [openssl_init] providers = provider_sect ssl_conf = ssl_section [ssl_section] system_default = system_default_section [system_default_section] options = ktls Ensure that the following requirements are met for connections to be offloaded: - TLS version should be v1.2 - Cipher should be AES128-GCM-SHA256 Inline TLS Offload ------------------ Configure TLS Offload and TOE Ports ----------------------------------- To configure Inline TLS Offload, connection offload policy should be used with the required TCP port numbers. Follow the steps mentioned below: 1. Create a new policy file and add the following line for each TCP port (to be TLS offloaded): src or dst port => offload tls mss 32 bind random . . all => offload The all => offload is added to ensure that rest of the TCP ports are regular TOE offloaded. Alternatively, portrange can be used to define a range of TCP ports (to be TLS offloaded). src or dst portrange => offload tls mss 32 bind random all => offload 2. Compile the policy. [root@host~]# cop -d -o 3. Apply the policy. [root@host~]# cxgbtool policy Upon applying the above policy, traffic on all the mentioned TCP ports are TLS offloaded, while traffic on other TCP ports are TOE offloaded. NOTE: The policy applied using cxgbtool is not persistent and should be applied every time drivers are reloaded or the machine is rebooted. The applied cop policies can be read using, [root@host~]# cat /proc/net/offload/toeX/read-cop Configuring and running Applications ------------------------------------ The below sections assume that openssl with kTLS support is installed by Unified Wire Installer at /usr/opensslv3. If you wish to use your own openssl with kTLS support, replace /usr/opensslv3 with the corresponding location. OpenSSL tool: 1. Start TLS offload Server. [root@host~]# cd /usr/opensslv3/bin [root@host~]# ./openssl s_server -key -cert -accept : -cipher AES128-GCM-SHA256 -WWW -4 -tls1_2 2. Start TLS offload Client. [root@host~]# cd /usr/opensslv3/bin [root@host~]# ./openssl s_time -connect : -www / In case of IPv6, the address should be specified within []. Custom Applications: To compile applications using Chelsio OpenSSL library: [root@host~]# gcc -g -o -lcrypto -lssl -L/usr/opensslv3/lib64/ Refer to the User's Guide to configure nginx server. Inline TLS Counters ------------------- To verify whether the Chelsio Inline is used, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4//tls Chelsio Inline TLS Stats TLS PDU Tx: 32661534 TLS PDU Rx: 231039210 TLS Keys (DDR) Count: 48 Co-processor ------------ To view the complete list of supported cryptographic algorithms, use the following command: [root@host~]# cat /proc/crypto|grep -i chcr The following applications can be offloaded by Chelsio Co-processor: Data at Rest - Dmcrypt - VeraCrypt TLS/SSL - nginx IPsec - Strongswan Refer to the User's Guide for configuring Co-processor TLS and IPsec offloads. To verify whether the Chelsio Co-processor is used by the applications, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4//crypto Chelsio Crypto Co-processor Stats aes_ops: 1016 digest_ops: 323 aead_ops: 2739611 comp: 2740950 error: 0 Fallback: 9 Performance Tuning ================== To tune your system for better performance, refer to the "Performance Tuning" section of the Crypto Offload chapter in the User's Guide. Driver Unloading ================ To unload Crypto Offload driver in Co-processor mode, run the following command: [root@host~]# rmmod chcr To unload Crypto Offload driver in Inline mode, unload the network driver in TOE mode, as mentioned under "Driver Unloading" in "Network (NIC/TOE)" section. NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ DCB ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Build and install all drivers with DCB support. [root@host~]# make dcbx=1 install NOTE: For more installation options, run "make help". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Before proceeding, ensure that Unified Wire is installed with DCB support as mentioned in the previous section. The switch ports need to be enabled with DCBX configuration s(Class mapping, ETS and PFC). Upon loading the network/storage driver and interface bringup, firmware completes DCBX negotiation with the switch. [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom [root@host~]# ifconfig ethX up [root@host~]# modprobe csiostor The negotiated DCBX parameters can be reviewed at /sys/kernel/debug/cxgb4//dcb_info The storage driver(FCoE Full Offload Initiator) uses the DCBX negotiated parameters (ETS, PFC etc.) Without any further configuration. The network drivers (cxgb4, t4_tom) and iSCSI drivers (cxgb4i, chiscsi) need further VLAN configuration to be setup, which is explained in the next section "Running NIC & iSCSI Traffic together with DCBx". Running NIC & iSCSI Traffic together with DCBx ============================================== NOTE: Refer to the "iSCSI PDU Offload Initiator" section to configure iSCSI Initiator. Use the following procedure to run NIC and iSCSI traffic together with DCBx enabled. 1. Identify the VLAN priority configured for NIC and iSCSI class of traffic on the switch. 2. Create VLAN interfaces for running NIC and iSCSI traffic, and configure corresponding VLAN priority. Example: Switch is configured with a VLAN priority of 2 and 5 for NIC and iSCSI class of traffic respectively. NIC traffic is run on VLAN10 and iSCSI traffic is run on VLAN20. Assign proper VLAN priorities on the interface (here eth5), using the following commands on the host machine: [root@host~]# vconfig set_egress_map eth5.10 0 2 [root@host~]# vconfig set_egress_map eth5.20 5 5 NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ FCoE Full Offload Initiator ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install FCoE full offload initiator driver. [root@host~]# make fcoe_full_offload_initiator_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user may fail. To load the driver, execute the following command: [root@host~]# modprobe csiostor Configuring the switch and Troubleshooting ========================================== Refer to the "Software Configuration and Fine-tuning" section in the User's Guide. Driver Unloading ================ To unload the driver, execute the following command: [root@host~]# modprobe -r csiostor NOTE: If multipath services are running, unload of FCoE driver is not possible. Stop the multipath service and then unload the driver. NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Offload Bonding driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install Chelsio Offload Bonding driver. [root@host~]# make bonding_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to loading the driver as a regular user may fail. To load the Bonding driver (with offload support), run the following command: [root@host~]# modprobe bonding Offloading TCP Traffic over a Bonded Interface ============================================== The Chelsio Offload Bonding driver supports all the bonding modes in NIC Mode. In offload mode (t4_tom loaded) however, only the balance-rr (mode=0), active-backup (mode=1), balance-xor (mode=2) and 802.3ad (mode=4) modes are supported. To offload TCP traffic over a bonded interface, use the following method: 1. Load the network driver with TOE support. [root@host~]# modprobe t4_tom 2. Create a bond interface. [root@host~]# modprobe bonding mode=1 miimon=100 max_bonds=1 3. Bring up the bond interface and enslave the interfaces to the bond. [root@host~]# ifconfig bond0 up [root@host~]# ifenslave bond0 ethX ethY NOTE: "ethX" and "ethY" are interfaces of the same adapter. 4. Assign IPv4/IPv6 address to the bond interface. [root@host~]# ifconfig bond0 X.X.X.X/Y [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up 5. Disable FRTO on the PEER. [root@host~]# sysctl -w net.ipv4.tcp_frto=0 6. Ping the PEER interface and verify the successful connectivity over the bond interface. All TCP traffic are offloaded over the bond interface now. Driver Unloading ================ To unload the bonding driver. [root@host~]# rmmod bonding NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Offload Multi-Adapter Failover (MAFO) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Multi-Adapter fail-over feature works for link down events caused by: - Cable unplug on bonded interface - Bringing corresponding switch port down NOTE: The feature does not work if the bonded interfaces are administratively taken down. IMPORTANT: - Portions of this software are covered under US Patent "Failover and migration for full-offload network interface devices: US 8346919 B1". - Use of the covered technology is strictly limited to Chelsio ASIC-based solutions. Installation ============ 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install the MAFO feature. [root@host~]# make bonding_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user may fail. To load the driver (with offload support), run the following command: [root@host~]# modprobe bonding Offloading TCP Traffic over a Bond Interface ============================================ The Chelsio MAFO driver supports only the active-backup (mode=1) mode. To offload TCP traffic over a bond interface, use the following method: 1. Load the network driver with TOE support. [root@host~]# modprobe t4_tom 2. Create a bond interface. [root@host~]# modprobe bonding mode=1 miimon=100 max_bonds=1 3. Bring up the bond interface and enslave the interfaces to the bond. [root@host~]# ifconfig bond0 up [root@host~]# ifenslave bond0 ethX ethY NOTE: "ethX" and "ethY" are interfaces of different adapters. 4. Assign IPv4/IPv6 address to the bond interface. [root@host~]# ifconfig bond0 X.X.X.X/Y [root@host~]# ifconfig bond0 inet6 add <128-bit IPv6 Address> up 5. Disable TCP timestamps. [root@host~]# sysctl -w net.ipv4.tcp_timestamps=0 6. Disable FRTO on the PEER. [root@host~]# sysctl -w net.ipv4.tcp_frto=0 7. Ping the PEER interface and verify the successful connectivity over the bond interface. All TCP traffic are offloaded over the bond interface now and fail-over happens in case of link-down event. Driver Unloading ================ To unload the driver, run the following command: [root@host~]# rmmod bonding NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ UDP Segmentation Offload and Pacing ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== The offload drivers support UDP Segmentation Offload with limited number of connections (1024 connections). To build and install UDP Offload drivers which support large number of offload connections (approx 10K): NOTE: 10K UDP Segmentation offload connections currently not supported on the T6 adapter. 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Run the following command: [root@host~]# make udp_offload_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user may fail. Run the following commands to load the driver: [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom Though normally associated with the Chelsio TCP Offload engine, the t4_tom module is required in order to allow for the proper redirection of UDP socket calls. Modifying the Application ========================= To use the UDP offload functionality, the application needs to be modified. Refer to the User's Guide for detailed steps. Configuring UDP Pacing ====================== Once the application is modified, traffic pacing can be set using cxgbtool. 1. Bring up the network interface. [root@host~]# ifconfig up 2. Run the following command. [root@host~]# cxgbtool sched-class params type packet level cl-rl mode flow rate-unit bits rate-mode absolute channel class max-rate pkt-size Here, ethX : Chelsio interface Channel No. : is the port on which data is flowing (0-3) scheduler-class-index : UDP traffic class (0-14 for T4/T5 adapters and 0-30 for T6 adapters) set in the SOL_SCHEDCLASS socket option in the application. maximum-rate : Bit rate (Kbps) for this UDP stream. This value should be in the range of 50 Kbps to 50 Mbps for T4 adapters. For T5/T6 adapters, it should be 100 kbps to 1 Gbps. Packet size : UDP packet payload size in bytes; it should be equal to the value set in the SO_FRAMESIZE socket option in the application. NOTE: To get an accurate bit rate per class, data sent by the application to the sockets should be a multiple of the value set for the “pkt-size” parameter. Enabling Offload ================ Load the offload drivers and bring up the Chelsio interface. [root@host~]# modprobe t4_tom [root@host~]# ifconfig ethX up The traffic is offloaded over the Chelsio interface now. To see the number of connections offloaded, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4//tids UOTID shows the number of UDP offload connections. Driver Unloading ================ Reboot the system to unload the driver. To unload without rebooting, refer to the "Unloading the driver" in Network (NIC/TOE) section. NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Offload IPv6 Driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Ensure that following requirements are met before driver installation: - IPv6 must be enabled in your system (enabled by default). - Unified Wire must be installed with IPv6 support as explained in the "Unified Wire" chapter in the User Guide. Installation ============ 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. IPv6 must be enabled in your system (enabled by default) to use the Offload IPv6 feature. Also, Unified Wire package must be installed with IPv6 support. [root@host~]# make install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. NIC & TOE Driver Loading ======================== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb After installing Unified Wire package and rebooting the host, load the NIC (cxgb4) and TOE (t4_tom) drivers. The drivers must be loaded by root user. Any attempt to load the drivers as a regular user may fail. [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom Configuration ============= 1. Load the Offload capable drivers. [root@host~]# modprobe t4_tom 2. Bring up the interface and ensure that IPv6 Link Local address is present. [root@host~]# ifconfig ethX up 3. Configure the required IPv6 address. [root@host~]# ifconfig ethX inet6 add On some distributions, ONBOOT="yes" should be added to interface network script for the interface to come up automatically with IPv6 Link Local address. 4. All the IPv6 traffic over the Chelsio interface will be offloaded now. To see the number of connections offloaded, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4//tids NIC and TOE Driver Unloading ========================== - To unload the NIC driver, [root@host~]# rmmod cxgb4 - To unload the TOE driver, Reboot the system to unload the TOE driver. To unload without rebooting, refer to the "Unloading the driver" in Network (NIC/TOE) section. NOTE: For more information on additional configuration options, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ WD Sniffing and Tracing ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The objective of these utilities (wd_sniffer and wd_tcpdump_trace) is to provide sniffing and tracing capabilities by making use of Terminator's hardware features. Sniffer- Involves targeting specific multicast traffic and sending it directly to user space. Tracer - All tapped traffic is forwarded to user space and also pushed back on the wire via the internal loop back mechanism In either mode the targeted traffic bypasses the kernel TCP/IP stack and is delivered directly to user space by means of a RX queue which is defined by the register MPS_TRC_RSS_CONTROL. Pre-requisites ============== Ensure that following requirement is met before driver installation: - IOMMU should be disabled by adding intel_iommu/amd_iommu=off to the grub/grub2 kernel command line. Installation ============ 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install Sniffer and Tracer utilities and iWARP driver. [root@host~]# make sniffer_install NOTE: For more information on additional configuration options, refer to the User's Guide. 3. Reboot your machine for changes to take effect. NOTE: For more information on usage, refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Classification and Filtering ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ LE-TCAM Filters =============== Creating Filter Rules --------------------- Network driver (cxgb4) must be installed before setting the filter rule. 1. If you have not done already, run the Unified Wire Installer with the appropriate configuration tuning option to install the Network Driver. 2. Load the network driver and bring up the Chelsio interface. [root@host~]# modprobe cxgb4 [root@host~]# ifconfig ethX up 3. Now, create filter rules using cxgbtool. [root@host~]# cxgbtool ethx filter action [pass/drop/switch] Where, ethX : Chelsio interface. index : positive integer set as filter id. 0-495 for T5 adapters; 0-559 for T6 adapters. action : Ingress packet disposition. pass : Ingress packets will be passed through set ingress queues. switch : Ingress packets will be routed to an output port with optional header rewrite. drop : Ingress packets will be dropped. prio 1 : Optional for T5. Mandatory for T6 indices 0-63; Should not be added for T6 indices 64-559. hitcnts 1 : To enable hit counts in cxgbtool filter show output. NOTE: In case of multiple filter rules, the rule with the lowest filter index takes higher priority. Listing Filter Rules -------------------- To list previously set filters, run the following command: [root@host~]# cxgbtool ethX filter show OR [root@host~]# cat /sys/kernel/debug/cxgb4//filters Removing Filter Rules --------------------- To remove a filter, run the following command with the corresponding filter rule index [root@host~]# cxgbtool ethX filter NOTE: For more information on additional parameters, refer to the cxgbtool manual by running the "man cxgbtool" command. Hash/DDR Filters ================ If you wish to create large number of filters, select one of the below configuration tuning options during Unified Wire installation: - High Capacity Hash Filter: Allows you to create ~0.5 million filter rules. Can run non-offload NIC traffic. - Unified Wire (Default): Allows you to create ~18k filter rules. Can run all offload traffic. You can create both LE-TCAM and Hash/DDR filters in the above configurations. NOTE: Refer to the User's Guide for choosing the appropriate filterMode and filterMask. Creating Filter Rules --------------------- Network driver (cxgb4) must be installed and loaded before setting the filter rule. 1. If you have not done already, run the Unified Wire Installer with the "High Capacity Hash Filter" or "Unified Wire" (Default) configuration tuning option to install the drivers. 2. Load the network driver with DDR filters support and bring up the Chelsio interface. [root@host~]# modprobe cxgb4 use_ddr_filters=1 [root@host~]# ifconfig ethX up 3. Now, create filter rules using cxgbtool. [root@host~]# cxgbtool ethX filter action [pass/drop/switch] fip lip fport lport proto Where, ethX : Chelsio interface. index : Filter index. For LE-TCAM filters, filter index should be 0-495 for T5 adapters and 0-559 for T6 adapters. In case of Hash/DDR filter, the index will be ignored and replaced by an automatically computed value, based on the hash (4-tuple). The index will be displayed after the filter rule is created successfully. action : Ingress packet disposition. pass : Ingress packets will be passed through set ingress queues. switch : Ingress packets will be routed to an output port with. optional header rewrite. drop : Ingress packets will be dropped. source_ip/port : Source IP/port of incoming packet. destination_ip/port : Destination IP/port of incoming packet. protocol : TCP by default. To change, specify the corresponding internet protocol number. E.g. for UDP, use 17. hitcnts 1 : To enable hit counts in cxgbtool filter show output. cap maskless : This is mandatory for hash filter. If not provided, LE-TCAM filter will be created at the specified index. NOTE: In case of Hash/DDR filters, source_ip, destination_ip, source_port and destination_port are mandatory, since the filters don't support masks and hence, 4-tuple must always be supplied. "Proto" is also a mandatory parameter. Listing Filter Rules -------------------- - To list the Hash/DDR filters set, run the following command: [root@host~]# cat /sys/kernel/debug/cxgb4//hash_filters - To list both LE-TCAM and Hash/DDR filters set, run the following command: [root@host~]# cxgbtool ethX filter show Removing Filter Rules ---------------------- To remove a filter, run the following command with cap maskless parameter and corresponding filter rule index: [root@host~]# cxgbtool ethX filter cap maskless NOTE: Filter rule index can be determined by referring the "hash_filters" file located in /sys/kernel/debug/cxgb4// NOTE: For more information on additional parameters, refer to the cxgbtool manual by running the man cxgbtool command. Filter Priority --------------- By default, Hash/DDR filter has priority over LE-TCAM filter. To override this, the LE-TCAM filter should be created with prio option. For example: [root@host~]# cxgbtool ethx filter action prio 1 Where index is a positive integer set as filter id. 0-495 for T5 adapters and 0-63 for T6 adapters. Swap MAC Feature ----------------- Chelsio’s T6/T5 Swap MAC feature swaps packet source MAC and destination MAC addresses. This is applicable only for switch filter rules. Here’s an example: [root@host~]# cxgbtool eth2 filter 100 action switch lip 102.2.2.1 fip 102.2.2.2 lport 5001 fport 14000 hitcnts 1 iport 1 eport 0 swapmac 1 proto 17 cap maskless Hash-Filter Index = 21936 The above example swaps source and destination MAC addresses of UDP packets (matching above 4 tuple) received on adapter port 1 and then switch them to port 0. Traffic Mirroring ----------------- Enabling Mirroring ------------------ To enable traffic mirroring, follow the steps mentioned below: 1. If not done already, install Unified Wire with the "High Capacity Hash Filter" or "Unified Wire" (Default) configuration tuning option as mentioned in the Unified Wire section. 2. Enable vnic_id match for filterMode in Hash filter config file, t5-config.txt, located in /lib/firmware/cxgb4/ filterMode = fragmentation, mpshittype, protocol, vnic_id, port, fcoe filterMask = port, protocol, vnic_id 3. Unload network driver (cxgb4) and reload it with mirroring enabled. [root@host~]# rmmod cxgb4 [root@host~]# modprobe cxgb4 enable_mirror=1 use_ddr_filters=1 4. The traffic will be mirrored now and received through mirror PF/VF corresponding to each port. Switch Filter with Mirroring ---------------------------- The following example explains the method to switch and mirror traffic simultaneously: 1. Obtain the PF and VF values of the incoming port from /sys/kernel/debug/cxgb4//mps_tcam 2. Create the desired switch filter rule. [root@host~]# cxgbtool ethX filter 0 fip 102.8.8.2 lip 102.8.8.1 fport 20000 lport 12865 proto 6 pf 4 vf 64 action switch iport 0 eport 1 cap maskless The hash filter rule switches TCP traffic matching the above 4-tuple received on port 0 to port 1. The traffic will be switched and simultaneously received on mirror queues and network stack of host as mirroring is enabled. Filtered Traffic Mirroring -------------------------- Once mirroring is enabled, all the traffic received on a physical port are duplicated. The following example explains the method to filter out the redundant traffic and receive only specific traffic on mirror queues: 1. Obtain the mirror PF and VF values from dmesg. [root@host~]# dmesg .... ... cxgb4 0000:02:00.4: Port 0 Traffic Mirror PF = 4; VF = 66 cxgb4 0000:02:00.4: Port 1 Traffic Mirror PF = 4; VF = 67 2. Create a DROP-ALL rule as below: [root@host~]# cxgbtool ethX filter 255 pf 4 vf 66 action drop Where, 255 is the last index of available TCAM filters. This creates a catch-all DROP filter for Mirror PF/VF of port 0. Similarly, create DROP filters for rest of Mirror PF/VF. 3. Create specific filter rules to allow specific traffic to be received on mirror queues as below: [root@host~]# cxgbtool ethX filter 1 lip 102.8.8.1 fip 102.8.8.2 lport 12865 fport 20000 pf 4 vf 66 action pass Now, the above specific traffic (from 102.8.8.2,20000 to 102.8.8.1,12865) will be received in Mirror receive queues and network stack of host. Packet Tracing and Hit Counters ------------------------------- For T5/T6 LE-TCAM and T6 Hash/DDR filters, hit counters work simply by adding hitcnts 1 parameter to the filter rule. However, for T5 Hash/DDR filters, make use of tracing feature and RSS queues. The following is step-by-step procedure to enable packet tracing and hit counters for Hash/DDR filter rules: 1. Load network driver with the following parameters: [root@host~]# modprobe cxgb4 use_ddr_filters=1 enable_traceq=1 2. Configure the required filter rules. 3. Enable tracing on adapter. [root@host~]# cxgbtool ethX reg 0x09800=0x13 4. Setup a trace filter. [root@host~]# echo tx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace0 Here, "snaplen" is the length in bytes to be captured. NOTE: Use "snaplen=60" in case of IPv6. The above step traces all the packets transmitting from port1(tx1) to trace filter 0. 5. Configure the RSS Queue to receive traced packets. Determine the "RspQ ID" of the queue by looking at "Trace" QType in /sys/kernel/debug/cxgb4//sge_qinfo file [root@host~]# cxgbtool ethX reg 0x0a00c= Now the traced packets can be seen in tcpdump and the hit counters also get incremented. Multi-tracing --------------- To enable packet capture or hit counters for multiple Chelsio ports in Tx/Rx direction enable Multi-tracing. Using this we can configure four different RSS Queues separately corresponding to 4 trace-filters. 1. Enable Tracing as well as MultiRSSFilter [root@host~]# cxgbtool ethX reg 0x09800=0x33 2. Setup a trace filter [root@host~]# echo tx0 snaplen=40 > /sys/kernel/debug/cxgb4//trace0 3. Configure the RSS Queue corresponding to trace0 filter configured above. Determine the "RspQ ID" of the queues by looking at "Trace" QType in /sys/kernel/debug/cxgb4//sge_qinfo file. [root@host~]# cxgbtool ethX reg 0x09808= 4. Similarly for other direction and for multiple ports run the follow commands: [root@host~]# echo rx0 snaplen=40 > /sys/kernel/debug/cxgb4//trace1 [root@host~]# echo tx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace2 [root@host~]# echo rx1 snaplen=40 > /sys/kernel/debug/cxgb4//trace3 [root@host~]# cxgbtool ethX reg 0x09ff4= [root@host~]# cxgbtool ethX reg 0x09ffc= [root@host~]# cxgbtool ethX reg 0x0a004= NOTE: Use "snaplen=60" in case of IPv6. NOTE: For more information on additional configuration options, refer to the User's Guide. NAT Filtering ============= T5/T6 adapters support offloading of stateless/static NAT functionality i.e. translating source/destination L3 IP addresses, and source/destination L4 port numbers. This feature is supported with both LE-TCAM and Hash filters. NOTE: This feature is only supported with filter action switch. Syntax: [root@host~]# cxgbtool ethX filter action switch fip lip fport lport nat nat_fip nat_lip nat_fport nat_lport Where, ethX : Chelsio interface. source_ip/port : Source IP/port of incoming packet. destination_ip/port : Destination IP/port of incoming packet. new_source_ip/port : Source IP/port to be translated to. new_destination_ip/port: Destination IP/port to be translated to. mode : Combination of IP/port to be translated. "all" will translate all 4-tuple fields. To see other modes, refer cxgbtool manual page. For more information and examples, refer to the User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ OVS Kernel Datapath Offload ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== The following dependencies must be installed: - GCC 4.6+ - Python 2.7+ - Python-six - Autoconf 2.63+ - Automake 1.10+ - libtool 2.4+ For the complete list, visit: http://docs.openvswitch.org/en/latest/intro/install/general/ Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install OVS modules and NIC driver. [root@host~]# make ovs_install NOTE: For more installation options, run "make help" or "install.py -h". 3. Reboot your machine for changes to take effect. Configuration ============= Configuring OVS Machine ----------------------- The following example explains the method to configure an OVS machine: ------------ -------- | OVS Switch | -------- | HOST A | ------->| eth2 eth3 |<------- | HOST B | -------- ------------ -------- "eth2" and "eth3" are Chelsio interfaces. 1. Ensure that Unified Wire in installed with "High Capacity Hash Filter" configuration tuning option. 2. Update the "filterMode" and "filterMask" in the hash config file in /lib/firmware/cxgb4/. Select a Filter Mode combination with fragmentation, ethertype, protocol and port from the supported list. Use "t6-config.txt" for T6 adapters and "t5-config.txt" for T5 adapters: filterMode = fragmentation, mpshittype, ethertype, protocol, tos, port, fcoe filterMask = fragmentation, ethertype, protocol, port NOTE: filterMask tuples can be subset of or equal to filterMode tuples. 3. Load NIC (cxgb4) driver with hash-filter support. [root@host~]# modprobe cxgb4 use_ddr_filters=1 4. Bring up the Chelsio interfaces in promiscuous mode. [root@host~]# ifconfig eth2 promisc up [root@host~]# ifconfig eth3 promisc up 5. Load Open vSwitch module. [root@host~]# modprobe openvswitch 6. Configure OVS. [root@host~]# ovs-appctl exit [root@host~]# pkill -9 ovs [root@host~]# rm -rf /usr/local/etc/ovs-vswitchd.conf [root@host~]# rm -rf /usr/local/var/run/openvswitch/db.sock [root@host~]# rm -rf /usr/local/etc/openvswitch/conf.db [root@host~]# touch /usr/local/etc/ovs-vswitchd.conf [root@host~]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db /src/openvswitch-x.x.x/vswitchd/vswitch.ovsschema [root@host~]# ovsdb-server /usr/local/etc/openvswitch/conf.db --remote=punix:/usr/local/var/run/openvswitch/db.sock \ --remote=db:Open_vSwitch,Open_vSwitch,manager_options --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach --log-file [root@host~]# ovs-vsctl --no-wait init [root@host~]# export DB_SOCK=/usr/local/var/run/openvswitch/db.sock [root@host~]# ovs-vswitchd --pidfile --detach 7. Create an OVS bridge and add Chelsio interfaces to it. [root@host~]# ovs-vsctl add-br br0 [root@host~]# sleep 2 [root@host~]# ifconfig br0 up [root@host~]# ovs-vsctl add-port br0 eth2 [root@host~]# sleep 5 [root@host~]# ovs-vsctl add-port br0 eth3 [root@host~]# sleep 5 [root@host~]# ovs-vsctl show NOTE: Ports on OVS bridge must be added in the same order as the adapter, since there's no mapping between OVS and physical ports. 8. Now ping from Host A to Host B to verify that OVS is configured successfully. 9. Stop the ping traffic and delete all the flows on switch. [root@host~]# ovs-ofctl del-flows br0 Creating OVS flows ------------------ It is mandatory to specify L2 Ethernet Type (dl_type) to offload OVS flows. There are two types of flows: - exact-match: Protocol and 4-tuple are mandatory to create an exact-match flow. ~0.5 million exact-match flows can be offloaded. - wild-card: If any of 4-tuple and protocol are absent, wild-card flow is created. 496 wild-card flows can be offloaded. NOTE: - T5/T6 SO adapters do not support exact-match flows. You can create 494 wild-card flows on these adapters. - To view OVS flow (and VXLAN flow) examples, refer to the User Guide. Verifying OVS Flow Dump ---------------------- OVS flow dump can be verified using: [root@host~]# ovs-ofctl dump-flows br0 Run traffic between hosts which matches the flow and verify if the 'n_packet' counter is incrementing. To check if the OVF Flows were offloaded, run the below command: [root@host~]# cxgbtool ethX filter show Wild-card flows are displayed as LE-TCAM Filters and Exact-match flows are displayed as Hash Filters. Hits and Hit-Bytes are incremented for the corresponding filters. Setting up ODL with OVS ----------------------- The following example explains the method to set up OpenDaylight (ODL) using OVS: ------------ | ODL | | Controller | ------------ ^ | |(private network) ------------ -------- | OVS_Switch | -------- | HOST A | ------->| eth2 eth3 |<------- | HOST B | -------- ------------ -------- On the ODL controller setup, 1. Download latest Java Development Kit. 2. Extract the tar file. 3. Create an entry in .bashrc which points to the extracted folder. export JAVA_HOME=/jdk1.8.0_92 export PATH=$PATH:$JAVA_HOME 4. Log out and log in again. 5. Download ODL controller pre-built zip package. 6. Unzip the package and change your working directory to "opendaylight" 7. Run the script "run.sh" and wait for ~3 minutes for the controller to be setup. 8. Open a web browser and enter the address http://localhost:8080 9. Login with admin keyword for both username and password. 10. On the OVS machine, add the bridge to the controller and disable in-band. [root@host ~]# ovs-vsctl set-controller br0 tcp::6633 [root@host ~]# ovs-vsctl set bridge br0 other-config:disable-in-band=true 11. Refresh the webpage on the ODL controller and you should see the OVS details. 12. Goto "Flows" tab, add and install a flow. 13. Verify the flow dump on the OVS machine. [root@host ~]# ovs-ofctl dump-flows br0 Run traffic between hosts which matches the flow and verify if the "n_packet" counter is incrementing. Driver Uninstallation ===================== [root@host~]# make ovs_uninstall ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Mesh Topology ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Mesh Connectivity ================= Each node should be connected to other node. Supported configs using this approach: N ports per node, N+1 node cluster. NOTE: Refer to the User's Guide for more information. Installation ============ Install Unified Wire on all the machines in the mesh topology. 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Install the drivers, tools and libraries. [root@host~]# make install NOTE: For more installation options, run make help or install.py -h 3. Reboot your machine for changes to take effect. Configuration ============= Configure all the machines in the mesh topology using the below steps. IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb 1. Load network driver. [root@host~]# modprobe cxgb4 2. Configure interfaces with required IPs and networking as mentioned in https://access.redhat.com/solutions/30564 article. You should be able to run traffic between the nodes. To run different protocol traffic, refer to their respective sections for protocol configuration. NOTE: Refer to the User's Guide for detailed example.  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Traffic Management ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb Traffic Management can be performed on non-offloaded connections as well as on offloaded connections. The drivers must be loaded by the root user. Any attempt to load the drivers as a regular user may fail. Run the following commands to load the TOE driver: [root@host~]# modprobe cxgb4 [root@host~]# modprobe t4_tom Usage ===== Traffic Management Rules ------------------------ Traffic Management supports the following types of scheduler hierarchy levels which can be configured using the cxgbtool utility: - Class Rate Limiting - Class Weighted Round Robin - Channel Rate Limiting For more information, refer to the "Traffic Management" chapter in the User's Guide. Traffic Management of Non-Offloaded Connections ----------------------------------------------- The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for Non-offload connections: 1. Load the network driver and bring up the interface. [root@host~]# modprobe cxgb4 [root@host~]# ifconfig eth0 up 2. Bind connections with destination IP address 192.168.5.3 to NIC TX queue 3. [root@host~]# tc qdisc add dev eth0 root handle 1: multiq [root@host~]# tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dst 192.168.5.3 action skbedit queue_mapping 3 3. Bind the NIC TX queue to class 0. [root@host~]# cxgbtool eth0 sched-queue 3 0 4. Set the appropriate rule for class 0. [root@host~]# cxgbtool eth0 sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 IMPORTANT: Flow mode is not supported for Non-Offloaded Connections. Traffic Management of Offloaded Connections ------------------------------------------- The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for offloaded connections: 1. Load the TOE driver and bring up the interface. [root@host~]# modprobe t4_tom [root@host~]# ifconfig eth0 up 2. Create a new policy file (say new_policy_file) and add the following line to associate connections with the given scheduling class. src host 102.1.1.1 => offload class 0 NOTE: If no specified rule matches a connection, a default setting is used which disables offload for that connection. That is, there always be a final implicit rule following all the rules in the input rule set of: all => !offload 3. Compile the policy file using COP. [root@host~]# cop -d -o 4. Apply the COP policy. [root@host~]# cxgbtool eth0 policy NOTE: The policy applied using cxgbtool is not persistent and should be applied every time drivers are reloaded or the machine is rebooted. The applied cop policies can be read using, [root@host~]# cat /proc/net/offload/toeX/read-cop 5. Set the appropriate rule for class 0 [root@host~]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 Traffic Management of Offloaded Connections with Modified Application --------------------------------------------------------------------- The following example demonstrates the method to rate limit all TCP connections on class 0 to a rate of 300 Mbps for offloaded connections with modified application. 1. Load the TOE driver and bring up the interface. [root@host~]# modprobe t4_tom [root@host~]# ifconfig eth0 up 2. Modify the application as mentioned in the Configuring Traffic Management section in the User's Guide. 3. Set the appropriate rule for class 0. [root@host~]# cxgbtool ethX sched-class params type packet level cl-rl mode class rate-unit bits rate-mode absolute channel 0 class 0 max-rate 300000 pkt-size 1460 NOTE: For more information on additional parameters, refer to the cxgbtool manual by running the "man cxgbtool" command. NOTE: For more information on additional configuration options, refer to the User's Guide. Traffic Management of Inline TLS Offload Connections ---------------------------------------------------- Refer to the Inline TLS Offload chapter to go through configuration steps. To rate limit Inline TLS Offload connections, follow the steps mentioned below: 1. Load the TOE driver and bring up the interface. [root@host~]# modprobe t4_tom [root@host~]# ifconfig eth0 up 2. Create a new policy file and add the following line for TCP port (to be TLS offloaded), 443 in this case. Bind the connections to class 0. src or dst port 443 => offload tls mss 32 bind random class 0 all => offload The all => offload is added to ensure that rest of the TCP ports are regular TOE offloaded. 3. Compile the policy file using COP. [root@host~]# cop -d –o 4. Apply the COP policy. [root@host~]# cxgbtool ethX policy NOTE: The policy applied using cxgbtool is not persistent and should be applied every time drivers are reloaded or the machine is rebooted. The applied cop policies can be read using, [root@host~]# cat /proc/net/offload/toeX/read-cop 5. Set the appropriate rule for class 0 with the required rate and burst size 16384. [root@host~]# cxgbtool ethX sched-class params type packet level cl-rl mode flow rate-unit bits rate-mode absolute channel 0 class 0 max-rate 5000 pkt-size 1460 burst-size 16384 This rule makes rate limit for all Inline TLS connections on class 0 to 5 Mbps per connection. Driver Unloading ================ Reboot the system to unload the driver. To unload without rebooting, refer to the "Unloading the driver" in Network (NIC/TOE) section. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Unified Boot Software ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== A DOS bootable USB flash drive or Floppy Disk is required for updating firmware, option ROM etc. Secure Boot =========== Secure Boot, a high-performance computing software solution is a method to restrict which binaries can be executed to boot the system. With Secure Boot, the system BIOS allows only the execution of boot loaders that carry the cryptographic signature of trusted entities. In other words, anything run in the BIOS must be “signed” with a key that the system knows is trustworthy. With each reboot of the server, every executed component is verified. The Chelsio Drivers are in-boxed in major Linux Distributions mentioned in the Software Requirements and can be used for OS installation after enabling Secure Boot in System BIOS. Flashing Firmware & Option ROM ============================== Depending on the boot mode selected, Chelsio Unified Boot provides the following methods to flash Firmware, Option ROM and boot configuration into the Chelsio adapters: Legacy mode: - cfut4 uEFI mode: - HII - drvcfg - Firmware Manager Protocol (FMP) OS Level: - cxgbtool These methods also provide the functionality to update/erase Hardware configuration and Phy Firmware files. IMPORTANT: It is highly recommended to use the same Option ROM (type and version) on all the Chelsio adapters present in the system. Preparing USB flash drive ------------------------- This document assumes that you are using an USB flash drive as a storage media for the necessary files. Follow the steps below to prepare the drive: 1. Create a DOS bootable USB flash drive. 2. Create a directory "CHELSIO" on USB flash drive. 3. If you have not done already, download the Chelsio Unified Wire driver package from Chelsio Download Center, service.chelsio.com 4. Extract the downloaded package and change your working directory to "OptionROM" directory. [root@host~]# tar zxvfm ChelsioUwire-x.x.x.x.tar.gz [root@host~]# cd ChelsioUwire-x.x.x.x/Uboot/OptionROM 5. Copy all the files and place them in the CHELSIO directory created on the USB flash drive. 6. Plug-in the USB flash drive in the system on which the Chelsio CNA is installed. 7. Reboot the system. Legacy ------ 1. Configure the system having Chelsio adapter to boot in Legacy mode. 2. Once the system boots from the USB flash drive, change your working directory to the Chelsio directory. C:\>cd CHELSIO 3. Run the following command to list all the Chelsio adapters present in the system. The list displays a unique index for each adapter found. C:\CHELSIO>cfut4 -l 4. Delete any previous version of Option ROM flashed onto the adapter. C:\CHELSIO>cfut4 -d -xb Here, idx is the adapter index found in step 3. 5. Delete any previous firmware using the following command. C:\CHELSIO>cfut4 -d -xh -xf 6. Delete any previous Option ROM settings. C:\CHELSIO>cfut4 -d -xc 7. Run the following command to flash the appropriate firmware. C:\CHELSIO>cfut4 -d -uf .bin 8. Flash the Unified Boot Option ROM using the following command. C:\CHELSIO>cfut4 -d -ub cubt4.bin Here, "cubt4.bin" is the Unified Boot Option ROM image file present in the CHELSIO directory. 9. Flash the boot configuration setting which enables PXE and disable iSCSI and FCoE. C:\CHELSIO>cfut4 -d -uc boot.cfg 10. In case of multiple adapters, repeat steps 4 through 9 to update/flash the firmware, Option ROM and boot configuration on all of them. 11. Reboot the system for changes to take effect. 12. To configure the base MAC address (optional), use the below command: C:\CHELSIO>cfut4 -d -um For example, C:\CHELSIO>cfut4 -d 0 -um 000743000123 uEFI ----- To configure Chelsio adapter using HII in uEFI mode, refer to the User's Guide. cxgbtool (OS Level) ------------------- Follow the steps mentioned below to flash the Option ROM onto Chelsio adapters using cxgbtool utility: 1. If not done already, install the Network driver and cxgbtool. [root@host~]# cd ChelsioUwire-x.x.x.x [root@host~]# make install 2. Load the Network driver. [root@host~]# modprobe cxgb4 3. Delete any previous version of Option ROM flashed onto the adapter. [root@host~]# cxgbtool ethX loadboot clear 4. Flash the Option ROM onto the Chelsio adapter. [root@host~]# cd ChelsioUwire-x.x.x.x/Uboot/OptionROM/ [root@host~]# cxgbtool ethX loadboot cubt4.bin 5. Flash the default boot configuration onto the adapter. [root@host~]# cd ChelsioUwire-x.x.x.x/Uboot/OptionROM/ [root@host~]# cxgbtool ethX loadboot-cfg boot.cfg 6. In case of multiple adapters in the system, repeat the steps from 3 through 5 to update/flash the Option ROM on all the adapters. 7. Reboot the system for changes to take effect. Update Option ROM Settings ========================== Default Settings ---------------- If you wish to restore Option ROM settings to their default values. That is PXE enabled, iSCSI and FCoE disabled, use any of the methods mentioned below: - Using Option ROM (boot level) For Legacy PXE, boot system into Chelsio’s Unified Boot Setup utility and press F8. For uEFI PXE, boot system into uEFI mode and press F3. - Using cxgbtool (OS level) Change your working directory to OptionROM directory and use cxgbtool to flash the default boot configuration onto the adapter: [root@host~]# cd ChelsioUwire-x.x.x.x/Uboot/OptionROM/ [root@host~]# cxgbtool loadboot-cfg boot.cfg The below command can be used to read the current settings. [root@host~]# cxgbtool readboot-cfg Custom Settings (using cxgbtool) -------------------------------- Use the below command to enable/disable PXE/FCoE/iSCSI boot for all the ports of the adapter. [root@host~]# cxgbtool modifyboot-cfg bios Where, ethX : Chelsio interface. value : Bitwise OR of boot types that need to be enabled. Ranging from 0x0 – 0x7. PXE (NIC) = 0x1 FCoE = 0x2 iSCSI = 0x4 Use the below command to enable/disable PXE (NIC) boot per port. [root@host~]# cxgbtool modifyboot-cfg port Where, ethX : Chelsio interface. port no. : Port number ranging from 0 – 3. param : en_nicboot to enable and dis_nicboot to disable NIC boot for the port. Use the below command to set the VLAN id for the port. [root@host~]# cxgbtool modifyboot-cfg port vlan Where, ethX : Chelsio interface. port no. : Port number ranging from 0 – 3. id : VLAN id ranging from 0 – 4095. iPXE ---- iPXE is the leading, free, open source network boot firmware. It provides a full PXE implementation enhanced with additional features such as boot from a web server via HTTP, automating boot commands etc. The following are supported: - TFTP Boot: Legacy & uEFI - HTTP Boot: uEFI - Linux (RHEL 9.2), Windows (10 Client, Server 2019/2022) - Chainload into iPXE To configure iPXE, refer to the User's Guide. 4. Support Documentation ================================================================================ The documentation for this release can be found inside the ChelsioUwire-x.x.x.x/docs folder. It contains: - README - Release Notes - User's Guide 6. Customer Support ================================================================================ Installer issues ---------------- In case of any failures while running the Chelsio Unified Wire Installer, collect the below: - install.log fille, if installed using install.py - Entire make command output, if installed using the makefile Logs collection --------------- For any other issues, run the below command to collect all the necessary log files: [root@host~]# chdebug A compressed tar ball, chelsio_debug_logs_with_cudbg.tar.bz2 will be created with all the logs. For kernel panics, following files need to be provided for analysis. vmcore, vmcore-dmesg.txt, vmlinux, System.map-$(uname -r), Chelsio modules .ko files   Contact the Chelsio support at: support@chelsio.com with relevant logs for any issues. ******************************************************************************** Copyright (C) 2024 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means without the express written consent of Chelsio Communications.