**************************************** README **************************************** Chelsio Unified Wire for Linux Version : 4.2.0.8-1 Date : 11/02/2026 Overview ================================================================================ Chelsio Unified Wire software for Linux is an easy to use utility developed to provide installation of 64-bit Linux based drivers and tools for Chelsio's Unified Wire adapters. The Chelsio Unified Wire package provides an interactive installer to install various drivers and utilities. Chelsio Unified Wire consists of the following components: - Network (NIC/TOE) - Port Forwarding Mode (Hardware Bypass Mode/Hairpin Mode/LAN Bypass) - Virtual Function Network (vNIC) - iWARP RDMA Offload - RoCEv2 - NVMe-oF iWARP (Target and Initiator) - NVMe-oF RoCEv2 (Target) - SPDK (user space) NVMe-oF iWARP (Target and Initiator) - SPDK (user space) NVMe-oF RoCEv2 (Target and Initiator) - NVMe-oF TOE (Target and Initiator) - SPDK (user space) NVMe-oF TOE (Target) - NVMe-oF/TCP PDU Offload (Target and Initiator) - SPDK (user space) NVMe/TCP PDU Offload (Target and Initiator) - SPDK (user space) NVMe/TCP over Memfree TOE (Target and Initiator) - SoftiWARP Initiator - LIO iSCSI Target Offload - iSCSI PDU Offload Initiator - SPDK (user space) iSCSI PDU Offload (Target and Initiator) - SPDK (user space) iSCSI over Memfree TOE (Target) - Crypto Offload - QUIC TX Offload - Inline IPSec Offload - iSER (Target and Initiator) - WD-UDP - Data Center Bridging (DCB) - FCoE Full Offload Initiator - Offload Bonding - UDP Segmentation Offload and Pacing - Offload IPV6 - WD Sniffing and Tracing - Classification and Filtering - Mesh Topology - Traffic Management (TM) - Unified Boot Software - Utility Tools (cop,cxgbtool,t4_perftune) - NC-SI (Network Controller Sideband Interface) Disclaimer: This initial release may contain corner-case and platform-specific issues. ================================================================================ CONTENTS ================================================================================ - 1. Supported Operating Systems - 2. Supported Hardware - 3. How To Use - 4. Support Documentation - 5. Customer Support - 6. Limitations 1. Supported Operating Systems ================================================================================ The Chelsio software has been developed to run on 64-bit Linux based platforms. Following is the list of supported Linux distribution/Kernel(s). |#############################|#################################################| | Linux Distribution | Driver/Software | |#############################|#################################################| |RHEL/Rocky/AlmaLinux 10.2, |Network(NIC/TOE),Precision Time Protocol (PTP), | |6.12.0-211.7.3.el10_2.x86_64 |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator)^,NVMe-oF TOE (Target and | | |Initiator), SPDK (user space) NVMe-oF TOE | | |(Target),SPDK(user space) NVMe/TCP PDU Offload | | |(Target and Initiator),SPDK (user space) NVMe/TCP| | |over Memfree TOE (Target and Initiator)*, | | |SPDK (user space) iSCSI PDU Offload | | |(Target and Initiator)*,SPDK (user space) iSCSI | | |over Memfree TOE (Target)^,LIO iSCSI Target | | |Offload,iSCSI PDU Offload Initiator,Crypto | | |Offload,QUIC TX Offload,Inline IPSec Offload, | | |iSER (Target and Initiator),WD-UDP,Data Center | | |Bridging (DCB),FCoE Full Offload Initiator, | | |Offload Bonding,UDP Segmentation Offload and | | |Pacing,Offload IPV6,WD Sniffing and Tracing, | | |Classification and Filtering,Mesh Topology, | | |Traffic Management feature(TM),Unified Boot | | |Software,Utility Tools (cop,cxgbtool, | | |t4_perftune),NC-SI (Network Controller Sideband | | |Interface) | |-----------------------------|-------------------------------------------------| |RHEL/Rocky/AlmaLinux 10.1, |Network(NIC/TOE),Precision Time Protocol (PTP), | |6.12.0-124.8.1.el10_1.x86_64 |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator)^,NVMe-oF TOE (Target and | | |Initiator), SPDK (user space) NVMe-oF TOE | | |(Target),SPDK(user space) NVMe/TCP PDU Offload | | |(Target and Initiator),SPDK (user space) NVMe/TCP| | |over Memfree TOE (Target and Initiator)*, | | |SPDK (user space) iSCSI PDU Offload | | |(Target and Initiator)*,SPDK (user space) iSCSI | | |over Memfree TOE (Target)^,LIO iSCSI Target | | |Offload,iSCSI PDU Offload Initiator,Crypto | | |Offload,QUIC TX Offload,Inline IPSec Offload,iSER| | |(Target and Initiator),WD-UDP,Data Center | | |Bridging (DCB),FCoE Full Offload Initiator, | | |Offload Bonding,UDP Segmentation Offload and | | |Pacing,Offload IPV6,WD Sniffing and Tracing, | | |Classification and Filtering,Mesh Topology, | | |Traffic Management feature(TM),Unified Boot | | |Software,Utility Tools (cop,cxgbtool, | | |t4_perftune),NC-SI (Network Controller Sideband | | |Interface) | |-----------------------------|-------------------------------------------------| |RHEL/Rocky/AlmaLinux 10.0, |Network(NIC/TOE),Precision Time Protocol (PTP), | |6.12.0-55.9.1.el10_0.x86_64 |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator)^,NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE | | |(Target),SPDK(user space) NVMe/TCP PDU Offload | | |(Target and Initiator),SPDK (user space) NVMe/TCP| | |over Memfree TOE (Target and Initiator)*, SPDK | | |(user space) iSCSI PDU Offload (Target and | | |Initiator)*,SPDK (user space) iSCSI over Memfree | | |TOE(Target)*,LIO iSCSI Target Offload,iSCSI PDU | | |Offload Initiator,Crypto Offload,QUIC TX Offload,| | |Inline IPSec Offload,iSER (Target and Initiator),| | |WD-UDP,Data Center Bridging (DCB), FCoE Full | | |Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing,Offload IPV6, | | |WD Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Traffic Management | | |feature(TM),Unified Boot Software,Utility Tools | | |(cop,cxgbtool,t4_perftune),NC-SI (Network | | |Controller Sideband Interface) | |-----------------------------|-------------------------------------------------| |RHEL/Rocky/AlmaLinux 9.8, |Network(NIC/TOE),Precision Time Protocol (PTP), | |5.14.0-687.5.3.el9_8.x86_64 |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator)^,NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE | | |(Target),SPDK(user space) NVMe/TCP PDU Offload | | |(Target and Initiator),SPDK (user space) NVMe/TCP| | |over Memfree TOE (Target and Initiator)*,SPDK | | |(user space) iSCSI PDU Offload (Target and | | |Initiator)*,SPDK (user space) iSCSI over Memfree | | |TOE (Target)*,LIO iSCSI Target Offload,iSCSI PDU | | |Offload Initiator,Crypto Offload,QUIC TX Offload,| | |Inline IPSec Offload,iSER (Target and Initiator),| | |WD-UDP,Data Center Bridging (DCB),FCoE Full | | |Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing,Offload IPV6,WD | | |Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Traffic Management | | |feature(TM),Unified Boot Software,Utility Tools | | |(cop,cxgbtool,t4_perftune),NC-SI (Network | | |Controller Sideband Interface) | |-----------------------------|-------------------------------------------------| |RHEL/Rocky/AlmaLinux 9.7, |Network(NIC/TOE),Precision Time Protocol (PTP), | |5.14.0-611.5.1.el9_7.x86_64 |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator)^,NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE | | |(Target),SPDK(user space) NVMe/TCP PDU Offload | | |(Target and Initiator),SPDK (user space) NVMe/TCP| | |over Memfree TOE (Target and Initiator)*,SPDK | | |(user space) iSCSI PDU Offload (Target and | | |Initiator)*,SPDK (user space) iSCSI over Memfree | | |TOE (Target)*,LIO iSCSI Target Offload,iSCSI PDU | | |Offload Initiator,Crypto Offload,QUIC TX Offload,| | |Inline IPSec Offload,iSER (Target and Initiator),| | |WD-UDP,Data Center Bridging (DCB),FCoE Full | | |Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing,Offload IPV6,WD | | |Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Traffic Management | | |feature(TM),Unified Boot Software,Utility Tools | | |(cop,cxgbtool,t4_perftune),NC-SI (Network | | |Controller Sideband Interface) | |-----------------------------|-------------------------------------------------| |RHEL/Rocky/AlmaLinux 9.6, |Network(NIC/TOE),Precision Time Protocol (PTP), | |5.14.0-570.12.1.el9_6.x86_64 |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |and Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator)^,NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE | | |(Target),SPDK(user space) NVMe/TCP PDU Offload | | |(Target and Initiator),SPDK (user space) NVMe/TCP| | |over Memfree TOE (Target and Initiator)*, SPDK | | |(user space) iSCSI PDU Offload (Target and | | |Initiator)*,SPDK (user space) iSCSI over Memfree | | |TOE(Target)*,LIO iSCSI Target Offload,iSCSI PDU | | |Offload Initiator,Crypto Offload,QUIC TX Offload,| | |Inline IPSec Offload,iSER (Target and Initiator),| | |WD-UDP,Data Center Bridging (DCB), FCoE Full | | |Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing,Offload IPV6,WD | | |Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Traffic Management | | |feature(TM),Unified Boot Software,Utility Tools | | |(cop,cxgbtool,t4_perftune),NC-SI (Network | | |Controller Sideband Interface) | |-----------------------------|-------------------------------------------------| |Ubuntu 24.04.4, |Network(NIC/TOE),Precision Time Protocol (PTP), | |6.8.0-136-generic |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |and Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator)^,NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE | | |(Target),SPDK(user space) NVMe/TCP PDU Offload | | |(Target and Initiator),SPDK (user space) NVMe/TCP| | |over Memfree TOE (Target and Initiator)*, SPDK | | |(user space) iSCSI PDU Offload (Target and | | |Initiator)*,SPDK (user space) iSCSI over Memfree | | |TOE (Target)*,LIO iSCSI Target Offload,iSCSI PDU | | |Offload Initiator,Crypto Offload,QUIC TX Offload,| | |Inline IPSec Offload,iSER (Target and Initiator),| | |WD-UDP,Data Center Bridging (DCB), FCoE Full | | |Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing,Offload IPV6,WD | | |Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Traffic Management | | |feature(TM),Unified Boot Software,Utility Tools | | |(cop,cxgbtool,t4_perftune),NC-SI (Network | | |Controller Sideband Interface) | |-----------------------------|-------------------------------------------------| |Kernel.org linux-6.18.39 |Network(NIC/TOE),Precision Time Protocol (PTP), | | |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target),SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator),NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE | | |(Target),NVMe-oF/TCP PDU Offload (Initiator), | | |SPDK(user space) NVMe/TCP PDU Offload (Target and| | |Host),SPDK (user space) NVMe/TCP over Memfree | | |TOE (Target and Initiator)*,SoftiWARP Initiator, | | |LIO iSCSI Target Offload,iSCSI PDU Offload | | |Initiator,SPDK (user space) iSCSI PDU Offload | | |(Target and Initiator)*,SPDK (user space) iSCSI | | |over Memfree TOE (Target)*,Crypto Offload,QUIC TX| | |Offload,Inline IPSec Offload,iSER (Target and | | |Initiator),WD-UDP,Data Center Bridging (DCB), | | |FCoE Full Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing,Offload IPV6, | | |WD Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Traffic Management | | |feature(TM),Unified Boot Software,Utility Tools | | |(cop,cxgbtool,t4_perftune),NC-SI (Network | | |Controller Sideband Interface) | |-----------------------------|-------------------------------------------------| |Kernel.org linux-6.12.104 |Network(NIC/TOE),Precision Time Protocol (PTP), | | |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target),SPDK | | |(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator),NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE | | |(Target),NVMe-oF/TCP PDU Offload (Target and | | |Initiator),SPDK(user space) NVMe/TCP PDU Offload | | |(Target and Initiator),SPDK (user space) NVMe/TCP| | |over Memfree TOE (Target and Initiator)*, | | |SoftiWARP Initiator,LIO iSCSI Target Offload, | | |iSCSI PDU Offload Initiator,SPDK (user space) | | |iSCSI PDU Offload(Target and Initiator)*,SPDK | | |(user space)iSCSI over Memfree TOE (Target)*, | | |Crypto Offload,QUIC TX Offload,Inline IPSec | | |Offload,iSER (Target and Initiator),WD-UDP,Data | | |Center Bridging (DCB),FCoE Full Offload | | |Initiator,Offload Bonding,UDP Segmentation | | |Offload and Pacing,Offload IPV6,WD Sniffing and | | |Tracing,Classification and Filtering,Mesh | | |Topology,Traffic Management feature(TM),Unified | | |Boot Software,Utility Tools (cop,cxgbtool, | | |t4_perftune),NC-SI (Network Controller Sideband | | |Interface) | |-----------------------------|-------------------------------------------------| |Kernel.org linux-6.6.144 |Network(NIC/TOE),Precision Time Protocol (PTP), | | |NC-SI (Network Controller Sideband Interface) | |-----------------------------|-------------------------------------------------| * Limited testing has been performed ^^Limited number of connections supported 2. Supported Hardware ================================================================================ Chelsio Drivers/Software and supported adapters =============================================== |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T72200-FH |Network(NIC/TOE),Precision Time Protocol (PTP), | | |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and Initiator), | | |SPDK (user space) NVMe-oF RoCEv2 (Target and | | |Initiator)^,NVMe-oF TOE (Target and Initiator), | | |NVMe-oF/TCP PDU Offload (Initiator), SPDK(user | | |space) NVMe/TCP PDU Offload (Target and Initiator), | | |SoftiWARP Initiator,SPDK (user space),iSCSI PDU | | |Offload (Target and Initiator)*,Crypto Offload, | | |QUIC TX Offload,Inline IPSec Offload,iSER (Target and| | |Initiator),Offload IPV6,Classification and Filtering,| | |Traffic Management feature(TM),Unified Boot Software,| | |Utility Tools(cop,cxgbtool,t4_perftune),NC-SI | | |(Network Controller Sideband Interface) | |------------------------|-----------------------------------------------------| |T72200 |Network(NIC/TOE),Precision Time Protocol (PTP), | | |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and Initiator), | | |SPDK (user space) NVMe-oF RoCEv2 (Target and | | |Initiator)^,NVMe-oF TOE (Target and Initiator), | | |NVMe-oF/TCP PDU Offload (Initiator), SPDK(user space)| | |NVMe/TCP PDU Offload (Target and Initiator),SoftiWARP| | |Initiator,SPDK (user space) iSCSI PDU Offload | | |(Target and Initiator)*, Crypto Offload,QUIC TX | | |Offload,Inline IPSec Offload,iSER (Target and | | |Initiator),Offload IPV6,Classification and Filtering,| | |Traffic Management feature(TM),Unified Boot | | |Software,Utility Tools (cop,cxgbtool,t4_perftune), | | |NC-SI (Network Controller Sideband Interface) | |------------------------|-----------------------------------------------------| |T7450 |Network(NIC/TOE),Precision Time Protocol (PTP), | | |Virtual Function Network (vNIC),iWARP RDMA | | |Offload,RoCEv2^,NVMe-oF iWARP (Target and | | |Initiator),NVMe-oF RoCEv2 (Target)^,SPDK | | |(user space) NVMe-oF iWARP (Target and Initiator), | | |SPDK (user space) NVMe-oF RoCEv2 (Target and | | |Initiator)^,NVMe-oF TOE (Target and Initiator), | | |NVMe-oF/TCP PDU Offload (Initiator), SPDK(user | | |space) NVMe/TCP PDU Offload (Target and Initiator), | | |SoftiWARP Initiator,SPDK (user space) iSCSI PDU | | |Offload (Target and Initiator)*,Crypto Offload, | | |QUIC TX Offload,Inline IPSec Offload,iSER | | |(Target and Initiator),Offload IPV6,Classification | | |and Filtering,Traffic Management feature(TM),Unified | | |Boot Software,Utility Tools (cop,cxgbtool, | | |t4_perftune),NC-SI (Network Controller Sideband | | |Interface) | |------------------------|-----------------------------------------------------| |S72200 |Network(NIC),Precision Time Protocol (PTP),RoCEv2^, | | |NVMe-oF RoCEv2 (Target)^,SPDK (user space) | | |NVMe-oF RoCEv2 (Target and Initiator),SPDK (user | | |space) NVMe/TCP over Memfree TOE (Target and | | |Initiator)*,SoftiWARP Initiator,SPDK (user space) | | |iSCSI PDU Offload (Target and Initiator),SPDK (user | | |space) iSCSI over Memfree TOE (Target)*,Crypto | | |Offload (Coprocessor Only),QUIC TX Offload,Unified | | |Boot Software,Utility Tools(cop,cxgbtool, | | |t4_perftune),NC-SI (Network Controller Sideband | | |Interface) | |------------------------|-----------------------------------------------------| |S71400 |Network(NIC),Precision Time Protocol (PTP),RoCEv2^, | | |NVMe-oF RoCEv2 (Target)^,SPDK (user space) | | |NVMe-oF RoCEv2 (Target and Initiator),SPDK | | |(user space) NVMe/TCP over Memfree TOE (Target | | |and Initiator)*,SoftiWARP Initiator,SPDK (user | | |space) iSCSI PDU Offload (Target and Initiator),SPDK | | |(user space) iSCSI over Memfree TOE (Target)*, | | |Crypto Offload (Coprocessor Only),Unified Boot | | |Software,Utility Tools (cop,cxgbtool,t4_perftune), | | |NC-SI (Network Controller Sideband Interface) | |------------------------|-----------------------------------------------------| |S7450 |Network(NIC),Precision Time Protocol (PTP), | | |RoCEv2^,NVMe-oF RoCEv2 (Target)^,SPDK (user space) | | |NVMe-oF RoCEv2 (Target and Initiator),SPDK (user | | |space) NVMe/TCP over Memfree TOE (Target and | | |Initiator)*,SoftiWARP Initiator,SPDK (user space) | | |iSCSI PDU Offload (Target and Initiator),SPDK (user | | |space) iSCSI over Memfree TOE (Target)*,Crypto | | |Offload (Coprocessor Only),Unified Boot Software, | | |Utility Tools (cop,cxgbtool,t4_perftune),NC-SI | | |(Network Controller Sideband Interface) | |------------------------|-----------------------------------------------------| |S7250 |Network(NIC),Precision Time Protocol (PTP),RoCEv2^, | | |NVMe-oF RoCEv2 (Target)^,SPDK (user space) | | |NVMe-oF RoCEv2 (Target and Initiator),SPDK (user | | |space) NVMe/TCP over Memfree TOE (Target and | | |Initiator)*,SoftiWARP Initiator,SPDK (user space) | | |iSCSI PDU Offload (Target and Initiator),SPDK (user | | |space) iSCSI over Memfree TOE (Target)*,Crypto | | |Offload (Coprocessor Only),Unified Boot Software, | | |Utility Tools (cop,cxgbtool,t4_perftune),NC-SI | | |(Network Controller Sideband Interface) | |------------------------|-----------------------------------------------------| |T62100-CR |Network(NIC/TOE),Virtual Function Network (vNIC), | | |iWARP RDMA Offload,NVMe-oF iWARP (Target | | |and Initiator),SPDK (user space) NVMe-oF iWARP | | |(Target and Initiator),NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE (Target), | | |SoftiWARP Initiator,LIO iSCSI Target Offload,iSCSI | | |PDU Offload Initiator,Crypto Offload,iSER (Target | | |and Initiator), WD-UDP,Data Center Bridging (DCB), | | |FCoE Full Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing Offload IPV6, | | |WD Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Classification and Filtering,| | |Traffic Management feature(TM),Utility Tools | | |(cop,cxgbtool,t4_perftune) | |------------------------|-----------------------------------------------------| |T62100-LP-CR |Network(NIC/TOE),Virtual Function Network (vNIC), | | |iWARP RDMA Offload,NVMe-oF iWARP (Target | | |and Initiator),SPDK (user space) NVMe-oF iWARP | | |(Target and Initiator),NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE (Target), | | |SoftiWARP Initiator,LIO iSCSI Target Offload,iSCSI | | |PDU Offload Initiator,Crypto Offload,iSER (Target | | |and Initiator), WD-UDP,Data Center Bridging (DCB), | | |FCoE Full Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing Offload IPV6, | | |WD Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Classification and Filtering,| | |Traffic Management feature(TM),Utility Tools | | |(cop,cxgbtool,t4_perftune) | |------------------------|-----------------------------------------------------| |T62100-SO-CR |Network(NIC),SoftiWARP Initiator,Crypto Offload | | |(Coprocessor Only),Utility Tools(cop,cxgbtool, | | |t4_perftune) | |------------------------|-----------------------------------------------------| |T62100-SO-OCP3 |Network(NIC),SoftiWARP Initiator,Crypto Offload | | |(Coprocessor Only),Utility Tools(cop,cxgbtool, | | |t4_perftune) | |------------------------|-----------------------------------------------------| |T6225-CR |Network(NIC/TOE),Virtual Function Network (vNIC), | | |iWARP RDMA Offload,NVMe-oF iWARP (Target | | |and Initiator),SPDK (user space) NVMe-oF iWARP | | |(Target and Initiator),NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE (Target), | | |SoftiWARP Initiator,LIO iSCSI Target Offload,iSCSI | | |PDU Offload Initiator,Crypto Offload,iSER (Target | | |and Initiator), WD-UDP,Data Center Bridging (DCB), | | |FCoE Full Offload Initiator,Offload Bonding,UDP | | |Segmentation Offload and Pacing Offload IPV6, | | |WD Sniffing and Tracing,Classification and | | |Filtering,Mesh Topology,Classification and Filtering,| | |Traffic Management feature(TM),Utility Tools | | |(cop,cxgbtool,t4_perftune) | |------------------------|-----------------------------------------------------| |T6225-SO-CR |Network(NIC),SoftiWARP Initiator,Crypto Offload | | |(Coprocessor Only),Utility Tools(cop,cxgbtool, | | |t4_perftune) | |------------------------|-----------------------------------------------------| |T540-LP-CR |Network(NIC/TOE),Virtual Function Network (vNIC), | | |iWARP RDMA Offload,NVMe-oF iWARP (Target | | |and Initiator),SPDK (user space) NVMe-oF iWARP | | |(Target and Initiator),NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE (Target), | | |SoftiWARP Initiator,LIO iSCSI Target Offload,iSCSI | | |PDU Offload Initiator,iSER (Target and Initiator), | | |WD-UDP,Data Center Bridging (DCB), FCoE Full Offload | | |Initiator,Offload Bonding,UDP Segmentation Offload | | |and Pacing Offload IPV6, WD Sniffing and Tracing, | | |Classification and Filtering,Mesh Topology, | | |Classification and Filtering, Traffic Management | | |feature(TM),Utility Tools (cop,cxgbtool,t4_perftune) | |------------------------|-----------------------------------------------------| |T520-CR |Network(NIC/TOE),Virtual Function Network (vNIC), | | |iWARP RDMA Offload,NVMe-oF iWARP (Target | | |and Initiator),SPDK (user space) NVMe-oF iWARP | | |(Target and Initiator),NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE (Target), | | |SoftiWARP Initiator,LIO iSCSI Target Offload,iSCSI | | |PDU Offload Initiator,iSER (Target and Initiator), | | |WD-UDP,Data Center Bridging (DCB), FCoE Full Offload | | |Initiator,Offload Bonding,UDP Segmentation Offload | | |and Pacing Offload IPV6, WD Sniffing and Tracing, | | |Classification and Filtering,Mesh Topology, | | |Classification and Filtering,Traffic Management | | |feature(TM),Utility Tools (cop,cxgbtool,t4_perftune) | |------------------------|-----------------------------------------------------| |T520-SO-CR |Network(NIC),SoftiWARP Initiator,Utility Tools | | |(cop,cxgbtool,t4_perftune) | |------------------------|-----------------------------------------------------| ^Limited number of connections supported 3. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chelsio Unified Wire ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== - Package Manager yum should be configured using any of the OS recommended ways to resolve and install missing packages. - python3 should be installed for Chelsio Unified Wire package scripts to run. - make, gcc, kernel-devel, kernel-headers, openssl-devel, libibverbs, libibverbs-devel, librdmacm, librdmacm-devel, rdma-core-devel, cmake, libnl3, libnl3-devel, fuse3, fuse3-devel, fuse3-libs, numactl-devel, libuuid-devel, libaio-devel, ncurses-devel, pciutils-devel, meson, patchelf, wheel, pyelftools, CUnit-devel, libpsl-devel, uv, and ninja-build packages should be installed for the compilation of drivers and utilities. - Uninstall any OFED present in the machine. - Installation must be attempted as 'root' user. - Compilation of drivers can be attempted as non-root user using below command: [user@host ~]$ make KSRC="" KOBJ="" OR [user@host ~]$ make KDIR="" Example #1 - [user@host ChelsioUwire-x.x.x.x]$ make KSRC=/home/linux-5.14.0-611.5.1.el9_7/ KOBJ=/lib/modules/5.14.0-611.5.1.el9_7.x86_64/build Example #2 - [user@host ChelsioUwire-x.x.x.x]$ make KDIR=/usr/src/kernels/linux-6.12.95/ Example #3 - For Ubuntu [user@host ChelsioUwire-x.x.x.x]$ make KSRC=/root/linux-source-6.8.0/ KOBJ=/lib/modules/6.8.0-136-generic/build Installing Chelsio Unified Wire =============================== Chelsio Unified Wire package can be installed using source only. The source package supports CLI. |#############################|################################################| |Configuration Tuning Option | Driver/Software installed | |#############################|################################################| |Unified Wire (Default) |Network(NIC/TOE),PTP,Virtual Function Network | | |(vNIC),iWARP RDMA Offload,RoCEv2,NVMe-oF iWARP | | |(Target and Initiator),NVMe-oF RoCEv2 (Target), | | |SPDK(user space) NVMe-oF iWARP (Target and | | |Initiator), SPDK (user space) NVMe-oF RoCEv2 | | |(Target and Initiator),NVMe-oF TOE (Target and | | |Initiator),SPDK (user space) NVMe-oF TOE | | (Target),|NVMe-oF/TCP PDU Offload (Target and | | |Initiator),SPDK(user space) NVMe/TCP PDU Offload| | |(Target and Initiator),SPDK (user space)NVMe/TCP| | |over Memfree TOE (Target and Initiator), | | |SoftiWARP Initiator,LIO iSCSI Target Offload, | | |iSCSI PDU Offload Initiator,SPDK (user space) | | |iSCSI PDU Offload(Target and Initiator),SPDK | | |(user space) iSCSI over Memfree TOE (Target), | | |Crypto Offload,QUIC TX Offload,Inline IPSec | | |Offload,iSER (Target and Initiator),WD-UDP,Data | | |Center Bridging (DCB),FCoE Full Offload | | |Initiator,Offload Bonding,UDPSegmentation | | |Offload and Pacing,Offload IPV6,WD Sniffing and | | |Tracing,Classification and Filtering,Mesh | | |Topology,Traffic Management feature(TM),Unified | | |Boot Software,Utility Tools(cop,cxgbtool, | | |t4_perftune),NC-SI (Network Controller Sideband | | |Interface) | |-----------------------------|------------------------------------------------| Installation of Chelsio Unified Wire ------------------------------------ Follow the steps mentioned below for installation using CLI. For GUI or Menu based installation, refer the User's Guide. 1.1. From source ---------------- 1. Download Chelsio Unified Wire driver package. 2. Extract the downloaded package. [root@host~]# tar zxvfm ChelsioUwire-x.x.x.x.tar.gz 3. Change your current working directory to the Chelsio Unified Wire package directory and build the source. [root@host~]# cd ChelsioUwire-x.x.x.x [root@host~]# make 4. Install the drivers, tools and libraries. [root@host~]# make install 5. The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= install NOTE: To view the different configuration tuning options, view help by typing [root@host~]# make help 6. Reboot your machine for changes to take effect. IMPORTANT: Steps 3 and 4 mentioned above does NOT install Crypto, DCB, OVS drivers and benchmark tools. They have to be installed manually. Refer to their respective sections for instructions on installing them. Installation using additional flags ----------------------------------- Provided here are steps to build and install drivers using additional flags. For the complete list, view help by running "make help". Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x - To build and install all drivers without IPv6 support. [root@host~]# make ipv6_disable=1 [root@host~]# make ipv6_disable=1 install - The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= - To build and install drivers along with benchmarks. [root@host~]# make BENCHMARKS=1 [root@host~]# make BENCHMARKS=1 install - The drivers will be installed as RPMs or Debian packages (for Ubuntu/Debian). To skip this and install drivers, [root@host~]# make SKIP_RPM=1 install - The installer removes the Chelsio specific drivers (inbox/outbox) from initramfs. To skip this and install drivers, [root@host~]# make SKIP_INIT=1 install - The installer checks for the required dependency packages and will install them if they are missing from the machine. To skip this and install drivers, [root@host~]# make SKIP_DEPS=1 install NOTE: - To view the different configuration tuning options, view the help by typing [root@host~]# make help - If IPv6 is disabled in the machine, the drivers will be built and installed without IPv6 Offload support by default. iWARP Driver Installation on Cluster Nodes ------------------------------------------ IMPORTANT: Ensure that you have enabled password less authentication with ssh on the peer nodes for this feature to work. Chelsio's Unified Wire package allows installing iWARP drivers on multiple Cluster nodes with a single command. Follow the procedure mentioned below: 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed. 3. Now, execute the following command: [root@host~]# ./install.py -C -m 4. Select the required configuration tuning option. The tuning options may vary depending on the Linux distribution. 5. Select the required Cluster Configuration. 6. The selected components will now be installed. The above command installs iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes listed in the file. 1.2. From RPM (tarball) ----------------------- NOTE: - IPv6 should be enabled in the machine to use the RPM Packages. - Drivers installed from RPM Packages do not have DCB support. 1. Download the tarball specific to your operating system and architecture. 2. Extract the downloaded package. Example: For RHEL/Rocky/AlmaLinux 9.7, extract using the following command: [root@host~]# tar zxvfm ChelsioUwire-x.x.x.x-RHEL9.7-x86_64.tar.gz 3. Change your current working directory to the Chelsio Unified Wire package directory and install the driver. [root@host~]# cd ChelsioUwire-x.x.x.x-- [root@host~]# ./install.py -i nic_toe : NIC and TOE drivers only. all : all Chelsio drivers. udpso : UDP segmentation offload capable NIC and TOE drivers only. wd : Wire Direct drivers and libraries only. crypto : Crypto drivers and OpenSSL modules. ovs : OVS modules and NIC driver. NOTE: The Installation options may vary depending on the Linux distribution. 4. The default configuration tuning option is Unified Wire. The configuration tuning can be selected using the following command: [root@host~]# ./install.py -i -c NOTE: To view the different configuration tuning options, view the help by typing. [root@host~]# ./install.py -h 5. Reboot your machine for changes to take effect. NOTE: If the installation aborts with the message "Resolve the errors/dependencies manually and restart the installation", go through the install.log to resolve errors/dependencies and then start the installation again. iWARP Driver Installation on Cluster Nodes ------------------------------------------ IMPORTANT: Ensure that you have enabled password less authentication with ssh on the peer nodes for this feature to work. 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x-- 2. Create a file (machinefilename) containing the IP addresses or hostnames of the nodes in the cluster. You can view the sample file, sample_machinefile, provided in the package to view the format in which the nodes have to be listed. 3. Install iWARP and TOE drivers. [root@host~]# ./install.py -C -m -i -c The above command installs iWARP (iw_cxgb4) and TOE (t4_tom) drivers on all the nodes listed in the file 4. Reboot your machine for changes to take effect. Firmware and init/vpd update ============================ 1. Download Chelsio Unified Wire driver package. 2. Extract the downloaded package. [root@host~]# tar zxvfm ChelsioUwire-x.x.x.x.tar.gz 3. Change your current working directory to the Chelsio Unified Wire package directory and build and install the drivers, tools and libraries from source. [root@host~]# cd ChelsioUwire-x.x.x.x [root@host~]# make [root@host~]# make install NOTE - If a previous version of driver package is installed, execute below commands before attempting installation - [root@host~]# make uninstall ; make clean 4. Unload NIC driver and its dependencies. [root@host~]# rmmod iw_cxgb4 cxgb4 5. Flash the updated init and vpd on the Chelsio T7 adapter. [root@host~]# t7seeprom -flash -b write -f:< path to seeprom.bin file > Example: - For T72200-FH: [root@host~]# t7seeprom -flash -b 11:00.0 write -f:/lib/firmware/cxgb4/config/t72200_fh_init_gen5x16_2_mem_2ch_4800_2x200g_vpd_mfg.bin - For T72200: [root@host~]# t7seeprom -flash -b 11:00.0 write -f:/lib/firmware/cxgb4/config/t72200_init_gen5x16_mem_2ch_4800_2x200g_vpd_mfg.bin - For T7450: [root@host~]# t7seeprom -flash -b 11:00.0 write -f:/lib/firmware/cxgb4/config/t7450_init_gen5x16_mem_2ch_4800_4x50g_vpd_mfg.bin - For S72200: [root@host~]# t7seeprom -flash -b 11:00.0 write -f:/lib/firmware/cxgb4/config/s72200_init_gen5x16_no_mem_2x200g_vpd_mfg.bin - For S71400: [root@host~]# t7seeprom -flash -b 11:00.0 write -f:/lib/firmware/cxgb4/config/s71400_init_gen5x16_no_mem_1x400g_vpd_mfg.bin - For S7450: [root@host~]# t7seeprom -flash -b 11:00.0 write -f:/lib/firmware/cxgb4/config/s7450_init_gen5x16_no_mem_4x40g_vpd_mfg.bin - For S7250: [root@host~]# t7seeprom -flash -b 11:00.0 write -f:/lib/firmware/cxgb4/config/s7250_init_gen5x16_no_mem_4x40g_vpd_mfg.bin 6. Update the firmware on the Chelsio T7 adapter. Run the following commands to update the firmware manually: [root@host~]# modprobe cxgb4 fw_attach=0 [root@host~]# cxgbtool ethX loadfw /lib/firmware/cxgb4/config/t7fwbootstrap-x.x.x.x.bin [root@host~]# cxgbtool ethX loadfw /lib/firmware/cxgb4/t7fw.bin 7. Power cycle your machine for changes to take effect. 8. Verify the firmware version (v2.1.32.0) after the host boots. The firmware (v2.1.x.x) is installed on the system, typically in /lib/firmware/cxgb4. The firmware version can be verified using: [root@host~]# ethtool -i ethX driver: cxgb4 ... firmware-version: 2.1.x.x,TP 0.1.30.0 ... Mounting debugfs ---------------- All the driver debug data is stored in debugfs, which will be mounted in most cases. If not, mount it manually. [root@host~]# mount -t debugfs none /sys/kernel/debug Collect all adapter logs using cudbg_app ---------------------------------------- To collect all adapter logs using cudbg_app, run the following command: [root@host~]# cudbg_app --collect all ethX logs Configuring IPv6 ---------------- The interfaces should come up with a link-local IPv6 address for complete and fully functional IPv6 configuration. Update the Interface network-script with ONBOOT="yes". Uninstallation of Chelsio Unified Wire ==================================== There are two methods to uninstall the Chelsio Unified Wire package: From source and from RPM. If you decide to use source, you can uninstall the package using CLI or GUI mode. Follow the steps mentioned below for uninstallation using CLI. For GUI based uninstallation, refer to the User's Guide. From source ----------- 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Uninstall the source. [root@host~]# make uninstall Uninstalling Individual Drivers/Software ---------------------------------------- You can also choose to uninstall drivers/software individually. Provided here are steps to uninstall few of them. For the complete list, view help by running "make help". Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x - To uninstall NIC driver, [root@host~]# make nic_uninstall - To uninstall drivers with offload support, [root@host~]# make toe_uninstall - To uninstall iWARP driver, [root@host~]# make iwarp_uninstall - To uninstall UDP Segmentation Offload driver, [root@host~]# make udp_offload_uninstall iWARP Driver Uninstallation on Cluster Nodes -------------------------------------------- 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x 2. Uninstall iWARP drivers on multiple Cluster nodes. [root@host~]# ./install.py -C -m -u all The above command removes the Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. From RPM (tar-ball) ------------------- 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x-- 2. Run the following command: [root@host~]# ./uninstall.py iWARP Driver Uninstallation on Cluster Nodes -------------------------------------------- To uninstall iWARP drivers on multiple Cluster nodes, 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x-- 2. Run the following command: [root@host~]# ./install.py -C -m -u The above command removes the Chelsio iWARP (iw_cxgb4) and TOE (t4_tom) drivers from all the nodes listed in the machinefilename file. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Network (NIC/TOE) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Installation =================== 1. Change your current working directory to the Chelsio Unified Wire package directory. [root@host~]# cd ChelsioUwire-x.x.x.x - To build and install NIC only driver (without offload support), [root@host~]# make nic_install - To build and install drivers with offload support, [root@host~]# make toe_install NOTE: For more installation options, run "make help" or "install.py -h". 2. Reboot your machine for changes to take effect. Driver Loading ============== IMPORTANT: Ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. [root@host~]# rmmod csiostor cxgb4i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user. Any attempt to load the driver as a regular user may fail. - To load the driver in NIC mode (without offload support), [root@host~]# modprobe cxgb4 - To load driver in TOE mode (with offload support), [root@host~]# modprobe t4_tom NOTE: Offload support needs to be enabled upon each reboot of the system. This can be done manually as shown above. In VMDirect Path environment, it is recommended to load the offload driver using the following command: [root@host~]# modprobe t4_tom vmdirectio=1 Enabling TCP Offload ==================== Load the offload drivers and bring up the Chelsio interface. [root@host~]# modprobe t4_tom [root@host~]# ifconfig ethX up All TCP traffic will be offloaded over the Chelsio interface now. To see the number of connections offloaded, [root@host~]# cat /sys/kernel/debug/cxgb4//tids Precision Time Protocol (PTP) ============================= ptp4l tool is used to synchronise clocks. NOTE: The "linuxptp" package must be installed before proceeding. 1. Load the network driver on all master and slave nodes. [root@host~]# modprobe cxgb4 2. Assign IP addresses and ensure that master and slave nodes are connected. 3. Start the ptp4l tool on a master using the Chelsio interface. [root@host~]# ptp4l -i -H -m 4. On the master, synchronize the PTP hardware clock with the system clock. [root@host~]# phc2sys -s CLOCK_REALTIME -c -w -m 5. Start the ptp4l tool on slave nodes. [root@host~]# ptp4l -i -H -m -s NOTE: To view the complete list of available options, refer to the ptp4l help manual. 6. On the slave, synchronize the system clock with the PTP hardware clock. [root@host~]# phc2sys -s -c CLOCK_REALTIME -w -m Enabling Busy Waiting ===================== Busy waiting/polling is a technique where a process repeatedly checks to see if an event has occurred, by spinning in a tight loop. By making use of similar technique, Linux kernel provides the ability for the socket layer code to poll directly on an Ethernet device's Rx queue. This eliminates the cost of interrupts and context switching, and with proper tuning allows to achieve latency performance similar to that of hardware. Chelsio's NIC and TOE drivers support this feature and can be enabled on Chelsio supported devices to attain improved latency. To make use of BUSY_POLL feature, follow the steps mentioned below: 1. Enable BUSY_POLL support in kernel config file by setting "CONFIG_NET_RX_BUSY_POLL=y". 2. Enable BUSY_POLL globally in the system by setting the values of following sysctl parameters depending on the number of connections: sysctl -w net.core.busy_read= sysctl -w net.core.busy_poll= Set the values of the above parameters to 50 for 100 or less connections; and 100 for more than 100 connections. NOTE: BUSY_POLL can also be enabled on a per-connection basis by making use of SO_BUSY_POLL socket option in the socket application code. Refer to the socket man-page for further details. Precision Time Protocol (PTP) ============================= Note: Supported only on T6 family of adapters. ptp4l tool (installed during Unified Wire installation) is used to synchronise clocks. 1. Load the network driver on all master and slave nodes. [root@host~]# modprobe cxgb4 2. Assign IP addresses and ensure that master and slave nodes are connected. 3. Start the ptp4l tool on master using the Chelsio interface. [root@host~]# ptp4l -i -H -m 4. Start the tool on slave nodes. [root@host~]# ptp4l -i -H -m -s NOTE: To view the complete list of available options, refer to the ptp4l help manual. 5. Synchronize the system clock to a PTP hardware clock (PHC) on slave nodes. [root@host~]# phc2sys -s -c CLOCK_REALTIME -w -m VXLAN Offload ============= The Chelsio adapters are uniquely capable of offloading the processing of VXLAN encapsulated frames such that all stateless offloads (checksums and TSO) are preserved, resulting in significant performance benefits. This is enabled by default on loading the driver. For information regarding the configuration, refer to the User Guide. HMA === To use HMA, ensure that the Unified Wire is installed using the "Unified Wire (Default)" configuration tuning option. Currently, 256 IPv4/128 IPv6 TOE Offload connections are supported on T6 25G SO adapters. To see the number of connections offloaded, [root@host~]# cat /sys/kernel/debug/cxgb4//tids Performance Tuning ================== TOE --- 1. Run the performance tuning script to update few kernel parameters using sysctl and map TOE queues to different CPUs. [root@host~]# t4_perftune.sh -i ethX -s 2. Disable Rx Coalesce and DDP using the following steps: a. Create a COP policy. [root@host~]# cat all => offload !ddp !coalesce b. Compile the policy. [root@host~]# cop -d -o c. Apply the policy. [root@host~]# cxgbtool ethX policy NIC --- 1. Run the performance tuning script to update few kernel parameters using sysctl and map NIC queues to different CPUs. [root@host~]# t4_perftune.sh -i ethX -s 2. Enable adaptive-rx. [root@host~]# ethtool -C enp2s0f4 adaptive-rx on Driver Unloading ================ - To unload the driver in NIC mode (without offload support), [root@host~]# rmmod cxgb4 - A reboot is required to unload the driver in TOE (with Offload support). To avoid rebooting, follow the steps mentioned below: 1. Load t4_tom driver with unsupported_allow_unload parameter. [root@host~]# modprobe t4_tom unsupported_allow_unload=1 2. Stop all the offloaded traffic, servers and connections. Check for the reference count. [root@host~]# cat /sys/module/t4_tom/refcnt If the reference count is 0, the driver can be directly unloaded. Skip to step 3. If the count is non-zero, load a COP policy which disables offload using the following procedure: i. Create a policy file which disables offload. [root@host~]# cat policy_file all => !offload ii. Compile and apply the output policy file. [root@host~]# cop –o no-offload.cop policy_file [root@host~]# cxgbtool ethX policy no-offload.cop 3. Unload the driver. [root@host~]# rmmod t4_tom [root@host~]# rmmod toecore [root@host~]# rmmod cxgb4 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Port Forwarding Mode (Hardware Bypass Mode/Hairpin Mode/LAN Bypass) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NOTE: To enable this feature please flash the latest init/vpd present in this package, refer "Firmware and init/vpd update" section in this chapter for steps. NOTE: This feature is supported on Chelsio T72200-FH, T72200, T7450 and S72200 adapters. NOTE: Port Forwarding Mode (Hardware Bypass Mode/Hairpin Mode/LAN Bypass) is provided as an Alpha quality release. NOTE: Supported has been added only for Kernel.org linux-6.12.93 and 6.6.142. Port forwarding enables automatic packet forwarding between ports of the same adapter through firmware, even when the driver is not loaded. This feature is intended to support packet forwarding during driver absence, driver unload, or driver failure. 100% hair-pin is supported by which all traffic from Port0 is sent to Port1, and traffic from Port1 is sent to Port0. Following modes are supported: 1. Hardware bypass on power-on 2. Hardware bypass with driver deadman watchdog 3. Hardware bypass with IOCTL-Based deadman watchdog Firmware and init/vpd update ============================ Note: The init/VPD files are only available in the source software package; while the 't7seeprom' utility used to flash init/VPD is available in both the source and RPM packages. 1. Download Chelsio Unified Wire driver package. 2. Extract the downloaded package. [root@host~]# tar zxvfm ChelsioUwire-x.x.x.x.tar.gz 3. Change your current working directory to the Chelsio Unified Wire package directory and build and install the drivers, tools and libraries from source. [root@host~]# cd ChelsioUwire-x.x.x.x [root@host~]# make [root@host~]# make install NOTE - If a previous version of driver package is installed, execute below commands before attempting installation - [root@host~]# make uninstall ; make clean 4. Unload NIC driver and its dependencies. [root@host~]# rmmod iw_cxgb4 cxgb4 5. Flash the updated init and vpd on the Chelsio T7 adapter. [root@host~]# t7seeprom -flash -b write -f:< path to seeprom.bin file > Example: - For T72200-FH: [root@host~]# cd ChelsioUwire-x.x.x.x/support/config5 For Gen5 - [root@host~]# t7seeprom -flash -b 11:00.0 write -f:t72200_fh_init_gen5x16_FW_wire_mode_2_mem_2ch_4800_2x200g_vpd_mfg.bin For Gen4 - [root@host~]# t7seeprom -flash -b 11:00.0 write -f:t72200_fh_init_gen4x16_FW_wire_mode_2_mem_2ch_4800_2x200g_vpd_mfg.bin - For T72200: [root@host~]# cd ChelsioUwire-x.x.x.x/support/config5 For Gen5 - [root@host~]# t7seeprom -flash -b 11:00.0 write -f:t72200_init_gen5x16_FW_wire_mode_mem_2ch_4800_2x200g_vpd_mfg.bin - For T7450: [root@host~]# cd ChelsioUwire-x.x.x.x/support/config5 For Gen5 - [root@host~]# t7seeprom -flash -b 11:00.0 write -f:t7450_init_gen5x16_FW_wire_mode_mem_2ch_4800_4x50g_vpd_mfg.bin For Gen4 - [root@host~]# t7seeprom -flash -b 11:00.0 write -f:t7450_init_gen4x16_FW_wire_mode_mem_2ch_4800_4x50g_vpd_mfg.bin - For S72200: [root@host~]# cd ChelsioUwire-x.x.x.x/support/config5 For Gen5 - [root@host~]# t7seeprom -flash -b 11:00.0 write -f:s72200_init_gen5x16_FW_wire_mode_no_mem_2x200g_vpd_mfg.bin 6. Update the firmware on the Chelsio T7 adapter. Run the following commands to update the firmware manually: [root@host~]# modprobe cxgb4 fw_attach=0 [root@host~]# cxgbtool ethX loadfw /lib/firmware/cxgb4/config/t7fwbootstrap-x.x.x.x.bin [root@host~]# cxgbtool ethX loadfw /lib/firmware/cxgb4/t7fw.bin 7. Power cycle your machine for changes to take effect. 8. Verify the firmware version (v2.1.x.x) after the host boots. The firmware (v2.1.x.x) is installed on the system, typically in /lib/firmware/cxgb4. The firmware version can be verified using: [root@host~]# ethtool -i ethX driver: cxgb4 ... firmware-version: 2.1.x.x,TP 0.1.30.0 ... 1. Hardware bypass on power-on ============================== Upon adapter poweron hardware bypass is enabled by default. After power-up: - Firmware will bring up the links automatically - Firmware will configure switch filters automatically - Firmware will start packet forwarding between ports without waiting for the driver to load as below: On a dual-port adapter: * Packets from Port0 are forwarded to Port1 * Packets from Port1 are forwarded to Port0 On a quad-port adapter: * Packets from Port0 are forwarded to Port1 * Packets from Port1 are forwarded to Port0 * Packets from Port2 are forwarded to Port3 * Packets from Port3 are forwarded to Port2 2. Hardware bypass with driver deadman watchdog =============================================== - Once the cxgb4 driver registers, hardware bypass is disabled and firmware-based packet forwarding is disabled. - Hardware bypass is re-enabled under below circumstances: - cxgb4 driver unregisters/unloads - cxgb4 driver stops sending deadman watchdog updates - A driver or firmware crash occurs In order to enable hardware bypass with deadman watchdog feature follow below steps: [root@host~]# modprobe -v cxgb4 deadman_watchdog=x,2 [root@host~]# ifconfig ethA up [root@host~]# ifconfig ethB up Here, x is the deadman watchdog timer interval in milliseconds ranging from 1000ms to 60000ms. ethA refers to Port0 of the adapter. ethB refers to Port1 of the adapter. Example 1: For a dual port adapter: [root@host~]# modprobe -v cxgb4 deadman_watchdog=2000,2 [root@host~]# ifconfig eth0 up [root@host~]# ifconfig eth1 up Here, driver sends watchdog sync every 1000ms (2000ms/2) to the firmware. If the firmware fails to receive a sync then it switches to hardware bypass mode and packets are forwarded from port0 to port1 and port1 to port0. Example 2: For a quad port adapter: [root@host~]# modprobe -v cxgb4 deadman_watchdog=2000,2 [root@host~]# ifconfig eth0 up [root@host~]# ifconfig eth1 up [root@host~]# ifconfig eth2 up [root@host~]# ifconfig eth3 up Here, driver sends watchdog sync every 1000ms (2000ms/2) to the firmware. If the firmware fails to receive a sync then it switches to hardware bypass mode and packets are forwarded from port0 to port1, port1 to port0 , port2 to port3 and port3 to port2. 3. Hardware bypass with IOCTL-Based deadman watchdog ==================================================== The driver provides a private IOCTL interface that enables a userspace application to configure, start, and periodically refresh the driver watchdog. The application configures the watchdog by issuing an ioctl() request using SIOCDEVPRIVATE on the target network interface. The watchdog configuration is passed through a dedicated structure that includes the watchdog operation command identifier (cmd) and the watchdog timeout value (timeout) in milliseconds. Once configured, the watchdog remains active as long as the application continues to periodically refresh it using the same IOCTL interface. To configure the watchdog, the application must first create a network socket and populate the watchdog configuration structure with the required command and timeout parameters. An ifreq structure must then be initialized with the target network interface name and a pointer to the watchdog configuration structure assigned to ifr_data. The application should invoke ioctl() with SIOCDEVPRIVATE to start the watchdog and continue issuing the same IOCTL periodically to refresh it. The refresh interval must be set to half of the configured watchdog timeout value to ensure the watchdog is refreshed well before expiration and to avoid unintended timeout events. For example, with a configured watchdog timeout of 20 seconds, the refresh IOCTL should be issued every 10 seconds. The 'cxgb4' driver provides support for IOCTL-based hairpin mode. Once the driver is loaded and the network interfaces are brought up, the application can configure the driver timer using 'set_timer' and periodically refresh it using 'refresh_timer'.If the application crashes or stops refreshing the timer, the timer will eventually expire, and the adapter will automatically switch to hairpin mode. cxgb4 driver parameter ---------------------- To enable hairpin mode immediately when the interface is brought up after driver load, the 'cxgb4' driver must be loaded with below driver parameter: [root@host~]# modprobe cxgb4 hairpin_enable=1 NOTE: The 'hairpin_enable=1' parameter is required only when hairpin mode needs to be enabled immediately during interface bring-up after driver load. This parameter is _not required_ for the timer-based application-failure mechanism. Even if 'hairpin_enable=1' is not specified, the application can configure 'set_timer'and periodically call 'refresh_timer'. If the application subsequently crashes or stops refreshing the timer, the timer will expire and the adapter will automatically switch to hairpin mode. Reference Application --------------------- The 'test_app_v2' is provided as an example/reference application to demonstrate how to use the driver-based hairpin functionality. End-users can use the 'test_app_v2' implementation and usage as a reference and modify their application accordingly to integrate and support driver-based hairpin functionality. Location: The 'test_app_v2.c' application can be found at: ChelsioUwire-x.x.x.x/support/config5/test_app_v2.c Compilation of the test application: [root@host ~]# gcc test_app_v2.c -o test_app_v2 Usage: [root@host~]# ./test_app_v2 set_timer