**************************************** README **************************************** Chelsio T4 Family 10/1Gb Unified Wire for Linux Version : 2.2.0.0 Date : 11/05/2012 Overview ================================================================================ Chelsio Unified Wire software for Linux is an easy to use utility developed to provide installation of 64-bit Linux based drivers and tools for Chelsio's T4 Unified Wire Adapters. The Chelsio Unified Wire Package provides an interactive installer to install various drivers and utilities. It consists of the following components: a. Network (NIC/TOE) b. Virtual Function Network (vNIC) c. iWARP (RDMA) d. WD-UDP e. iSCSI PDU Offload Target f. iSCSI PDU Offload Initiator g. FCoE Full Offload Initiator h. Offload Bonding driver i. Offload IPv6 driver j. Bypass driver k. UDP Segmentation Offload and Pacing l. tools (cop,cxgbtool,t4_perftune,sniffer & tracer) m. libs (iWARP and WD-UDP libraries) ================================================================================ CONTENTS ================================================================================ - 1. Requirements - 2. Supported Operating System - 3. Supported Cards - 4. How To Use - 5. Support Documentation - 6. Customer Support 1. Requirements ================================================================================ 1.1. Network (NIC/TOE) ====================== - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. 1.2. Virtual Function Network (vNIC) ==================================== - Chelsio T4 10/1Gb adapter and an SR-IOV enabled system with supported platforms mentioned in section 2. 1.3. iWARP (RDMA) ================= - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. - To use the iWARP functionality with Chelsio adapters, user needs to install the iWARP drivers as well as the libcxgb4, libibverbs, and librdmacm libraries. Chelsio provides the iWARP drivers and libcxgb4 library as part of the driver package. The other libraries are provided as part of the Open Fabrics Enterprise Distribution (OFED) package. The OFED package is composed of several software components and is generally intended to use on a computer cluster built with an iWARP network or InfiniBand network. Please visit http://www.openfabrics.org/downloads/OFED to download the appropriate OFED package. NOTE: If you are planning to upgrade the OFED package on one member of the cluster, the upgrade needs to be installed on all the members. 1.4. WD-UDP ========== - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. - To use WD-UDP with Chelsio adapters, user needs to install the iWARP drivers as well as the libcxgb4, libcxgb4_sock, libcxgb4_udp, libibverbs, and librdmam libraries. Chelsio provides the iWARP drivers and libcxgb4, libcxgb4_sock, and libcxgb4_udp libraries as part of its driver package. The other drivers are provided as part of the Open Fabrics Enterprise Distribution (OFED) package 1.5. iSCSI PDU Offload Target ============================== - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. - chiscsi_base.ko is iSCSI non-offload target mode driver and chiscsi_t4.ko is iSCSI PDU offload target mode driver. cxgb4, toecore, t4_tom and chiscsi_base modules are required by chiscsi_t4.ko module to work in offload mode. Whereas in iscsi non-offload target(NIC) mode, only cxgb4 is needed by chiscsi_base.ko module. 1.6. iSCSI PDU Offload Initiator ================================= - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. - cxgb4i runs on top of cxgb4 and open-iscsi-2.0.872 only on a T4 card. 1.7. FCoE full offload Initiator ================================= - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. 1.8. Offload Bonding driver =========================== - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. 1.9. Offload IPv6 driver ============================ - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. 1.10. Bypass driver ============================ - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. 1.11. UDP Segmentation Offload and Pacing driver ================================================= - Chelsio T4 10/1Gb adapter and system with supported platforms mentioned in section 2. 2. Supported Operating System ================================================================================ The Chelsio T4 Unified Wire software has been developed to run on 64-bit Linux based platforms. Following is the list of Drivers/Software and supported Linux distributions. ########################|######################################################| Linux Distribution | Driver/Software | ########################|######################################################| RHEL5.3,2.6.18-128.el5 |NIC/TOE*,vNIC*,iWARP,WD-UDP,iSCSI Target,Bonding, | |Sniffer & Tracer | ------------------------|------------------------------------------------------| RHEL5.4,2.6.18-164.el5 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,Bonding,Sniffer & Tracer | ------------------------|------------------------------------------------------| RHEL5.5,2.6.18-194.el5 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,Bonding,IPv6,Sniffer & Tracer | ------------------------|------------------------------------------------------| RHEL5.6,2.6.18-238.el5 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,Bonding,IPv6,Sniffer & Tracer | ------------------------|------------------------------------------------------| RHEL5.7,2.6.18-274.el5 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target*, | |iSCSI Initiator*,Bonding,IPv6,Sniffer & Tracer | ------------------------|------------------------------------------------------| RHEL5.8,2.6.18-308.el5 |NIC/TOE,vNIC,iWARP,iSCSI Target*, | |iSCSI Initiator,Bonding,IPv6,Sniffer & Tracer | ------------------------|------------------------------------------------------| RHEL6.0,2.6.32-71.el6 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,FCoE Initiator,Bonding, | |IPv6,Bypass*,Sniffer & Tracer,UDP-SO | ------------------------|------------------------------------------------------| RHEL6.1, |NIC/TOE*,vNIC*,iWARP,WD-UDP,iSCSI Target* | 2.6.32-131.0.15.el6 |iSCSI Initiator*,FCoE Initiator*, | |Bonding,IPv6,Bypass*,Sniffer & Tracer,UDP-SO | ------------------------|------------------------------------------------------| RHEL6.2, |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target*, | 2.6.32-220.el6 |iSCSI Initiator*,FCoE Initiator, | |Bonding,IPv6,Bypass*,Sniffer & Tracer,UDP-SO | ------------------------|------------------------------------------------------| RHEL6.3, |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target*, | 2.6.32-279.el6 |iSCSI Initiator*,FCoE Initiator, | |Bonding,IPv6,Bypass*,Sniffer & Tracer,UDP-SO | ------------------------|------------------------------------------------------| SLES10SP3, |NIC/TOE,vNIC | 2.6.16.60-0.54.5 | | ------------------------|------------------------------------------------------| SLES11,2.6.27.19-5 |NIC/TOE*,vNIC*,iWARP,WD-UDP,iSCSI Target*, | |iSCSI Initiator*,Bonding,Bypass*,Sniffer & Tracer | ------------------------|------------------------------------------------------| SLES11SP1,2.6.32.12-0.7 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,FCoE Initiator, | |Bonding,IPv6,Bypass*,Sniffer & Tracer,UDP-SO | ------------------------|------------------------------------------------------| SLES11SP2,3.0.13-0.27 |NIC/TOE,vNIC,iWARP*,WD-UDP,FCoE Initiator,Bonding, | |IPv6,Bypass*,Sniffer & Tracer* | ------------------------|------------------------------------------------------| Fedora release 13 |NIC/TOE,vNIC*,iWARP*,WD-UDP*,Bypass*, | (FC 13),2.6.33.3-85.fc13|Sniffer & Tracer*,UDP-SO | ------------------------|------------------------------------------------------| Fedora release 14 |NIC/TOE*,vNIC*,iWARP*,WD-UDP*,iSCSI Target*, | (FC 14),2.6.35.6-45.fc14|Bypass*,Sniffer & Tracer*,UDP-SO | ------------------------|------------------------------------------------------| Ubuntu 12.04, 3.2.0-23 |NIC/TOE,iWARP,iSCSI Initiator, | |Bonding,Bypass,IPv6 | ------------------------|------------------------------------------------------| Kernel.org linux-2.6.34 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target*, | |iSCSI Initiator*,FCoE Initiator*,Bypass, | |IPv6,Sniffer & Tracer,UDP-SO | ------------------------|------------------------------------------------------| Kernel.org linux-2.6.35 |NIC/TOE*,vNIC,iWARP,WD-UDP,iSCSI Target*, | |iSCSI Initiator*,Bypass*,Sniffer & Tracer,UDP-SO | ------------------------|------------------------------------------------------| Kernel.org linux-2.6.36 |NIC/TOE*,vNIC,iWARP,WD-UDP,iSCSI Target*, | |iSCSI Initiator*,Bypass,Sniffer & Tracer | ------------------------|------------------------------------------------------| Kernel.org linux-2.6.37 |NIC/TOE*,vNIC,iWARP,WD-UDP,iSCSI Target*, | |iSCSI Initiator*,Bypass,Sniffer & Tracer | ------------------------|------------------------------------------------------| Kernel.org linux-2.6.39 |NIC/TOE,vNIC*,iWARP,WD-UDP,iSCSI Target*, | |iSCSI Initiator*,Bypass*,Sniffer & Tracer | ------------------------|------------------------------------------------------| Kernel.org linux-3.1 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,Bonding,Bypass,IPv6,Sniffer & Tracer | ------------------------|------------------------------------------------------| Kernel.org linux-3.5 |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Initiator, | |FCoE Initiator,Bonding,IPv6,Bypass,Sniffer & Tracer | -------------------------------------------------------------------------------| NOTE: - Other kernel versions have not been tested and are not guaranteed to work. *Limited QA performed 3. Supported Cards ================================================================================ Following is the list of Drivers/Software and supported Chelsio T4 adapters. ########################|######################################################| Chelsio T4 Adapter | Driver/Software | ########################|######################################################| T420-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,FCoE Initiator,Bonding, | |IPv6,Sniffer & Tracer, UDP-SO | ------------------------|------------------------------------------------------| T420-LL-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Initiator, | |Bonding,IPv6,Sniffer & Tracer, UDP-SO | ------------------------|------------------------------------------------------| T440-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,FCoE Initiator,Bonding, | |IPv6,Sniffer & Tracer, UDP-SO | ------------------------|------------------------------------------------------| T440-LP-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target,iSCSI Initiator| |,FCoE Initiator,Bonding,IPv6, | |Sniffer & Tracer, UDP-SO | ------------------------|------------------------------------------------------| T420-BCH |NIC/TOE,iWARP,iSCSI Target,iSCSI Initiator,Bonding, | |IPv6,Sniffer & Tracer | ------------------------|------------------------------------------------------| T422-CR |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,FCoE Initiator,Bonding, | |IPv6,Sniffer & Tracer, UDP-SO | ------------------------|------------------------------------------------------| T420-SO-CR |NIC/TOE,vNIC,Bonding,IPv6 | ------------------------|------------------------------------------------------| T420-CX |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Initiator,Bonding, | |IPv6,Sniffer & Tracer, UDP-SO | ------------------------|------------------------------------------------------| T420-BT |NIC/TOE,vNIC,iSCSI Target,iSCSI Initiator,iWARP, | |WD-UDP,Bonding,IPv6,Sniffer & Tracer | -------------------------------------------------------------------------------| T404-BT |NIC/TOE,vNIC,iWARP,WD-UDP,iSCSI Target, | |iSCSI Initiator,FCoE Initiator,Bonding, | |IPv6,Sniffer & Tracer, UDP-SO | ------------------------|------------------------------------------------------| B420-SR |Bypass | ------------------------|------------------------------------------------------| B404-BT |Bypass | ------------------------|------------------------------------------------------| 4. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.1. Chelsio Unified Wire ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ There are two main methods to install the Chelsio Unified Wire package: from source and from RPM. If you decide to use source, you can install the package using CLI or GUI mode. If you decide to use RPM, you can install the package using Menu or CLI mode. Follow the steps mentioned below for installation using CLI. For GUI or Menu based installation, refer the User's Guide. 4.1.1. Installing Chelsio Unified Wire ====================================== a. From source -------------- i. Download the tarball ChelsioUwire-x.x.x.x.tar.gz ii. Untar the tarball [root@host]# tar zxvfm ChelsioUwire-x.x.x.x.tar.gz iii. Navigate to the 'ChelsioUwire-x.x.x.x' directory. Build the source: [root@host]# make iv. Install the drivers, tools and libraries: [root@host]# make install v. The default T4 configuration tuning option is Unified Wire (includes FCoE Initiator). The configuration tuning can be selected using the following commands: [root@host]# make CONF= [root@host]# make CONF= install NOTE: To view the different T4 configuration tuning options, view the help by typing [root@host]#make help vi. Reboot your machine for changes to take effect. IMPORTANT: Steps iii,iv and v mentioned above will NOT install UDP Segmentation Offload and Bypass drivers. These drivers will have to be installed manually. Please refer to section "4.1.2. Installing individual drivers" for instructions on installing them individually. Installation on updated kernels ---------------------------------------- If the kernel version on your Linux distribution is updated, please execute the following command to install the Unified Wire package: [root@host]# make UNAME_R= Where kernel_version is the nearest supported kernel version. For example, if you want to install the package on a RHEL 6 distribution updated to 2.6.32-131.17.1.el6 kernel, run the following commands: [root@host]# make UNAME_R=2.6.32-131.0.15.el6 [root@host]# make UNAME_R=2.6.32-131.0.15.el6 install To view the list of the supported kernel versions, run the following command: [root@host]# make list_kernels Reboot your machine for changes to take effect. b. From RPM (tarball) ---------------------- i. Download the tarball specific to your operating system and architecture. ii. Untar the tarball E.g. For RHEL 6.0, untar using the following command: [root@host]# tar zxvfm ChelsioUwire-x.x.x.x-RHEL6.0_x86_64.tar.gz iii. Navigate to the ChelsioUwire-x.x.x.x directory. Run the following command: [root@host]# ./install.py -i nic_toe :NIC and TOE drivers only all :all Chelsio drivers (including FCoE Initiator) built against inbox OFED all_ofed :all Chelsio drivers (including FCoE Initiator) built against OFED-1.5.4.1. bypass :bypass drivers and tools udpso :UDP segmentation offload capable NIC and TOE drivers only NOTE: The Installation options may vary depending on the Linux distribution. iv. The default T4 configuration tuning option is Unified Wire (includes FCoE Initiator). The configuration tuning can be selected using the following command: [root@host]# ./install.py -i -c NOTE: To view the different T4 configuration tuning options, view the help by typing [root@host]#make help v. Reboot your machine for changes to take effect. NOTE: i. If the installation aborts with the message "Resolve the errors/dependencies manually and restart the installation", please go through the install.log to resolve errors/dependencies and then start the installation again. 4.1.2. Installing individual drivers ====================================== You can also choose to install drivers individually. Provided here are steps to build and install NIC,TOE,iWARP,Bypass and UDP Segmentation Offload drivers. To know about other drivers, access help by running "make help". i. To build and install NIC driver without offload support : [root@host]# make nic [root@host]# make nic_install ii. To build and install NIC driver with offload support and Offload drivers: [root@host]# make toe [root@host]# make toe_install iii. To build and install iWARP driver against outbox OFED: [root@host]# make iwarp [root@host]# make iwarp_install iv. To build and install all drivers except Bypass against outbox OFED: [root@host]# make [root@host]# make install v. To build and install Bypass driver: [root@host]# make bypass [root@host]# make bypass_install vi. To build and install UDP Segmentation Offload driver: [root@host]# make udp_offload [root@host]# make udp_offload_install vii. The default T4 configuration tuning option is Unified Wire (includes FCoE Initiator). The configuration tuning can be selected using the following commands: [root@host]# make CONF= [root@host]# make CONF= NOTE: To view the different T4 configuration tuning options, view the help by typing [root@host]#make help 4.1.3. Firmware Update ======================================== The T4 firmware(1.6.2.0) is installed on the system, typically in /lib/firmware/cxgb4, and the driver will auto-load the firmware if an update is required. The kernel must be configured to enable userspace firmware loading support: Device Drivers -> Generic Driver Options -> Userspace firmware loading support The firmware version can be verified using ethtool: [root@host]# ethtool -i 4.1.4. Uninstalling Chelsio Unified Wire ======================================== There are two methods to uninstall the Chelsio Unified Wire package: from source and from RPM. If you decide to use source, you can uninstall the package using CLI or GUI mode. Follow the steps mentioned below for uninstallation using CLI. For GUI based uninstallation, refer to the User Guide. a. From source -------------- Navigate to the ChelsioUwire-x.x.x.x directory. Uninstall the source using the following command: [root@host]# make uninstall b. From RPM (tar-ball) ---------------------- Go to the ChelsioUwire-x.x.x.x directory. Run the following command: [root@host]# ./uninstall.py Here, inbox : for removing all Chelsio drivers. ofed : for removing OFED and Chelsio drivers. NOTE: i. If the Chelsio Unified Wire is installed using one method and you decide to reinstall the package using the other, please make sure that the package is first uninstalled using the method it was installed with. E.g. If the package is installed using the source, uninstall it using the source only and not RPMs and vice-versa. 4.1.5. Uninstalling individual drivers ====================================== You can also choose to uninstall drivers individually. Provided here are steps to uninstall NIC, TOE, iWARP,Bypass and UDP Segmentation Offload drivers. To know about other drivers, access help by running "make help". i. To uninstall NIC driver: [root@host]# make nic_uninstall ii. To uninstall NIC driver with offload support: [root@host]# make toe_uninstall iii. To uninstall iWARP driver: [root@host]# make iwarp_uninstall iv. To uninstall Bypass driver: [root@host]# make bypass_uninstall v. To uninstall UDP Segmentation Offload driver: [root@host]# make udp_offload_uninstall ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.2. Network (NIC/TOE) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.2.1. Loading/Unloading the driver =================================== a. Loading the driver ---------------------- The driver must be loaded by the root user. Any attempt to loading the driver as a regular user will fail. i. To load the driver in NIC mode(without offload support) [root@host]# modprobe cxgb4 ii. To load driver in TOE mode(with offload support) [root@host]# modprobe t4_tom NOTE: Offload support needs to be enabled upon each reboot of the system. This can be done manually as shown above. b. Unloading the driver ----------------------- i. To unload the NIC driver. [root@host]# rmmod cxgb4 ii. To unload the TOE driver. Please reboot the system to unload the TOE driver. c. Instantiate Virtual Functions ------------------------------ To instantiate the Virtual functions, load the cxgb4 driver with 'num_vf' parameter with a non-zero value. For example: [root@host]# modprobe cxgb4 num_vf=1,0,0,0 Each number instantiates the specified number of Virtual Functions for the corresponding Physical Function. The Virtual Functions can be assigned to Virtual Machines(Guest OS). A maximum of 64 Virtual Functions can be instantiated with 16 Virtual Functions per Physical Function. Loading the cxgb4 driver with "num_vf" parameter loads the cxgb4vf driver by default. Hence unload the cxgb4vf driver (on the host) before assigning Virtual Functions to the Virtual Machines(Guest OS), using the following command: [root@host]# rmmod cxgb4vf 4.2.2. Performance tuning =================================== In order to auto tune the system and TOE devices for best performance, Chelsio recommends installing the tools which will copy t4_perftune.sh script to /sbin directory. Run the script by using the following command: [root@host]# t4_perftune.sh This script will configure RSS and enables Interrupt Coalescing. 4.2.3. Network Device Configuration =================================== Please refer to the operating system documentation for administration and configuration of network devices. NOTE: Some operating systems may attempt to auto-configure the detected hardware and some may not detect all ports on a multi-port adapter. If this happens, please refer to the operating system documentation for manually configuring the network device. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.3. Virtual Function Network (vNIC) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.3.1. Instantiate Virtual Functions =================================== To instantiate Chelsio Virtual Functions, please refer to the Network (NIC/TOE) section [section 4.2.1.(c)]. 4.3.2. Loading/Unloading the Driver ================================== The cxgb4vf driver must be loaded on the Guest OS by the root user. Any attempt to load the driver as a regular user will fail. a. Loading the driver --------------------- To load the driver execute the following command: [root@host]# modprobe cxgb4vf b. Unloading the Driver ----------------------- To unload the driver execute the following command: [root@host]# rmmod cxgb4vf ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.4. iWARP (RDMA) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.4.1. Installing iWARP with OFED package ========================================= i. Download and untar the tarball ChelsioUwire-x.x.x.x.tar.gz ii. Go to the ChelsioUwire-x.x.x.x directory. Build the source using : [root@host]# make iii. Install the drivers, tools and libraries: [root@host]# make install iv. Reboot your machine for changes to take effect. NOTE: WD-UDP libraries are pre-compiled binaries against OFED-1.5.4 package. 4.4.2. Loading/Unloading the driver ================================== a. Loading the driver --------------------- The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the iWARP driver we need to load the NIC driver & core RDMA drivers first: [root@host]#modprobe cxgb4 [root@host]#modprobe iw_cxgb4 [root@host]#modprobe rdma_ucm b. Unloading the driver ----------------------- To unload the iWARP driver, run the following command: [root@host]#rmmod iw_cxgb4 NOTE: Set Chelsio driver option for MPI connection changes. Give the below command on all systems: [root@host]# echo 1 > /sys/module/iw_cxgb4/parameters/peer2peer For more information on iWARP and MPI usage please refer to the User Guide. IMPORTANT: On RHEL6.2,SLES11sp2 and later distributions, openmpi-1.4.3 included in OFED-1.5.x can cause IMB benchmark stalls due to a shared memory BTL issue. This is fixed in openmpi-1.4.5 which can be downloaded at: http://www.open-mpi.org/software/ompi/v1.4/downloads/openmpi-1.4.5.tar.bz2 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.5. WD-UDP ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.5.1. Compiling & Loading Chelsio WD-UDP Drivers ------------------------------------------------- InfiniBand modules from the OFED package should be loaded before proceeding. Load the cxgb4, iw_cxgb4 and rdma_ucm drivers: [root@host]#modprobe cxgb4 [root@host]#modprobe iw_cxgb4 [root@host]#modprobe rdma_ucm 4.5.2. Unloading the driver ---------------------------------------- a. To unload the iWARP driver. [root@host]#rmmod iw_cxgb4 4.5.3. Configuring WD-UDP ---------------------------------------- a. The libcxgb4_sock interface requires a configuration file on each node that defines which UDP port numbers will be accelerated and which T4 interface it will use. The configuration file to use is /etc/libcxgb4_sock.conf, and here is an example: [root@host]# cat /etc/libcxgb4_sock.conf Syntax: endpoint { attributes } ... where attributes include: interface = interface-name port = udp-port-number E.g: endpoint {interface=eth2 port=8888} endpoint {interface=eth3 port=9999} The above file defines 2 accelerated endpoints, port 8888 which will use eth2, and port 9999 which will use eth3. b. Currently the only way to enable WD-UDP sockets is to preload libcxgb4_sock.so when running the desired udp application. E.g. for running netperf: server: [root@host]# LD_PRELOAD=libcxgb4_sock.so netserver -p OR [root@host]# wdload netserver -p client: [root@host]# LD_PRELOAD=libcxgb4_sock.so netperf -H -p -t UDP_RR -- -P, OR [root@host]# wdload netperf -H -p -t UDP_RR -- -P, The WD-UDP endpoint is the port number registered for the respective T4 interface in the /etc/libcxgb4_sock.conf file. **NOTE: i. In WD-UDP only one application can be run per T4 device per udp port number. For running 2 concurrent netperf UDP_RR tests, each must use a unique UDP port number or T4 device. ii. In order to offload IPv6 UDP sockets, please select "low latency networking" as T4 configuration tuning option during installation. iii. Jumbo frames of 9000B are supported only on kernel 2.6.32 and above. E.g.: endpoint {interface=eth2 port=8888} endpoint {interface=eth2 port=9000} The above file defines 2 accelerated endpoints, port 8888 and port 9000 both will be used by eth2 only. i. install.py script will automatically uninstall the previous versions of OFED and Chelsio WD-UDP. ii. To install/uninstall you will require root privileges. iii. For more information on WD-UDP configuration please refer to the User Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.6. iSCSI PDU Offload Target ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.6.1. Loading the module ========================================== To load the module for RHEL distributions, run modprobe as follows: [root@host]# modprobe chiscsi_t4 For SLES11 SP1: [root@host]# modprobe chiscsi_t4 --allow-unsupported 4.6.2. Obtaining the iSCSI software license ========================================== A license file is required for each copy of the Chelsio iSCSI software installed. The license is tied to a selected NIC interface in the machine.The license file will be generated depending on your requirement for a Chelsio iSCSI Target. a. Prepare License information file ----------------------------------- A license information file is required for each license. Run the following command to obtain the license information file: [root@host]# chinfotool The chinfotool scans and lists all the NICs in the system, and prompts the user to select a NIC to which the keyfile will be tied. At the end, it prints out a summary of the license file. For example: ++++++++++++++++++++++++++++++++++++ [root@host]# chinfotool Scanning System for network devices.... License key will be tied to any of the following interfaces. Please select the interface 1. Interface eth1 with INTEL Adapter Linkspeed is 1000 Mbps/s MAC is: 00:30:48:00:00:10. 2. Interface eth2 with CHELSIO Adapter Linkspeed is 10000 Mbps/s MAC is: 00:07:43:00:00:10. ENTER NUMBER:2 SUMMARY: Interface with MAC Address 00:07:43:00:00:10 (10000 Mbps) is selected Please send info file "/etc/xyz_chelsio_infofile" to Chelsio Communications at support@chelsio.com Chelsio Support will send back the keyfile. The keyfile unlocks the software as long as the above interface is present ++++++++++++++++++++++++++++++++++++ For the selected NIC, the following information is gathered: -PCI vendor ID -MAC address -Link speed The above information is used strictly by Chelsio Communications for the iSCSI license management. Chelsio Communications Inc. does not share, sell, rent, or lease the information to any third party. The generated file hostname_chelsio_infofile needs to be sent back to Chelsio at support@chelsio.com as instructed by the chinfotool. NOTE: Please make sure that the selected NIC card is present in the system all the time. If it is removed the license will be invalid and you will need to use chinfotool to rescan the system and get a new keyfile from our support team. b. Install the license file received from Chelsio ------------------------------------------------- Once the information file (generated by chinfotool in the previous step) is processed by Chelsio, the user will receive a license file tailored to the system. Copy the file to /etc/chelsio-iscsi/chiscsi.key [root@host]# cp -f chiscsi.key /etc/chelsio-iscsi/chiscsi.key Now the iSCSI software is ready for use. 4.6.3. Configuration =================== a. iSCSI configuration file --------------------------- Please see the sample iSCSI configuration file installed in /etc/chelsio-iscsi/chiscsi.conf. This file contains configurations for target. The file contains the following types of entity blocks marked in the file as: global: target: You can specify multiple targets blocks as required. This file should be edited using a standard text editor and customized to fit your environment. b. Configuring Chelsio iSCSI Target ----------------------------------- Chelsio's implementation of the target iSCSI stack has flexibility to accommodate a large range of configurations. To configure an iSCSI target, there are three minimum required parameters (in the form of key=value pairs) needed to be configured. target: TargetName=iqn.2006-02.com.chelsio.diskarray.san1 TargetDevice=/dev/sda PortalGroup=1@10.192.166.178:3260 -TargetName is worldwide unique iSCSI target name. -TargetDevice is a storage device served up by the associated target. This target storage device can be any of the device types that are supported (block, virtual block, RAM disk, regular file). Multiple storage devices are allowed by configuring the devices to one target or multiple targets. The software allows multiple targets to share the same device but use caution when doing this. -PortalGroup is the portal group tag associated with a list of target IP address(es) and port number(s) that service the login request. Please see the sample file for other iSCSI protocol specific and Chelsio specific key=value parameters and modify them as required. For quick testing, a RAM Disk or a regular file as the block storage device can be used easily. For deployment in a production environment a more sophisticated storage configuration would be needed. 4.6.4. Chelsio target service scripts ===================================== Chelsio iSCSI software has two service scripts, for target. -chelsio-target is target specific service script The chelsio-iscsi service does basic checks before starting the service, loads the kernel module, and starts all targets configured by default. It can also be used to stop the iSCSI Target, and restart/reload configuration. The service scripts are installed to /etc/init.d and the parameters for the scripts are installed at /etc/sysconfig/chelsio. The scripts are installed as a system service. To auto-start the iSCSI services, at a certain runlevel, e.g. runlevel 3, chkconfig can be used. [root@host]# chkconfig --level 3 chelsio-target on NOTE:For the script to execute properly, make sure the following flag is set on all kernel.org kernels. # CONFIG_MODULE_FORCE_LOAD=y 4.6.5. Chelsio "iscsictl" Command-line Configuration Utility ============================================================ The Chelsio iSCSI control tool, iscsictl, is a Command Line utility which is installed in /sbin/ and allows administration of the Chelsio iSCSI driver Target node . It can perform following functions: -Start / Stop the iSCSI node(Target) -Get / Set the iSCSI driver global settings -Get / Set/ Remove the iSCSI node configuration settings -Retrieve active session's information of a target -Reload the iSCSI configuration file -Start / Stop the iSNS client -Save the current iSCSI configuration to file For help run: /sbin/iscsictl -h i. Using iscsictl for Global Parameter control Please refer User Guide to find all options and parameters available. a. Get the current global settings [root@host]# iscsictl -g b. Set the new global settings. [root@host]# iscsictl -G iscsi_offload_mode=ULP ii. Using iscsictl for Target control Please refer User Guide to find all options and parameters available. a. Start a target or ALL targets. [root@host]# iscsictl -S target=iqn.2004-05.com.chelsio.target [root@host]# iscsictl -S target=ALL b. Stop a target or ALL targets. [root@host]# iscsictl -s target=iqn.2004-05.com.chelsio.target [root@host]# iscsictl -s target=ALL c. Get all or a particular target's current configuration settings [root@host]# iscsictl -c [root@host]# iscsictl -c target=iqn.2004-05.com.chelsio.target d. Retrieve active session's information of a target [root@host]# iscsictl -r target=iqn.2004-05.com.chelsio.target 4.6.6. Unloading the module ======================================== Use the following command to unload the module: [root@host]# rmmod chiscsi_t4 NOTE: i. While using rpm-tar-ball for installation a. Uninstallation will result into chiscsi.conf file renamed into chiscsi.conf.rpmsave. b. It is advised to take a backup of chiscsi.conf file before you do an uninstallation and installation of new/same unified wire package. As re-installing/upgrading unified-wire package may lead to loss of chiscsi.conf file. ii. Installation/uninstallation using source-tar-ball will neither remove the conf file nor rename it. It will always be intact. However it's recommended to always take a backup of your configuration file for both methods of installation. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.7. iSCSI PDU Offload Initiator ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.7.1. Loading/Unloading the cxgb4i driver ======================================== a. Loading the driver ---------------------- The driver must be loaded by the root user. Any attempt to loading the driver as a regular user will fail. Load cxgb4i driver using the following command: [root@host]#modprobe cxgb4i The cxgb4i module registers a new transport class "cxgb4i". On SLES11sp1, load the driver with the '--allow' option: [root@host]#modprobe cxgb4i --allow If loading of cxgb4i displays "unknown symbols found" error in dmesg, then kill iscsid, unload all open iSCSI modules and related/dependent modules (like multipath, bnx2i etc) and then load cxgb4i driver. NOTE: i. Sometimes open-iSCSI fails to compile because of dependencies on libopenssl-devel (for SLES11SP1) or openssl-devel(for RHEL 6) packages, so install these packages first and then install the Chelsio Unified Wire package (this openssl-devel dependency will not be seen while using rpm.tar.gz to install package). ii. On SLES, if source package is used and make && make install doesn't install cxgb4i package then please follow these steps to configure your source tree and then re-run make && make install:- [root@host]# cd /usr/src/linux [root@host]# make clean [root@host]# make oldconfig [root@host]# make scripts If loading the driver fails, then follow the steps mentioned below and try again. i. Run the following command to view all the loaded iSCSI modules [root@host]# lsmod | grep iscsi vii. Now, unload them using the following command: [root@host]# rmmod b. Unloading the driver -------------------------- [root@host]#rmmod cxgb4i [root@host]#rmmod libcxgbi ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.8. FCoE Full Offload Initiator ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.8.1. Loading/Unloading Driver ======================================== a. Loading the driver ------------------------------ The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the driver, execute the following command: [root@host]# modprobe csiostor NOTE: To load the driver on SLES11sp1, execute the following command: [root@host]# modprobe csiostor --allow b. Unloading the driver ------------------------------ To unload the driver, execute the following command: [root@host]# modprobe -r csiostor NOTE: If multipath services are running, unload of FCoE driver is not possible. Stop the multipath service and then unload the driver. 4.8.2. Configuring the switch and Troubleshooting =============================================== Please refer to the User Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.9. Offload Bonding driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.9.1. Loading/Unloading Driver ======================================== The driver must be loaded by the root user. Any attempt to loading the driver as a regular user will fail. a. Loading the driver ------------------------------ To load the Bonding driver (with offload support) [root@host]# modprobe bonding b. Unloading the driver ------------------------------ To unload the bonding driver. [root@host]# rmmod bonding c. Creating a Bond Interface ------------------------------ The Chelsio Offload Bonding driver supports all bond modes for NICs. In case of Chelsio adapters using TOE, only the active-backup (mode=1) and 802.3ad (mode=4) modes are supported. To create a bond, use the following command: [root@host]# modprobe bonding mode=1 miimon=100 This would create a bond interface (bondX) in active-backup mode. The interfaces can be added to bond using: [root@host]# ifconfig bond0 up [root@host]# ifenslave bond0 ethX ethY Load the t4_tom driver to offload the traffic over the bond interface. [root@host] # modprobe t4_tom For more information on the parameters supported by the bonding driver, [root@host]# modinfo bonding 4.9.2. Network Device Configuration ======================================== Please refer to the operating system documentation for administration and configuration of network devices. NOTE: Some operating systems may attempt to auto-configure the detected hardware and some may not detect all ports on a multi-port adapter. If this happens, please refer to the operating system documentation for manually configuring the network device. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.10. Offload IPv6 driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.10.1. Loading/Unloading Driver ======================================== The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. a. Loading the driver ------------------------------ After installing the drivers and rebooting the host, the IPv6 driver loads automatically. However, it is possible to manually load the driver by executing the command below: [root@host]# modprobe ipv6 b. Unloading the driver ------------------------------ To unload the ipv6 driver. [root@host]# rmmod ipv6 c. Offloading IPv6 traffic ------------------------------ Once the IPv6 Offload driver is loaded successfully, load the t4_tom driver, [root@host]# modprobe t4_tom All the IPv6 TCP traffic will be offloaded now. 4.10.2. Network Device Configuration ======================================== Please refer to the operating system documentation for administration and configuration of network devices. NOTE: Some operating systems may attempt to auto-configure the detected hardware and some may not detect all ports on a multi-port adapter. If this happens, please refer to the operating system documentation for manually configuring the network device. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.11. Bypass Driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.11.1. Loading/Unloading Driver ======================================== The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. a. Loading the driver ------------------------------ Run the following command to load the Bypass driver: [root@host]# modprobe cxgb4 b. Unloading the driver ------------------------------ Run the following command to unload the Bypass driver: [root@host]# rmmod cxgb4 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.12. WD Sniffing and Tracing ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The objective of these utilities (wd_sniffer and wd_tcpdump_trace) is to provide sniffing and tracing capabilities by making use of T4's hardware features. Sniffer- Involves targeting specific multicast traffic and sending it directly to user space. Tracer - All tapped traffic is forwarded to user space and also pushed back on the wire via the internal loop back mechanism In either mode the targeted traffic bypasses the kernel TCP/IP stack and is delivered directly to user space by means of a RX queue which is defined by the register MPS_TRC_RSS_CONTROL. For more information,please refer to the User Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.13. UDP Segmentation Offload and Pacing ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The diver is designed to provide UDP segmentation offload and pacing. 4.13.1. Loading/Unloading Driver ======================================== The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. a. Loading the driver ------------------------------ Run the following commands to load the driver: [root@host]# modprobe cxgb4 [root@host]# modprobe t4_tom Though normally associated with the Chelsio TCP Offload engine, the t4_tom module is required in order to allow for the proper redirection of UDP socket calls. b. Unloading the driver ------------------------------ Reboot the system to unload the driver. 5. Support Documentation ================================================================================ The documentation for this release can be found inside the ChelsioUwire-x.x.x.x/docs folder. It contains: - README - Release Notes - User's Guide 6. Customer Support ================================================================================ Please contact Chelsio support at support@chelsio.com for any issues regarding the product. ******************************************************************************** Copyright (C) 2012 Chelsio Communications. All Rights Reserved The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system,or transmitted in any form or by any means without the express written consent of Chelsio Communications.