**************************************** README **************************************** Chelsio T5 Family Memory Free Driver for Linux Version : 1.0.0.0 (Alpha) Date : 05/24/2016 Overview ================================================================================ Chelsio Memory Free Driver package for Linux is an easy to use utility developed to provide installation of 64-bit Linux based drivers and tools for Chelsio's T5 Unified Wire Adapters. The driver package provides an interactive installer to install various drivers and utilities. It consists of the following components: a. Network (NIC/TOE) b. iWARP (RDMA) c. iSCSI PDU Offload Target d. iSCSI PDU Offload Initiator e. Utility Tools (cop,cxgbtool,t4_perftune,benchmark tools) ================================================================================ CONTENTS ================================================================================ - 1. Requirements - 2. Supported Operating Systems - 3. Supported Cards - 4. How To Use - 5. Support Documentation - 6. Customer Support 1. Requirements ================================================================================ 1.1. Network (NIC/TOE) ====================== - Chelsio T5 40/10Gb adapter and system with supported platforms mentioned in section 2. 1.2. iWARP (RDMA) ================= - Chelsio T5 40/10Gb adapter and system with supported platforms mentioned in section 2. - If you are planning to upgrade OFED on one member of the cluster, the upgrade needs to be installed on all the members. 1.3. iSCSI PDU Offload Target ============================= - Chelsio T5 40/10Gb adapter and system with supported platforms mentioned in section 2. - iSCSI PDU offload target driver (chiscsi_t4.ko) requires NIC(cxgb4), TOE(t4_tom & toecore) and iSCSI non-offload(chiscsi_base.ko) modules to work. Whereas the iSCSI non-offload target driver requires only NIC module. 1.4. iSCSI PDU Offload Initiator ================================ - Chelsio T5 40/10Gb adapter and system with supported platforms mentioned in section 2. - The iSCSI PDU Offload Initiator driver (cxgb4i) runs on top of NIC module(cxgb4) and open-iscsi-2.0-872/873 only, on a T5 card. - If you're planning to install iSCSI PDU Offload Initiator, please install openssl-devel package. 2. Supported Operating Systems ================================================================================ The Memory Free Driver package has been developed to run on 64-bit Linux based platforms. Following is the list of supported Linux distributions: - RHEL 7.2, 3.10.0-327.el7 - RHEL 6.7, 2.6.32-573.el6 - RHEL 6.6, 2.6.32-504.el6 - Kernel.org linux-4.1 - Kernel.org linux-3.18 - Kernel.org linux-3.17 NOTE:Other kernel versions have not been tested and are not guaranteed to work. 3. Supported Adapters ================================================================================ - T580-OCP-SO - T580-SO-CR - T520-OCP-SO - T520-SO-CR - T580-LP-CR* - T580-CR* - T540-CR* - T520-BT* - T520-LL-CR* - T520-CR* *Not supported with memory free configuration 4. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.1. Memory Free Driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The following table describes the various "configuration tuning options" available during installation and drivers/software installed with each option: |#######################################|######################################| | Configuration Tuning Option | Driver/Software installed | |#######################################|######################################| |Memory Free TOE & iWARP (Default) |NIC/TOE,iWARP | |---------------------------------------|--------------------------------------| |Memory Free iSCSI |NIC,iSCSI Initiator | |---------------------------------------|--------------------------------------| |Unified Wire |iSCSI Target | |------------------------------------------------------------------------------| The following are the maximum number of IPv4 offload connections supported with each protocol in memory free configuration: |#######################################|######################################| | Offload Protocol | # Max Connections | |#######################################|######################################| |TOE |256 | |---------------------------------------|--------------------------------------| |iWARP |255* | |---------------------------------------|--------------------------------------| |iSCSI Initiator |128 | |------------------------------------------------------------------------------| *The actual number of connections offloaded will depend on the I/O size used. IMPORTANT: Running multiple protocols simultaneously is not supported in the current release. 4.1.1. Installation =================== Uninstall any existing version of Chelsio Unified Wire drivers in the system. Follow the steps mentioned below to install the driver package: a. Download the tarball Chelsio-Memfree-x.x.x.x.tar.gz b. Untar the tarball [root@host~]# tar zxvfm Chelsio-Memfree-x.x.x.x.tar.gz c. Change your current working directory to "Chelsio-Memfree-x.x.x.x" directory. Build the source: [root@host~]# make d. Install the drivers and tools: [root@host~]# make install e. The default configuration tuning option is "Memory Free TOE & iWARP". The configuration tuning can be selected using the following commands: [root@host~]# make CONF= [root@host~]# make CONF= install NOTE: To view the different configuration tuning options, view help by typing [root@host~]# make help f. Reboot your machine for changes to take effect. Installation on updated kernels ------------------------------- If the kernel version on your Linux distribution is updated, please execute the following command to install the Unified Wire package: [root@host~]# make UNAME_R= Where kernel_version is the nearest supported kernel version. For example, if you want to install the package on a RHEL 6 distribution updated to 2.6.32-431.20.3.el6 kernel, run the following commands: [root@host~]# make UNAME_R=2.6.32-431.el6 [root@host~]# make UNAME_R=2.6.32-431.el6 install To view the list of the supported kernel versions, run the following command: [root@host~]# make list_kernels Reboot your machine for changes to take effect. 4.1.2. Firmware Update ====================== The T5 firmware(v1.15.37.0) is installed on the system, typically in /lib/firmware/cxgb4, and the driver will auto-load the firmwares if an update is required. The kernel must be configured to enable userspace firmware loading support: Device Drivers -> Generic Driver Options -> Userspace firmware loading support The firmware version can be verified using ethtool: [root@host~]# ethtool -i 4.1.3. Uninstallation ====================== Change your working directory to "Chelsio-Memfree-x.x.x.x" directory. Uninstall using the following command: [root@host~]# make uninstall NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.2. Network (NIC/TOE) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ IMPORTANT: Please ensure that only SO adapters are installed on server and client machines with "Memory Free TOE & iWARP" configuration tuning option. Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding. The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. - To load the driver in NIC mode(without offload support) [root@host~]# modprobe cxgb4 - To load driver in TOE mode(with offload support) [root@host~]# modprobe t4_tom NOTE: For Memory free configuration, both server and client machines must be in TOE mode. Driver Unloading ================ - To unload the driver in NIC mode(without offload support): [root@host~]# rmmod cxgb4 - A reboot is required to unload the driver in TOE(with Offload support). To avoid rebooting, follow the steps mentioned below: a. Load t4_tom driver with unsupported_allow_unload parameter. [root@host~]# modprobe t4_tom unsupported_allow_unload=1 b. Stop all the offloaded traffic, servers and connections. Check for the reference count. [root@host~]# cat /sys/module/t4_tom/refcnt If the reference count is 0, the driver can be directly unloaded. Skip to step (c). If the count is non-zero, load a COP policy which disables offload using the following procedure: i. Create a policy file which will disable offload [root@host~]# cat policy_file all => !offload ii. Compile and apply the output policy file [root@host~]# cop –o no-offload.cop policy_file [root@host~]# cxgbtool ethX policy no-offload.cop c. Unload the driver: [root@host~]# rmmod t4_tom [root@host~]# rmmod toecore [root@host~]# rmmod cxgb4 NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.3. iWARP (RDMA) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ IMPORTANT: Please ensure that only SO adapters are installed on server and client machines with "Memory Free TOE & iWARP" configuration tuning option. Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. The driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. To load the iWARP driver we need to load the NIC driver & core RDMA drivers first: [root@host~]# modprobe cxgb4 [root@host~]# modprobe iw_cxgb4 snd_win=16384 rcv_win=262144 [root@host~]# modprobe rdma_ucm Driver Unloading ================ To unload the iWARP driver, run the following command: [root@host~]# rmmod iw_cxgb4 IMPORTANT:openmpi-1.4.3 can cause IMB benchmark stalls due to a shared memory BTL issue. This issue is fixed in openmpi-1.4.5 and later releases. Hence, it is recommended that you download and install the latest stable release from Open MPI's official website, http://www.open-mpi.org NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.4. iSCSI PDU Offload Target ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ IMPORTANT: Please ensure that only non-SO adapters are installed on target machine with "Unified Wire" configuration tuning option. Run the following command: [root@host~]# make CONF=UNIFIED_WIRE install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. To load the module, run modprobe as follows: [root@host~]# modprobe chiscsi_t4 iso_on=0 lro_on=0 Configuration for Memory Free Initiator ======================================= The target needs to be configured with below settings in order to be used with memory free iSCSI PDU Offload initiator: a. Configure MTU of 9000 bytes on the Chelsio interface using the command below: [root@host~]# ifconfig ethX mtu 9000 b. Edit the following entries in the target configuration file "chiscsi.conf" located in "/etc/chelsio/": target: HeaderDigest=None DataDigest=None ImmediateData=No InitialR2T=Yes c. Start the target: [root@host~]# iscsictl -S target=ALL d. Set the iSCSI performance parameters: [root@host~]# iscsictl -G iscsi_perf_params=0x1 Driver Unloading ================ Use the following command to unload the module: [root@host~]# rmmod chiscsi_t4 NOTE:Installation/uninstallation will neither remove the conf file nor rename it. It will always be intact. However it is recommended to always take a backup of your configuration file. NOTE: For more information on additional configuration options, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.5. iSCSI PDU Offload Initiator ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ IMPORTANT: Please ensure that only SO adapters are installed on initiator machine with "Memory Free iSCSI" configuration tuning option. Run the following command: [root@host~]# make CONF=MEMORY_FREE_ISCSI install Driver Loading ============== IMPORTANT:Please ensure that all inbox drivers are unloaded before proceeding with unified wire drivers. The driver must be loaded by the root user. Any attempt to loading the driver as a regular user will fail. Load cxgb4i driver using the following command: [root@host~]# modprobe cxgb4i The cxgb4i module registers a new transport class "cxgb4i". If loading of cxgb4i displays "unkown symbols found" error in dmesg, follow the steps mentioned below: a. Kill iSCSI daemon "iscsid" b. View all the loaded iSCSI modules [root@host~]# lsmod | grep iscsi c. Now, unload them using the following command: [root@host~]# rmmod d. Finally reload the cxgb4i driver. Memory Free Configuration ========================= NOTE: Please ensure that the target to which the initiator connects, is Chelsio PDU Offload Target with a non-SO adapter installed. Also, the target should be configured as instructed in the iSCSI PDU Offload Target section. a. Configure MTU of 9000 bytes on the Chelsio interfaces using the command below: [root@host~]# ifconfig ethX mtu 9000 b. Discover target from initiator: [root@host~]# iscsiadm -m discovery -t st -p :3260 -I cxgb4i. c. Login to target: [root@host~]# iscsiadm -m node -p :3260 -I cxgb4i. -l -T d. Logout from target: [root@host~]# iscsiadm -m node -p :3260 -I cxgb4i. -u Driver Unloading ================ [root@host~]# rmmod cxgb4i [root@host~]# rmmod libcxgbi NOTE: For more information on additional configuration options, please refer User's Guide. 5. Support Documentation ================================================================================ The documentation for this release can be found inside the Chelsio-Memfree-x.x.x.x/docs folder. It contains: - README - Release Notes - User's Guide 6. Customer Support ================================================================================ Please contact Chelsio support at support@chelsio.com for any issues regarding the product. ******************************************************************************** Copyright (C) 2016 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system,or transmitted in any form or by any means without the express written consent of Chelsio Communications.