**************************************** README **************************************** Chelsio Unified Wire for FreeBSD Version : 3.0.0.2 (Beta) Date : 12/05/2016 Overview ================================================================================ Chelsio Unified Wire driver package for FreeBSD contains Network, iWARP and iSCSI drivers for Chelsio's Unified Wire adapters. It consists of the following components: - Network (NIC-TOE) - iWARP - iSCSI PDU Offload Target - Offload IPv6 ================================================================================ CONTENTS ================================================================================ - 1. Requirements - 2. Supported Operating System - 3. Supported Cards - 4. How to Use - 5. Support Documentation - 6. Customer Support 1. Requirements ================================================================================ Chelsio adapter and system with supported platforms mentioned in section 2. 2. Supported Operating System ================================================================================ - FreeBSD 10.3-RELEASE - FreeBSD 11.0-RELEASE Other versions have not been tested and are not guaranteed to work. 3. Supported Cards ================================================================================ ########################|######################################################| Chelsio Adapter | Driver/Software | ########################|######################################################| T62100-LP-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T6225-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T580-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T580-LP-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T580-SO-CR |NIC | ------------------------|------------------------------------------------------| T540-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T520-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T520-LL-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T520-SO-CR |NIC | ------------------------|------------------------------------------------------| T420-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T420-LL-CR |NIC/TOE,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T440-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T440-LP-CR |NIC/TOE,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T422-CR |NIC/TOE,iWARP,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T420-SO-CR |NIC | ------------------------|------------------------------------------------------| T420-CX |NIC/TOE,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T420-BT |NIC/TOE,iSCSI,IPv6 | ------------------------|------------------------------------------------------| T404-BT |NIC/TOE,iWARP,iSCSI,IPv6 | -------------------------------------------------------------------------------| 4. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.1. Compiling and installing the kernel and driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ i. Install FreeBSD 10.X/11.x from ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/ ii. Download the tar-ball ChelsioUwire-FBSD-x.x.x.x.tar.gz from Chelsio Download Center, http://service.chelsio.com iii. Untar the package. [root@host~]# tar xvf ChelsioUwire-FBSD-x.x.x.x.tar.gz iv. This package has the entire FreeBSD world (operating system). You need to build entire FreeBSD world since infiniband core driver compiles only within FreeBSD kernel and not separately as a module. Build the FreeBSD world(operating system) from the package as follows: a. Add the following entry in /etc/src.conf file. WITH_OFED='yes' b. Create a backup of existing kernel source. [root@host~]# mv /usr/src /usr/src_bkp c. Copy the contents of ChelsioUwire-FBSD-x.x.x.x/kernel-head to /usr/src/ d. Build and install the world as follows: [root@host~]# cd usr/src/ [root@host~]# make buildworld [root@host~]# make installworld e. Build and install the kernel as follows: [root@host~]# make buildkernel [root@host~]# make installkernel f. Add the following entries to /boot/loader.conf to enable capabilities: hw.cxgbe.config_file="flash" hw.cxgbe.rdmacaps_allowed=0xff hw.cxgbe.iscsicaps_allowed=0xf g. Reboot the machine for changes to take effect. h. After reboot, check if you have booted to the right kernel by executing "uname -a". You should see a similar output: FreeBSD host 12.0-CURRENT FreeBSD 12.0-CURRENT #0 f061800bf4e8 (freebsd-head) tip: Thu Nov 10 20:02:53 IST 2016 root@host:/usr/obj/usr/src/sys/GENERIC amd64 i. Verify that the kernel supports TOE: [root@host~]# config -x /boot/kernel/kernel | grep TCP_OFFLOAD options TCP_OFFLOAD 4.1.1. Userland Tools --------------------- Build and install cxgbetool: [root@host~]# cd /usr/src/tools/tools/cxgbetool [root@host~]# make && make install 4.1.2. Firmware --------------- Firmware (v1.16.22.0) will have to be loaded manually. Run the following commands for T6 adapters: [root@host~]# kldload if_cxgbe [root@host~]# cxgbetool t6nexX loadfw ChelsioUwire-FBSD-x.x.x.x/firmware/t6fw-x.xx.xx.x.bin [root@host~]# cxgbetool t6nexX loadcfg ChelsioUwire-FBSD-x.x.x.x/firmware/t6-config.txt Reboot system for changes to take effect. Firmware version can be verified using: [root@host~]# sysctl -n dev.t6nex.X.firmware_version NOTE: For T5 adapters, use "t5nex" as the parent device, "t5fw-x.xx.xx.x.bin" as the firmware binary file and "t5-config.txt" as the firmware configuration file. Similarly, for T4 adapters, use "t4nex", "t4fw-x.xx.xx.x.bin" and "t4-config.txt" respectively. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.2. Network (NIC-TOE) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.2.1. Loading/Unloading the driver ==================================== a. Loading the driver ---------------------- The driver must be loaded by the root user. Any attempt to load as a regular user will fail. i. Load the NIC module (if_cxgbe) using the command given below: [root@host~]# kldload if_cxgbe ii. The TOE driver module (t4_tom.ko) is needed only if you plan to use the TOE capability and must be loaded manually. [root@host~]# kldload t4_tom The NIC module (if_cxgbe) by itself can be loaded / unloaded at any time, but once the TOE driver module (t4_tom) is loaded, it will prevent both the modules from being unloaded. b. Unloading the driver ------------------------ i. To unload the NIC driver. [root@host~]# kldunload if_cxgbe NOTE: NIC driver cannot be unloaded if TOE module is loaded. ii. To unload the TOE driver, reboot the system. 4.2.2. Driver Configuration ============================ i. Enabling network interface: Enable any Chelsio T6 network interface using the following command: [root@host~]# ifconfig ccX up For T5 interfaces, run the following command: [root@host~]# ifconfig cxlX up For T4 interfaces, run the following command: [root@host~]# ifconfig cxgbeX up ii. Enable TCP offload on a T6 interface using ifconfig. An error usually indicates that t4_tom is not loaded. [root@host~]# ifconfig ccX toe For T5 interfaces, run the following command: [root@host~]# ifconfig cxlX toe For T4 interfaces, run the following command: [root@host~]# ifconfig cxgbeX toe If successful, TOE4 and TOE6 will be displayed in the list of the enabled options as shown in example below: [root@host~]# ifconfig cc0 toe [root@host~]# ifconfig cc0 cc0: flags=8802 metric 0 mtu 1500 options=ecc7bb ether 00:07:43:04:b2:30 nd6 options=29 media: Ethernet 40Gbase-CR4 status: active When a TOE capable interface is the outbound path to a destination to which a new connection is being offloaded, the connection should be offloaded automatically. Verify with netstat: [root@host~]# netstat -np tcp | grep toe iii. Enabling DDP ------------------ DDP is disabled by default. To enable it, run the "sysctl" command as: For T6 adapters: [root@host~]# sysctl -w dev.t6nex.X.toe.ddp=1 For T5 adapters: [root@host~]# sysctl -w dev.t5nex.X.toe.ddp=1 For T4 adapters: [root@host~]# sysctl -w dev.t4nex.X.toe.ddp=1 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.3. iWARP ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.3.1. Loading/Unloading the driver ==================================== a. Loading the driver ---------------------- The driver must be loaded by the root user. Any attempt to load as a regular user will fail. i. If not done already, load NIC and TOE drivers: [root@host~]# kldload if_cxgbe [root@host~]# kldload t4_tom ii. Bring up the interface and enable TOE: [root@host~]# ifconfig ccX up [root@host~]# ifconfig ccX toe iii. Load the iWARP driver using the following command: [root@host~]# kldload iw_cxgbe b. Unloading the driver ------------------------ Reboot the system to unload the iWARP driver. 4.3.2. Verifying iWARP functionality ===================================== i. Load krping module using the following command: [root@host~]# kldload krping ii. Start krping server on one machine as: [root@host~]# echo "verbose,count=5000,server,addr=,port=" > /dev/krping iii. Start krping client on PEER machine as: [root@host~]# echo "verbose,count=5000,client,addr=,port=" > /dev/krping iv. After starting the client, read the /dev/krping on server as: [root@host~]# cat /dev/krping krping: num device snd bytes snd msgs rcv bytes rcv msgs wr bytes wr msgs rd bytes rd msgs krping: 1 listen 0 0 16 1 60085 70624 917184 0 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.4. iSCSI PDU Offload Target ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.4.1. Module Installation ========================== i. Compile and install iSCSI module [root@host~]# cd /usr/src/sys/modules/iscsi [root@host~]# make [root@host~]# make install ii. Compile and install ctl drivers [root@host~]# cd /usr/src/sys/modules/ctl [root@host~]# make [root@host~]# make install iii. Compile and install ctld [root@host~]# cd /usr/src/usr.sbin/ctld [root@host~]# make all install iv. Compile and install iSCSI PDU offload target module [root@host~]# cd /usr/src/sys/modules/cxgbe/cxgbei [root@host~]# make [root@host~]# make install 4.4.2. Module Loading ===================== i. If not done already, load NIC and TOE drivers: [root@host~]# kldload if_cxgbe [root@host~]# kldload t4_tom ii. Bring up the interface and enable TOE: [root@host~]# ifconfig ccX up [root@host~]# ifconfig ccX toe iii. Load iSCSI PDU offload target module: [root@host~]# kldload cxgbei 4.4.3. Starting iSCSI Target ============================ i. Make necessary changes in the iSCSI configuration file located in /etc/ctl.conf. NOTE: Refer to the official FreeBSD documentation for more information. ii. Start iSCSI service: [root@host~]# service ctld onestart 4.4.4. Unloading the module ======================================== Use the following command to unload the iSCSI PDU offload target module: [root@host~]# kldunload cxgbei +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4.5. Offload IPv6 driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ i. Add the following entry in /etc/rc.conf file: ipv6_prefer="YES" ii. Reboot the system. iii. TOE driver (t4_tom) must be installed and loaded in order to use the Offload IPv6 feature. Also, the driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. Now, load the TOE driver as: [root@host~]# kldload t4_tom All IPv6 traffic will be offloaded now. 5. Support Documentation ================================================================================ The documentation for this release can be found inside the ChelsioUwire-FBSD-x.x.x.x/docs folder. It contains: - README - Release Notes - User's Guide 6. Customer Support ================================================================================ Please contact Chelsio support at support@chelsio.com for any issues regarding the product. ******************************************************************************** Copyright (C) 2016 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system,or transmitted in any form or by any means without the express written consent of Chelsio Communications.