**************************************** README **************************************** Chelsio Unified Wire for FreeBSD Version : 3.2.0.0 (Beta) Date : 02/24/2017 Overview ================================================================================ Chelsio Unified Wire driver package for FreeBSD contains drivers and utilities for Chelsio's Unified Wire adapters. It consists of the following components: - Network (NIC-TOE) - iWARP - iSCSI PDU Offload Target - Classification and Filtering - Offload IPv6 ================================================================================ CONTENTS ================================================================================ - 1. Requirements - 2. Supported Operating System - 3. Supported Cards - 4. How to Use - 5. Support Documentation - 6. Customer Support 1. Requirements ================================================================================ Chelsio adapter and system with supported platforms mentioned in section 2. 2. Supported Operating System ================================================================================ - FreeBSD 10.3-RELEASE - FreeBSD 11.0-RELEASE Other versions have not been tested and are not guaranteed to work. 3. Supported Cards ================================================================================ |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T62100-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T62100-LP-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T62100-SO-CR |NIC/TOE,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T6225-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T6225-SO-CR |NIC/TOE,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T580-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T580-LP-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T580-SO-CR |NIC,Filtering | |------------------------|-----------------------------------------------------| |T540-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T520-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T520-LL-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T520-SO-CR |NIC,Filtering | |------------------------|-----------------------------------------------------| |T420-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T420-LL-CR |NIC/TOE,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T440-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T440-LP-CR |NIC/TOE,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T422-CR |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T420-SO-CR |NIC,Filtering | |------------------------|-----------------------------------------------------| |T420-CX |NIC/TOE,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T420-BT |NIC/TOE,iSCSI,Filtering,IPv6 | |------------------------|-----------------------------------------------------| |T404-BT |NIC/TOE,iWARP,iSCSI,Filtering,IPv6 | |------------------------------------------------------------------------------| 4. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Compiling and installing the kernel and driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ a) Install FreeBSD 10.X/11.x from ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/ b) Download the tar-ball ChelsioUwire-FBSD-x.x.x.x.tar.gz from Chelsio Download Center, http://service.chelsio.com c) Untar the package. [root@host~]# tar xvf ChelsioUwire-FBSD-x.x.x.x.tar.gz d) This package has the entire FreeBSD world (operating system). You need to build entire FreeBSD world since infiniband core driver compiles only within FreeBSD kernel and not separately as a module. Build the FreeBSD world (operating system) from the package as follows: i. Add the following entry in /etc/src.conf file. WITH_OFED='yes' ii. Create a backup of existing kernel source. [root@host~]# mv /usr/src /usr/src_bkp iii. Copy the contents of ChelsioUwire-FBSD-x.x.x.x/kernel-head to /usr/src/ iv. Build and install the world as follows: [root@host~]# cd usr/src/ [root@host~]# make buildworld [root@host~]# make installworld v. Build and install the kernel as follows: [root@host~]# make buildkernel [root@host~]# make installkernel vi. Add the following entries to /boot/loader.conf to enable capabilities: hw.cxgbe.config_file="flash" hw.cxgbe.iscsicaps_allowed=0xf vii. Reboot the machine for changes to take effect. viii. After reboot, check if you have booted to the right kernel by executing "uname -a". You should see a similar output: FreeBSD host 12.0-CURRENT FreeBSD 12.0-CURRENT #0 f061800bf4e8 (freebsd-head) tip: Thu Nov 10 20:02:53 IST 2016 root@host:/usr/obj/usr/src/sys/GENERIC amd64 Verify that the kernel supports TOE: [root@host~]# config -x /boot/kernel/kernel | grep TCP_OFFLOAD options TCP_OFFLOAD 1. Userland Tools ----------------- Build and install cxgbetool: [root@host~]# cd /usr/src/tools/tools/cxgbetool [root@host~]# make && make install 2. Firmware ------------ Firmware (v1.16.33.0) will have to be loaded manually. Run the following commands for T6 adapters: [root@host~]# kldload if_cxgbe [root@host~]# cxgbetool t6nexX loadfw ChelsioUwire-FBSD-x.x.x.x/firmware/t6fw-x.xx.xx.x.bin [root@host~]# cxgbetool t6nexX loadcfg ChelsioUwire-FBSD-x.x.x.x/firmware/t6-config.txt Reboot system for changes to take effect. Firmware version can be verified using: [root@host~]# sysctl -n dev.t6nex.X.firmware_version NOTE: For T5 adapters, use "t5nex" as the parent device, "t5fw-x.xx.xx.x.bin" as the firmware binary file and "t5-config.txt" as the firmware configuration file. Similarly, for T4 adapters, use "t4nex", "t4fw-x.xx.xx.x.bin" and "t4-config.txt" respectively. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Network (NIC-TOE) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Loading ============== The driver must be loaded by the root user. Any attempt to load as a regular user will fail. - Load the NIC module (if_cxgbe) using the command given below: [root@host~]# kldload if_cxgbe - The TOE driver module (t4_tom.ko) is needed only if you plan to use the TOE and iWARP capabilities, and must be loaded manually. [root@host~]# kldload t4_tom The NIC module (if_cxgbe) by itself can be loaded / unloaded at any time, but once the TOE driver module (t4_tom) is loaded, it will prevent both the modules from being unloaded. Driver Configuration ==================== Enabling network interface: --------------------------- Enable any Chelsio T6 network interface using the following command: [root@host~]# ifconfig ccX up For T5 interfaces, run the following command: [root@host~]# ifconfig cxlX up For T4 interfaces, run the following command: [root@host~]# ifconfig cxgbeX up Enabling TCP offload --------------------- Enable TCP offload on a T6 interface using ifconfig. An error usually indicates that t4_tom is not loaded. [root@host~]# ifconfig ccX toe For T5 interfaces, run the following command: [root@host~]# ifconfig cxlX toe For T4 interfaces, run the following command: [root@host~]# ifconfig cxgbeX toe If successful, TOE4 and TOE6 will be displayed in the list of the enabled options as shown in example below: [root@host~]# ifconfig cc0 toe [root@host~]# ifconfig cc0 cc0: flags=8802 metric 0 mtu 1500 options=ecc7bb ether 00:07:43:04:b2:30 nd6 options=29 media: Ethernet 40Gbase-CR4 status: active When a TOE capable interface is the outbound path to a destination to which a new connection is being established, the connection should be offloaded automatically. Verify with netstat: [root@host~]# netstat -np tcp | grep toe T6 SO (Server Offload) adapters support limited number of offload connections. To see the maximum number of offload connections, run the following command: [root@host~]# sysctl dev.t6nex.X.misc.tids dev.t6nex.0.misc.tids: ATID range: 0-127, in use: 0 TID range: 0-255, in use: 0 STID range: 256-319, in use: 0 FTID range: 320-815 HW TID usage: 0 IP users, 0 IPv6 users Where, TID is the number of offload connections. STID is the number of offload servers. Enabling DDP ------------- DDP is disabled by default. To enable it, run the "sysctl" command as: For T6 adapters: [root@host~]# sysctl -w dev.t6nex.X.toe.ddp=1 For T5 adapters: [root@host~]# sysctl -w dev.t5nex.X.toe.ddp=1 For T4 adapters: [root@host~]# sysctl -w dev.t4nex.X.toe.ddp=1 Driver Unloading ================ - To unload the NIC driver, stop all the running traffic and run the following command: [root@host~]# kldunload if_cxgbe NOTE: NIC driver cannot be unloaded if TOE module is loaded. - To unload the TOE driver, reboot the system. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iWARP ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver Loading ============== The driver must be loaded by the root user. Any attempt to load as a regular user will fail. a) If not done already, load NIC and TOE drivers: [root@host~]# kldload if_cxgbe [root@host~]# kldload t4_tom b) For T6, bring up the interface and enable TOE: [root@host~]# ifconfig ccX up [root@host~]# ifconfig ccX toe NOTE: For T5 and T4, use "cxlX" and "cxgbeX" respectively. c) Load the iWARP driver using the following command: [root@host~]# kldload iw_cxgbe Verifying iWARP functionality ============================= a) Load krping module using the following command: [root@host~]# kldload krping b) Start krping server on one machine as: [root@host~]# echo "verbose,count=5000,server,addr=,port=" > /dev/krping c) Start krping client on PEER machine as: [root@host~]# echo "verbose,count=5000,client,addr=,port=" > /dev/krping d) After starting the client, read the /dev/krping on server as: [root@host~]# cat /dev/krping krping: num device snd bytes snd msgs rcv bytes rcv msgs wr bytes wr msgs rd bytes rd msgs krping: 1 listen 0 0 16 1 60085 70624 917184 0 Driver Unloading ================ - To unload the iWARP driver, stop all the running traffic and run the following command: [root@host~]# kldunload iw_cxgbe ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSCSI PDU Offload Target ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NOTE: This product will be supported in the next release. Module Installation =================== a) Compile and install iSCSI module [root@host~]# cd /usr/src/sys/modules/iscsi [root@host~]# make [root@host~]# make install b) Compile and install ctl drivers [root@host~]# cd /usr/src/sys/modules/ctl [root@host~]# make [root@host~]# make install c) Compile and install ctld [root@host~]# cd /usr/src/usr.sbin/ctld [root@host~]# make all install d) Compile and install iSCSI PDU offload target module [root@host~]# cd /usr/src/sys/modules/cxgbe/cxgbei [root@host~]# make [root@host~]# make install Module Loading ============== a) If not done already, load NIC and TOE drivers: [root@host~]# kldload if_cxgbe [root@host~]# kldload t4_tom b) For T6, bring up the interface and enable TOE: [root@host~]# ifconfig ccX up [root@host~]# ifconfig ccX toe NOTE: For T5 and T4, use "cxlX" and "cxgbeX" respectively. c) Load iSCSI PDU offload target module: [root@host~]# kldload cxgbei Starting iSCSI Target ===================== a) Make necessary changes in the iSCSI configuration file located in /etc/ctl.conf. NOTE: Refer to the official FreeBSD documentation for more information. b) Start iSCSI service: [root@host~]# service ctld onestart Module Unloading ================ Use the following command to unload the iSCSI PDU offload target module: [root@host~]# kldunload cxgbei ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Classification and Filtering ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Usage ===== Creating Filter Rules --------------------- Network driver (if_cxgbe) and "cxgbetool" must be installed before setting the filter rule. Next, run the following command to load the network driver: [root@host~]# kldload if_cxgbe Now, create filter rules using cxgebtool: [root@host~]# cxgbetool t6nexY filter action [pass/drop/switch] Where, t6nexY: Chelsio T6 interface. Use t5nexY and t4nexY for T5 and T4 adapters respectively. index : positive integer ranging from 0-495 set as filter id. action: Ingress packet disposition. pass : Ingress packets will be passed through set ingress queues. switch: Ingress packets will be routed to an output port with optional header rewrite. drop : Ingress packets will be dropped. NOTE: In case of multiple filter rules, the rule with the lowest filter index takes higher priority. Listing Filter Rules -------------------- To list previously set filters, run the following command: [root@host~]# cxgbetool t6nexY filter list Removing Filter Rules ---------------------- To remove a filter, run the following command with the corresponding filter rule index [root@host~]# cxgbetool t6nexY filter delete NOTE: For more information on additional parameters, refer cxgbetool manual by running the "man cxgbetool" command. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Offload IPv6 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ a) Add the following entry in /etc/rc.conf file: ipv6_prefer="YES" b) Reboot the system. c) TOE driver (t4_tom) must be installed and loaded in order to use the Offload IPv6 feature. Also, the driver must be loaded by the root user. Any attempt to load the driver as a regular user will fail. Now, load the TOE driver as: [root@host~]# kldload t4_tom d) For T6, bring up the interface and enable TOE: [root@host~]# ifconfig ccX up [root@host~]# ifconfig ccX toe NOTE: For T5 and T4, use "cxlX" and "cxgbeX" respectively. All IPv6 traffic over Chelsio interfaces will be offloaded now. 5. Support Documentation ================================================================================ The documentation for this release can be found inside the ChelsioUwire-FBSD-x.x.x.x/docs folder. It contains: - README - Release Notes - User's Guide 6. Customer Support ================================================================================ Please contact Chelsio support at support@chelsio.com for any issues regarding the product. ******************************************************************************** Copyright (C) 2017 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system,or transmitted in any form or by any means without the express written consent of Chelsio Communications.