**************************************** README **************************************** Chelsio Unified Wire for VMware ESXi 6.7 Version : 3.1.0.22 (Beta) Date : 03/29/2019 Overview ================================================================================ The Chelsio Unified Wire package installs various drivers and utilities and consists of the following software: - Native Network (NIC) Driver with SR-IOV support - iSCSI/iSER Offload Initiator Driver NOTE: Drivers are not VMware certified. ================================================================================ CONTENTS ================================================================================ - 1. Supported Operating Systems - 2. Supported Cards - 3. How to Use - 4. Support Documentation - 5. Customer Support 1. Supported Operating Systems ================================================================================ Host ==== - ESXi 6.7 - ESXi 6.7 U1 Virtual Machine (with VFs): =========================== - RHEL 7.6, 3.10.0-957.el7 - RHEL 7.5, 3.10.0-862.el7 - RHEL 6.10, 2.6.32-754.el6 - RHEL 6.9, 2.6.32-696.el6 - SLES 15, 4.12.14-23-default - SLES 12 SP4, 4.12.14-94.41-default - Ubuntu 18.04.1, 4.15.0-29-generic - Ubuntu 16.04.5, 4.4.0-131-generic - Kernel.org linux-4.9 (minimum 4.9 kernel version supported is 4.9.13) - Kernel.org linux-4.14.105 - Kernel.org linux-4.19.27 NOTE: Windows Guest is not supported with SR-IOV. 2. Supported Cards ================================================================================ |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T61100-OCP |NIC | |------------------------|-----------------------------------------------------| |T62100-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T62100-LP-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T62100-SO-CR |NIC | |------------------------|-----------------------------------------------------| |T6425-CR* |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T6225-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T6225-LL-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T580-SO-CR |NIC | |------------------------|-----------------------------------------------------| |T580-LP-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T580-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T540-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T540-LP-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T540-SO-CR |NIC | |------------------------|-----------------------------------------------------| |T520-SO-CR |NIC | |------------------------|-----------------------------------------------------| |T520-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T520-LL-CR |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T520-BT |NIC,iSCSI,iSER | |------------------------|-----------------------------------------------------| |T540-BT |NIC,iSCSI,iSER | |------------------------------------------------------------------------------| * All 4-ports of T6425-CR adapter will be functional only if PCIe x8 -> 2x PCIe x4 slot bifurcation is supported by the system and enabled in BIOS. Otherwise, only 2-ports will be functional. Memory-free Adapters ==================== |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T6225-SO-CR |NIC,iSCSI*,iSER* | |------------------------------------------------------------------------------| * 256 IPv4/128 IPv6 offload connections supported. 3. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Native Network Driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== To use SR-IOV feature: - SR-IOV should be enabled in the BIOS. - Intel Virtualization Technology for Directed I/O (VT-d) should be enabled in the BIOS. - PCI Express Slot should be ARI capable. Driver Installation =================== a) Download the driver package from https://service.chelsio.com. b) Unzip the driver package: [root@host:~] unzip .zip c) Put the host in maintenance mode using the vSphere Client (Inventory->Host->Enter Maintenance Mode) d) Install the driver: [root@host:~] cp cxl-*.vib /productLocker/ [root@host:~] cp cxl-*.vib /var/log/vmware/ [root@host:~] esxcli software vib install -v /productLocker/cxl-*.vib --no-sig-check NOTE: If there is an older version of the driver installed, use "update" instead of install. [root@host:~] esxcli software vib update -v /productLocker/cxl-*.vib --no-sig-check e) After installation/update completes successfully, exit from maintenance mode and reboot the system. Driver Loading ============== After rebooting the ESXi Host, the driver will load automatically. However, it is possible to manually load the driver by using the command below: [root@host:~] vmkload_mod cxl Execute the below command so that device manager performs a rescan: [root@host:~] kill -SIGHUP $(cat /var/run/vmware/vmkdevmgr.pid) cxgbtool ======== cxgbtool will be installed by default on installing the driver VIB. Usage: ------ To use cxgbtool, use the syntax: [root@host:~] /opt/chelsio/bin/cxgbtool NOTE: For information on available parameters and their usage, refer to cxgbtool help by running the "/opt/chelsio/bin/cxgbtool -h" command. Adapter Configuration ===================== The adapter's configuration should be updated for optimal performance in ESXi environment. Follow the steps mentioned below: a) Run the following cxgbtool command: [root@host:~] /opt/chelsio/bin/cxgbtool –c esxcfg –set NOTE: Not supported on T6 adapters. b) Reboot the system for changes to take effect. Firmware Update =============== Driver will auto-load the firmware (v1.23.4.0) if an update is required. The version can be verified using: [root@host:~] /opt/chelsio/bin/cxgbtool -c version Connecting a virtual machine ============================ To connect Chelsio adapter to a virtual machine: a) Create a new virtual switch: [root@host:~] esxcfg-vswitch -a vSwitchN b) Link a chelsio NIC to the newly created virtual switch: [root@host:~] esxcfg-vswitch -L vmnicN vSwitchN c) Create a new portgroup on the vSwitch: [root@host:~] esxcfg-vswitch -A vSwitchN d) From the vSphere client, right-click on the virtual machine, select the virtual network adapter to be used, and attach the newly created port group. Tuning vMotion for vSAN ======================== While creating a vSAN you may encounter out of memory issues, resulting in vMotions timing out. To avoid this, follow the steps mentioned below on all ESXi hosts in the vSAN: a) Set the low-memory packet heap value to the maximum supported value of 512: [root@host:~] esxcli system settings kernel set -s netPktHeapMaxMBPerGB -v 512 b) Set the networking packet buffer pool size to the maximum supported value of 200: [root@host:~] esxcli system settings kernel set -s netPktPoolMaxMBPerGB -v 200 c) Reboot the host. d) Enable maximum RSS queues: [root@host:~] vmkload_mod cxl rss_q=1 Virtual Functions (SR-IOV) ========================== Instantiate Virtual Functions ----------------------------- Follow the steps mentioned below to instantiate virtual functions: a) Load the native network driver (cxl) with "max_vfs" parameter and set it to a comma separated value to specify the number of VFs per port. In case of multiple adapters, use ',,' to separate the number of VFs per adapter: [root@host:~] esxcfg-module cxl -s max_vfs=W,X,,Y,Z Where, W: Number of VFs per port 0 of adapter 0. X: Number of VFs per port 1 of adapter 0. Y: Number of VFs per port 0 of adapter 1. Z: Number of VFs per port 1 of adapter 1. NOTE: A maximum of 16 VFs can be instantiated per port. b) Verify max_vfs setting using the -g option: [root@host:~] esxcfg-module -g cxl c) Reboot the ESXi host for changes to take effect. d) Check if VFs were instantiated successfully on the PCI bus by either using the shell prompt (using lspci) or vSphere GUI (under Host >Configuration>Advanced setting) NOTE: - Unloading driver when VFs are attached to VMs is not supported by VMware. - VMs with SR-IOV interface might not power on with "out of MSI-X vectors" message in vmkernel.log. To resolve this issue, you need to add "pciPassthru.maxMSIXvectors" parameter to VMs configuration file. Maximum value allowed for this param is 31. It is recommended to set the value according to the following equation: pciPassthru.maxMSIXvectors = + 2 For more information refer to: http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.networking.doc/GUID-880A9270-807F-4F2A-B443-71FF01DCC61D.html - Windows Guest is not supported with SR-IOV. Assigning VFs to VM -------------------- For instructions on how to assign virtual functions to a virtual machine, please refer to VMware's official documentation at http://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-CC021803-30EA-444D-BCBE-618E0D836B9F.html Using VFs in Linux VM ---------------------- For instructions on how to use the VFs in a Linux VM, please refer the User's Guide. VXLAN ===== For step-by-step instructions on how to set up a VXLAN using a Chelsio adapter, refer User's Guide. Driver Unloading ================ Execute the command below to unload the Native Network driver: [root@host:~] vmkload_mod -u cxl NOTE: If iSCSI/iSER Offload Initiator Driver (cheiscsi) is loaded, it needs to be unloaded before unloading the native network driver. Driver Uninstallation ===================== a) Put the host in maintenance mode using the vSphere Client (Inventory->Host->Enter Maintenance Mode). b) Uninstall the driver: [root@host:~] esxcli software vib remove -n cxl c) Reboot the host: [root@host:~] reboot NOTE: If iSCSI/iSER Offload Initiator Driver (cheiscsi) is installed, it needs to be uninstalled before uninstalling the native network driver. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSCSI/iSER Offload Initiator Driver ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The Chelsio Unified Wire series of adapters are Independent Hardware iSCSI adapters. They also support iSER protocol, a translation layer for operating iSCSI over RDMA transports, such as iWARP RDMA. Driver Installation =================== Native Network driver (cxl) is required for iSCSI/iSER Offload Initiator driver to work. Hence, ensure that the driver is installed and loaded as described in the "Native Network Driver" chapter before proceeding. a) Put the host in maintenance mode using the vSphere (desktop or web) Client. b) Install the iSCSI/iSER driver components "cheiscsi" and " chelsio-esx-libcheiscsi-ima.so": [root@host:~] cp *.vib /productLocker/ [root@host:~] cp *.vib /var/log/vmware/ [root@host:~] esxcli software vib install -v /productLocker/cheiscsi-*.vib --no-sig-check [root@host:~] esxcli software vib install -v /productLocker/chelsio-esx-libcheiscsi-ima.so-*.vib --no-sig-check NOTE: If there is an older version of the driver installed, use the below commands to "update" so that persistent values like IP address, target lists etc. can be retained. [root@host:~] esxcli software vib update -v /productLocker/cxl-*.vib --no-sig-check [root@host:~] esxcli software vib update -v /productLocker/cheiscsi-*.vib --no-sig-check [root@host:~] esxcli software vib install -v /productLocker/chelsio-esx-libcheiscsi-ima.so-*.vib --no-sig-check [root@host:~] esxcli software vib remove -n chelsio-esx-libcheiscsi_ima.so c) After installation/update completes successfully, exit from maintenance mode and reboot the host. Driver Loading ============== After rebooting the ESXi Host, the driver will load automatically. However, it is possible to manually load the driver by using the command below: [root@host:~] vmkload_mod cheiscsi Execute the below command so that device manager performs a rescan: [root@host:~] kill -SIGHUP $(cat /var/run/vmware/vmkdevmgr.pid) iSCSI or iSER Offload Initiator will be used automatically based on the target configuration (iSCSI or iSER mode). Driver Configuration ==================== For information on how to configure iSCSI/iSER Offload Initiator, please refer User's Guide. Driver Unloading ================ Logout all the existing iSCSI/iSER sessions. Execute the command below to unload the iSCSI/iSER Offload Initiator driver: [root@host:~] vmkload_mod -u cheiscsi Driver Uninstallation ===================== NOTE: Before proceeding, please ensure that no iSCSI/iSER session or connection is active and running. Follow the steps mentioned below to uninstall the iSCSI/iSER Offload Initiator driver: a) Put the host in maintenance mode using the vSphere (desktop or web) Client. b) Uninstall the driver: [root@host:~] esxcli software vib remove -n cheiscsi [root@host:~] esxcli software vib remove -n chelsio-esx-libcheiscsi-ima.so c) Reboot the host: [root@host:~] reboot  5. Support Documentation ================================================================================ The documentation for this release can be found inside the /doc folder. It contains: - README - Release Notes - User's Guide 6. Customer Support ================================================================================ Please contact Chelsio support at support@chelsio.com for any issues regarding the product. ******************************************************************************** Copyright (C) 2019 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system,or transmitted in any form or by any means without the express written consent of Chelsio Communications.