**************************************** README **************************************** Chelsio DPDK driver for Linux Version : 2.2.0.0 (Beta) Date : 02/24/2017 Overview ================================================================================ Chelsio Data Plane Development Kit (DPDK) driver package is a collection of data plane libraries and NIC drivers optimized for running in the Linux user space to boost packet processing. The driver has support for: - Multiple queues for Tx and Rx - Receive Side Scaling (RSS) - VLAN filtering - Checksum offload - Promiscuous mode - All multicast mode - Port hardware statistics - Jumbo frames (only for UIO) - Classification and filtering (only for UIO) NOTE: Chelsio DPDK driver is built using open-source DPDK driver v16.04 with added support for Chelsio adapters. ================================================================================ CONTENTS ================================================================================ - 1. Requirements - 2. Supported Operating Systems - 3. Supported Cards - 4. How to Use - 5. Support Documentation - 6. Customer Support 1. Requirements ================================================================================ - Chelsio adapter and system with supported platforms mentioned in section 2. - flex, bison, byacc, patch, patchutils, autoconf, automake and rpm-build packages must be present in the machine (required for libpcap installation). 2. Supported Operating Systems ================================================================================ The Chelsio DPDK driver has been developed to run on 64-bit Linux platforms. Following is the list of supported distributions: 1. x86_64 Architecture ====================== |########################|#####################################################| | Linux Distribution | Driver/Software | |########################|#####################################################| |RHEL 7.2, |UIO,VFIO,Filtering | |3.10.0-327.el7 | | |------------------------|-----------------------------------------------------| |RHEL 7.1, |UIO,VFIO,Filtering | |3.10.0-229.el7 | | |------------------------------------------------------------------------------| NOTE: Other kernel versions have not been tested and are not guaranteed to work. 2. POWERPC64 Architecture ========================= |########################|#####################################################| | Linux Distribution | Driver/Software | |########################|#####################################################| |RHEL 7.2 (POWER8), |VFIO* | |3.10.0-327.el7.ppc64le | | |------------------------------------------------------------------------------| * Alpha release. Will be fully functional in the next release. NOTE: Other kernel versions have not been tested and are not guaranteed to work. 3. Supported Cards ================================================================================ - T62100-CR - T62100-LP-CR - T62100-SO-CR - T6225-CR - T6225-SO-CR - T580-CR - T580-LP-CR - T580-SO-CR - T540-CR - T520-CR - T520-LL-CR - T520-SO-CR - T520-BT 4. How to Use ================================================================================ IMPORTANT: Please ensure that any existing version of DPDK driver is uninstalled before proceeding. Driver Installation =================== a. Download the driver package b. Untar the tarball: [root@host~]# tar zxvf Chelsio-DPDK-x.x.x.x.tar.gz c. Change your working directory to Chelsio-DPDK-x.x.x.x [root@host~]# cd Chelsio-DPDK-x.x.x.x d. Compile and install the driver using one of the following options: - To install DPDK source, testpmd and Pktgen tool only: [root@host~]# make dpdk_install - To install DPDK source and all the tools provided in the DPDK repository: [root@host~]# make install NOTE: - The above commands will install the driver with the following build configuration: arch: x86_64/ppc_64 machine: native/power8 execenv: linuxapp toolchain: gcc - The above commands will create a target environment directory "x86_64-native-linuxapp-gcc/ppc_64-power8-linuxapp-gcc" in the driver package directory. - The pktgen application has been modified to disable packet classification, to reflect correct Rx rate as packet classification is an expensive operation. It will not be installed on POWERPC64 architecture. e. Reboot the system for changes to take effect. Firmware Update =============== Firmware (v1.16.33.0) is installed on the system, typically in /lib/firmware/cxgb4, and the kernel mode driver (cxgb4) will auto-load the firmware if an update is required. Load the kernel mode NIC driver (cxgb4): [root@host~]# modprobe cxgb4 Verify the firmware version using ethtool: [root@host~]# ethtool -i Flashing Firmware Configuration File ==================================== a. If not done already, load the kernel mode NIC driver (cxgb4): [root@host~]# modprobe cxgb4 b. Flash the firmware configuration file (for T6 adapters): [root@host~]# cxgbtool loadcfg /lib/firmware/cxgb4/t6-config.txt NOTE: For T5 adapters, use t5-config.txt. Driver Loading ============== NOTE: Before proceeding, unload all Chelsio inbox and outbox kernel mode drivers using the following commands: [root@host~]# rmmod cxgb4 [root@host~]# rmmod csiostor The driver must be loaded by the root user. Any attempt to load as a regular user will fail. UIO support ----------- Follow the steps mentioned below to load DPDK driver with UIO support: a. Disable Intel VT-d in system BIOS. b. Turn off Intel iommu by adding below entry in Kernel grub/grub2 menu: intel_iommu=off c. Reboot the system for changes to take effect. d. Load the UIO module: [root@host~]# modprobe uio [root@host~]# insmod Chelsio-DPDK-x.x.x.x/src/DPDK/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko VFIO support ------------- Follow the steps mentioned below to load DPDK driver with VFIO support: a. Enable Intel VT-d in system BIOS. b. Add the following entry to grub/grub2 menu: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 E.g.: kernel /vmlinuz-3.13.0-32-generic ro root=UUID=5149fae1-c52b-42a9-a48c-b0a70937e8fb intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet c. Reboot the system for changes to take effect. d. Load the VFIO module: [root@host~]# modprobe vfio-pci Configuration ============= Huge Pages ----------- Using script ------------ Run the "res_hugepages.sh" shell script (copied to /sbin during installation): [root@host~]# res_hugepages.sh Manual ------ a. Mount hugetlbfs: [root@host~]# mkdir -p /mnt/huge [root@host~]# mount -t hugetlbfs nodev /mnt/huge b. Reserve Huge Page memory manually: - x86_64: [root@host~]# echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages - POWERPC64: [root@host~]# echo 512 > /sys/devices/system/node/node0/hugepages/hugepages-16384kB/nr_hugepages In case of dual socket machines, run the above command for other CPU nodes. Binding network ports --------------------- NOTE: Please make sure that adapter's physical function 4 is used, since it is assigned for all NIC functions of the adapter. - Run the following command to bind network ports to DPDK environment with UIO support: [root@host~]# dpdk_nic_bind.py --bind=igb_uio Now, Verify using: [root@host~]# dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ 0000:03:00.4 'T6225-CR Unified Wire Ethernet Controller' drv=igb_uio unused= - Run the following command to bind network ports to DPDK environment with VFIO support: [root@host~]# dpdk_nic_bind.py --bind=vfio-pci Now, Verify using: [root@host~]# dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ 0000:03:00.4 'T6225-CR Unified Wire Ethernet Controller' drv=vfio-pci unused= Unbinding network ports ----------------------- a. Run the following command to unbind network ports from DPDK environment: [root@host~]# dpdk_nic_bind.py -u b. Run the following command to verify: [root@host~]# dpdk_nic_bind.py --status Running DPDK Test Applications ============================== For information(syntax,examples) regarding test applications Testpmd and Pktgen, and configuring RSS, Flow Control and Jumbo frames, please refer user's guide. Classification and Filtering ============================ NOTE: Currently supported on T5 adapters only. Classification and Filtering feature enhances network security by controlling incoming traffic as they pass through network interface based on source and destination addresses, protocol, source and receiving ports, or the value of some status bits in the packet. DPDK driver currently supports LE-TCAM (maskfull) filters which enable specifying masks to the accept criteria. The masks will allow specifying a match for a range of values. You can create up to 496 LE-TCAM filters. For more information regarding the feature, please refer the User's Guide. Driver Unloading ================ UIO support ------------ Run the following commands to unload DPDK driver with UIO support: [root@host~]# rmmod igb_uio [root@host~]# rmmod uio If unloading "uio" module reports an error, unload the following dependent modules and try again: [root@host~]# rmmod bnx2fc [root@host~]# rmmod bnx2i [root@host~]# rmmod cnic VFIO support ------------ Run the following command to unload DPDK driver with VFIO support: [root@host~]# rmmod vfio-pci Driver Uninstallation ===================== a. Change your working directory to Chelsio-DPDK-x.x.x.x directory: [root@host~]# cd Chelsio-DPDK-x.x.x.x b. Uninstall the DPDK driver using the following command: [root@host~]# make uninstall 5. Support Documentation ================================================================================ The documentation for this release can be found in the Chelsio-DPDK-x.x.x.x\docs\ directory. It contains: - README - Release Notes - User's Guide 6. Customer Support ================================================================================ Please contact Chelsio support at support@chelsio.com for any issues regarding the product. ******************************************************************************** Copyright (C) 2017 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system,or transmitted in any form or by any means without the express written consent of Chelsio Communications.