**************************************** README **************************************** Chelsio Unified Wire for Windows Version : 6.15.35.0_WIN_006.0.76 Date : 08/31/2020 Overview ================================================================================ Chelsio Unified Wire is an easy to use utility developed to provide installation of 64-bit Windows based drivers and tools for Chelsio's adapters. It consists of the following components: - NDIS Function Driver (NIC) v6.15.35.0 - NVGRE Offload - VXLAN Task Offload - Timestamps (Hardware and Software) - PacketDirect (Kernel Mode and ARM Moderation) - VMMQ/vRSS - iWARP (ND) - SMB Direct - RDMA with Virtual Switch (Mode 2) - Storage Spaces Direct (S2D) - Storage Replica (SR) - Mesh Topology - NDIS SR-IOV - Guest RDMA (Mode 3) - iSCSI Storport Miniport Driver v6.15.35.0 - iSER Storport Miniport Driver v6.15.35.0 - NVMe Storport Miniport Driver v6.15.35.0 - Data Center Bridging (DCB) ================================================================================ CONTENTS ================================================================================ - 1. Supported Operating Systems - 2. Supported Cards - 3. How to Use - 4. Support Documentation - 5. Customer Support 1. Supported Operating Systems ================================================================================ The Chelsio Unified Wire software has been developed to run on Windows based platforms. Following is the list of Drivers/Software and supported Windows version(s). |########################|#####################################################| | Windows Version | Driver/Software | |########################|#####################################################| |Server 2019 |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |Server 2016 |NIC,NVGRE,VXLAN,PacketDirect,VMMQ/vRSS,ND,SMB Direct*| | |S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,NVMe,DCB | |------------------------|-----------------------------------------------------| |10 Client |NIC,ND^,SMB Direct†^,Mesh,iSCSI,iSER,NVMe^,DCB | |------------------------|-----------------------------------------------------| |Server 2012 R2 |NIC,NVGRE,ND,SMB Direct†,Mesh,SR-IOV,iSCSI,DCB | |------------------------------------------------------------------------------| * SMB Direct and RDMA with Virtual Switch (Mode 2). † SMB Direct Mode 2 not supported. ^ Enterprise and "Pro for Workstations" editions. NOTE: Drivers are WHQL certified for Windows Server 2019, 2016, 2012 R2 and 10 Client. Other versions have not been tested and are not guaranteed to work. 2. Supported Cards ================================================================================ Following is the list of drivers/software and supported Chelsio adapters: |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T62100-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T62100-LP-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T62100-SO-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS, | | |Mesh(NIC),SR-IOV,DCB | |------------------------|-----------------------------------------------------| |T6425-CR** |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T6225-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T6225-LL-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T580-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T580-LP-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T580-SO-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS, | | |Mesh(NIC),SR-IOV,DCB | |------------------------|-----------------------------------------------------| |T580-OCP-SO |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS, | | |Mesh(NIC),SR-IOV,DCB | |------------------------|-----------------------------------------------------| |T540-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T540-LP-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T540-SO-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS, | | |Mesh(NIC),SR-IOV,DCB | |------------------------|-----------------------------------------------------| |T540-BT |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T520-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T520-LL-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------|-----------------------------------------------------| |T520-SO-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS, | | |Mesh(NIC),SR-IOV,DCB | |------------------------|-----------------------------------------------------| |T520-OCP-SO |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS, | | |Mesh(NIC),SR-IOV,DCB | |------------------------|-----------------------------------------------------| |T520-BT |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND,| | |SMB Direct*,S2D,SR,Mesh,SR-IOV,Guest RDMA,iSCSI,iSER,| | |NVMe,DCB | |------------------------------------------------------------------------------| * SMB Direct and RDMA with Virtual Switch (Mode 2) ** All 4-ports of T6425-CR adapter will be functional only if PCIe x8 -> 2x PCIe x4 slot bifurcation is supported by the system and enabled in BIOS. Otherwise, only 2-ports will be functional. NOTE: T4 family of adapters are not supported. Memory-free Adapters ==================== |########################|#####################################################| | Chelsio Adapter | Driver/Software | |########################|#####################################################| |T6225-SO-CR |NIC,NVGRE,VXLAN,Timestamps,PacketDirect,VMMQ/vRSS,ND^| | |SMB Direct*^,S2D^,Mesh^,SR-IOV,Guest RDMA^,iSCSI^, | | |iSER^,NVMe^,DCB | |------------------------------------------------------------------------------| * SMB Direct and RDMA with Virtual Switch (Mode 2) ^ Limited number of offload connections supported: 256 IPv4/128 IPv6 SMB Direct, ND, NVMe Initiator 128 IPv4/IPv6 iSCSI Initiator 32 IPv4/IPv6 iSER Initiator 3. How to Use ================================================================================ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chelsio Unified Wire ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== Please make sure that the following requirements are met before installation: - Latest Microsoft Visual C++ Redistributable Packages for Visual Studio (x86 and x64) are required if you are planning to use installer. - Windows update KB2685811 should be installed on Windows 7 Client and 2008 R2 Server. It will be installed if "Automatic Updates" is configured or can be installed manually from https://www.catalog.update.microsoft.com/Search.aspx?q=KB2685811 Installation ============ Chelsio Unified Wire can be installed using the Installer or the Zip package. Refer the appropriate section below depending on the method of installation. IMPORTANT: If you are planning to install Unified Wire in a cluster, it should be done on all the members. Using different versions of Unified Wire is not supported. Unified Wire Installer (Windows GUI Machine) -------------------------------------------- Enabling Test Signing ---------------------- In case the drivers in the package are not WHQL certified, follow the steps mentioned below to enable test signing on your system: a) Goto Start->Run command option, enter "cmd" and press OK. This will open the command prompt utility. b) Run the following command: C:\Users\Administrator> bcdedit /set testsigning on c) Reboot the machine for the changes to take effect. GUI Based Installation ---------------------- a) Run the "ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe" application. b) Click the Next button for the Chelsio End User License Agreement Window. c) Select the radio button "I accept the terms of the license agreement" and click on "Next" button. d) Select "Complete" for complete package installation or else select "Custom" radio button to customize the installation. NOTE: Complete option will not install iSCSI, iSER and NVMe Storport Miniport drivers. Use the Custom option to install them. VBD and NIC drivers are required for iSER and NVMe Storport Miniport drivers to work, whereas iSCSI Storport Miniport needs VBD to function. e) Now clicking on "Install" button will start installation with the selected options. f) Click on the "Finish" button to exit from the installer. CLI Based (Silent) Installation ------------------------------- To install drivers/software using this feature, open command prompt and execute the following command: C:\Users\Administrator>ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe -in E.g.: C:\Users\Administrator>ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe -in all The above command will install all drivers except iSCSI, iSER and NVMe Storport Miniport drivers. To install a particular driver (custom installation), execute the following command: C:\Users\Administrator>ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe -in NIC The above command will install the Network driver. To add a driver to an existing list of already installed drivers, use the following command: C:\Users\Administrator>ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe -add To know more about other parameters and options, execute the following command: C:\Users\Administrator>ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe -help The above command will create a help file, ChelsioUwire_x.x.x.x_WIN_yyy.y.yy_help.txt, in the present working directory which contains the complete list of command line syntax required for performing all the necessary CLI operations. NOTE: A log file, ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.log, is created in the same directory which keeps a record of all the commands executed and their results. Zip Package ----------- Enabling Test Signing --------------------- In case of non-WHQLed drivers, please follow the steps mentioned below to enable test signing before installing Unified Wire: a) Open command prompt and run the following command to enable test signing: C:\Users\Administrator>bcdedit /set testsigning on b) Change your working directory to "ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\Selfsign\" and run the following commands: C:\ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\Selfsign> .\certmgr.exe -add .\chelsiocert1.cer -s -r LocalMachine root -all C:\ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\Selfsign> .\certmgr.exe -add .\chelsiocert1.cer -s -r LocalMachine trustedpublisher -all c) Reboot the machine for the changes to take effect. Drivers location ---------------- The driver files should be manually picked from: - ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\drivers\Win19 for Server 2019. - ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\drivers\Win10 for Server 2016 and 10 Client. - ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\drivers\Win7 for Server 2012 R2. Manual Installation ------------------- In Manual method, the driver components will have to be individually installed using the Device Manager. Please refer User's Guide for instructions. The components must be installed strictly in the order specified in the document. Any other order is not supported. Offline Windows Image --------------------- You can use Windows Deployment Image Servicing and Management (DISM) tool to add Chelsio driver components (.inf files) to an offline Windows image. For more information, visit the following links: - http://technet.microsoft.com/en-us/library/hh825070.aspx - http://blogs.technet.com/b/heyscriptingguy/archive/2012/09/27/use-the-powershell-dism-cmdlets-to-manage-windows-8.aspx Firmware Update =============== The driver will auto-load firmware (v1.24.17.0) if an update is required. The firmware version can be verified using cxgbtool: C:\Users\Administrator>cxgbtool nic0 debug version In case of installer, the firmware binaries will be copied to \ChelsioUwire\Firmware\ folder during installation. For the zip package, the binaries will be present in ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\firmware\ folder. Driver Update ============= Chelsio Unified Wire can be updated using Windows update, installer or zip package. Please refer the user's guide for a detailed explanation regarding each method. IMPORTANT: If you are planning to upgrade Unified Wire in a cluster, it should be done on all the members. Using different versions of Unified Wire is not supported. Configuring 40G Adapters ======================== The "chelsio_adapter_config.ps1" configuration script will configure the Chelsio 40G adapters in any of the following three operational modes: - DEFAULT (2X40G): This is the default mode of operation where each port functions as 40Gbps link. The port nearest to the motherboard will appear as the first network interface (Port 0). - SPIDER (4X10G): In this mode, port 0 functions as 4 10Gbps links and port 1 is disabled. - QSA (2X10G): This mode adds support for QSA (QSFP to SFP+) modules, enabling smooth, cost-effective, connections between 40 Gigabit Ethernet adapters and 1 or 10 Gigabit Ethernet networks using existing SFP+ based cabling. The port farthest from the motherboard will appear as the first network interface (Port 0). Windows GUI Machine ------------------- a) In case of Installer, the configuration script and cxgbtool.exe will be copied to "\Windows\System32" during installation. If you are using the zip package, copy cxgbtool.exe from ChelsioUwire_x.x.x.x_WIN_yyy.y.yy folder to "\Windows\System32" and change your working directory to "ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\Adapter Configuration". b) Open PowerShell with administrative privileges. c) Run the adapter configuration script and enter the index of the 40G adapter for which the configuration needs to be updated. Hit [Enter]. Installer: PS C:\Users\Administrator>chelsio_adapter_config.ps1 Zip Package: PS D:\ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\Adapter Configuration> .\chelsio_adapter_config.ps1 d) Select "Port settings" (option 3) as the configuration type. Hit [Enter]. e) Select the operation mode and then enter "y" to confirm. Hit [Enter]. f) Open Device Manager, click on System Devices and right-click on the 40G Chelsio adapter selected in step (c). g) Select Disable and then Enable for changes to take effect. Deadman Watchdog Timer ====================== In some cases when a port is receiving traffic, if the host machine hits a BSOD or crashes, the port cleanup will not be complete, and the port will send indefinite pause frames. This might cause few switch ports to go into an error state. Using the Deadman Watchdog timer feature, the firmware will bring down the ports on the host machine after the specified time (user configurable). This feature is disabled by default. It can be configured using the Set-DeadmanWatchdogTimer.ps1 script. a) In case of Installer, the configuration script will be copied to "\Windows\System32" during installation. If you are using the zip package, change your working directory to "ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\DeadmanWatchdogTimer". b) Open PowerShell with administrative privileges. c) Run the configuration script specifying the time (in milliseconds). Installer: PS C:\Users\Administrator>Set-DeadmanWatchdogTimer.ps1 -WatchdogTimer 30000 Zip Package: PS D:\ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\DeadmanWatchdogTimer> .\Set-DeadmanWatchdogTimer.ps1 -WatchdogTimer 30000 The command will set the Deadman Watchdog Timer value for all the adapters in the machine. To set only to a specific adapter, specify the Device Path using the -DeviceId option. NOTE: For more help and detailed examples, execute the below command: PS C:\Users\Administrator>get-help Set-DeadmanWatchdogTimer.ps1 -full d) To disable the feature, set 0 as the timer value. Chelsio Management ================== Chelsio Management Feature allows to update the following properties of Chelsio adapters and to collect Debug logs. - FW Config file - Serial Init - Performance tuning Open the Device Manager, click on the System Devices, double click on the Chelsio Bus Enumerator, and then click the Chelsio Management tab. Please refer User's Guide for detailed steps to use this feature. Un-installation =============== Installer --------- GUI --- a) Run the "ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe" application. b) Select the radio button "Remove" and click "Next". c) Click on the "Finish" button to exit from the installer. NOTE: This method of uninstallation is possible only if the drivers were installed using Unified Wire Installer. CLI --- To uninstall all the drivers, execute the following command: C:\Users\Administrator>ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe -un all To uninstall a particular driver, execute the following command: C:\Users\Administrator>ChelsioUwire_x.x.x.x_WIN_yyy.y.yy.exe -rm NOTE: Uninstalling Unified Wire package using the above method will not uninstall Unified Wire Manager. You will have to manually uninstall it using "Programs and Features" in the Control Panel. Zip Package ----------- If the driver components were installed using the zip package, they will have to be uninstalled manually. Please refer to User's Guide for step-by-step instructions. IMPORTANT: The driver components must be uninstalled manually and strictly in the order specified in the document. Any other order is not supported. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NDIS Function ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chelsio’s Terminator series of Unified Wire adapters provide extensive support for NIC operation, including all stateless offload mechanisms for both IPv4 and IPv6. There are various advanced configuration options for Chelsio devices. To see or change the tunable NDIS Driver parameters, open the Device Manager, click on the Network adapters, double click on any Chelsio Network Adapter, and then click the Advanced tab. They can also be changed from CLI using the below commands, especially in case of Server Core installations where there is no GUI. PS C:\Users\Administrator> Get-NetAdapterAdvancedProperty -Name PS C:\Users\Administrator> Set-NetAdapterAdvancedProperty -Name -DisplayName -DisplayValue Please refer to User's Guide for description and possible values of all the NDIS Advanced Properties. NVGRE Offload ============= Chelsio’s Terminator based adapters are uniquely capable of offloading the processing of NVGRE encapsulated frames such that all stateless offloads(checksums, LSO, VMQ, RSS and filtering) are preserved, resulting in significant performance benefits. This feature is enabled by default. Configure the relevant customer and provider network settings on the host. VXLAN Task Offload ================== Chelsio’s Terminator based adapters are capable of offloading the processing of VXLAN encapsulated frames such that all stateless offloads (checksums, LSO, VMQ, RSS and filtering) are preserved, resulting in significant performance benefits. This feature is enabled by default. Timestamps ========== Timestamping feature is used to maintain a precise and accurate time for the workloads. NOTE: This feature is currently supported on Windows Server 2019 only. Hardware Timestamps is disabled by default. Use NDIS Miniport Driver Parameters (Device manager -> Chelsio Network Adapter Properties -> Advanced Tab -> PTP Hardware Timestamp) to enable it. Chelsio NDIS function driver supports generating software timestamps for the packets. This feature is disabled by default. Use NDIS Miniport Driver Parameters (Device manager -> Chelsio Network Adapter Properties -> Advanced Tab -> Software Timestamp) to enable it. Currenlty, only RxAll and TaggedTx are supported. PacketDirect ============ PacketDirect Provider Interface (PDPI), an extension to the existing NDIS miniport driver model, gives applications complete control over buffers, poll processors, and sending and receiving packets over a miniport adapter. Chelsio’s Unified Wire adapters offer extensive support for this feature leading to a higher throughput performance. This feature is disabled by default. Please refer User's Guide to configure the PacketDirect feature. NOTE: This feature is available for experimental usage only. Virtual Environments ==================== Network throughput of a multi-core virtual machine is limited by the processing power of a single virtual processor, since only one processor handles all the interrupts from a virtual network adapter. Virtual Machine Multi-Queue (VMMQ)/ Virtual Switch Receive-side Scaling (vRSS) and Dynamic VMMQ/RSSv2 solve this bottleneck by distributing the network traffic across multiple virtual processors. Dynamic VMMQ/RSSv2, makes RSS algorithm dynamic by taking CPU measurement and making balancing decision at DISPATCH level. The feature also avoids taking global locks by changing only the subset of indirection table which affects current CPUs only. If NDIS function driver is installed using Unified Wire installer, these features will be enabled by default. However, if the driver is installed manually (using zip package), they will be disabled. Use NDIS Miniport Driver Parameters (Device manager -> Chelsio Network Adapter Properties -> Advanced Tab -> Virtual Switch RSS) to enable them. IMPORTANT: This is currently supported for 28 VM Network Adapters per Chelsio adapter. VMMQ/vRSS --------- Additionally, VMMQ/vRSS must be enabled for each Virtual Machine using the below command on the host: PS C:\Users\Administrator> Set-VMNetworkAdapter -VMName -VrssEnabled:$true -VmmqEnabled:$true -VmmqQueuePairs Example: PS C:\Users\Administrator> Set-VMNetworkAdapter -VMName VM -VrssEnabled:$true -VmmqEnabled:$true -VmmqQueuePairs 8 Dynamic VMMQ/RSSv2 ------------------ NOTE: Available on Windows Server 2016 version 1709 and later. Additionally, Dynamic VMMQ/RSSv2 must be enabled for each VM on the host using: PS C:\Users\Administrator> Set-VMNetworkAdapter -VMName -VrssEnabled 1 -VmmqEnabled 1 -VmmqQueuePairs -VrssQueueSchedulingMode Dynamic ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iWARP (ND) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chelsio Unified Wire Installer registers Chelsio iWARP provider automatically during installation. In case of zip package, run the following command to register: C:\Users\Administrator>chinstallsp.exe -i The iWARP provider should show up using: C:\Users\Administrator>chinstallsp.exe -l You should see a similar output: 0000001019 - Chelsio Network Direct provider chrping application =================== chrping application can be used to test RDMA connectivity between the hosts. Configure the Chelsio interfaces on the hosts with the appropriate IP Address, netmask etc. Use the basic ping command to verify connectivity via the Terminator interface. In case of Installer, chrping.exe will be copied to \Windows\System32 folder during installation. If you are using the zip package, copy chrping.exe from ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\chrping folder to \Windows\System32 The following steps verify RDMA connectivity between a Windows and a Linux host. IMPORTANT: Disable iWARP Port Mapper (iwpmd) service on Linux hosts. On RHEL7.X machines, use the below command. [root@host~]# systemctl stop iwpmd a. Start rping server on Linux host. [root@host~]# rping -s -a -p -S b. Start chrping client on Windows host. C:\Users\Administrator>chrping.exe c t S HMA === The Host Memory Access(HMA) implements a bridge between the Memory Arbiter (MA) and the PCIE so that the Host Memory is available to all clients. The translation from the MA memory map to the Host Memory map is performed based on the client number and the address of the transaction, both of which are used as index to look up a 64-bit offset within the Host Memory. Currently 256 IPv4/128 IPv6 Network Direct offload connections are supported on T6225-SO-CR adapter. You can view the number of connections offloaded by using C:\Users\Administrator> cxgbtool hardware tid_info ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ SMB Direct ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Enabling SMB Direct =================== Windows Server 2019/2016/2012 R2 -------------------------------- SMB Direct is installed and enabled by default on Windows Server 2019/2016/2012 R2. RDMA functionality on Chelsio adapters will be enabled after installing the driver from the package. Windows 10 Client ----------------- On Windows 10 Client systems, SMB Direct feature is installed but disabled by default. To enable it, follow the steps mentioned below: a) Open Control Panel and click Programs. b) Under Programs and Features, click Turn Windows features on or off. c) In the Windows Features window, scroll down and select the SMB Direct checkbox. d) Click OK. RDMA functionality on Chelsio adapters will be enabled after installing the driver from the package. Configuration -------------- You can use the PowerShell command "Get-NetAdapterRdma" to get a list of RDMA capable interfaces in your system. At least one adapter should indicate "RDMA Capable" true to take advantage of SMB RDMA capability. The next step is to configure your network so that SMB server and client machines can communicate with each other. Please configure the Microsoft firewall to allow traffic between the SMB clients and server. IMPORTANT: When using with Linux machines, disable iWARP Port Mapper (iwpmd) service on them. On RHEL7.X machines, use the below command. [root@host~]# systemctl stop iwpmd RDMA with Virtual Switch (Mode 2) ================================= In this mode, you can run RDMA traffic using a virtual switch. This feature is enabled by default in the driver, but must be enabled on the Hyper-V Virtual Ethernet adapter. NOTE: For more information on the features mentioned above, please refer to User's Guide. Storage Replica =============== Currently, Unified Wire supports Server to Server replication mode. In this mode, data is synced between two servers and each server keeps an identical copy of the same volume. For more information on how to setup and configure in this mode, please refer to Microsoft’s official documentation at https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-replica/server-to-server-storage-replication Storage Spaces Direct (S2D) =========================== Chelsio’s high performance iWARP RDMA adapters prove to be a best fit for both networking and virtualization requirements, as well as hyper-converged scalable storage solutions like Storage Spaces Direct (S2D), a core storage feature for Azure Stack solution. Windows SMB Direct over iWARP RDMA provides higher performance by giving direct access to the data residing on hyper-converged or disaggregated storage, while the CPU reduction enables a larger number of VMs per Hyper-V server, enabling savings in power dissipation, system configuration and deployment scale throughout the life of the installation. Chelsio iWARP RDMA adapters do not require any configuration of DCB, PFC, ETC, ECN etc. on the switch and support the ability to work with any standard Ethernet switch. It is recommended to disable them and enable regular flow control on switch ports. This way, they enable incremental, non-disruptive server installs, delivering a brownfield strategy to enable high performance, low cost, scalable Azure Stack Hyper Converged Infrastructure (HCI) deployments. Please refer to User's Guide for steps on configuring S2D. HMA === The Host Memory Access (HMA) implements a bridge between the Memory Arbiter (MA) and the PCIE so that the Host Memory is available to all clients. The translation from the MA memory map to the Host Memory map is performed based on the client number and the address of the transaction, both of which are used as index to look up a 64-bit offset within the Host Memory. Currently 256 IPv4/128 IPv6 SMB Direct offload connections are supported on T6225-SO-CR adapter. You can view the number of connections offloaded by using C:\Users\Administrator> cxgbtool hardware tid_info Once the offload connections limit is reached, connections will continue to establish in regular NIC mode. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Mesh Topology ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Mesh topology involves connecting each node to every other node. Supported configs using this approach: N ports per node, N+1 node cluster. Installation ============ Install Unified Wire on all the machines in the Mesh Topology (Refer Chelsio Unified Wire Chapter). Configuration ============= Configure the required IP addresses on all the interfaces. You should be able to run NIC and SMB Direct traffic between the nodes of mesh topology. NOTE: For detailed example, please refer User's Guide. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NDIS SR-IOV ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Pre-requisites ============== SR-IOV should be enabled in the machine. Installation ============ Please ensure that Unified Wire is installed on the host before proceeding. Refer Unified Wire's Installation section for step-by-step instructions. Enabling SR-IOV =============== The adapter's configuration should be updated to make use of the SR-IOV feature. Change the Serial Init to SR-IOV using Chelsio Management feature. IMPORTANT: To use the adapter in non-SRIOV scenarios in Windows or with other operating systems, the configuration type must be changed to "NON-SRIOV (Default)". Host Configuration ================== a) Create a vSwitch with SR-IOV enabled using Chelsio adapter. b) Assign a virtual network adapter with SR-IOV enabled to the VM. c) Bring up the VM. NOTE: For more detailed information, refer User's Guide. Guest (VM) Configuration ======================== a) Use VM Connect as described in https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/learn-more/use-local-resources-on-hyper-v-virtual-machine-with-vmconnect and access the Unified Wire package on the Guest (VM). Proceed with the installation as described in the "Software/Driver Installation" section of the "Chelsio Unified Wire" chapter. b) Verify if SR-IOV was enabled successfully on the host: PS C:\Users\Administrator> Get-VMNetworkAdapter | fl The VFDataPathActive parameter must display "True". This, however, is possible only if the physical port has an active link. To uninstall Unified Wire, please refer Unified Wire's "Un-installation" section. Guest RDMA (Mode 3) =================== In this mode, you can run RDMA traffic on Guest (VM) using a virtual switch. This feature is enabled by default in the driver, but must be enabled on the Microsoft Hyper-V Virtual Ethernet adapter. Please refer User's Guide for step-by-step instructions on how to enable this feature. NOTE: Both Host and Guest should be installed with Windows Server 2016 version 1709 and later. IMPORTANT: When using with Linux machines, disable iWARP Port Mapper (iwpmd) service on them. On RHEL7.X machines, use the below command. [root@host~]# systemctl stop iwpmd ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSCSI Storport Miniport ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Configuring iSCSI Initiator =========================== Configuring IP -------------- There are two ways to assign IP to Chelsio iSCSI interface: DHCP: NIC driver must be installed (see Software/Driver Installation). Upon installing the driver, DHCP will be enabled by default and the NIC interface will be assigned an IP automatically. The same IP can be used by the iSCSI interface as well. Static: To assign a static IP to the iSCSI interface use the Device Manager, as explained below: a. Open Device Manager, right click on Chelsio iSCSI Initiator and click on Properties. b. Click on Advanced tab and select IP option and add IP. Similarly, add subnet mask and default gateway. Configuring Optional Parameters ------------------------------- Configure the below optional parameters based on your requirement. - The Max PDU Length parameter can be used to improve iSCSI performance. The default value of the parameter (indicated by 0) is set based on the MTU: 1024 for MTU 1500, 8192 for MTU 9000. However, these default values can be overridden by entering the value in increments of 512. - Set the maximum iSCSI queue sets. This parameter can be used to improve performance. - Select MTU and assign value as per requirement. - VLAN is disabled by default. To enable it, click on Vlan Insertion and select value as Enabled. - Assign the required VLAN ID and click OK. Repeat the above mentioned steps to assign these driver properties on other ports. iSCSI Target Discovery and Login ================================ Before target discovery, make sure an IP address, either static or using DHCP, is assigned to the respective Chelsio iSCSI node (see previous section). NOTE: - Connecting to more than 64 targets on single port using Chelsio iSCSI Initiator is currently not supported. - In case of Microsoft target, a maximum of 6 targets can be discovered by default. To increase this, change the maximum iSCSI PDU length to 8192 in Device Manager as explained in the "Configuring iSCSI Initiator" section. Following are the set of instructions to discover the target and login to it using the Chelsio iSCSI interface: a. To start the Initiator configuration, go to Control Panel and click on the iSCSI Initiator option. b. Choose the Discovery tab and click on Discover Portal. c. Click on the Advanced button. d. In the Advanced Settings window, select Chelsio iSCSI interface as the local adapter. e. Select the corresponding Chelsio iSCSI Interface IP as the Initiator IP and click OK. f. Enter the IP address (DNS name not supported) of the target machine and the corresponding port number and click OK on the Discover Target Portal Window. Note that the default port number for iSCSI traffic is 3260. g. Once target portal is added, details like target and initiator machine IP, Port number and Chelsio iSCSI interface IP will be displayed. h. Click on the Targets tab to see the list of targets available, choose a particular target and click on Connect. i. A window pops up showing the Target Name, Click OK. j. After logging in to the target, the state of the target will change from Inactive to Connected. k. After successful login, go to Disk Management and make the respective volume online. HMA === The Host Memory Access(HMA) implements a bridge between the Memory Arbiter (MA) and the PCIE so that the Host Memory is available to all clients. The translation from the MA memory map to the Host Memory map is performed based on the client number and the address of the transaction, both of which are used as index to look up a 64-bit offset within the Host Memory. Currently 128 IPv4/IPv6 iSCSI initiator offload connections are supported on T6225-SO-CR adapter. You can view the number of connections offloaded by using C:\Users\Administrator> cxgbtool hardware tid_info ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ iSER Storport Miniport ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Configuring iSER Target ----------------------- The iSER target must be configured before establishing connection. For more information, please refer Unified Wire for Linux documentation available at the Chelsio Download Center, service.chelsio.com IMPORTANT: Disable iWARP Port Mapper (iwpmd) service on the target machine. On RHEL7.X machines, use the below command. [root@host~]# systemctl stop iwpmd Configuring iSER Initiator -------------------------- The iSER interface uses the same IP as the NIC interface. If not already done, assign an IP to Chelsio interface as described in the "Assigning IP Address" section in the User Guide, before proceeding. If a vSwitch is created on the physical port, disable and enable the physical port using the below command for the iSER driver to pick up the IP address. C:\Users\Administrator> restart-netadapter iSER Target Discovery and Login -------------------------------- Please refer the User Guide for step-by-step instructions on how to discover and login to iSER target using the Chelsio iSER interface. HMA --- The Host Memory Access(HMA) implements a bridge between the Memory Arbiter (MA) and the PCIE so that the Host Memory is available to all clients. The translation from the MA memory map to the Host Memory map is performed based on the client number and the address of the transaction, both of which are used as index to look up a 64-bit offset within the Host Memory. Currently 32 IPv4/IPv6 iSER Initiator offload connections are supported on T6225-SO-CR adapter. You can view the number of connections offloaded by using C:\Users\Administrator> cxgbtool hardware tid_info ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ NVMe Storport Miniport ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Connecting to NVMe target ------------------------- Configure the NVMe target machine with the IP Address, Target name, disks etc. IMPORTANT: Disable iWARP Port Mapper (iwpmd) service on the target machine. On RHEL7.X machines, use the below command. [root@host~]# systemctl stop iwpmd Follow the below procedure on NVMe Initiator machine to connect to the target. i. Configure the Initiator with DHCP or static IP Address. If it is connected to a network with a DHCP server, Chelsio network interfaces will acquire DHCP IPs automatically. To assign static IPs, follow the steps mentioned below: a. Double click on the Network Connections icon and choose the Chelsio card entry and double click it. b. Click on the Properties button from the Local Area Connection X Status. c. Select “Internet Protocol (TCP/IP)” from the list and click on Properties button below it. d. From the Internet Protocol (TCP/IP) Properties window, assign an IP Address (e.g. 192.169.1.10) and subnet mask (e.g. 255.255.255.0). e. Click on Ok and close on the other window. Check to see if you can ping to the target machine. ii. "nvmetool" utility should be used to connect to the target. In case of Installer, it will be copied to \Windows\System32 folder during installation. If you are using the zip package, copy nvmetool.exe from ChelsioUwire_x.x.x.x_WIN_yyy.y.yy folder to \Windows\System32 folder. iii. Scan the system for available NVMe initiator devices. PS C:\Users\Administrator> nvmetool.exe findnvmehosts iv. Discover the target by specifying the PortId and Target IP Address. PS C:\Users\Administrator> nvmetool.exe DiscoverTargets v. Connect to the target by specifying the PortId, Target Name and Target IP Address. PS C:\Users\Administrator> nvmetool.exe ConnectTarget vi. List the successful connections. PS C:\Users\Administrator> nvmetool.exe ListTargets vii. For targets to be persistent across reboots, execute the below command: PS C:\Users\Administrator> schtasks /create /sc ONLOGON /tn "addnvmerestore" /tr "nvmetool restorepreviousconfig" Disconnecting from NVMe target ------------------------------ To disconnect from the NVMe target, use the below command. PS C:\Users\Administrator> nvmetool.exe RemoveTarget HMA --- The Host Memory Access(HMA) implements a bridge between the Memory Arbiter (MA) and the PCIE so that the Host Memory is available to all clients. The translation from the MA memory map to the Host Memory map is performed based on the client number and the address of the transaction, both of which are used as index to look up a 64-bit offset within the Host Memory. Currently 256 IPv4/128 IPv6 NVMe Initiator offload connections are supported on T6225-SO-CR adapter. You can view the number of connections offloaded by using C:\Users\Administrator> cxgbtool hardware tid_info Configure the number of IoQueues while connecting to more number of targets. By default, number of IoQueues will be equal to the number of CPU cores. PS C:\Users\Administrator> nvmetool.exe ConnectTarget [] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Data Center Bridging (DCB) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chelsio network adapters work seamlessly with any legacy switches and do not require the complexity of configuring Data Center Bridging (DCB) protocols either on adapter or network switch. In case DCB is already configured on the switch, the adapter will automatically negotiate the PFC settings configured on the switch. NOTE: Only IEEE configuration supported. Network QoS (Optional) ========================= Network Quality of Service (QoS) is an advanced Windows feature that can be used to distribute bandwidth between different kinds of outgoing traffic. This feature ensures efficient usage of resources and minimizes the impact of bandwidth congestion. The bandwidth percentage can be configured either on switch or host. Enabling QoS on Host -------------------- Network QoS is disabled by default. Use NDIS Miniport Driver Parameters (Device manager -> Chelsio Network Adapter Properties -> Advanced Tab -> NDIS QoS) to enable it. NOTE: Chelsio QoS requires a minimum of 8 cores to work. In case of multiple sockets, where the number of cores per socket is less than 8, Chelsio adapters must be configured to utilize all the available cores before enabling QoS. Run the following command on all Chelsio ports: PS C:\Users\Administrator> Set-NetAdapterRss -Name -MaxProcessorGroup 3 Configuring Bandwidth Allocation -------------------------------- Switch ------- Here is an example of setting bandwidth allocation on a Dell Force10 switch: a) Log in to the switch and enter the configuration mode: Login: admin Password: Force10#enable Force10#configure b) Enable DCB and LLDP: Force10(conf)#dcb enable Force10(conf)#protocol lldp Force10(conf-lldp)#no disable c) Set DCBx version to auto: Force10(conf-lldp)#dcbx version auto d) Specify the intervals between hello packets: Force10(conf-lldp)#hello 5 e) Create a DCB map. For example, here we are configuring 2 groups/classes: - Group 0 has BW of 80% with pfc enabled. - Group 1 had BW of 20% with pfc disabled. - Assigning priority groups: Priority 5-7 are under group 0 and 0-4 are under group 1. Force10(conf)#dcb-map win_dcb Force10(conf-dcbmap-win_dcb)# priority-group 0 bandwidth 80 pfc on Force10(conf-dcbmap-win_dcb)# priority-group 1 bandwidth 20 pfc off Force10(conf-dcbmap-win_dcb)# priority-pgid 1 1 1 1 1 0 0 0 f) Enter interface configuration mode and configure protocol LLDP: Force10(conf)#interface tengigabitethernet 0/17 Force10(conf-if-te-0/17)#protocol lldp Force10(conf-if-te-0/17-lldp)#no disable g) Set DCBx version to auto: Force10(conf-if-te-0/17-lldp)#dcbx version auto h) Apply the DCB-MAP created: Force10(conf-if-te-0/17)#dcb-map win_dcb The host adapter will honor the bandwidth allocation settings configured on the switch. Host ---- Here is an example of setting bandwidth allocation on the host: a) Open PowerShell with administrative privileges and run the following command: PS C:\Users\Administrator> Install-WindowsFeature -Name "Data-Center-Bridging" -IncludeAllSubFeature -IncludeManagementTools b) Create new policy: PS C:\Users\Administrator> New-NetQosPolicy -Name rdma -PriorityValue 5 -NetDirectPortMatchCondition 445 c) Create new rule with bandwidth percentage: PS C:\Users\Administrator> New-NetQosTrafficClass -Name rdma -Priority 5 -Algorithm ETS -BandwidthPercentage 80 -Verbose The above command allocates 80% of the bandwidth to the policy named 'rdma'. To change the percentage of an existing ETS rule: PS C:\Users\Administrator> Set-NetQosTrafficClass -Name rdma -Priority 5 -Algorithm ETS -BandwidthPercentage 10 -Verbose 4. Support Documentation ================================================================================ The documentation for this release can be found in the following directory in the system: Windows Unified Wire Installer (after installation): \ChelsioUwire\docs\ Zip Package: ChelsioUwire_x.x.x.x_WIN_yyy.y.yy\docs\ It contains: - README - Release Notes - User's Guide 5. Customer Support ================================================================================ Installer issues ---------------- In case of any failures while running the Chelsio Unified Wire Installer, please collect the below: a) Screenshot of the failure. b) Installer logs from \Windows\Temp\ChelsioUwire_x.x.x.x_WIN_yyy.y.y_logs directory. Compress the directory to a zip package. Logs collection --------------- In case of any other issues, follow the below procedure to collect the standard logs: a) Download Get-NetView.psm1 file from https://github.com/microsoft/Get-NetView and run the below commands: PS C:\Users\Administrator\Desktop> Set-ExecutionPolicy Unrestricted PS C:\Users\Administrator\Desktop> Import-Module Get-NetView.psm1 b) Run the below command and the logs will be collected in a zip package. PS C:\Users\Administrator\Desktop> get-netview c) Driver Event tracing logs can be collected using the below procedure: - Start the tracing. PS C:\Users\Administrator\Desktop> cxgbtool debug trace nic 0xffff 5 vbd 0xffff 5 iscsi 0xffff 5 iser 0xffff 5 nvme 0xffff 5 - Rerun the test to hit the observed issue. - Stop tracing. PS C:\Users\Administrator\Desktop> cxgbtool debug trace stop cxgbtrace.etl file will be collected in the same path. Please contact Chelsio support at support@chelsio.com with relevant logs for any issues. ******************************************************************************** Copyright (C) 2020 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means without the express written consent of Chelsio Communications.