****************************************
                                   README

                    ****************************************

                       Chelsio T5 NVMe-oF Driver for Linux 
                               

                             Version : 1.0.0.2 (Alpha)
                             Date    : 08/26/2016



Overview
================================================================================

Chelsio T5 NVMe-oF driver package for Linux is for Chelsio's 10/40Gb Unified Wire 
adapters. The package provides an installer to install the following components:

- Kernel 4.8 (RC3) with NVMe Over Fabrics support 
- iWARP libraries
- nvme-cli utility (for initiator)
- nvmetcli utility (for target)

   

================================================================================
CONTENTS
================================================================================

- 1. Requirements
- 2. Supported Operating Systems
- 3. Supported Cards
- 4. How To Use
- 5. Support Documentation
- 6. Customer Support



1. Requirements
================================================================================

- Chelsio T5 10/40G adapter and system with supported platforms mentioned in 
  section 2.
- openssl-devel,bzip2-devel,zlib-devel,ncurses-devel,sqlite-devel packages for
  setuptools (nvmetcli utility) and kernel.
- Python v2.7 or above.


2. Supported Operating Systems
================================================================================

- RHEL7.2, 3.10.0-327.el7 
- RHEL7.1, 3.10.0-229.el7 
- RHEL6.7, 2.6.32-573.el6 
- RHEL6.6, 2.6.32-504.el6

Other versions have not been tested and are not guaranteed to work.



3. Supported Hardware
================================================================================

- T580-CR
- T580-LP-CR
- T540-CR
- T520-CR
- T520-LL-CR
- T520-BT



4. How to Use
================================================================================

4.1. Pre-requisites
===================

- If Python v2.7 is not already present in the system, or if an older version 
  exists, v2.7.10 provided in the package will be installed along with the kernel.

- openssl-devel,bzip2-devel,zlib-devel,ncurses-devel,sqlite-devel packages 
  should be installed in the machine.


4.2. Installation
=================

Follow the steps mentioned below to install NVMeF supported kernel and related 
components on both target and initiator machines:

a) Download the driver package, Chelsio-NVMeoF-x.x.x.x.tar.gz

b) Untar the tarball and change your working directory to Chelsio-NVMeoF-x.x.x.x

   [root@host~]# tar zxvfm Chelsio-NVMeoF-x.x.x.x.tar.gz
   [root@host~]# cd Chelsio-NVMeoF-x.x.x.x
   
c) On target, run the following command:

   [root@host~]# ./setup.sh -t
   
   On initiator, run the following command:

   [root@host~]# ./setup.sh -i

   The installer will compile and install the NVMe-oF kernel. During the compilation, 
   if any prompts appear, please type y or m and continue.  
	
d) Reboot system to installed 4.8 (RC3) kernel.


4.3. Driver Loading 
===================

Follow the steps mentioned below on both target and initiator machines:
 
a) Load the following modules:

   [root@host~]# modprobe iw_cxgb4
   [root@host~]# modprobe rdma_ucm    

b) Bring up the Chelsio interface(s):

   [root@host~]# ifconfig ethX x.x.x.x up

c) Mount configfs by running the below command:

   [root@host~]# mount -t configfs none /sys/kernel/config
  
d) On target, run the following commands:

   [root@host~]# modprobe null_blk
   [root@host~]# modprobe nvmet
   [root@host~]# modprobe nvmet-rdma 

   On initiator, run the following commands:
   
   [root@host~]# modprobe nvme
   [root@host~]# modprobe nvme-rdma



4.4. Firmware Update
====================

The T5 firmware (v1.15.37.0) is installed on the system, typically in 
/lib/firmware/cxgb4, and the driver will auto-load the firmware if an 
update is required. The kernel must be configured to enable userspace 
firmware loading support:

Device Drivers -> Generic Driver Options -> Userspace firmware loading support

The firmware version can be verified using ethtool:

   [root@host~]# ethtool -i <iface>


4.5. Configuration
===================

4.5.1. Target
-------------

a) The following commands will configure target using nvmetcli with a LUN:

   [root@host~]# nvmetcli
   /> cd subsystems
   /subsystems> create nvme-ram0
   /subsystems> cd nvme-ram0/namespaces
   /subsystems/n...m0/namespaces> create nsid=1
   /subsystems/n...m0/namespaces> cd 1
   /subsystems/n.../namespaces/1> set device path=/dev/ram1
   /subsystems/n.../namespaces/1> cd ../..
   /subsystems/nvme-ram0> set attr allow_any_host=1
   /subsystems/nvme-ram0> cd namespaces/1
   /subsystems/n.../namespaces/1> enable
   /subsystems/n.../namespaces/1> cd ../../../..
   /> cd ports
   /ports> create 1
   /ports> cd 1/
   /ports/1> set addr adrfam=ipv4.
   /ports/1> set addr trtype=rdma
   /ports/1> set addr trsvcid=4420
   /ports/1> set addr traddr=102.1.1.102
   /ports/1> cd subsystems
   /ports/1/subsystems> create nvme-ram0
     
b) Save the target configuration to a file: 

   /ports/1/subsystems> saveconfig /root/nvme-target_setup
   /ports/1/subsystems> exit

c) Clear the targets: 

   [root@host~]# nvmetcli clear


4.5.2. Initiator
----------------

a) Discover the target
   
   [root@host~]# nvme discover -t rdma -a <target_ip> -s 4420 

b) Connect to target

Connecting to a specific target:
   
   [root@host~]# nvme connect -t rdma -a <target_ip> -s 4420 -n <target_name>

Connecting to all targets configured on a portal  

   [root@host~]# nvme connect-all -t rdma -a <target_ip> -s 4420 

c) List the connected targets

   [root@host~]# nvme list

d) Format and mount the NVMe disks shown with the above command.

e) Disconnect from the target and unmount the disk:
   
   [root@host~]# nvme disconnect -d <nvme_disk_name>

   NOTE: nvme_disk_name is the name of the device (Ex:nvme0n1) and not the device 
         path. 


4.6. Driver Unloading 
=====================

Follow the steps mentioned below to unload the drivers:

On target, run the following commands:

   [root@host~]# rmmod nvmet-rdma 
   [root@host~]# rmmod nvmet
   [root@host~]# rmmod iw_cxgb4 

On initiator, run the following commands:
   
   [root@host~]# rmmod nvme-rdma
   [root@host~]# rmmod nvme
   [root@host~]# rmmod iw_cxgb4



5. Support Documentation
================================================================================

The documentation for this release can be found inside the 
Chelsio-NVMeoF-x.x.x.x/docs folder. It contains:

- README
- Release Notes



6. Customer Support
================================================================================

Please contact Chelsio support at support@chelsio.com for any issues regarding
the product.








********************************************************************************
Copyright (C) 2016 Chelsio Communications. All Rights Reserved.

The information in this document is furnished for informational use only, is
subject to change without notice, and should not be construed as a commitment by
Chelsio Communications. Chelsio Communications assumes no responsibility or
liability for any errors or inaccuracies that may appear in this document or any
software that may be provided in association with this document. Except as
permitted by such license, no part of this document may be reproduced, stored in
a retrieval system,or transmitted in any form or by any means without the
express written consent of Chelsio Communications.