Quick and Dirty steps to NFS-RDMA over T3
Author: Steve Wise
Date: 12/10/2008
1) configure/build/install 2.6.28-rc7
- enable the Chelsio T3 10GbE driver (Device Drivers->Network device support->Ethernet (10000 Mbit)->Chelsio Communications T3 10Gb Ethernet support)
- enable RDMA support (Device Drivers->Infiniband support)
- enable the Chelsio RDMA driver (Device Drivers->Infiniband support->Chelsio RDMA Driver
- enable NFS Client (File systems->Network File Systems->NFS client support
- enable NFS Server (File systems->Network File Systems->NFS server support
- see Documentation/filesystems/nfs-rdma.txt for instructions
- one you build and install 2.6.28-rc7, reboot and verify that these modules exist (assuming you configured them as modules): iw_cxgb3, cxgb3, xprtrdma, and svcrdma
2) configure/build/install nfs-utils.
- wget http://www.kernel.org/pub/linux/utils/nfs/nfs-utils-1.1.4.tar.bz2
- on sles10sp2, I ended up installing tcpd-devel, e2fsprogs-devel and openldap2-devel as prereqs
- sles10sp2 configure opts: --disable-gss --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib64
- once installed, verify that '/sbin/mount.nfs -V' shows nfs-utils 1.1.4
3) Once up on 2.6.28-rc7, configure the chesio rnic:
- get chelsio 7.0.0 (t3fw-7.0.0.bin) firmware from http://service.chelsio.com
- cp t3fw-7.0.0.bin into /lib/firmware
- modprobe iw_cxgb3
- ifconfig ethX ipaddr up
- verify you can ping over the T3 interface.
4) On the NFS server, setup an /etc/exports file. EG:
/usr/local/src 192.168.70.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)
5) Load NFS RDMA modules and restart nfs
- modprobe xprtrdma
- modprobe svcrdma
- /etc/init.d/nfs restart
- /etc/init.d/nfsserver restart (sles only)
6) On the server, enable listening for NFSRDMA mounts
- echo rdma 2050 > /proc/fs/nfsd/portlist
- verify via 'cat /proc/fs/nfsd/portlist'
7) On the client, mount via NFSRDMA:
- /sbin/mount.nfs
- verify via 'mount' - should show 'rdma,port=2050' for the /mnt mount
On the server, can you see NFSRDMA protocol stats via /proc/sys/sunrpc/svc_rdma/*
On both systems, you can view T3 offload protocol stats via /sys/class/cxgb3_0/proto_stats/*
This is also explained in the kernel Documentation tree under: linux/Documentation/filesystems/nfs-rdma.txt