Chelsio T3 Solaris Driver README (for version 1.1 of the driver)
================================================================
(Best viewed in a fixed width font)
Section 1: What's New
=====================
1) Support for new Chelsio Cards
2) Support for new Solaris and OpenSolaris releases
3) Support for project Crossbow
4) Many bug fixes and enhancements
Section 2: Supported Operating Systems and Cards
================================================
The driver is GLDv3 based but for backward compatibility it can work in GLDv2
mode on older Solaris releases. The driver automatically determines the best
mode of operation and uses it.
OpenSolaris
-----------
OpenSolaris 2009.06 (GLDv3 + Project Crossbow)
OpenSolaris 2008.11 (GLDv3)
The driver was not tested for the following Solaris versions but is expected to
work.
Solaris 10
----------
Solaris 10 update 1 (GLDv2)
Solaris 10 update 2 (GLDv2)
Solaris 10 update 3 (GLDv2)
Solaris 10 update 4 (GLDv3)
Solaris 10 update 5 (GLDv3)
Solaris 10 update 6 (GLDv3)
Solaris 10 update 7 (GLDv3)
Solaris 10 update 8 (GLDv3)
NOTE: Using the driver on onnv "Nevada" builds is NOT supported. Cursory
testing indicates that the driver will install and may function on these builds
but this is NEITHER RECOMMENDED NOR SUPPORTED.
Chelsio Cards
-------------
This release of the Chelsio Solaris Ethernet driver supports the following
Chelsio network adapters:
N320E (SFP+)
S320E (SFP+)
N320E-CXA
S320E-CXA
S320E-LP-CR
N310E (SFP+)
S310E (SFP+)
S310E-CR
N310E-CXA
S310E-CXA
S302E
N320E-G2
Section 3: Driver Installation
==============================
The driver includes two kernel modules - cxgen and cxge.
cxgen - is the bus nexus driver that attaches to the Chelsio T3 controller.
cxge - is the Ethernet driver that attaches to the cxgen bus nexus.
The Chelsio T3 network interfaces appear as cxgeX by default, where X is a
number assigned by the kernel.
Use the following steps to install the driver.
1) Remove the old driver (1.0) packages if they are installed on the system.
On x86 systems:
# pkgrm CHLSt3ei
# pkgrm CHLSt3eix
On SPARC systems:
# pkgrm CHLSt3es
# pkgrm CHLSt3esx
2) Extract the contents of the driver tarball. There are 4 packages in the
tarball:
CHLSt3ei - The 32 bit driver set for x86.
CHLSt3eix - The 64 bit driver set for x86.
CHLSt3es - The 32 bit driver set for SPARC.
CHLSt3esx - The 64 bit driver set for SPARC.
3) Install the driver packages.
On x86 systems:
# pkgadd -d . CHLSt3ei CHLSt3eix
On SPARC systems:
# pkgadd -d . CHLSt3es CHLSt3esx
4) Reboot the system.
5) Verify that the ports on all Chelsio Ethernet cards in the system are
visible. See the "Troubleshooting" section towards the end of this README if
they are not.
# prtconf | grep cxge
# dladm show-link
Section 4: Driver Configuration and Tuning
==========================================
NOTE: Jumbo frames and other driver private properties can be set via ndd(1M) or
dladm(1M), depending on the version of Solaris or OpenSolaris installed. This
README contains instructions for dladm(1M) ONLY. See the README included with
the previous version of the driver (1.0) for ndd(1M) usage. Use these commands
to determine which method is supported on a particular system:
# ndd -get /dev/cxge0 \?
# dladm show-linkprop -p mtu cxge0
Jumbo Frames
------------
Set the MTU of a cxge interface as shown in these examples:
# dladm set-linkprop -p mtu=9000 cxge0
# dladm set-linkprop -p mtu=1500 cxge0
A "try again later" error means the interface is plumbed and should be unplumbed
before changing its MTU.
Use this to display the current MTU of all interfaces on the system:
# dladm show-link
Driver Private Properties
-------------------------
The driver tunables (private properties) are listed here for the sake of
completeness. The driver ships with reasonable defaults and the use of these
properties is generally discouraged.
_hw_csum hardware checksum capability enabled or not.
_hw_lso hardware LSO capability enabled or not.
_intr_coalesce interrupt coalescing timer.
_desc_budget processing budget in each iteration of rx.
_frame_budget processing budget in each iteration of rx.
You can inspect and modify these properties as shown in the following example.
See dladm(1M) for details.
# dladm show-linkprop -p _intr_coalesce cxge0
# dladm set-linkprop -p _intr_coalesce=10 cxge0
Kernel Tuning
-------------
The driver's behaviour and performance can be influenced by a variety of kernel
tunables. General information about TCP/IP tuning on Solaris can be found here:
http://www.solarisinternals.com/wiki/index.php/Networks
The use of MSI-X interrupts is recommended. This can be done by adding one of
these sets of lines to /etc/system (depending on the Solaris version).
set ddi_msix_alloc_limit=16
---------- OR -------------
set ddi_msix_alloc_limit=16
set pcplusmp:apic_multi_msi_max=16
set pcplusmp:apic_msix_max=16
set pcplusmp:apic_intr_policy=1
Section 5: Troubleshooting
==========================
Run the commands in this section in order, save their output, and contact
support@chelsio.com if you have problems installing or running the driver.
1) What system is this?
# uname -a
# cat /etc/release
2) Is the driver installed? Is it the right version?
# pkginfo | grep CHLS
# pkginfo -l CHLSt3ei CHLSt3eix (on x86 systems)
# pkginfo -l CHLSt3es CHLSt3esx (on SPARC systems)
# grep cxge /etc/driver_aliases
# grep cxge /etc/name_to_major
3) Are there any Chelsio T3 cards in the system? (Look for 1425 in the output
of these commands.)
# scanpci (or /usr/X11/bin/scanpci)
4) Is the driver loaded? Did it attach to the Chelsio cards in the system?
(Look for 1425, cxge, cxgen in the output of these commands.)
# modinfo | grep cxge
# dmesg
# prtconf
5) Are the Ethernet ports registered with the kernel?
# dladm show-link
# dladm show-linkprop
# ifconfig -a
6) Is a link established? Can packets be transmitted and received?
# ifconfig plumb cxge0 a.b.c.d/nm up
# kstat cxge:0:mac:link_state
# netstat -I cxge0 1 (run this while attempting to send/receive traffic)
7) Detailed information about the driver state and statistics.
# kstat cxgen
# kstat cxge