                           Installation Notes
                       Broadcom bnx2x Linux Driver

                          Broadcom Corporation
                         5300 California Avenue,
                            Irvine, CA 92617

                 Copyright (c) 2007-2008 Broadcom Corporation
                           All rights reserved


Table of Contents
=================

  Introduction
  Limitations
  Packaging
  Installing Source RPM Package
  Building Driver From TAR File
  Driver Dependencies
  Driver Settings
  Driver Parameters
  Driver Defaults
  Unloading and Removing Driver
  Driver Messages


Introduction
============

This file describes the bnx2x Linux driver for the Broadcom NetXtreme II
BCM57710/BCM57711/BCM57711E 10/100/1000/2500/10000 Mbps PCIE Ethernet Network
Controllers.


Limitations
===========

The current version of the driver has been tested on 2.6.x kernels starting
from 2.6.9. The driver may not compile on kernels older than 2.6.9. Testing
is concentrated on i386 and x86_64 architectures. Only limited testing has
been done on some other architectures.

Minor changes to some source files and Makefile may be needed on some
kernels.

IP Forwarding (bridging) cannot be used with TPA on kernels older than
2.6.26. Please disable TPA with either ethtool (if available) or driver
parameter (see "Driver Parameters" section below)

Packaging
=========

The driver is released in two packaging formats: source RPM and compressed tar
formats. The file names for the two packages are bnx2x-<version>.src.rpm and
bnx2x-<version>.tar.gz respectively. Identical source files to build the
driver are included in both packages.


Installing Source RPM Package
=============================

The following are general guidelines for installing the driver.

1. Install the source RPM package:

   rpm -ivh bnx2x-<version>.src.rpm

2. CD to the RPM path and build the binary driver for your kernel:

   cd /usr/src/{redhat,OpenLinux,turbo,packages,rpm ..}

   rpm -bb SPECS/bnx2x.spec

or

   rpmbuild -bb SPECS/bnx2x.spec (for RPM version 4.x.x)

Note that the RPM path is different for different Linux distributions.

The driver will be compiled for the running kernel by default. To build
the driver for a kernel different than the running one, specify the
kernel by defining it in KVER:

   rpmbuild -bb SPECS/bnx2x.spec --define "KVER <kernel version>"

where <kernel version> in the form of 2.6.y-z is the version of another
kernel that is installed on the system.

3. Install the newly built package (driver and man page):

   rpm -ivh RPMS/<arch>/bnx2x-<version>.<arch>.rpm

where <arch> is the machine architecture such as i386:

   rpm -ivh RPMS/i386/bnx2x-<version>.i386.rpm

Note that the --force option may be needed on some Linux distributions
if conflicts are reported.

The driver will be installed in the following path:

    /lib/modules/<kernel_version>/kernel/drivers/net/bnx2x.ko

4. Unload existing driver if necessary:

   rmmod bnx2x

5. Load the driver:

   insmod bnx2x.ko

or

   modprobe bnx2x

6. To configure network protocol and address, refer to various Linux
documentations.


Building Driver From TAR File
=============================

The following are general guidelines for installing the driver.

1. Create a directory and extract the files:

   tar xvzf bnx2x-<version>.tar.gz

2. Build the driver bnx2x.ko as a loadable module for the running kernel:

   cd src

   make

The driver will be compiled for the running kernel by default. To build
the driver for a kernel different than the running one, specify the
kernel by defining it in KVER:

   make KVER=<kernel version>

where <kernel version> in the form of 2.6.y-z is the version of another
kernel that is installed on the system.

3. Test the driver by loading it (unload existing driver first if necessary):

   rmmod bnx2x

   insmod bnx2x.ko

4. Install the driver and man page:

   make install

See RPM instructions above for the location of the installed driver.

5. To configure network protocol and address, refer to various Linux
documentations.


Driver Dependencies
===================

The driver uses library functions in the crc32 and zlib_inflate libraries.
On most kernels, these libraries are already built into the kernel. In
some cases, it may be necessary to load these library modules before the
driver or unresolved symbol errors will appear. Using modprobe will
resolve the dependencies automatically.

In rare cases where the crc32 and zlib_inflate libraries are not enabled
in the kernel, it will be necessary to compile the kernel again with the
libraries enabled.

The driver uses also library functions in the crc32c library. On new kernels,
this library is already built into the kernel. In some old kernels,
it may be necessary to load this library module before the driver or
unresolved symbol errors will appear. Using modprobe will resolve the
dependencies automatically.


Driver Settings
===============

The bnx2x driver settings can be queried and changed using ethtool. The
latest ethtool can be downloaded from http://sourceforge.net/projects/gkernel
if it is not already installed. The following are some common examples on how
to use ethtool. See the ethtool man page for more information. ethtool settings
do not persist across reboot or module reload. The ethtool commands can be put
in a startup script such as /etc/rc.local to preserve the settings across a
reboot. On Red Hat distributions, "ethtool -s" parameters can be specified
in the ifcfg-ethx scripts using the ETHTOOL_OPTS keyword. The specified
ethtool parameters will be set during ifup. Example:
/etc/sysconfig/network-scripts/ifcfg-eth0:

ETHTOOL_OPTS="wol g speed 100 duplex half autoneg off"

Some ethtool examples:

1. Show current speed, duplex, and link status:

   ethtool eth0

2. Change speed, duplex, autoneg:

Example: 100Mbps half duplex, no autonegotiation:

   ethtool -s eth0 speed 100 duplex half autoneg off

Example: Autonegotiation with full advertisement:

   ethtool -s eth0 autoneg on

Example: Autonegotiation with 100Mbps full duplex advertisement only:

   ethtool -s eth0 speed 100 duplex full autoneg on

3. Show flow control settings:

   ethtool -a eth0

4. Change flow control settings:

Example: Turn off flow control

   ethtool -A eth0 autoneg off rx off tx off

Example: Turn flow control autonegotiation on with tx and rx advertisement:

   ethtool -A eth0 autoneg on rx on tx on

   Note that this is only valid if speed is set to autonegotiation.

5. Show offload settings:

   ethtool -k eth0

6. Change offload settings:

Example: Turn off TSO (TCP segmentation offload)

   ethtool -K eth0 tso off

7. Get statistics:

   ethtool -S eth0

8. Perform self-test:

   ethtool -t eth0

   Note that the interface (eth0) must be up to do all tests.

9. See ethtool man page for more options.


Driver Parameters
=================

Several optional parameters can be supplied as a command line argument
to the insmod or modprobe command. These parameters can also be set in
modprobe.conf. See the man page for more information.

The optional parameter "int_mode" is used to force using an interrupt mode
other than MSI-X. By default, the driver will try to enable MSI-X if it is
supported by the kernel. In case MSI-X is not attainable, the driver will try
to enable MSI if it is supported by the kernel. In case MSI is not attainable,
the driver will use legacy INTx mode.

Set the "int_mode" parameter to 1 as shown below to force using the legacy
INTx mode on all NetXtreme II NICs in the system.

   insmod bnx2x.ko int_mode=1

or

   modprobe bnx2x int_mode=1

Set the "int_mode" parameter to 2 as shown below to force using MSI mode
on all NetXtreme II NICs in the system.

   insmod bnx2x.ko int_mode=2

or

   modprobe bnx2x int_mode=2


The optional parameter "disable_tpa" can be used to disable the
Transparent Packet Aggregation (TPA) feature. By default, the driver will
aggregate TCP packets, but if a user would like to disable this advanced
feature - it can be done.

Set the "disable_tpa" parameter to 1 as shown below to disable the TPA
feature on all NetXtreme II NICs in the system.

   insmod bnx2x.ko disable_tpa=1

or

   modprobe bnx2x disable_tpa=1

Use ethtool (if available) to disable TPA (LRO) for a specific NetXtreme II NIC.


The optional parameter "multi_mode" is for use on systems that support
multi queue networking. Multi queue networking is supported on kernels
starting from 2.6.24. By default, this parameter is set to 1 thus, the driver
will allocate one queue per-CPU and up to the HW limit.

The "multi_mode" optional parameter can also be used to enable SAFC
(Service Aware Flow Control) by differentiating the traffic to up to 3
priority queues in the HW according to the VLAN PRI value or according to the
IP DSCP value (least 3 bits).

The optional parameter "pri_map" is used to map the VLAN PRI value or the
IP DSCP value to a different or same priority queue in the HW. This 32 bits
parameter is evaluated by the driver as a 8 values of 4 bits each. Each nibble
sets the desired HW queue number for that priority.

For example, set the "pri_map" parameter to 0x22221100 to map priority 0
and 1 to HW queue 0, map priority 2 and 3 to HW queue 1, and map priority 4
to 7 to HW queue 2.

Set the "multi_mode" parameter to 2 as shown below to differentiate the
traffic according to the VLAN PRI value.

   insmod bnx2x.ko multi_mode=2 pri_map=0x22221100

or

   modprobe bnx2x multi_mode=2 pri_map=0x22221100

Set the "multi_mode" parameter to 4 as shown below to differentiate the
traffic according to the IP DSCP value.

   insmod bnx2x.ko multi_mode=4 pri_map=0x22221100

or

   modprobe bnx2x multi_mode=4 pri_map=0x22221100


There are some more optional parameters that can be supplied as a command line
argument to the insmod or modprobe command. These optional parameters are
mainly to be used for debug and may be used only by an expert user.

The debug optional parameter "poll" can be used for timer based polling.
Set the "poll" parameter to the timer polling interval on all NetXtreme
II NICs in the system.

The debug optional parameter "mrrs" can be used to override the MRRS
(Maximum Read Request Size) value of the HW. Set the "mrrs" parameter to
the desired value (0..3) for on all NetXtreme II NICs in the system.

The debug optional parameter "debug" can be used to set the default
msglevel on all NetXtreme II NICs in the system. Use "ethtool -s" to set
the msglevel for a specific NetXtreme II NIC.


Driver Defaults
===============

Speed :                    Autonegotiation with all speeds advertised

Flow control :             Autonegotiation with rx and tx advertised

MTU :                      1500 (range 46 - 9000)

Rx Ring Size :             4078 (range 0 - 4078)

Tx Ring Size :             4078 (range (MAX_SKB_FRAGS+4) - 4078)

                            MAX_SKB_FRAGS varies on different kernels and
                            different architectures. On a 2.6 kernel for
                            x86, MAX_SKB_FRAGS is 18.

Coalesce rx usecs :          25 (range 0 - 3000)

Coalesce tx usecs :          50 (range 0 - 12288)

Coalesce stats usecs :   999936 (aprox. 1 sec.)
                                (range 0 - 16776960 in 256 increments)

MSI-X :                    Enabled (if supported by 2.6 kernel and interrupt
                                    test passes)

TSO :                      Enabled

WoL :                      Disabled


Unloading and Removing Driver
=============================

To unload the driver, do the following:

   rmmod bnx2x

If the driver was installed using rpm, do the following to remove it:

   rpm -e bnx2x


If the driver was installed using make install from the tar file, the driver
bnx2x.ko has to be manually deleted from the system. Refer to the section
"Installing Source RPM Package" for the location of the installed driver.


Driver Messages
===============

The following are the most common sample messages that may be logged in the file
/var/log/messages. Use dmesg -n <level> to control the level at which messages
will appear on the console. Most systems are set to level 6 by default. To see
all messages, set the level higher.

Driver signon:
-------------

Broadcom NetXtreme II 5771x 10Gigabit Ethernet Driver bnx2x 0.40.15 ($DateTime: 2007/11/22 05:32:40 $)


NIC detected:
------------

eth0: Broadcom NetXtreme II BCM57710 XGb (A1) PCI-E x8 2.5GHz found at mem e8800000, IRQ 16, node addr 001018360012


MSI-X enabled successfully:
--------------------------

bnx2x: eth0: using MSI-X


Link up and speed indication:
----------------------------

bnx2x: eth0 NIC Link is Up, 10000 Mbps full duplex, receive & transmit flow control ON


Link down indication:
--------------------

bnx2x: eth0 NIC Link is Down



