User guide

User Guide NetXtreme II
September 2013
Broadcom Corporation
Document INGSRVT78-CDUM100-R Interrupt Management Page 171
INTERRUPT MANAGEMENT
If you have a system with many interfaces, it is possible to reach the allocation limit of MSIX interrupts. By default, Solaris
limits each driver to 2 MSIX allocations, and there is an issue with the pcplusmp module where only a maximum of 31 MSIX
interrupts are available per interrupt priority level.
If your system has four Broadcom BCM57711 network adapter ports, each running in multifunction mode, Solaris will
enumerate 16 bnxe interfaces. The last interface attached will fail to allocate its second MSIX interrupt and revert to Fixed.
This in turn can eventually expose an issue in the system regarding interrupt management resulting in interrupts never being
received on the interface that reverted back to Fixed.
To ensure all interfaces are able to allocate their two MSIX interrupts, the workaround is to change the priority levels of
specific interfaces. Network drivers are automatically assigned an interrupt priority level of 6, so changing an interface's
priority level to 5 is common.
1. First read the driver.conf man page for a background primer.
2. Find out the driver instance paths assigned on your system.
% grep bnxe /etc/path_to_inst
"/pci@0,0/pci8086,2779@1/pci14e4,1650@0" 0 "bnxe"
"/pci@0,0/pci8086,2779@1/pci14e4,1650@0,1" 1 "bnxe"
3. Normally, the name of the driver is the last portion of the path, but you should use the most appropriate PCI ID found in
/etc/driver_aliases. Depending on how the hardware is layered, there are cases where the name identified in path_to_inst
will not work. To figure out which name to use, examine the output from prtconf -v and match against the IDs specified
in the driver_aliases file.
% grep bnxe /etc/driver_aliases
bnxe "pci14e4,164e"
bnxe "pci14e4,164f"
bnxe "pci14e4,1650"
bnxe "pciex14e4,164e"
bnxe "pciex14e4,164f"
bnxe "pciex14e4,1650"
4. The parent of the driver is the entire path leading up to the name.
5. The unit-address is located after the final @ in the path.
6. Therefore, change both of the bnxe interfaces found in path_to_inst to interrupt priority 5 and use the following config
lines to bnxe.conf:
name = "pciex14e4,1650" parent = "/pci@0,0/pci8086,2779@1" unit-address = "0" interrupt-
priorities = 5;
name = "pciex14e4,1650" parent = "/pci@0,0/pci8086,2779@1" unit-address = "0,1" interrupt-
priorities = 5;
7. After modifying the config, either reboot the system or unplumb all interfaces and run the update_drv command.
8. When the system has been reconfigured and the interfaces plumbed back up, verify the new interrupt priority settings by
running the following command as root:
% echo "::interrupts -d" | mdb -k