Segment Servers Stuck in Down, InFailover State IPMI Unable to Est. lan Session

Segment Servers Stuck in Down, InFailover State IPMI Unable to est.
LAN Session
Solution ID: 55
Created: 8/25/2008
DETAILS:
Segment Servers stuck in "Down, InFailover" status
From the logs it was determined IPMI failed to power off the server with the following error:
"Aug 20, 2008 3:03:23 PM com.ibrix.ias.server.OperationQueue doRun
WARNING: com.ibrix.ias.model.BusinessException: command : [/usr/bin/ipmitool -I lan -H
192.9.202.127 -U ibrix -P ibrix chassis power off] terminated with status 1, output is: Error: Unable to
establish LAN session ipmi_lan_send_cmd failed to open intf Unable to set Chassis Power Control to
Down/Off"
For this case we also intermittently experienced the following error when trying to do ipmitool status
commands to the server from the FM.
“WARNING: com.ibrix.ias.model.BusinessException: power status verification failed”
Resolution:
In this case there were 2 configuration problems:
1. channel bonding mode =4 on host and switches not 100% compatible with IPMI
2. Absence of PortFast protocol or (STP ) will cuase port recovery latency issues with IPMI
There is an issue with auto-negotiation or PortFast (Spanning Tree Protocol) on Cisco, Extreme
Summit and other 3rd Party Ethernet switches. If the bond goes down and/or eth0 goes down IPMI on
eth0 comes back up at 100MB/Full duplex by default. However, if the switch ports are fixed for
1000MB, then IPMI will fail to connect. The switch must be set for auto negotiate for all these
eth0/bond ports and PortFast (STP) should also be enabled. When your switch is fixed for 1000MB
speed the IPMI is unable to log into the port on the switch unless it is set to auto-negotiate explaining
why failover is not able to make lan session connection to the Segment Server to complete a failover.
The corrective action needed to fix the issue is:
1. Enable STP and ensure all ports were set to auto negotiate
2. Disable 802.3ad (mode 4) channel bonding on switches
Ibrix_S0000055.pdf