Description of problem: grant-01 is attempting to fence grant-03 but is unable to due to connection errors. Jun 25 11:35:04 grant-01 fenced[12237]: grant-03 not a cluster member after 30 sec post_fail_delay Jun 25 11:35:04 grant-01 fenced[12237]: fencing node "grant-03" Jun 25 11:37:23 grant-01 fenced[12237]: agent "fence_ipmilan" reports: Rebooting machine @ IPMI:grant-03-ipmi...ipmilan: Failed to connect after 30 seconds Failed Jun 25 11:37:23 grant-01 ccsd[12215]: Attempt to close an unopened CCS descriptor (216180). Jun 25 11:37:23 grant-01 ccsd[12215]: Error while processing disconnect: Invalid request descriptor Jun 25 11:37:23 grant-01 fenced[12237]: fence "grant-03" failed Jun 25 11:37:28 grant-01 fenced[12237]: fencing node "grant-03" Jun 25 11:39:48 grant-01 fenced[12237]: agent "fence_ipmilan" reports: Rebooting machine @ IPMI:grant-03-ipmi...ipmilan: Failed to connect after 30 seconds Failed Jun 25 11:39:48 grant-01 ccsd[12215]: Attempt to close an unopened CCS descriptor (216630). Jun 25 11:39:48 grant-01 ccsd[12215]: Error while processing disconnect: Invalid request descriptor Jun 25 11:39:48 grant-01 fenced[12237]: fence "grant-03" failed Jun 25 11:39:53 grant-01 fenced[12237]: fencing node "grant-03" Jun 25 11:42:12 grant-01 fenced[12237]: agent "fence_ipmilan" reports: Rebooting machine @ IPMI:grant-03-ipmi...ipmilan: Failed to connect after 30 seconds Failed Jun 25 11:42:12 grant-01 ccsd[12215]: Attempt to close an unopened CCS descriptor (217080). Jun 25 11:42:12 grant-01 ccsd[12215]: Error while processing disconnect: Invalid request descriptor Jun 25 11:42:12 grant-01 fenced[12237]: fence "grant-03" failed Jun 25 11:42:17 grant-01 fenced[12237]: fencing node "grant-03" Jun 25 11:44:36 grant-01 fenced[12237]: agent "fence_ipmilan" reports: Rebooting machine @ IPMI:grant-03-ipmi...ipmilan: Failed to connect after 30 seconds Failed Jun 25 11:44:36 grant-01 ccsd[12215]: Attempt to close an unopened CCS descriptor (217560). Jun 25 11:44:36 grant-01 ccsd[12215]: Error while processing disconnect: Invalid request descriptor Jun 25 11:44:36 grant-01 fenced[12237]: fence "grant-03" failed Version-Release number of selected component (if applicable): 2.6.18-92.el5
The CCS errors here are a red herring. Because the fencing takes to long to fail, the ccs handle has been expired.
This problem is caused by long timeout of IPMI fence agent. It's duplicate of older bug, so I'm closing this one. *** This bug has been marked as a duplicate of bug 276541 ***