Bug 160646 - GFS cluster node does not shutdown: CMANsendmsg failed: -101
Summary: GFS cluster node does not shutdown: CMANsendmsg failed: -101
Alias: None
Product: Fedora
Classification: Fedora
Component: cman
Version: 4
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Jonathan Earl Brassow
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2005-06-16 10:54 UTC by Axel Thimm
Modified: 2007-11-30 22:11 UTC (History)
1 user (show)

Fixed In Version: RHEL3 U5
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2005-10-04 17:17:32 UTC
Type: ---

Attachments (Terms of Use)

Description Axel Thimm 2005-06-16 10:54:43 UTC
Description of problem:
Trying to shutdown a GFS node in a 3 cluster node hangs with repeated

CMANsendmsg failed: -101

lines. -101 seems to mean the network is down, so perhaps this is a race between
network and cluster shutdown?

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.create a GFS cluster
2.try rebooting one node
Actual results:
reboot fails on shutdown as described above

Expected results:
shutdown should complete

Additional info:

Comment 1 Jonathan Earl Brassow 2005-06-16 14:33:32 UTC
Are the init scripts active so the system shuts down in the right order?

Shutting down the network before shutting cman would cause a problem like this.

Comment 2 Axel Thimm 2005-06-17 08:04:41 UTC
Yes, all GFS related init scripts have ben chkconfig-enabled. The shutdown is
performed by normal init scripts in the script-given ordering.

Perhaps cman shutdown fails for any reason and later on cman holds the final
rebooting? Then there would be two bugs, one for not having cman properly
shutdown (and I can imagine fencing to take part in this), and another one for
the not-stopped cman not allowing a system to shutdown/reboot.

Comment 3 Jonathan Earl Brassow 2005-07-20 00:10:07 UTC
I believe this was solved by alewis by altering the clvm init script.

Comment 4 AJ Lewis 2005-08-03 14:49:42 UTC
hrm...not sure - are there any initscript errors before this happens?

Comment 5 Jonathan Earl Brassow 2005-08-03 15:48:29 UTC
I don't think there were any errors that were reported by the clvmd init script.  Previously, it would 
shutdown volumes, but not kill the clvmd daemon.  Since the daemon was still logged into cman, cman 
would refuse to shutdown and start spitting out errors like described above....

The way to get to the bottom of this hypothesis is to have the user attach their clvmd init script and 
check to make sure that it is killing off the daemon during all shutdown cases.

Comment 6 Jonathan Earl Brassow 2005-10-04 17:17:32 UTC
The clvmd init script now kills off the daemon when shutting down

Comment 7 Axel Thimm 2005-10-04 21:18:16 UTC
The resolution of this bug is RHEL3 (aka RHCS 3), while the bug was opened
against FC4 which is more like RHEL4 wrt to RHCS/RHGFS.

Also there seems to still be some racing in RHEL4 in shutting down the cluster with

for service in rgmanager gfs clvmd fenced cman ccsd; do
   service $service stop

Sometimes cman fails to stop, and service cman stop needs to be reissued.

Note You need to log in before you can comment on or make changes to this bug.