Description of problem:
Trying to shutdown a GFS node in a 3 cluster node hangs with repeated
CMANsendmsg failed: -101
lines. -101 seems to mean the network is down, so perhaps this is a race between
network and cluster shutdown?
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.create a GFS cluster
2.try rebooting one node
reboot fails on shutdown as described above
shutdown should complete
Are the init scripts active so the system shuts down in the right order?
Shutting down the network before shutting cman would cause a problem like this.
Yes, all GFS related init scripts have ben chkconfig-enabled. The shutdown is
performed by normal init scripts in the script-given ordering.
Perhaps cman shutdown fails for any reason and later on cman holds the final
rebooting? Then there would be two bugs, one for not having cman properly
shutdown (and I can imagine fencing to take part in this), and another one for
the not-stopped cman not allowing a system to shutdown/reboot.
I believe this was solved by alewis by altering the clvm init script.
hrm...not sure - are there any initscript errors before this happens?
I don't think there were any errors that were reported by the clvmd init script. Previously, it would
shutdown volumes, but not kill the clvmd daemon. Since the daemon was still logged into cman, cman
would refuse to shutdown and start spitting out errors like described above....
The way to get to the bottom of this hypothesis is to have the user attach their clvmd init script and
check to make sure that it is killing off the daemon during all shutdown cases.
The clvmd init script now kills off the daemon when shutting down
The resolution of this bug is RHEL3 (aka RHCS 3), while the bug was opened
against FC4 which is more like RHEL4 wrt to RHCS/RHGFS.
Also there seems to still be some racing in RHEL4 in shutting down the cluster with
for service in rgmanager gfs clvmd fenced cman ccsd; do
service $service stop
Sometimes cman fails to stop, and service cman stop needs to be reissued.