Red Hat Bugzilla – Bug 210359
Cluster nodes hang in vgscan at reboot time
Last modified: 2009-09-03 12:51:07 EDT
Description of problem:
During a "revolver" cluster recovery test on the smoke cluster,
three of the five nodes (camel, merit, and salem) were deliberately
killed by the test, leaving kool and winston to recover when they
rebooted. Instead of recovering, all three killed nodes hung
at reboot time in vgscan, and it looks to me like some kind
of strange cman kernel communication error.
Version-Release number of selected component (if applicable):
RHEL5 beta 1 plus latest CVS code from 10 Oct 2006.
Unknown - happened once so far.
Steps to Reproduce:
1. cd /root/sts-test on "smoke"
2. ../gfs/bin/revolver -f var/share/resource_files/smoke.xml -l $PWD -i 0 -L
LITE -I -t 1
vgscan hangs forever.
Console has error messages such as:
dlm: Error sending to node 4 -32
dlm: clvmd: dlm_wait_function aborted
(See attached file.)
No vgscan hang
See attached files for console messages from all nodes
and group_tool dump information.
Created attachment 138261 [details]
console output, /proc/net/sctp/assocs and group_tool -v for all nodes
Created attachment 138262 [details]
Time-adjusted, sorted and collated cman_tool dump from all nodes
This is the output from a tool I wrote called grimoire.
Its function is to figure out all nodes in a cluster from cluster.conf,
collect daemon information from each (group_tool -dump), time-adjust
them all and collate them together. The result is a timeline of what
happened from a groupd daemon point of view.
The DLM sctp messages are the clue.
Here's a patch to fix, I'll send this upstream.
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 7bcea7c..867f93d 100644
@@ -548,7 +548,7 @@ static int receive_from_sock(void)
len = iov.iov_len + iov.iov_len;
- r = ret = kernel_recvmsg(sctp_con.sock, &msg, iov, 1, len,
+ r = ret = kernel_recvmsg(sctp_con.sock, &msg, iov, msg.msg_iovlen, len,
MSG_NOSIGNAL | MSG_DONTWAIT);
if (ret <= 0)
DLM kernel module change required to pass the cluster beta2 release criteria.
Changing the component to dlm-kernel and rhel beta product. Devel ACK.
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux release. Product Management has requested further review
of this request by Red Hat Engineering. This request is not yet committed for
inclusion in release.
yes, this patch is in RHEL5B2.