Description of problem: During a "revolver" cluster recovery test on the smoke cluster, three of the five nodes (camel, merit, and salem) were deliberately killed by the test, leaving kool and winston to recover when they rebooted. Instead of recovering, all three killed nodes hung at reboot time in vgscan, and it looks to me like some kind of strange cman kernel communication error. Version-Release number of selected component (if applicable): RHEL5 beta 1 plus latest CVS code from 10 Oct 2006. How reproducible: Unknown - happened once so far. Steps to Reproduce: 1. cd /root/sts-test on "smoke" 2. ../gfs/bin/revolver -f var/share/resource_files/smoke.xml -l $PWD -i 0 -L LITE -I -t 1 Actual results: vgscan hangs forever. Console has error messages such as: dlm: Error sending to node 4 -32 dlm: clvmd: dlm_wait_function aborted (See attached file.) Expected results: No vgscan hang Additional info: See attached files for console messages from all nodes and group_tool dump information.
Created attachment 138261 [details] console output, /proc/net/sctp/assocs and group_tool -v for all nodes
Created attachment 138262 [details] Time-adjusted, sorted and collated cman_tool dump from all nodes This is the output from a tool I wrote called grimoire. Its function is to figure out all nodes in a cluster from cluster.conf, collect daemon information from each (group_tool -dump), time-adjust them all and collate them together. The result is a timeline of what happened from a groupd daemon point of view.
The DLM sctp messages are the clue. Here's a patch to fix, I'll send this upstream. diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index 7bcea7c..867f93d 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -548,7 +548,7 @@ static int receive_from_sock(void) } len = iov[0].iov_len + iov[1].iov_len; - r = ret = kernel_recvmsg(sctp_con.sock, &msg, iov, 1, len, + r = ret = kernel_recvmsg(sctp_con.sock, &msg, iov, msg.msg_iovlen, len, MSG_NOSIGNAL | MSG_DONTWAIT); if (ret <= 0) goto out_close;
DLM kernel module change required to pass the cluster beta2 release criteria. Changing the component to dlm-kernel and rhel beta product. Devel ACK.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux release. Product Management has requested further review of this request by Red Hat Engineering. This request is not yet committed for inclusion in release.
yes, this patch is in RHEL5B2.