Description of problem: Cluster still locks up on recovery after several rounds of killing master and slave gulm servers. Version-Release number of selected component (if applicable): How reproducible: Takes awhile. Steps to Reproduce: 1. Kill Master and Slave lots of times. 2. 3. Actual results: Cluster eventually hangs. Expected results: Cluster recovers successfully. Additional info:
Taking off blocker list, some of the issues have been fixed, but there still might be problems outstanding.
I hit this today during RHEL4-U3 errata testing. I was running gulm-1.0.6-0. 2 of 3 server nodes were shot. It doesn't appear that the server that rejoined to form quorum expired the locks it had prior to being shot.
I'm still hitting this in RHEL4-U4 testing.
Problem occurs if you kill enough masters for the remaining gulm server to lose quorum. It then may not fence all of the killed gulm servers resulting in an inconsistent lock state. The problem can be easily fixed by fencing the lock servers that were killed but not fence previously. I'm working on a solution.
I'm still hitting this in RHEL4-U4 testing. x86 cluster.
Hit this over the weekend on x86_64 during the "GULM kill Master and all but one Slave" revolver senario.
Devel ACK.
Ok, so it appears that gulm was not properly propagating all of the slaves/clients to the slaves. This should fix one type of lockup, and hopefully the lockup that was occurring in this bug. The fix is built in gulm-1.0.9-2.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2007-0145.html