Created attachment 892840 [details] glusterd log Description of problem: While doing rolling upgrade, when one of the machines is 2.1U2 and other 3.5qa2; peer detach force is executed on the updated machine (3.5qa2) from 2.1U2. glusterd on 2.1U2 crashes. Peer detach is never successful. Version-Release number of selected component (if applicable): glusterfs 3.4.0.59rhs built on Feb 4 2014 08:44:13 glusterfs 3.5qa2 built on May 2 2014 06:17:44 How reproducible: Always. Steps to Reproduce: 1. Create a 2x2 cluster (Version: 2.1U2) 2. Perform rolling upgrade (IO is not stopped); Upgrade one of the machines to 3.5qa2. From 3.5qa2 peer probe the 2.1U2 machine. 3. From 2.1U2 execute `peer detach force' to detach the 3.5qa2 node 4. glusterd will crash Actual results: glusterd crashes Expected results: glusterd should not crash Additional info:
Core dumps can be found at: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1094716/
Sac, The cores don't seem to be working for me. I'm trying to get a backtrace with glusterfs-3.4.0.59rhs with them, and all I get are '??'. This could be due to an issue with my environment. But I just want to confirm that they are from glusterfs-3.4.0.59rhs. Also, it will be helpful if you could provide logs from the crashed node.
Thanks Sac for giving access to the systems, it's gonna help identify the problem quickly. Meanwhile, I've got concerns on this bugs status as a possible release blocker for rhs-3.0. We currently don't have officially documented steps for rhs-2.1 to rhs-3.0 upgrades. Proper upgrade documents would prevent the user from performing a combination of operations, as in this bug, which could lead failures. So, with proper upgrade documentation, we will not be hitting this issue. Also, the crash was observed in rhs-2.1, not in rhs-3.0, so having this bug as a blocker for rhs-3.0, doesn't feel right. Because of the above two reasons, I believe this bug should be removed from the scope of the rhs-3.0 release and be targeted for a future release. I'll still continue my investigation as I've got access to the systems currently and I'd most likely not be able to get it later.
Kaushal, we are documenting rolling upgrade for Denali. Which means, during upgrades we will have both 2.1U2 and 3.0 machines. And in the case above: * We upgrade one of the machines and probe 3.0 into the cluster. Is there any other way we are planning to do this? When can I expect the officially documented steps? I'm blocked on upgrade testing...
(In reply to Sachidananda Urs from comment #7) > Kaushal, we are documenting rolling upgrade for Denali. Which means, during > upgrades we will have both 2.1U2 and 3.0 machines. And in the case above: > > * We upgrade one of the machines and probe 3.0 into the cluster. > > Is there any other way we are planning to do this? When can I expect the > officially documented steps? I'm blocked on upgrade testing... The steps you are following with along with saving and restoring /var/lib/glusterd should be the required steps for upgrade and should ideally work without needing any more intervention. But it isn't the case now. rhs-3.0 has added some new fields into the volinfo struct which are needed for snapshot. As it is currently, the import_volinfo code which imports the volumes during peer handshaking, expects these fields to be present and will fail the import if they are not present. Failure of this import leads to the different behaviours depending on the original starting point. 1. If this import was being done on a new peer on a peer probe, its failure would lead to the probe process being stuck. The peer status would be in an intermediate 'Probe sent to peer' state. Doing a peer detach in this case could lead to glusterd crashing, as it happened in this bug report. 2. If this import was happening during the startup of a Glusterd, this failure could lead to the peer entering a peer rejected state (I still need to validate if this is correct). So even if /var/lib/glusterd was saved and restored during the upgrade, this could still lead to peer being rejected, thus causing upgrade problems. So we still need to fix rhs-3.0 as it is not backwards compatible enough yet. But we also need to still come up with the correct documentation for upgrades. I'm removing the rhs-future flag and giving devel_ack for this bug. Please mark this bug as a blocker, as we shouldn't be breaking backwards compatibility as we are doing now.
Patch merged upstream. Hence moving it to POST.
Patch merged downstream : https://code.engineering.redhat.com/gerrit/#/c/26430/
Verified on glusterfs 3.6.0.15 I no longer see the crash, with 2.1U2 and 3.0 machines in the cluster.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-1278.html