Description of problem: When we do a rolling upgrade of the cluster from 3.12, 4.1 or 5.5 to 6, the upgraded node goes into disconnected state. Version-Release number of selected component (if applicable): 6.0 How reproducible: 100% Steps to Reproduce: 1.create a replica 3 cluster 2.kill gluster process on one node 3.upgrade the node and start glusterd Actual results: the upgrade node goes into disconnected state Expected results: the peer shouldnt get disconnected. Additional info:
To over come this issue the following steps were performed. Upgrade all the node in the cluster one after other. Once all the nodes are upgraded, kill glusterd process alone. Let other process keep running. Now do a "iptables -F" And then restart the glusterd on all the nodes Try gluster peer state after this to check if the nodes are connected.
Please explain why this happens and how the workaround solves the issue.
> When we do a rolling upgrade of the cluster from 3.12, 4.1 or 5.5 to 6, the upgraded node goes into disconnected state. Isn't this only seen from 3.12 to 6 upgrade?
Hi Nithya, The RCA for this is yet to be done. I didn't find anything fishy in the logs. As I had to move forward with the testing, I tried the usually way of flushing the iptables to check if it fixes the disconnects and it yes, it did connect the peers back. The reason why this is happening is yet to be discovered. Regards, Hari.
(In reply to Atin Mukherjee from comment #3) > > When we do a rolling upgrade of the cluster from 3.12, 4.1 or 5.5 to 6, the upgraded node goes into disconnected state. > > Isn't this only seen from 3.12 to 6 upgrade? No, Atin. I issue happened with all the versions. It could as well be some network issue with the machines I tried it on. Not sure of it. The point to note here is: some times just a glusterd restart fixed and in some scenarios it needed a iptables flush followed with glusterd restart. But I found that the iptables flush with glusterd restart fixed it in every scenario i tried. I could find time to debug this further.
FYI.. I tested the rolling upgrade from glusterfs 3.12 latest to glusterfs-6 with out any issues. Can some one else please try as well?
I've tested rolling upgrade from 3.12 to 6, but haven't seen any issue. The cluster is in a healthy state and all peers are in connected state. Based on my experience and comment 6, I'm closing this as not a bug. Please, feel free to re-open the bug if you face it. Thanks, Sanju
(In reply to Sanju from comment #7) > I've tested rolling upgrade from 3.12 to 6, but haven't seen any issue. The > cluster is in a healthy state and all peers are in connected state. Based on > my experience and comment 6, I'm closing this as not a bug. Please, feel > free to re-open the bug if you face it. > > Thanks, > Sanju What about the upgrades from the other versions? This BZ refers to upgrades to release 6 from 3.12, 4 and 5.
(In reply to Nithya Balachandran from comment #8) > What about the upgrades from the other versions? This BZ refers to upgrades > to release 6 from 3.12, 4 and 5. I did test upgrade to release 6 from 4 and 5. Haven't seen any issue. Thanks, Sanju