Description of problem: ======================= Performed the glusterfs update from glusterfs-3.4.0.2rhs-1 to glusterfs-3.4.0.3rhs-1 and performed reboot of all the system in cluster. Once all the machines are UP, one of the system shows multiple entry of same machine with same UUID in gluster peer status. [root@rhs-client11 ~]# gluster peer status Number of Peers: 4 Hostname: 10.70.36.38 <<<---------------Note Uuid: e815f97d-4b98-4501-a19f-c9f1d0a9cd2e State: Peer in Cluster (Connected) Hostname: 10.70.36.36 Uuid: 91d06293-a9d4-467f-bc09-a0d929ebacac State: Peer in Cluster (Connected) Hostname: 10.70.36.38 <<<---------------Note Uuid: e815f97d-4b98-4501-a19f-c9f1d0a9cd2e State: Peer in Cluster (Connected) Hostname: 10.70.36.37 Uuid: 53ded8aa-05eb-4d57-a4d4-3db78bbde921 State: Peer in Cluster (Connected) [root@rhs-client11 ~]# [root@rhs-client13 ~]# gluster peer status Number of Peers: 3 Hostname: 10.70.36.38 Uuid: e815f97d-4b98-4501-a19f-c9f1d0a9cd2e State: Peer in Cluster (Connected) Hostname: 10.70.36.36 Uuid: 91d06293-a9d4-467f-bc09-a0d929ebacac State: Peer in Cluster (Connected) Hostname: 10.70.36.35 Uuid: c6efd0c0-e6cc-4eda-af93-eea9eeef14ea State: Peer in Cluster (Connected) [root@rhs-client13 ~]# Version-Release number of selected component (if applicable): ============================================================= [root@rhs-client11 ~]# rpm -qa | grep gluster | grep 3.4 glusterfs-3.4.0.3rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.3rhs-1.el6rhs.x86_64 glusterfs-fuse-3.4.0.3rhs-1.el6rhs.x86_64 glusterfs-devel-3.4.0.3rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.3rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.3rhs-1.el6rhs.x86_64 glusterfs-debuginfo-3.4.0.3rhs-1.el6rhs.x86_64 [root@rhs-client11 ~]# Steps Carried: ============== 1. Upgraded all the four systems in server from 3.4.0.2rhs to 3.4.0.3rhs 2. Rebooted all the four systems in cluster 3. Noticed one of the server in cluster shows multiple entry for another peer with same UUID Additional info: ================ All the commands like start/stop of volume from this machine (rhs-client11) fails with the reason "volume start: vol-dis-rep: failed: Another transaction could be in progress. Please try again after sometime." The same command from other system in cluster is successful
This bug is seen because of glusterd's persistent store being left in an inconsistent state due to node shutdown. The persistent store in rhs-client11 is as below, kp@trantor ~/b/9/r/v/l/glusterd> ls peers/ 53ded8aa-05eb-4d57-a4d4-3db78bbde921 91d06293-a9d4-467f-bc09-a0d929ebacac e815f97d-4b98-4501-a19f-c9f1d0a9cd2e e815f97d-4b98-4501-a19f-c9f1d0a9cd2e.tmp The shutdown killed glusterd after it had made the changes to the state of peer with uuid e815f97d-4b98-4501-a19f-c9f1d0a9cd2e in a temporary file and before it had renamed the tmp file to the original file, to 'commit' the update. To avoid these issues, glusterd should use a different directory for making temporary updates to its persistent store.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html