Bug 959869

Summary:	gluster peer status shows two entries with same UUID for same machine on one of the peer in cluster
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	glusterd	Assignee:	krishnan parthasarathi <kparthas>
Status:	CLOSED ERRATA	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	2.1	CC:	amarts, kparthas, nsathyan, rhs-bugs, surs, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0.10rhs	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	971796 (view as bug list)		Environment:
Last Closed:	2013-09-23 22:39:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	971796

Description Rahul Hinduja 2013-05-06 07:16:03 UTC

Description of problem:
=======================

Performed the glusterfs update from glusterfs-3.4.0.2rhs-1 to glusterfs-3.4.0.3rhs-1 and performed reboot of all the system in cluster.

Once all the machines are UP, one of the system shows multiple entry of same machine with same UUID in gluster peer status.



[root@rhs-client11 ~]# gluster peer status
Number of Peers: 4

Hostname: 10.70.36.38                  <<<---------------Note
Uuid: e815f97d-4b98-4501-a19f-c9f1d0a9cd2e
State: Peer in Cluster (Connected)

Hostname: 10.70.36.36
Uuid: 91d06293-a9d4-467f-bc09-a0d929ebacac
State: Peer in Cluster (Connected)

Hostname: 10.70.36.38                  <<<---------------Note
Uuid: e815f97d-4b98-4501-a19f-c9f1d0a9cd2e
State: Peer in Cluster (Connected)

Hostname: 10.70.36.37
Uuid: 53ded8aa-05eb-4d57-a4d4-3db78bbde921
State: Peer in Cluster (Connected)
[root@rhs-client11 ~]# 




[root@rhs-client13 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.36.38
Uuid: e815f97d-4b98-4501-a19f-c9f1d0a9cd2e
State: Peer in Cluster (Connected)

Hostname: 10.70.36.36
Uuid: 91d06293-a9d4-467f-bc09-a0d929ebacac
State: Peer in Cluster (Connected)

Hostname: 10.70.36.35
Uuid: c6efd0c0-e6cc-4eda-af93-eea9eeef14ea
State: Peer in Cluster (Connected)
[root@rhs-client13 ~]# 



Version-Release number of selected component (if applicable):
=============================================================

[root@rhs-client11 ~]# rpm -qa | grep gluster | grep 3.4
glusterfs-3.4.0.3rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.3rhs-1.el6rhs.x86_64
glusterfs-fuse-3.4.0.3rhs-1.el6rhs.x86_64
glusterfs-devel-3.4.0.3rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.3rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.3rhs-1.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.3rhs-1.el6rhs.x86_64
[root@rhs-client11 ~]# 



Steps Carried:
==============
1. Upgraded all the four systems in server from 3.4.0.2rhs to 3.4.0.3rhs
2. Rebooted all the four systems in cluster
3. Noticed one of the server in cluster shows multiple entry for another peer with same UUID
  
Additional info:
================

All the commands like start/stop of volume from this machine (rhs-client11) fails with the reason "volume start: vol-dis-rep: failed: Another transaction could be in progress. Please try again after sometime."

The same command from other system in cluster is successful

Comment 3 krishnan parthasarathi 2013-05-23 10:01:00 UTC

This bug is seen because of glusterd's persistent store being left in an inconsistent state due to node shutdown.

The persistent store in rhs-client11 is as below,

kp@trantor ~/b/9/r/v/l/glusterd> ls peers/
53ded8aa-05eb-4d57-a4d4-3db78bbde921  91d06293-a9d4-467f-bc09-a0d929ebacac  e815f97d-4b98-4501-a19f-c9f1d0a9cd2e  e815f97d-4b98-4501-a19f-c9f1d0a9cd2e.tmp

The shutdown killed glusterd after it had made the changes to the state of peer with uuid e815f97d-4b98-4501-a19f-c9f1d0a9cd2e in a temporary file and before it had renamed the tmp file to the original file, to 'commit' the update.

To avoid these issues, glusterd should use a different directory for making temporary updates to its persistent store.

Comment 6 Scott Haines 2013-09-23 22:39:38 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Comment 7 Scott Haines 2013-09-23 22:43:46 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html