Bug 962343 - glusterd: 'gluster peer status' is not showing all peers on few nodes of cluster(once you detach peer and then again probe that peer) - peer is in 'Connected to Peer' or ' Probe Sent to Peer' state and not able to use that peer in volume creation
glusterd: 'gluster peer status' is not showing all peers on few nodes of clus...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.1
x86_64 Linux
high Severity urgent
: ---
: ---
Assigned To: Kaushal
amainkar
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-13 05:08 EDT by Rachana Patel
Modified: 2015-04-20 07:57 EDT (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.8rhs-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-23 18:39:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rachana Patel 2013-05-13 05:08:04 EDT
Description of problem:
glusterd: 'gluster peer status' is not showing all peers on few nodes of cluster - peer is in 'Connected to Peer' or ' Probe Sent to Peer' state from almost 2 hours

Version-Release number of selected component (if applicable):
3.4.0.6rhs-1.el6rhs.x86_64


How reproducible:


Steps to Reproduce:
1.had a cluster of 4 RHS nodes and all had same rpm - 3.4.0.6rhs-1.el6rhs.x86_64 and same RHS (RHS-2.1-20131510)
[root@fred ~]# gluster peer status
Number of Peers: 3

Hostname: cutlass.lab.eng.blr.redhat.com
Port: 24007
Uuid: dd350929-45c7-484e-8254-c2f3439c7052
State: Peer in Cluster (Disconnected)

Hostname: fan.lab.eng.blr.redhat.com
Uuid: 92377c1e-cae1-493d-a0dc-7dc030fb10c6
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Uuid: e9bec39c-e388-4ce7-a597-027fa372f7f3
State: Peer in Cluster (Connected)



2. on one server(cutlass) re-install RHS (RHS-2.1-20131510) so detach that node from cluster using force and confirm the same

[root@fred ~]# gluster peer detach cutlass.lab.eng.blr.redhat.com force
peer detach: success
[root@fred ~]# gluster peer status
Number of Peers: 2

Hostname: fan.lab.eng.blr.redhat.com
Uuid: 92377c1e-cae1-493d-a0dc-7dc030fb10c6
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Uuid: e9bec39c-e388-4ce7-a597-027fa372f7f3
State: Peer in Cluster (Connected)

3.after installing RHS on that node upgrade rpm to rpm - 3.4.0.6rhs-1.el6rhs.x86_64

4. from cluster peer probe that RHS server again
[root@fred ~]# gluster peer probe cutlass.lab.eng.blr.redhat.com

it didn't say success or fail

5. execute 'gluster peer status' on all server to verify

mia:-

[root@mia ~]# hostname
mia.lab.eng.blr.redhat.com
[root@mia ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.34.80
Uuid: c35f2e3c-2c1d-4ad7-acfd-9f68f5f69466
State: Peer in Cluster (Connected)

Hostname: fan.lab.eng.blr.redhat.com
Uuid: 92377c1e-cae1-493d-a0dc-7dc030fb10c6
State: Peer in Cluster (Connected)
[root@mia ~]# rpm -qa | grep gluster
gluster-swift-doc-1.4.8-4.el6.noarch
glusterfs-fuse-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.6rhs-1.el6rhs.x86_64
gluster-swift-plugin-1.0-5.noarch
gluster-swift-object-1.4.8-4.el6.noarch
gluster-swift-container-1.4.8-4.el6.noarch
glusterfs-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.6rhs-1.el6rhs.x86_64
vdsm-gluster-4.10.2-4.0.qa5.el6rhs.noarch
gluster-swift-proxy-1.4.8-4.el6.noarch
gluster-swift-account-1.4.8-4.el6.noarch
glusterfs-devel-3.4.0.6rhs-1.el6rhs.x86_64
gluster-swift-1.4.8-4.el6.noarch
[root@mia ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.34.80
Uuid: c35f2e3c-2c1d-4ad7-acfd-9f68f5f69466
State: Peer in Cluster (Connected)

Hostname: fan.lab.eng.blr.redhat.com
Uuid: 92377c1e-cae1-493d-a0dc-7dc030fb10c6
State: Peer in Cluster (Connected)


fred:-

[root@fred ~]# hostname
fred.lab.eng.blr.redhat.com
[root@fred ~]# gluster peer status
Number of Peers: 3

Hostname: fan.lab.eng.blr.redhat.com
Uuid: 92377c1e-cae1-493d-a0dc-7dc030fb10c6
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Uuid: e9bec39c-e388-4ce7-a597-027fa372f7f3
State: Peer in Cluster (Connected)

Hostname: cutlass.lab.eng.blr.redhat.com
Port: 24007
Uuid: 8e8fd82c-31fb-4a8c-93db-827a5e9ce289
State: Probe Sent to Peer (Connected)
[root@fred ~]# rpm -qa | grep gluster
gluster-swift-doc-1.4.8-4.el6.noarch
glusterfs-fuse-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.6rhs-1.el6rhs.x86_64
gluster-swift-plugin-1.0-5.noarch
gluster-swift-object-1.4.8-4.el6.noarch
gluster-swift-container-1.4.8-4.el6.noarch
glusterfs-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.6rhs-1.el6rhs.x86_64
vdsm-gluster-4.10.2-4.0.qa5.el6rhs.noarch
gluster-swift-proxy-1.4.8-4.el6.noarch
gluster-swift-account-1.4.8-4.el6.noarch
glusterfs-devel-3.4.0.6rhs-1.el6rhs.x86_64
gluster-swift-1.4.8-4.el6.noarch
[root@fred ~]# gluster peer status
Number of Peers: 3

Hostname: fan.lab.eng.blr.redhat.com
Uuid: 92377c1e-cae1-493d-a0dc-7dc030fb10c6
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Uuid: e9bec39c-e388-4ce7-a597-027fa372f7f3
State: Peer in Cluster (Connected)

Hostname: cutlass.lab.eng.blr.redhat.com
Port: 24007
Uuid: 8e8fd82c-31fb-4a8c-93db-827a5e9ce289
State: Probe Sent to Peer (Connected)


fan:-
[root@fan ~]# hostname
fan.lab.eng.blr.redhat.com
[root@fan ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.34.80
Uuid: c35f2e3c-2c1d-4ad7-acfd-9f68f5f69466
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Uuid: e9bec39c-e388-4ce7-a597-027fa372f7f3
State: Peer in Cluster (Connected)
[root@fan ~]# rpm -qa | grep gluster
gluster-swift-doc-1.4.8-4.el6.noarch
glusterfs-fuse-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.6rhs-1.el6rhs.x86_64
gluster-swift-plugin-1.0-5.noarch
gluster-swift-object-1.4.8-4.el6.noarch
gluster-swift-container-1.4.8-4.el6.noarch
glusterfs-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.6rhs-1.el6rhs.x86_64
vdsm-gluster-4.10.2-4.0.qa5.el6rhs.noarch
gluster-swift-proxy-1.4.8-4.el6.noarch
gluster-swift-account-1.4.8-4.el6.noarch
glusterfs-devel-3.4.0.6rhs-1.el6rhs.x86_64
gluster-swift-1.4.8-4.el6.noarch
[root@fan ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.34.80
Uuid: c35f2e3c-2c1d-4ad7-acfd-9f68f5f69466
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Uuid: e9bec39c-e388-4ce7-a597-027fa372f7f3
State: Peer in Cluster (Connected)


cutlass:-

[root@cutlass ~]# hostname
cutlass.lab.eng.blr.redhat.com
[root@cutlass ~]# gluster peer status
Number of Peers: 1

Hostname: 10.70.34.80
Port: 24007
Uuid: c35f2e3c-2c1d-4ad7-acfd-9f68f5f69466
State: Connected to Peer (Connected)
[root@cutlass ~]# rpm -qa | grep gluster
gluster-swift-doc-1.4.8-4.el6.noarch
glusterfs-fuse-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.6rhs-1.el6rhs.x86_64
gluster-swift-plugin-1.0-5.noarch
gluster-swift-object-1.4.8-4.el6.noarch
gluster-swift-container-1.4.8-4.el6.noarch
glusterfs-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.6rhs-1.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.6rhs-1.el6rhs.x86_64
vdsm-gluster-4.10.2-4.0.qa5.el6rhs.noarch
gluster-swift-proxy-1.4.8-4.el6.noarch
gluster-swift-account-1.4.8-4.el6.noarch
glusterfs-devel-3.4.0.6rhs-1.el6rhs.x86_64
gluster-swift-1.4.8-4.el6.noarch

[root@cutlass ~]# gluster peer status
Number of Peers: 1

Hostname: 10.70.34.80
Port: 24007
Uuid: c35f2e3c-2c1d-4ad7-acfd-9f68f5f69466
State: Connected to Peer (Connected)


  
Actual results:
'gluster peer status' is not showing all peers on few nodes of cluster - peer is in 'Connected to Peer' or ' Probe Sent to Peer' state from almost 2 hours
Comment 2 Rachana Patel 2013-05-14 05:56:50 EDT
also coming in -3.4.0.7rhs-1.el6.x86_64

steps:-

1. had a cluster pr four peer

[root@cutlass peers]# gluster peer status
Number of Peers: 3

Hostname: 10.70.34.80
Port: 24007
Uuid: daa7bdb9-de87-4b6e-9f86-e8bff3d47fc0
State: Peer in Cluster (Connected)

Hostname: fan.lab.eng.blr.redhat.com
Uuid: fdcb0533-eeb3-4054-8265-26558e92e65a
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Uuid: d665808d-a42a-4eac-bf05-ca53c595486d
State: Peer in Cluster (Connected)

2.detach one peer from cluster.
[root@fred peers]# gluster peer detach cutlass.lab.eng.blr.redhat.com
peer detach: success
[root@fred peers]# gluster peer status
Number of Peers: 2

Hostname: fan.lab.eng.blr.redhat.com
Port: 24007
Uuid: fdcb0533-eeb3-4054-8265-26558e92e65a
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Port: 24007
Uuid: d665808d-a42a-4eac-bf05-ca53c595486d
State: Peer in Cluster (Connected)


3. after its detached successful, peer probe that server again

fan:-
[root@fan peers]#  gluster peer probe cutlass.lab.eng.blr.redhat.com
[root@fan peers]# gluster peer status
Number of Peers: 3

Hostname: 10.70.34.80
Port: 24007
Uuid: daa7bdb9-de87-4b6e-9f86-e8bff3d47fc0
State: Peer in Cluster (Connected)

Hostname: mia.lab.eng.blr.redhat.com
Uuid: d665808d-a42a-4eac-bf05-ca53c595486d
State: Peer in Cluster (Connected)

Hostname: cutlass.lab.eng.blr.redhat.com
Port: 24007
Uuid: 27860513-fee2-445d-9467-901a3a704d33
State: Probe Sent to Peer (Connected)

cutlass:-
[root@cutlass ~]# gluster peer status
Number of Peers: 1

Hostname: 10.70.34.91
Port: 24007
Uuid: fdcb0533-eeb3-4054-8265-26558e92e65a
State: Connected to Peer (Connected)

mia:-
[root@mia peers]# gluster peer status
Number of Peers: 2

Hostname: 10.70.34.80
Port: 24007
Uuid: daa7bdb9-de87-4b6e-9f86-e8bff3d47fc0
State: Peer in Cluster (Connected)

Hostname: fan.lab.eng.blr.redhat.com
Uuid: fdcb0533-eeb3-4054-8265-26558e92e65a
State: Peer in Cluster (Connected)
Comment 3 Amar Tumballi 2013-05-17 02:46:28 EDT
Updated from Kaushal, on RPM issue faced till 3.4.0.8rhs (which has all the
right fixes).

------------------
Hi all,
Another small update on the steps to be taken when updating from build 7 to any
newer releases.

1. Backup /var/lib/glusterd
2. Upgrade
3. Stop gluster
4. Restore /var/lib/glusterd
5. Delete the /var/lib/glusterd/options file if empty. This will be recreated
by glusterd.
6. Start gluster and continue with your testing.

The /var/lib/glusterd/options file being empty causes syncing problems on
glusterd restart. Build7 cleared this file. If you hadn't done any
server-quorum test with build7, this file is most probably still empty.

So, if anyone is facing any volume syncing issues, do step 5 and restart
glusterd.

Thanks,
Kaushal

----- Original Message -----
> From: "Kaushal M" <kaushal@redhat.com>
> To: storage-qa@redhat.com
> Sent: Wednesday, May 15, 2013 12:10:08 PM
> Subject: Re: Warning on upgrade from gluster v3.4.0.7 to v3.4.0.8
>
> A small clarification. The upgrade will not delete all the files in
> /var/lib/glusterd. Only some files/directories like glusterd.info and nfs
> directory can be deleted. This is due to a packaging bug in build 7, in
> which these files/directories were a part of the package itself.
> This may be avoided by uninstalling and installing, instead of and upgrade (I
> haven't tested this). But to be on the safer side, backup and restore the
> /var/lib/glusterd directory.
>
> - Kaushal
>
> ----- Original Message -----
>> From: "Kaushal M" <kaushal@redhat.com>
>> To: storage-qa@redhat.com
>> Sent: Wednesday, May 15, 2013 11:48:05 AM
>> Subject: Warning on upgrade from gluster v3.4.0.7 to v3.4.0.8
>>
>> Hi all,
>>
>> Because of bugs in packaging of build 7, an upgrade from build 7 tp build 8
>> will cause files /var/lib/glusterd/ to be deleted. As you can probably
>> guess
>> this will lead to all sorts of problems.
>> So, before upgrading, backup your /var/lib/glusterd directory. Follow the
>> below steps to make sure you don't break your existing setup,
>>
>> 1. Backup /var/lib/glusterd
>> 2. Upgrade
>> 3. Stop gluster
>> 4. Restore /var/lib/glusterd
>> 5. Start gluster and continue with your testing.
>>
>>
>> Regards,
>> Kaushal
>
Comment 4 Rachana Patel 2013-05-22 09:13:25 EDT
verified with 3.4.0.8rhs (not upgrade, remove old version and install new version)

not able to reproduce hence moving to verified
Comment 5 Scott Haines 2013-09-23 18:39:46 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html
Comment 6 Scott Haines 2013-09-23 18:43:48 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.