Bug 1754483

Summary: Peer wrongly shown as Connected.
Product: [Community] GlusterFS Reporter: Pushkar <r41291>
Component: glusterdAssignee: bugs <bugs>
Status: CLOSED WORKSFORME QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 5CC: bugs, pasik, srakonde
Target Milestone: ---   
Target Release: ---   
Hardware: other   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-06 11:15:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Pushkar 2019-09-23 11:51:30 UTC
Description of problem:
We use glusterfs to sync configuration on active/standby servers (A/B).
So, we have 2 virtual machines running debian with glusterfs setup.
We want to support upgrade/revert for the vms.
Upgrade will involve shutting down vm, detaching existing vmdk disk image, loading another vmdk disk and starting up the vm.
Revert will involve shutting down the vm and simply reattaching the earlier used vmdk disk image.
Chnaging the vmdk disk image results in peer connected(rejected) which we are able to handle.
The issue is with revert when we bring back older vmdk image.


Version-Release number of selected component (if applicable):
5.3

How reproducible:
Always.
Reproducible on vmware, virtualbox.
Reproducible on glusterfs-3.8.8 and 5.3

Steps to Reproduce:
1. Setup replicated glusterfs on 2 virtual machines - A and B (vmware/virtualbox) using vmdk as disk format.
2. Do a peer probe and observe that they form a trusted pool and show as connected.
3. Turn off virtual machine B.
4. Remove the vmdk disk currently attached to B and add a new fresh vmdk image. Proceed to setup glusterfs on the newer setup.
5. On A, we now see B as connected(rejected).
6. On A, gluster detach B.
7. On A, gluster peer probe B, to again form a trusted pool.
8. Shutdown B, remove the curently attached vmdk disk image and reattach the older vmdk diskimage and start B.
9. Once B is up, A still shows B as in cluster and connected.
10. B correctly shows A as disconnected.
The IPs used by the vms do not change during the procedure.

Actual results:
A shows B as connected at the end of all the steps.

Expected results:
A should show B as disconnected/rejected at the end of all the steps.


Additional info:

===================================================
On A (after the final step):

pushkar@pushkar-VirtualBox1:~$ sudo gluster pool list
UUID					Hostname    	State
95455fbb-b26b-4071-b9a5-4f8f98decf12	10.70.52.128	Connected 
c1c2be16-6979-47e2-9560-0d02b19a3eef	localhost   	Connected 

pushkar@pushkar-VirtualBox1:~$ sudo gluster peer status
Number of Peers: 1

Hostname: 10.70.52.128
Uuid: 95455fbb-b26b-4071-b9a5-4f8f98decf12
State: Peer in Cluster (Connected)

===================================================
On B (after the final step):

pushkar@pushkar-VirtualBox2:~$ sudo gluster pool list
UUID					Hostname    	State
95455fbb-b26b-4071-b9a5-4f8f98decf12	10.70.52.128	Disconnected 
c1c2be16-6979-47e2-9560-0d02b19a3eef	10.70.52.86 	Disconnected 
536c921f-236d-4d28-a6b9-cd250c49b1f3	localhost   	Connected 

pushkar@pushkar-VirtualBox2:~$ sudo gluster peer status
Number of Peers: 2

Hostname: 10.70.52.128
Uuid: 95455fbb-b26b-4071-b9a5-4f8f98decf12
State: Peer in Cluster (Disconnected)

Hostname: 10.70.52.86
Uuid: c1c2be16-6979-47e2-9560-0d02b19a3eef
State: Peer in Cluster (Disconnected)

===================================================

I am aware that replacing the vmdk disk images results in A and B going out of sync of the knowledge they have of their peers.
But should not A show B as not connected/rejected?

Comment 1 Sanju 2019-12-05 10:18:57 UTC
I think it is a firewall issue, can you please verify?

Comment 2 Sanju 2020-01-06 11:15:11 UTC
Closing this bug. If you hit this issue again, please feel free to re-open.