Bug 1756704

Summary:	Peer Rejected (Connected) after instance recreation
Product:	[Community] GlusterFS	Reporter:	Alex <alexettelis>
Component:	glusterd	Assignee:	Sanju <srakonde>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6	CC:	bugs, pasik, srakonde
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-09-30 04:26:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alex 2019-09-29 08:28:00 UTC

Description of problem:
When an instance dies in GCP, it is recreated with the same, but different boot disk and different data drive. The other node see that node as Peer Rejected (Connected)

Version-Release number of selected component (if applicable):

Steps to Reproduce:
1. Create 3 node cluster
2. Recreate one of the instance (new boot and data disk) with the same name, IP
3. Run gluster peer status


Actual results:
On old nodes: Peer Rejected (Connected)
On new node: no peers

Expected results:
Every peer should be connected


Additional info:
The UUID of the new node is changed.
I tried editing glusterd.info with the new node UUID. 
I tried restarting all of the nodes, deleting data on the broken node.

Comment 1 Sanju 2019-09-30 04:26:59 UTC

This is not a bug rather a an expected behaviour. To bring back your cluster to a normal state, you can edit the UUID of the node in /var/lib/glusterd.info to old UUID and copy /var/lib/glusterd/ contents from a good node to re-created node. You also need to delete the information related to own node from /var/lib/glusterd/peers/ and add data related to the peer from which the data is copied.

HTH,
Sanju

Comment 2 Sanju 2019-09-30 04:28:21 UTC

Once the data is copied you need to restart the glusterd on re-created node.

Comment 3 Alex 2019-10-02 05:16:04 UTC

Thank you so much Sanju,  this has recovered the state!