Bug 1234725 - [New] - Bricks fail to restore when a new node is added to the cluster and rebooted when having management and data on two different interfaces
Summary: [New] - Bricks fail to restore when a new node is added to the cluster and re...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: RHGS 3.1.0
Assignee: Kaushal
QA Contact: RamaKasturi
URL:
Whiteboard: GlusterD
Depends On: 1234842 1234843 1241882
Blocks: 1202842
TreeView+ depends on / blocked
 
Reported: 2015-06-23 06:49 UTC by RamaKasturi
Modified: 2016-09-17 14:42 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.7.1-9
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-07-29 05:06:52 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:1495 0 normal SHIPPED_LIVE Important: Red Hat Gluster Storage 3.1 update 2015-07-29 08:26:26 UTC

Description RamaKasturi 2015-06-23 06:49:47 UTC
Description of problem:
Had a cluster with two nodes in a cluster with two interfaces one for management and one for data.

gluster peer status from node1:
=================================
Hostname: rhs-client33.lab.eng.blr.redhat.com
Uuid: f4ed464c-ce35-4a4c-865c-b88c1c5852ae
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

gluster peer status from node2:
===============================
Hostname: 10.70.36.48
Uuid: c7f020c3-f360-43e3-a0c7-655a75683021
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Now added a third node to the cluster which has two interfaces one for management and one for data.

gluster peer status from node1 after adding third node:
======================================================
[root@rhs-client24 ~]# gluster peer status
Number of Peers: 2

Hostname: rhs-client33.lab.eng.blr.redhat.com
Uuid: f4ed464c-ce35-4a4c-865c-b88c1c5852ae
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

Hostname: 10.70.37.45
Uuid: 902c46ce-e6ce-4e64-b9b0-40d6b26cbce8
State: Peer in Cluster (Connected)
Other names:
10.70.37.110

gluster peer status from node2 after adding third node:
======================================================
[root@rhs-client33 ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.36.48
Uuid: c7f020c3-f360-43e3-a0c7-655a75683021
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Hostname: 10.70.37.45
Uuid: 902c46ce-e6ce-4e64-b9b0-40d6b26cbce8
State: Peer in Cluster (Connected)
Other names:
10.70.37.110

gluster peer status on third node:
====================================
[root@dhcp37-45 ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.36.48
Uuid: c7f020c3-f360-43e3-a0c7-655a75683021
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Hostname: rhs-client33.lab.eng.blr.redhat.com
Uuid: f4ed464c-ce35-4a4c-865c-b88c1c5852ae
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

Now Rebooted the third node.Below is the gluster peer status from all the three nodes:

Node1:
==============
[root@rhs-client24 ~]# gluster peer status
Number of Peers: 2

Hostname: rhs-client33.lab.eng.blr.redhat.com
Uuid: f4ed464c-ce35-4a4c-865c-b88c1c5852ae
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

Hostname: 10.70.37.45
Uuid: 902c46ce-e6ce-4e64-b9b0-40d6b26cbce8
State: Peer in Cluster (Disconnected)
Other names:
10.70.37.110

Node2:
==============
[root@rhs-client33 ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.36.48
Uuid: c7f020c3-f360-43e3-a0c7-655a75683021
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Hostname: 10.70.37.45
Uuid: 902c46ce-e6ce-4e64-b9b0-40d6b26cbce8
State: Peer in Cluster (Disconnected)
Other names:
10.70.37.110

Node3:
==============================
 gluster peer status
Connection failed. Please check if gluster daemon is operational.
peer status: failed

glusterd fails to start because it could not resolve the bricks.

peer info from both the nodes node1 and node2:
===============================================

cat c7f020c3-f360-43e3-a0c7-655a75683021 
uuid=c7f020c3-f360-43e3-a0c7-655a75683021
state=3
hostname1=10.70.36.48

cat f4ed464c-ce35-4a4c-865c-b88c1c5852ae
uuid=f4ed464c-ce35-4a4c-865c-b88c1c5852ae
state=3
hostname1=rhs-client33.lab.eng.blr.redhat.com
hostname2=10.70.33.221


Version-Release number of selected component (if applicable):
glusterfs-3.7.1-4.el6rhs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Add two nodes in a cluster with two interfaces one for data and one for mgmt.
2. Now add a third node to the cluster with two interfaces one for data and one for mgmt
3. once the peer probe is done successfully, reboot the third node.


Actual results:
glusterd fails to start on third node and logs says that brick restore fails.

Expected results:
glusterd should not fail to start and brick should be resolved correctly.

Additional info:

Comment 6 Kaushal 2015-06-23 11:21:47 UTC
This is caused because glusterd wasn't saving the peerinfo data after getting an update on it. This was a regression introduced with the multi-network peer probe support which came in with  better peer identification changes.

Comment 8 Atin Mukherjee 2015-06-24 03:51:53 UTC
Upstream patch http://review.gluster.org/#/c/11365/ posted for review

Comment 10 Kaushal 2015-06-25 06:01:40 UTC
Backported change on review on https://code.engineering.redhat.com/gerrit/51547

Comment 11 RamaKasturi 2015-07-09 09:57:07 UTC
I tried verifying this bug but i am still able to reproduce the issue. Below are the steps i performed:

Had a cluster with two nodes in a cluster with two interfaces one for management and one for data.

gluster peer status from node1:
=================================
gluster peer status
Number of Peers: 1

Hostname: 10.70.36.57
Uuid: e31034cd-75e0-49aa-8b40-362f7e94dd3d
State: Peer in Cluster (Connected)
Other names:
10.70.33.221


gluster peer status from node2:
===============================
gluster peer status
Number of Peers: 1

Hostname: 10.70.36.48
Uuid: f823b629-4f70-4fdb-84dc-dfd900aea8b7
State: Peer in Cluster (Connected)
Other names:
10.70.33.238


Now added a third node to the cluster which has two interfaces one for management and one for data.

gluster peer status from node1 after adding third node:
======================================================
gluster peer status
Number of Peers: 2

Hostname: 10.70.36.57
Uuid: e31034cd-75e0-49aa-8b40-362f7e94dd3d
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

Hostname: 10.70.37.87
Uuid: 6d64313d-dc09-4de5-8c81-35a9b01a6082
State: Peer in Cluster (connected)
Other names:
10.70.37.200


gluster peer status from node2 after adding third node:
======================================================
gluster peer status
Number of Peers: 2

Hostname: 10.70.36.59
Uuid: f823b629-4f70-4fdb-84dc-dfd900aea8b7
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Hostname: 10.70.37.87
Uuid: 6d64313d-dc09-4de5-8c81-35a9b01a6082
State: Peer in Cluster (Connected)
Other names:
10.70.37.200


gluster peer status on third node:
====================================
[root@dhcp37-45 ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.36.48
Uuid: c7f020c3-f360-43e3-a0c7-655a75683021
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Hostname: rhs-client33.lab.eng.blr.redhat.com
Uuid: f4ed464c-ce35-4a4c-865c-b88c1c5852ae
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

Now Rebooted the third node.Below is the gluster peer status from all the three nodes:

Node1:
==============
[root@rhs-client24 ~]# 
gluster peer status
Number of Peers: 2

Hostname: 10.70.36.57
Uuid: e31034cd-75e0-49aa-8b40-362f7e94dd3d
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

Hostname: 10.70.37.87
Uuid: 6d64313d-dc09-4de5-8c81-35a9b01a6082
State: Peer in Cluster (Disconnected)
Other names:


Node2:
==============
[root@rhs-client33 ~]#
gluster peer status
Number of Peers: 2

Hostname: 10.70.36.59
Uuid: f823b629-4f70-4fdb-84dc-dfd900aea8b7
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Hostname: 10.70.37.87
Uuid: 6d64313d-dc09-4de5-8c81-35a9b01a6082
State: Peer in Cluster (Disconnected)
Other names:
10.70.37.200


Node3:
==============================
 gluster peer status
Connection failed. Please check if gluster daemon is operational.
peer status: failed

glusterd fails to start because it could not resolve the bricks.

snippet from  /var/log/glusterfs logs.
[2015-07-09 09:50:29.338782] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-07-09 09:50:58.383097] E [glusterd-op-sm.c:254:glusterd_get_txn_opinfo] (--> 0-management: Unable to get transaction opinfo for transaction ID : 56ce6ad6-1c64-4019-b251-7f1b0da5038b
[2015-07-09 09:51:05.585444] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (15), shutting down
[2015-07-09 09:51:47.856326] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2015-07-09 09:51:47.966616] I [MSGID: 106478] [glusterd.c:1377:init] 0-management: Maximum allowed open file descriptors set to 65536
[2015-07-09 09:51:47.966804] I [MSGID: 106479] [glusterd.c:1426:init] 0-management: Using /var/lib/glusterd as working directory
[2015-07-09 09:51:48.088966] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2015-07-09 09:51:48.089028] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device
[2015-07-09 09:51:48.089042] W [rpc-transport.c:358:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2015-07-09 09:51:48.089129] W [rpcsvc.c:1595:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-07-09 09:51:48.089155] E [MSGID: 106243] [glusterd.c:1656:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2015-07-09 09:51:53.439596] I [MSGID: 106513] [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30702
[2015-07-09 09:51:53.440563] I [MSGID: 106544] [glusterd.c:157:glusterd_uuid_init] 0-management: retrieved UUID: 6d64313d-dc09-4de5-8c81-35a9b01a6082
[2015-07-09 09:51:53.440791] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2015-07-09 09:51:53.442081] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2015-07-09 09:51:53.445742] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2015-07-09 09:51:53.446831] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2015-07-09 09:51:53.447506] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2015-07-09 09:51:53.457560] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2015-07-09 09:51:53.581465] I [MSGID: 106498] [glusterd-handler.c:3556:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-07-09 09:51:53.584863] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-07-09 09:51:53.597092] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-07-09 09:51:53.584729] I [MSGID: 106498] [glusterd-handler.c:3556:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-07-09 09:51:53.604496] E [MSGID: 106187] [glusterd-store.c:4253:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
[2015-07-09 09:51:53.604585] E [MSGID: 101019] [xlator.c:428:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2015-07-09 09:51:53.604612] E [MSGID: 101066] [graph.c:326:glusterfs_graph_init] 0-management: initializing translator failed
[2015-07-09 09:51:53.604641] E [MSGID: 101176] [graph.c:672:glusterfs_graph_activate] 0-graph: init failed
[2015-07-09 09:51:53.607544] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (0), shutting down

Comment 12 RamaKasturi 2015-07-10 07:07:05 UTC
Moving this back to assigned as i am still facing the issue.

Comment 13 RamaKasturi 2015-07-10 07:07:42 UTC
Moving this back to assigned as i am still facing the issue.

Comment 14 Kaushal 2015-07-10 10:05:45 UTC
This is caused by race which hadn't been encountered before. The analysis is as follows.

Assuming A, B and C as the peers. A and B are a cluster and have probed each other on the 2 networks. C is probed from A.

During the probe, C is first validate by A. Once C is accepted, A sends and update to both B and C to inform them of the each other. The update C gets from A doesn't have A's second network information. C can only get this information when B sends an update to C.

The problem faced here was that B didn't send an update to C. This happens because B sending an update to C depends on the ordering of connection establishment between B and C.

B and C both try to establish connections to each other once they receive A's update and get to know of each other. If B establishes the connection first then it sends and update to C. But if C establishes the connection first, B will not send an update to C.

This is the first time this situation was observed. This doesn't happen always.

Comment 15 Kaushal 2015-07-11 07:20:10 UTC
Downstream change posted for review at https://code.engineering.redhat.com/gerrit/52811

Comment 16 Kaushal 2015-07-11 09:24:57 UTC
Correct change is https://code.engineering.redhat.com/gerrit/52812

Comment 18 RamaKasturi 2015-07-14 06:29:25 UTC
Verified and works fine with build glusterfs-3.7.1-9.el6rhs.x86_64.

Followed the steps below to verify:
===================================
1) Added two nodes to the cluster with separate mgmt and data traffic.
2) Created a volume with the two nodes added to the cluster.
3) Now added a third node to the cluster and rebooted the node.

Third node comes back up successfully and there is no host which is in disconnected state in the cluster.

Peer status from node1:
=======================================
[root@rhs-client24 ~]# gluster peer status
Number of Peers: 2

Hostname: rhs-client33.lab.eng.blr.redhat.com
Uuid: e31034cd-75e0-49aa-8b40-362f7e94dd3d
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

Hostname: 10.70.37.87
Uuid: 6d64313d-dc09-4de5-8c81-35a9b01a6082
State: Peer in Cluster (Connected)
Other names:
10.70.37.200

peer status from node2:
===========================================
[root@rhs-client33 ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.36.48
Uuid: f823b629-4f70-4fdb-84dc-dfd900aea8b7
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Hostname: 10.70.37.87
Uuid: 6d64313d-dc09-4de5-8c81-35a9b01a6082
State: Peer in Cluster (Connected)
Other names:
10.70.37.200

peer status from node3:
=================================================
[root@dhcp37-87 ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.36.48
Uuid: f823b629-4f70-4fdb-84dc-dfd900aea8b7
State: Peer in Cluster (Connected)
Other names:
10.70.33.238

Hostname: rhs-client33.lab.eng.blr.redhat.com
Uuid: e31034cd-75e0-49aa-8b40-362f7e94dd3d
State: Peer in Cluster (Connected)
Other names:
10.70.33.221

Above tested scenario does not work if there are any snapshots in the cluster. Bug has been logged for the same. https://bugzilla.redhat.com/show_bug.cgi?id=1238111

Comment 19 errata-xmlrpc 2015-07-29 05:06:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1495.html


Note You need to log in before you can comment on or make changes to this bug.