Bug 1218273

Summary:	[Tiering] : Attaching another node to the cluster which has a tiered volume times out
Product:	[Community] GlusterFS	Reporter:	senaik
Component:	tiering	Assignee:	Mohammed Rafi KC <rkavunga>
Status:	CLOSED CURRENTRELEASE	QA Contact:	bugs <bugs>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	amukherj, bugs, rkavunga, sankarshan
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-4.1.4	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1229266 (view as bug list)		Environment:
Last Closed:	2018-10-08 09:52:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1186580, 1229266, 1260923

Description senaik 2015-05-04 13:18:13 UTC

Description of problem:
=======================
Attaching another node to a cluster which has tiered volumes times out.


Version-Release number of selected component (if applicable):
=============================================================
glusterfs 3.7.0beta1 built on May  1 2015 

How reproducible:
================
3/3


Steps to Reproduce:
==================
1.Create a 4 node cluster

2.Create a tiered volume. Fuse and NFS mount the volume and create some IO

3.Attach another node to the cluster 

gluster peer probe  snapshot11.lab.eng.blr.redhat.com
Error : Request timed out

-----Part of glusterd log from  snapshot11.lab.eng.blr.redhat.com------------

[2015-05-04 12:42:45.846735] E [glusterd-svc-mgmt.c:29:glusterd_svc_create_rundir] 0-management: Unable to create rundir /var/lib/glusterd/vols/gluster_shared_storage/run
[2015-05-04 12:42:45.846797] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2015-05-04 12:42:45.849728] E [glusterd-utils.c:4273:glusterd_brick_start] 0-management: Could not find peer on which brick rhs-arch-srv2.lab.eng.blr.redhat.com:/rhs/brick8/b8 resides
[2015-05-04 12:42:45.849774] E [glusterd-op-sm.c:1840:glusterd_start_bricks] 0-management: Failed to start rhs-arch-srv2.lab.eng.blr.redhat.com:/rhs/brick8/b8 for gluster_shared_storage
[2015-05-04 12:42:46.259992] E [glusterd-svc-mgmt.c:29:glusterd_svc_create_rundir] 0-management: Unable to create rundir /var/lib/glusterd/vols/vol0/run
[2015-05-04 12:42:46.260124] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2015-05-04 12:42:46.262278] E [glusterd-utils.c:4273:glusterd_brick_start] 0-management: Could not find peer on which brick rhs-arch-srv4.lab.eng.blr.redhat.com:/rhs/brick4/b4 resides
[2015-05-04 12:42:46.262363] E [glusterd-op-sm.c:1840:glusterd_start_bricks] 0-management: Failed to start rhs-arch-srv4.lab.eng.blr.redhat.com:/rhs/brick4/b4 for vol0
[2015-05-04 12:42:46.262588] E [glusterd-volgen.c:959:volgen_write_volfile] 0-management: No such file or directory
[2015-05-04 12:42:46.262616] E [glusterd-volgen.c:1003:volgen_write_volfile] 0-management: failed to create volfile /var/lib/glusterd/vols/vol0/vol0-snapd.vol
[2015-05-04 12:42:46.262638] E [glusterd-snapd-svc.c:244:glusterd_snapdsvc_start] 0-management: Couldn't create snapd volfile for volume: vol0
[2015-05-04 12:42:47.493470] E [glusterd-volgen.c:4976:glusterd_create_volfiles] 0-management: Could not generate trusted client volfiles
[2015-05-04 12:42:47.493497] E [glusterd-sm.c:1224:glusterd_friend_sm] 0-glusterd: handler returned: -1

--------------------------------------------------------------------

[root@snapshot11 ~]# gluster peer status
Number of Peers: 1

Hostname: 10.70.34.50
Uuid: 1c1b2314-8bf5-4b64-9b58-7a5fd1eca21c
State: Connected to Peer (Connected)





[root@inception ~]# gluster peer status
Number of Peers: 4

Hostname: rhs-arch-srv2.lab.eng.blr.redhat.com
Uuid: 54cda762-d199-4ee4-be57-50a0b3313bcf
State: Peer in Cluster (Connected)

Hostname: rhs-arch-srv3.lab.eng.blr.redhat.com
Uuid: 419a4609-e9e8-4cc6-b1c5-58a5efaf3ecf
State: Peer in Cluster (Connected)

Hostname: rhs-arch-srv4.lab.eng.blr.redhat.com
Uuid: 961b6d3a-9534-4d28-a4c7-9c1d3296cd60
State: Peer in Cluster (Connected)

Hostname: snapshot11.lab.eng.blr.redhat.com
Uuid: 429914f4-486d-4eb7-ac27-593335d76592
State: Probe Sent to Peer (Connected)

Actual results:


Expected results:


Additional info:

Comment 2 senaik 2015-05-05 06:57:21 UTC

Scaling up the cluster is not successful when tiered volumes are present. 

Proposing this bug as a blocker

Comment 3 Mohammed Rafi KC 2015-05-06 10:37:45 UTC

Fixed as part of the following changes, 

http://review.gluster.org/#/c/10339/
http://review.gluster.org/#/c/10449/

Comment 4 Joseph Elwin Fernandes 2015-05-12 06:30:38 UTC

http://review.gluster.org/#/c/10339/ merged
http://review.gluster.org/#/c/10449/ needs review

Comment 6 Vivek Agarwal 2015-09-24 14:51:57 UTC

3.7 bug 1219846, patch review.gluster.org/10449

Comment 7 Amar Tumballi 2018-10-08 09:52:54 UTC

This bug was ON_QA status, and on GlusterFS product in bugzilla, we don't have that as a valid status. We are closing it as 'CURRENT RELEASE ' to indicate the availability of the fix, please reopen if found again.