Bug 1356050

Summary:	brick dir permission changed after node reboot
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Neha <nerawat>
Component:	rhgs-server-container	Assignee:	Humble Chirammal <hchiramm>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Sachin <sashinde>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	rhgs-3.1	CC:	amainkar, hchiramm, lpabon, mliyazud, mzywusko, rcyriac, rhs-bugs, sankarshan, sashinde
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS Container Converged 1.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rhgs-server-docker-3.1.3-12	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-10-14 13:44:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1332128

Description Neha 2016-07-13 10:30:11 UTC

By default during volume creation all dirs are created with permission drwxr-xr-x. 


After rebooting one of the RHGS nodes

drw-------. 3 root root 23 Jul 13 05:26 /var/lib/heketi/mounts/vg_5cfcc9f2d07deb5e47fc6757fa026814/brick_e84eeb7bde5c91f6099e599780f8d1de/brick/


permission have changed for the brick dir


Also status of process:

sh-4.2# gluster v status
Status of volume: heketidbstorage
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.42.101:/var/lib/heketi/mounts/v
g_3f822ad7b9165474b6aae796e9af1cd5/brick_fc
3d86440289b7324e68c5dee821ae53/brick        49166     0          Y       6212 
Brick 10.70.43.26:/var/lib/heketi/mounts/vg
_0865fe8d497b8f2189e81a68645093b4/brick_2a4
266fde07828b776c9e216d68b9c96/brick         49166     0          Y       309  
Brick 10.70.43.30:/var/lib/heketi/mounts/vg
_2507b80a719f0e15930a7c114bd2f3cd/brick_dd3
ade914a1c347833317de12f7ee0fa/brick         49166     0          Y       306  
Brick 10.70.43.30:/var/lib/heketi/mounts/vg
_2507b80a719f0e15930a7c114bd2f3cd/brick_bf7
ed7e9f49706ca10d239a0fdc8be20/brick         N/A       N/A        N       N/A  
Brick 10.70.43.26:/var/lib/heketi/mounts/vg
_5cfcc9f2d07deb5e47fc6757fa026814/brick_e84
eeb7bde5c91f6099e599780f8d1de/brick         N/A       N/A        N       N/A  
Brick 10.70.42.101:/var/lib/heketi/mounts/v
g_41e96e9310ac486d6f49849cda78ad07/brick_44
accff87ae66310b0c8f230dfc31367/brick        49167     0          Y       6231 
NFS Server on localhost                     2049      0          Y       330  
Self-heal Daemon on localhost               N/A       N/A        Y       338  
NFS Server on 10.70.43.30                   2049      0          Y       330  
Self-heal Daemon on 10.70.43.30             N/A       N/A        Y       338  
NFS Server on 10.70.42.101                  2049      0          Y       7075 
Self-heal Daemon on 10.70.42.101            N/A       N/A        Y       7083 
 
Task Status of Volume heketidbstorage
------------------------------------------------------------------------------
There are no active volume tasks

Comment 2 Neha 2016-07-13 10:33:52 UTC

Also during initial volume creation I dont see any "brick" dir under brick path on host system, though its there in container.

But after node reboot it has created empty brick dir under host system.

ls -lR
.:
total 0
drwxr-xr-x. 3 root root 18 Jul 13 14:56 brick_22c7313b5da3e04aeb5a6ced1be31f5a
drwxr-xr-x. 2 root root  6 Jul 13 14:43 brick_2a4266fde07828b776c9e216d68b9c96

./brick_22c7313b5da3e04aeb5a6ced1be31f5a:
total 0
drw-------. 3 root root 23 Jul 13 14:56 brick

./brick_22c7313b5da3e04aeb5a6ced1be31f5a/brick:
total 0

./brick_2a4266fde07828b776c9e216d68b9c96:
total 0

Not true for all the brick paths.

Comment 3 Luis Pabón 2016-07-13 12:55:51 UTC

Humble, could you look at this?

Comment 4 Humble Chirammal 2016-07-14 18:40:13 UTC

@neha, 

*) Is the permission change happening for all the bricks or only on some of them? 

*) Have you noticed this behaviour for any other volume other than 'heketidbstorage' ?

Comment 5 Neha 2016-07-15 04:02:07 UTC

(In reply to Humble Chirammal from comment #4)
> @neha, 
> 
> *) Is the permission change happening for all the bricks or only on some of
> them? 

some of them , not for all the bricks.

> 
> *) Have you noticed this behaviour for any other volume other than
> 'heketidbstorage' ?

Yes I noticed it for other volumes also.

Comment 6 Humble Chirammal 2016-07-16 09:09:51 UTC

@Neha, Can you provide youe setup details ? so that we can do further analysis on the same?

Comment 10 Neha 2016-07-20 09:37:44 UTC

Humble,

I believe this is the same image we tested yesterday and facing issues.

Comment 12 Humble Chirammal 2016-07-20 13:53:32 UTC

(In reply to Neha from comment #10)
> Humble,
> 
> I believe this is the same image we tested yesterday and facing issues.

@Neha, we couldnt properly test the image due to the setup issue. Any way the root cause of the bug is available here https://bugzilla.redhat.com/show_bug.cgi?id=1355801#c21 and we have to open a gluster bug as mentioned in the bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1355801#c30. However the triggering point of this reported issue is same as https://bugzilla.redhat.com/show_bug.cgi?id=1355801. We have made some changes in "heketi" via https://bugzilla.redhat.com/show_bug.cgi?id=1356065 and in RHGS container image via  https://bugzilla.redhat.com/show_bug.cgi?id=1355801 to resolve this issue.

Comment 13 Neha 2016-07-23 11:39:09 UTC

Don't see this issue with latest build.

rhgs-server-docker-3.1.3-13
rhgs3-rhgs-volmanager-rhel7-3.1.3-12

Moving it to verified.