Bug 1410091

Summary:	seeing weird or junk owner and group id when doing IOs from multiple ganesha clients on disperse volume
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Nag Pavan Chilakam <nchilaka>
Component:	nfs-ganesha	Assignee:	Kaleb KEITHLEY <kkeithle>
Status:	CLOSED WONTFIX	QA Contact:	Manisha Saini <msaini>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	rhgs-3.2	CC:	arjsharm, jthottan, kkeithle, pasik, rhs-bugs, skoduri, storage-qa-internal
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-06 11:51:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nag Pavan Chilakam 2017-01-04 12:41:06 UTC

Description of problem:
========================
Note:this is inconsistent, but I have seen this atleast 3 times.
I am finding a way to reproduce

I was creating files on ganesha mount say c1 ( by scp'ing files from another machine into this)
From another client c2 i was issuing ll and sometime making file modifications like changing permissions

I observed that c2 was always seeing all the files(irrespective of which client created) owner/group  as root/root 
However, I saw that c1 was publishing weird number as below for the files modified or created by c2 as below (owner groupid is changed to 4294967294)


c1:
===
[root@rhs-client45 dir45]# ll
total 25463910
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d1
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d10
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d2
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d3
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d4
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d5
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d6
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d7
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d8
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 d9
drwxr-xr-x. 2 4294967294 4294967294       4096 Jan  4  2017 dir107
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:43 file.006
-rwxrwxrwx. 1 root       root       2500000256 Jan  4  2017 file.1
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:48 file.10
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:48 file.11
-rwxrwxrwx. 1 root       root       1121507540 Jan  4  2017 file.12
-rwxrwxrwx. 1 root       root       1121507540 Jan  4  2017 file.13
-rwxrwxrwx. 1 root       root       1121507540 Jan  4  2017 file.14
-rwxrwxrwx. 1 root       root       1121507540 Jan  4  2017 file.15
-rwxrwxrwx. 1 root       root       1121507540 Jan  4  2017 file.16
-rw-r--r--. 1 root       root       1121507540 Jan  4  2017 file.17
-rw-r--r--. 1 4294967294 4294967294 1121507540 Jan  4  2017 file.18
-rw-r--r--. 1 4294967294 4294967294 1121507540 Jan  4  2017 file.19
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:44 file.2
-rw-r--r--. 1 4294967294 4294967294          0 Jan  4  2017 file.20
-rw-r--r--. 1 4294967294 4294967294 1121507540 Jan  4  2017 file.21
-rw-r--r--. 1 4294967294 4294967294 1121507540 Jan  4  2017 file.22
-rw-r--r--. 1 4294967294 4294967294 1121507540 Jan  4  2017 file.23
-rw-r--r--. 1 4294967294 4294967294 1121507540 Jan  4  2017 file.24
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:45 file.3
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:45 file.4
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:46 file.5
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:46 file.6
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:46 file.7
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:47 file.8
-rwxrwxrwx. 1 root       root       1121507540 Jan  4 17:47 file.9
-rw-r--r--. 1 4294967294 4294967294          3 Jan  4  2017 sample



c2:
==
root@dhcp35-107 dir45]# ll
total 21630686
-rw-r--r--. 1 root root           0 Jan  4 17:52 d1
-rw-r--r--. 1 root root           0 Jan  4 17:52 d10
-rw-r--r--. 1 root root           0 Jan  4 17:52 d2
-rw-r--r--. 1 root root           0 Jan  4 17:52 d3
-rw-r--r--. 1 root root           0 Jan  4 17:52 d4
-rw-r--r--. 1 root root           0 Jan  4 17:52 d5
-rw-r--r--. 1 root root           0 Jan  4 17:52 d6
-rw-r--r--. 1 root root           0 Jan  4 17:52 d7
-rw-r--r--. 1 root root           0 Jan  4 17:52 d8
-rw-r--r--. 1 root root           0 Jan  4 17:52 d9
drwxr-xr-x. 2 root root        4096 Jan  4 17:51 dir107
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:43 file.006
-rwxrwxrwx. 1 root root 10000000000 Jan  4 17:53 file.1
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:48 file.10
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:48 file.11
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:49 file.12
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:49 file.13
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:49 file.14
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:50 file.15
-rwxrwxrwx. 1 root root   280377344 Jan  4 17:50 file.16
-rw-r--r--. 1 root root   280377344 Jan  4 17:51 file.17
-rw-r--r--. 1 root root  1121507540 Jan  4 17:51 file.18
-rw-r--r--. 1 root root  1121507540 Jan  4 17:52 file.19
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:44 file.2
-rw-r--r--. 1 root root           0 Jan  4 17:53 file.20
-rw-r--r--. 1 root root   280377344 Jan  4 17:52 file.21
-rw-r--r--. 1 root root  1121507540 Jan  4 17:53 file.22
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:45 file.3
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:45 file.4
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:46 file.5
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:46 file.6
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:46 file.7
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:47 file.8
-rwxrwxrwx. 1 root root  1121507540 Jan  4 17:47 file.9



However after some time the issue fades away, assuming it to be the invalidation of mdcache timing(ie 10min)

[root@dhcp35-37 ~]# gluster v info disperse
 
Volume Name: disperse
Type: Distributed-Disperse
Volume ID: ea0b329a-f2ff-462d-a3a2-457a80b35e08
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.35.37:/rhs/brick2/disperse
Brick2: 10.70.35.116:/rhs/brick2/disperse
Brick3: 10.70.35.239:/rhs/brick2/disperse
Brick4: 10.70.35.135:/rhs/brick2/disperse
Brick5: 10.70.35.8:/rhs/brick2/disperse
Brick6: 10.70.35.196:/rhs/brick2/disperse
Brick7: 10.70.35.37:/rhs/brick3/disperse
Brick8: 10.70.35.116:/rhs/brick3/disperse
Brick9: 10.70.35.239:/rhs/brick3/disperse
Brick10: 10.70.35.135:/rhs/brick3/disperse
Brick11: 10.70.35.8:/rhs/brick3/disperse
Brick12: 10.70.35.196:/rhs/brick3/disperse
Options Reconfigured:
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
ganesha.enable: on
features.cache-invalidation: off
performance.readdir-ahead: on
nfs.disable: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable


Version-Release number of selected component (if applicable):
==================================================================
[root@dhcp35-37 ~]# rpm -qa|grep gluster
glusterfs-geo-replication-3.8.4-10.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.1-3.el7rhgs.x86_64
glusterfs-3.8.4-10.el7rhgs.x86_64
python-gluster-3.8.4-10.el7rhgs.noarch
glusterfs-cli-3.8.4-10.el7rhgs.x86_64
glusterfs-rdma-3.8.4-10.el7rhgs.x86_64
glusterfs-api-3.8.4-10.el7rhgs.x86_64
glusterfs-server-3.8.4-10.el7rhgs.x86_64
glusterfs-api-devel-3.8.4-10.el7rhgs.x86_64
glusterfs-libs-3.8.4-10.el7rhgs.x86_64
glusterfs-devel-3.8.4-10.el7rhgs.x86_64
glusterfs-fuse-3.8.4-10.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-10.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-10.el7rhgs.x86_64
glusterfs-events-3.8.4-10.el7rhgs.x86_64



1)ganesha mount a disperse volume on two diffferent clients using differrent VIPs
2)from one client say c1 keep creating files
3) from another client say c2 change permission of existing files and create new files on same location as c1
4)you may notice the above issue

Comment 3 Sunil Kumar Acharya 2017-02-08 06:37:19 UTC

The owner/group value displayed on c1 is root-sqaush value. Discussed the issue with Soumya, moving the bug to nfs-ganesh team for investigation.

Comment 5 Kaleb KEITHLEY 2017-08-21 12:47:15 UTC

needs RCA