Bug 863100

Summary: DHT :- inconsistent 'custom extended attributes',uid and gid, Access permission (for directories) if User set/modifies it after bringing one or more sub-volume down
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rachana Patel <racpatel>
Component: distributeAssignee: Nithya Balachandran <nbalacha>
Status: CLOSED DEFERRED QA Contact: Matt Zywusko <mzywusko>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.0CC: mzywusko, nsathyan, rgowdapp, rhs-bugs, rwheeler, sdharane, spalai, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.4.0qa8, glusterfs-3.3.0.5rhs-42 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1064147 1286036 (view as bug list) Environment:
Last Closed: 2015-11-27 10:06:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1064147    

Description Rachana Patel 2012-10-04 12:43:26 UTC 
Description of problem:
DHT :-  inconsistent 'custom  extended attributes',uid and gid, Access permission (for directories) if User set/modifies it after bringing one or more  sub-volume down

Version-Release number of selected component (if applicable):
3.3.0.3rhs-32.el6rhs.x86_64	

How reproducible:
always


Steps to Reproduce:
1. Create a Distributed volume having 3 or more sub-volumes on multiple server and start that volume.


2. Fuse Mount the volume from the client-1 using “mount -t glusterfs  server:/<volume> <client-1_mount_point>”

3. From mount point create some dirs and files inside it.

4. Bring on sub-volume down
[root@Rhs3 t1]# gluster volume status test
Status of volume: test
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 10.70.35.81:/home/t1				24009	Y	18564
Brick 10.70.35.85:/home/t1				24211	Y	16174
Brick 10.70.35.86:/home/t1				24212	Y	2360
NFS Server on localhost					38467	Y	2366
NFS Server on 10.70.35.81				38467	Y	12929
NFS Server on 10.70.35.85				38467	Y	10226
 
[root@Rhs3 t1]# kill -9 2360


5. 
Custom atribute:-
from mount point set custom attribute for directory and verify it on all server

client
[root@client test]# setfattr -n user.foo -v bar2 d1
[root@client test]# getfattr -n user.foo d1
# file: d1
user.foo="bar2"

server1:-
[root@Rhs1 t1]# getfattr -n user.foo d1
# file: d1
user.foo="bar2"

server2:-
[root@Rhs2 t1]# getfattr -n user.foo d1
# file: d1
user.foo="bar2"

server3:-
[root@Rhs3 t1]# getfattr -n user.foo d1
d1: user.foo: No such attribute

6.from mount point verify owner and group of dir and then modify it

[root@client test]# stat d1
  File: `d1'
  Size: 12        	Blocks: 2          IO Block: 131072 directory
Device: 15h/21d	Inode: 10442536925251715313  Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:57.006871636 +0530
Modify: 2012-10-04 12:10:57.006871636 +0530
Change: 2012-10-04 12:10:57.007864913 +0530

[root@client test]# chown u1 d1
[root@client test]# chgrp t1 d1
[root@client test]# stat d1
  File: `d1'
  Size: 12        	Blocks: 2          IO Block: 131072 directory
Device: 15h/21d	Inode: 10442536925251715313  Links: 2
Access: (0755/drwxr-xr-x)  Uid: (  500/      u1)   Gid: (  500/      t1)
Access: 2012-10-04 12:10:57.006871636 +0530
Modify: 2012-10-04 12:10:57.006871636 +0530
Change: 2012-10-04 12:13:05.168865621 +0530

7.verify that it is updated on all sub-volume except the down one

server1:-
[root@Rhs1 t1]# stat d1
  File: `d1'
  Size: 6         	Blocks: 8          IO Block: 4096   directory
Device: fc05h/64517d	Inode: 403116740   Links: 2
Access: (0755/drwxr-xr-x)  Uid: (  500/ UNKNOWN)   Gid: (  500/ UNKNOWN)
Access: 2012-10-04 12:10:57.006871636 +0530
Modify: 2012-10-04 12:10:57.006871636 +0530
Change: 2012-10-04 12:13:05.168865621 +0530

server2:-
[root@Rhs2 t1]# stat d1
  File: `d1'
  Size: 6         	Blocks: 8          IO Block: 4096   directory
Device: fc05h/64517d	Inode: 134423062   Links: 2
Access: (0755/drwxr-xr-x)  Uid: (  500/ UNKNOWN)   Gid: (  500/ UNKNOWN)
Access: 2012-10-04 12:10:56.807630951 +0530
Modify: 2012-10-04 12:10:56.807630951 +0530
Change: 2012-10-04 12:13:04.970323409 +0530

server3:-
[root@Rhs3 t1]# stat d1
  File: `d1'
  Size: 6         	Blocks: 8          IO Block: 4096   directory
Device: fc05h/64517d	Inode: 402655089   Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:51.424531338 +0530
Modify: 2012-10-04 12:10:51.424531338 +0530
Change: 2012-10-04 12:10:51.610007436 +0530

8.from mount point verify Dir permission and modify it

client
[root@client test]# stat d2
  File: `d2'
  Size: 12        	Blocks: 2          IO Block: 131072 directory
Device: 15h/21d	Inode: 9860248238918728119  Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:57.281276686 +0530
Modify: 2012-10-04 12:10:57.281276686 +0530
Change: 2012-10-04 12:10:57.282865329 +0530

[root@client test]# chmod 444 d2

[root@client test]# stat d2
  File: `d2'
  Size: 12        	Blocks: 2          IO Block: 131072 directory
Device: 15h/21d	Inode: 9860248238918728119  Links: 2
Access: (0444/dr--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:57.281276686 +0530
Modify: 2012-10-04 12:10:57.281276686 +0530
Change: 2012-10-04 15:39:42.509942805 +0530

9. verify that it is updated on all sub-volume except the down one
server1
[root@Rhs1 t1]# stat d2
  File: `d2'
  Size: 6         	Blocks: 8          IO Block: 4096   directory
Device: fc05h/64517d	Inode: 134694359   Links: 2
Access: (0444/dr--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:57.281276686 +0530
Modify: 2012-10-04 12:10:57.281276686 +0530
Change: 2012-10-04 15:39:42.509942805 +0530

server2
[root@Rhs1 t1]# stat d2
  File: `d2'
  Size: 6         	Blocks: 8          IO Block: 4096   directory
Device: fc05h/64517d	Inode: 134694359   Links: 2
Access: (0444/dr--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:57.281276686 +0530
Modify: 2012-10-04 12:10:57.281276686 +0530
Change: 2012-10-04 15:39:42.509942805 +0530

server3:-
[root@Rhs3 t1]# stat d1
  File: `d1'
  Size: 6         	Blocks: 8          IO Block: 4096   directory
Device: fc05h/64517d	Inode: 402655089   Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:51.424531338 +0530
Modify: 2012-10-04 12:10:51.424531338 +0530
Change: 2012-10-04 12:10:51.610007436 +0530

10. now bring all sub-volumes up and perform lookup from client.

Verify updated 'custom  extended attributes',uid and gid, Access permission from client

11.verify access permission from sub-volume which was down previously
[root@Rhs3 t1]# stat d2
  File: `d2'
  Size: 6         	Blocks: 8          IO Block: 4096   directory
Device: fc05h/64517d	Inode: 134219638   Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:51.700461206 +0530
Modify: 2012-10-04 12:10:51.700461206 +0530
Change: 2012-10-04 12:10:51.759989493 +0530

 verify gid and uid on sub-volume which was down previously
[root@Rhs3 t1]# stat d1
  File: `d1'
  Size: 6         	Blocks: 8          IO Block: 4096   directory
Device: fc05h/64517d	Inode: 402655089   Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-10-04 12:10:51.424531338 +0530
Modify: 2012-10-04 12:10:51.424531338 +0530
Change: 2012-10-04 12:10:51.610007436 +0530

Verify custom  extended attributes for that directory.


server3:-
[root@Rhs3 t1]# getfattr -n user.foo d1
d1: user.foo: No such attribute


  
Actual results:
mount point shows modified values but on subvolumes values are not consistent

Expected results:
Once sub-volume is up it should update values for 'custom  extended attributes',uid and gid, Access permission (for directories).

Additional info:
Comment 2 shishir gowda 2012-10-19 04:21:57 UTC 
There are multiple issues in this bug.
1. For user related xattrs, we can not handle healing as in dht, we would not be able to identify the correct copy. The work around for this is a subsequent setxattr for the same key, which will fix the xattr mis-match

2. UID/GID: A fix is in progress (862967)
3. mis-matching permission: Will investigate it respond back
Comment 4 Amar Tumballi 2013-02-15 12:11:59 UTC 
https://code.engineering.redhat.com/gerrit/#/c/1895/
Comment 5 Rachana Patel 2013-03-19 13:09:46 UTC 
(In reply to comment #2)
> There are multiple issues in this bug.
> 1. For user related xattrs, we can not handle healing as in dht, we would
> not be able to identify the correct copy. The work around for this is a
> subsequent setxattr for the same key, which will fix the xattr mis-match
> 
> 2. UID/GID: A fix is in progress (862967)
> 3. mis-matching permission: Will investigate it respond back

1. If it is the case then, it should get documented.

2. It is dependent of bug 86296 and for that buf fixed in version is glusterfs-3.4.0qa5. So the same fix is available in latest build?

3. Could you please update on 3rd issue? what is the decision?
Comment 6 Scott Haines 2013-09-27 17:07:27 UTC 
Targeting for 3.0.0 (Denali) release.
Comment 15 Susant Kumar Palai 2015-11-27 10:06:05 UTC 
Cloning this bug to 3.1. Will be fixed in future release.