Description of problem: DHT :- inconsistent 'custom extended attributes',uid and gid, Access permission (for directories) if User set/modifies it after bringing one or more sub-volume down Version-Release number of selected component (if applicable): 3.3.0.3rhs-32.el6rhs.x86_64 How reproducible: always Steps to Reproduce: 1. Create a Distributed volume having 3 or more sub-volumes on multiple server and start that volume. 2. Fuse Mount the volume from the client-1 using “mount -t glusterfs server:/<volume> <client-1_mount_point>” 3. From mount point create some dirs and files inside it. 4. Bring on sub-volume down [root@Rhs3 t1]# gluster volume status test Status of volume: test Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.81:/home/t1 24009 Y 18564 Brick 10.70.35.85:/home/t1 24211 Y 16174 Brick 10.70.35.86:/home/t1 24212 Y 2360 NFS Server on localhost 38467 Y 2366 NFS Server on 10.70.35.81 38467 Y 12929 NFS Server on 10.70.35.85 38467 Y 10226 [root@Rhs3 t1]# kill -9 2360 5. Custom atribute:- from mount point set custom attribute for directory and verify it on all server client [root@client test]# setfattr -n user.foo -v bar2 d1 [root@client test]# getfattr -n user.foo d1 # file: d1 user.foo="bar2" server1:- [root@Rhs1 t1]# getfattr -n user.foo d1 # file: d1 user.foo="bar2" server2:- [root@Rhs2 t1]# getfattr -n user.foo d1 # file: d1 user.foo="bar2" server3:- [root@Rhs3 t1]# getfattr -n user.foo d1 d1: user.foo: No such attribute 6.from mount point verify owner and group of dir and then modify it [root@client test]# stat d1 File: `d1' Size: 12 Blocks: 2 IO Block: 131072 directory Device: 15h/21d Inode: 10442536925251715313 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:57.006871636 +0530 Modify: 2012-10-04 12:10:57.006871636 +0530 Change: 2012-10-04 12:10:57.007864913 +0530 [root@client test]# chown u1 d1 [root@client test]# chgrp t1 d1 [root@client test]# stat d1 File: `d1' Size: 12 Blocks: 2 IO Block: 131072 directory Device: 15h/21d Inode: 10442536925251715313 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 500/ u1) Gid: ( 500/ t1) Access: 2012-10-04 12:10:57.006871636 +0530 Modify: 2012-10-04 12:10:57.006871636 +0530 Change: 2012-10-04 12:13:05.168865621 +0530 7.verify that it is updated on all sub-volume except the down one server1:- [root@Rhs1 t1]# stat d1 File: `d1' Size: 6 Blocks: 8 IO Block: 4096 directory Device: fc05h/64517d Inode: 403116740 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 500/ UNKNOWN) Gid: ( 500/ UNKNOWN) Access: 2012-10-04 12:10:57.006871636 +0530 Modify: 2012-10-04 12:10:57.006871636 +0530 Change: 2012-10-04 12:13:05.168865621 +0530 server2:- [root@Rhs2 t1]# stat d1 File: `d1' Size: 6 Blocks: 8 IO Block: 4096 directory Device: fc05h/64517d Inode: 134423062 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 500/ UNKNOWN) Gid: ( 500/ UNKNOWN) Access: 2012-10-04 12:10:56.807630951 +0530 Modify: 2012-10-04 12:10:56.807630951 +0530 Change: 2012-10-04 12:13:04.970323409 +0530 server3:- [root@Rhs3 t1]# stat d1 File: `d1' Size: 6 Blocks: 8 IO Block: 4096 directory Device: fc05h/64517d Inode: 402655089 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:51.424531338 +0530 Modify: 2012-10-04 12:10:51.424531338 +0530 Change: 2012-10-04 12:10:51.610007436 +0530 8.from mount point verify Dir permission and modify it client [root@client test]# stat d2 File: `d2' Size: 12 Blocks: 2 IO Block: 131072 directory Device: 15h/21d Inode: 9860248238918728119 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:57.281276686 +0530 Modify: 2012-10-04 12:10:57.281276686 +0530 Change: 2012-10-04 12:10:57.282865329 +0530 [root@client test]# chmod 444 d2 [root@client test]# stat d2 File: `d2' Size: 12 Blocks: 2 IO Block: 131072 directory Device: 15h/21d Inode: 9860248238918728119 Links: 2 Access: (0444/dr--r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:57.281276686 +0530 Modify: 2012-10-04 12:10:57.281276686 +0530 Change: 2012-10-04 15:39:42.509942805 +0530 9. verify that it is updated on all sub-volume except the down one server1 [root@Rhs1 t1]# stat d2 File: `d2' Size: 6 Blocks: 8 IO Block: 4096 directory Device: fc05h/64517d Inode: 134694359 Links: 2 Access: (0444/dr--r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:57.281276686 +0530 Modify: 2012-10-04 12:10:57.281276686 +0530 Change: 2012-10-04 15:39:42.509942805 +0530 server2 [root@Rhs1 t1]# stat d2 File: `d2' Size: 6 Blocks: 8 IO Block: 4096 directory Device: fc05h/64517d Inode: 134694359 Links: 2 Access: (0444/dr--r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:57.281276686 +0530 Modify: 2012-10-04 12:10:57.281276686 +0530 Change: 2012-10-04 15:39:42.509942805 +0530 server3:- [root@Rhs3 t1]# stat d1 File: `d1' Size: 6 Blocks: 8 IO Block: 4096 directory Device: fc05h/64517d Inode: 402655089 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:51.424531338 +0530 Modify: 2012-10-04 12:10:51.424531338 +0530 Change: 2012-10-04 12:10:51.610007436 +0530 10. now bring all sub-volumes up and perform lookup from client. Verify updated 'custom extended attributes',uid and gid, Access permission from client 11.verify access permission from sub-volume which was down previously [root@Rhs3 t1]# stat d2 File: `d2' Size: 6 Blocks: 8 IO Block: 4096 directory Device: fc05h/64517d Inode: 134219638 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:51.700461206 +0530 Modify: 2012-10-04 12:10:51.700461206 +0530 Change: 2012-10-04 12:10:51.759989493 +0530 verify gid and uid on sub-volume which was down previously [root@Rhs3 t1]# stat d1 File: `d1' Size: 6 Blocks: 8 IO Block: 4096 directory Device: fc05h/64517d Inode: 402655089 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-10-04 12:10:51.424531338 +0530 Modify: 2012-10-04 12:10:51.424531338 +0530 Change: 2012-10-04 12:10:51.610007436 +0530 Verify custom extended attributes for that directory. server3:- [root@Rhs3 t1]# getfattr -n user.foo d1 d1: user.foo: No such attribute Actual results: mount point shows modified values but on subvolumes values are not consistent Expected results: Once sub-volume is up it should update values for 'custom extended attributes',uid and gid, Access permission (for directories). Additional info:
There are multiple issues in this bug. 1. For user related xattrs, we can not handle healing as in dht, we would not be able to identify the correct copy. The work around for this is a subsequent setxattr for the same key, which will fix the xattr mis-match 2. UID/GID: A fix is in progress (862967) 3. mis-matching permission: Will investigate it respond back
https://code.engineering.redhat.com/gerrit/#/c/1895/
(In reply to comment #2) > There are multiple issues in this bug. > 1. For user related xattrs, we can not handle healing as in dht, we would > not be able to identify the correct copy. The work around for this is a > subsequent setxattr for the same key, which will fix the xattr mis-match > > 2. UID/GID: A fix is in progress (862967) > 3. mis-matching permission: Will investigate it respond back 1. If it is the case then, it should get documented. 2. It is dependent of bug 86296 and for that buf fixed in version is glusterfs-3.4.0qa5. So the same fix is available in latest build? 3. Could you please update on 3rd issue? what is the decision?
Targeting for 3.0.0 (Denali) release.
Cloning this bug to 3.1. Will be fixed in future release.