Bug 1670259

Summary: New GFID file recreated in a replica set after a GFID mismatch resolution
Product: [Community] GlusterFS Reporter: Nithya Balachandran <nbalacha>
Component: distributeAssignee: bugs <bugs>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: mainline   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1661258
: 1684969 (view as bug list) Environment:
Last Closed: 2019-02-02 03:10:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1661258    
Bug Blocks: 1684969    

Description Nithya Balachandran 2019-01-29 04:46:27 UTC
If the gfid is not set for a directory on one brick, a lookup on that directory will cause a different GFID to be set on it, leading to a GFID mismatch.

Comment 1 Nithya Balachandran 2019-01-29 04:56:49 UTC
Steps to reproduce the problem:

1. Create a 3 brick distribute volume
2. Fuse mount the volume and create some directories.
cd /mnt/fuse1
mkdir gfid-mismatch
mkdir gfid-mismatch/dir-1

3. delete the gfid and .glusterfs handle from the hashed brick

[root@rhgs313-6 brick1]# setfattr -x trusted.gfid vol1-1/gfid-mismatch/dir-1
[root@rhgs313-6 brick1]# unlink vol1-1/.glusterfs/8e/6c/8e6c686c-93e9-4bd7-ac5e-98bbf852a62b
[root@rhgs313-6 brick1]# 
[root@rhgs313-6 brick1]# 
[root@rhgs313-6 brick1]# 
[root@rhgs313-6 brick1]# getx vol1-*/gfid-mismatch/dir-1
# file: vol1-1/gfid-mismatch/dir-1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.glusterfs.dht=0x00000000000000000000000055555554
trusted.glusterfs.dht.mds=0x00000000
trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3

# file: vol1-2/gfid-mismatch/dir-1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b
trusted.glusterfs.dht=0x000000000000000055555555aaaaaaa9
trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3

# file: vol1-3/gfid-mismatch/dir-1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b
trusted.glusterfs.dht=0x0000000000000000aaaaaaaaffffffff
trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3


4. Unmount and remount the fuse mount and list the contents of gfid-mismatch

[root@rhgs313-6 ~]# umount -f /mnt/fuse1; mount -t glusterfs -s 192.168.122.6:/vol1 /mnt/fuse1
[root@rhgs313-6 ~]# cd /mnt/fuse1
[root@rhgs313-6 fuse1]# cd gfid-mismatch/
[root@rhgs313-6 gfid-mismatch]# l
total 20K
drwxr-xr-x. 2 root root 4.0K Jan 29 09:20 dir-1


5. Check the gfid for dir-1 on the backend bricks.

[root@rhgs313-6 brick1]# getx vol1-*/gfid-mismatch/dir-1
# file: vol1-1/gfid-mismatch/dir-1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x0c3a4860f93f416cb5261c3b2b06f52d    < ---- GFID is DIFFERENT!!
trusted.glusterfs.dht=0x00000000000000000000000055555554
trusted.glusterfs.dht.mds=0x00000000
trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3

# file: vol1-2/gfid-mismatch/dir-1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b
trusted.glusterfs.dht=0x000000000000000055555555aaaaaaa9
trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3

# file: vol1-3/gfid-mismatch/dir-1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b
trusted.glusterfs.dht=0x0000000000000000aaaaaaaaffffffff
trusted.glusterfs.mdata=0x010000000000000000000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3000000005c4fcd7500000000017863f3



The GFID on brick vol1-1 is set to 
trusted.gfid=0x0c3a4860f93f416cb5261c3b2b06f52d 

The original GFID was:
trusted.gfid=0x8e6c686c93e94bd7ac5e98bbf852a62b

Comment 2 Nithya Balachandran 2019-01-29 04:58:11 UTC
Upstream patch:
https://review.gluster.org/#/c/glusterfs/+/22112/

Comment 3 Worker Ant 2019-02-02 03:10:15 UTC
REVIEW: https://review.gluster.org/22112 (cluster/dht: Do not use gfid-req in fresh lookup) merged (#7) on master by Amar Tumballi

Comment 4 Shyamsundar 2019-03-25 16:33:11 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-6.0, please open a new bug report.

glusterfs-6.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2019-March/000120.html
[2] https://www.gluster.org/pipermail/gluster-users/