Description of problem: While renaming directories from a to b and from b to a, file handle seems to be broken resulting in directories not getting listed properly from mountpoint. [This bug is raised based on https://bugzilla.redhat.com/show_bug.cgi?id=1118762#c20] - mv abcd dcba - mv dcba abcd - From the mountpoint, only dir 'abcd' is seen. dcba is not seen from the mountpoint [root@dhcp46-9 yamaha]# ll total 20582408 drwxr-xr-x. 3 root root 4096 May 24 13:56 abcd [root@dhcp46-9 yamaha]# ll abcd/ total 0 - However, from the backend bricks following directory structure is seen from all the bricks. [root@dhcp46-103 ~]# ll /bricks/brick0/yamaha/abcd/ total 0 drwxr-xr-x. 2 root root 6 May 24 08:24 dcba - directory structure seems to be broken [root@dhcp47-171 glusterfs]# getfattr -d -m . -e hex /bricks/brick0/yamaha/abcd/ getfattr: Removing leading '/' from absolute path names # file: bricks/brick0/yamaha/abcd/ security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.gfid=0xfa38d55bedd1400d8f2504d766bd372a trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc [root@dhcp47-171 glusterfs]# getfattr -d -m . -e hex /bricks/brick0/yamaha/abcd/dcba/ getfattr: Removing leading '/' from absolute path names # file: bricks/brick0/yamaha/abcd/dcba/ security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.gfid=0x1f8b735b340447f68f35f3deac835b12 trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff [root@dhcp47-171 glusterfs]# ls -l /bricks/brick0/yamaha/.glusterfs/1f/8b total 0 lrwxrwxrwx. 1 root root 53 May 24 08:26 1f8b735b-3404-47f6-8f35-f3deac835b12 -> ../../fa/38/fa38d55b-edd1-400d-8f25-04d766bd372a/dcba Version-Release number of selected component (if applicable): glusterfs-3.7.9-6.el7rhgs.x86_64 How reproducible: 1/5 (approx) with breakpoints using gdb Steps to Reproduce: 1. mount a gluster volume from two different mount points [Although, this issue was seen with DHT, this can be seen with single sub-volume as well] 2. using gdb have breakpoint on dht_rename_hashed_dir_cbk from both the mountpoints 3. from mountpoint-1, run 'mv abcd dcba' 4. from mountpoint-2, run 'mv dcba abcd' 5. Once breakpoint is hit, allow gdb to continue from both mountpoints Actual results: From the mountpoint, only dir 'abcd' is seen. dcba is not seen from the mountpoint [root@dhcp46-9 yamaha]# ll total 20582408 drwxr-xr-x. 3 root root 4096 May 24 13:56 abcd [root@dhcp46-9 yamaha]# ll abcd/ total 0 - However, from the backend bricks following directory structure is seen from all the bricks. [root@dhcp46-103 ~]# ll /bricks/brick0/yamaha/abcd/ total 0 drwxr-xr-x. 2 root root 6 May 24 08:24 dcba - directory structure seems to be broken [root@dhcp47-171 glusterfs]# getfattr -d -m . -e hex /bricks/brick0/yamaha/abcd/ getfattr: Removing leading '/' from absolute path names # file: bricks/brick0/yamaha/abcd/ security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.gfid=0xfa38d55bedd1400d8f2504d766bd372a trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc [root@dhcp47-171 glusterfs]# getfattr -d -m . -e hex /bricks/brick0/yamaha/abcd/dcba/ getfattr: Removing leading '/' from absolute path names # file: bricks/brick0/yamaha/abcd/dcba/ security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.gfid=0x1f8b735b340447f68f35f3deac835b12 trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff [root@dhcp47-171 glusterfs]# ls -l /bricks/brick0/yamaha/.glusterfs/1f/8b total 0 lrwxrwxrwx. 1 root root 53 May 24 08:26 1f8b735b-3404-47f6-8f35-f3deac835b12 -> ../../fa/38/fa38d55b-edd1-400d-8f25-04d766bd372a/dcba Expected results: Directory structure has to be consistent Additional info: Pasting Dev's analysis from https://bugzilla.redhat.com/show_bug.cgi?id=1118762#c19 and https://bugzilla.redhat.com/show_bug.cgi?id=1118762#c20 Raghavendra G 2016-05-25 00:02:59 EDT On two of the bricks, gfid handle <brick-export/.glusterfs/fa/38/fa38d55b-edd1-400d-8f25-04d766bd372a was not present (ls said ENOENT). One of those bricks was hashed-subvolume of directory /mnt/glusterfs/abcd/dcba. So, stat sent along with dentry "dcba" from hashed-subvol was invalid (zeroed out). This resulted in readdirp not listing the directory and hence rmdir never came on "dcba". When rm crawled up to parent directory "abcd", rmdir failed with ENOTEMPTY as "dcba" was present on all bricks. I assume the reason for absence of gfid handle for fa38d55b-edd1-400d-8f25-04d766bd372a was due to a race in handling gfid handles during rename at brick. I confirm the hypothesis once I figure exact sequence of steps that might result in such a loss of handle. Raghavendra G 2016-05-25 01:23:12 EDT From one of the bricks on which handle fa38d55b-edd1-400d-8f25-04d766bd372a is missing I could see following error messages: [2016-05-24 08:26:22.422505] E [MSGID: 113071] [posix.c:2414:posix_rename] 0-yamaha-posix: rename of /bricks/brick0/yamaha/abcd to /bricks/brick0/yamaha/abcd/dcba/abcd failed [Invalid argument] [2016-05-24 08:26:22.422670] W [MSGID: 113001] [posix.c:2464:posix_rename] 0-yamaha-posix: modification of parent gfid xattr failed (gfid:fa38d55b-edd1-400d-8f25-04d766bd372a) [2016-05-24 08:26:22.422745] I [MSGID: 115061] [server-rpc-fops.c:1032:server_rename_cbk] 0-yamaha-server: 385: RENAME /abcd (00000000-0000-0000-0000-000000000000/abcd) -> /abcd/dcba/abcd (00000000-0000-0000-0000-000000000000/abcd) ==> (Invalid argument) [Invalid argument] in posix_rename (storage/posix), for rename (src dst) we: 1. unset gfid handle of src. 2. call sys_rename. 3. If failure, unwind As can be seen when sys_rename fails, gfid handle of src is never recreated. The rename logs above give an indication of sys_rename failure with fa38d55b-edd1-400d-8f25-04d766bd372a as src. So, incomplete failure handling in posix_rename is the root cause of this issue. Please note that this issue can happen even on single brick non-dht setup. So, I assume we can file a new bug on this (also this is a failure path, unlike this bug which uncovers inconsistencies even in success path).