Bug 1339501 - renaming directories from src to dst and dst to src leads into inconsistent directory structure
Summary: renaming directories from src to dst and dst to src leads into inconsistent d...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: posix
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Pranith Kumar K
QA Contact: Anoop
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-25 08:13 UTC by krishnaram Karthick
Modified: 2018-04-16 18:17 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-16 18:17:26 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description krishnaram Karthick 2016-05-25 08:13:51 UTC
Description of problem:

While renaming directories from a to b and from b to a, file handle seems to be broken resulting in directories not getting listed properly from mountpoint.

[This bug is raised based on https://bugzilla.redhat.com/show_bug.cgi?id=1118762#c20]

 - mv abcd dcba
 - mv dcba abcd
 - From the mountpoint, only dir 'abcd' is seen. dcba is not seen from the mountpoint

[root@dhcp46-9 yamaha]# ll
total 20582408
drwxr-xr-x. 3 root root       4096 May 24 13:56 abcd

[root@dhcp46-9 yamaha]# ll abcd/
total 0

 - However, from the backend bricks following directory structure is seen from all the bricks.

[root@dhcp46-103 ~]# ll /bricks/brick0/yamaha/abcd/
total 0
drwxr-xr-x. 2 root root 6 May 24 08:24 dcba

 - directory structure seems to be broken

[root@dhcp47-171 glusterfs]# getfattr -d -m . -e hex /bricks/brick0/yamaha/abcd/
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick0/yamaha/abcd/
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0xfa38d55bedd1400d8f2504d766bd372a
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc

[root@dhcp47-171 glusterfs]# getfattr -d -m . -e hex /bricks/brick0/yamaha/abcd/dcba/
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick0/yamaha/abcd/dcba/
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x1f8b735b340447f68f35f3deac835b12
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff


[root@dhcp47-171 glusterfs]# ls -l /bricks/brick0/yamaha/.glusterfs/1f/8b
total 0
lrwxrwxrwx. 1 root root 53 May 24 08:26 1f8b735b-3404-47f6-8f35-f3deac835b12 -> ../../fa/38/fa38d55b-edd1-400d-8f25-04d766bd372a/dcba

Version-Release number of selected component (if applicable):
glusterfs-3.7.9-6.el7rhgs.x86_64

How reproducible:
1/5 (approx) with breakpoints using gdb

Steps to Reproduce:
1. mount a gluster volume from two different mount points
[Although, this issue was seen with DHT, this can be seen with single sub-volume as well]
2. using gdb have breakpoint on dht_rename_hashed_dir_cbk from both the mountpoints
3. from mountpoint-1, run 'mv abcd dcba'
4. from mountpoint-2, run 'mv dcba abcd'
5. Once breakpoint is hit, allow gdb to continue from both mountpoints

Actual results:
From the mountpoint, only dir 'abcd' is seen. dcba is not seen from the mountpoint

[root@dhcp46-9 yamaha]# ll
total 20582408
drwxr-xr-x. 3 root root       4096 May 24 13:56 abcd

[root@dhcp46-9 yamaha]# ll abcd/
total 0

 - However, from the backend bricks following directory structure is seen from all the bricks.

[root@dhcp46-103 ~]# ll /bricks/brick0/yamaha/abcd/
total 0
drwxr-xr-x. 2 root root 6 May 24 08:24 dcba

 - directory structure seems to be broken

[root@dhcp47-171 glusterfs]# getfattr -d -m . -e hex /bricks/brick0/yamaha/abcd/
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick0/yamaha/abcd/
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0xfa38d55bedd1400d8f2504d766bd372a
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc

[root@dhcp47-171 glusterfs]# getfattr -d -m . -e hex /bricks/brick0/yamaha/abcd/dcba/
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick0/yamaha/abcd/dcba/
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.gfid=0x1f8b735b340447f68f35f3deac835b12
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff


[root@dhcp47-171 glusterfs]# ls -l /bricks/brick0/yamaha/.glusterfs/1f/8b
total 0
lrwxrwxrwx. 1 root root 53 May 24 08:26 1f8b735b-3404-47f6-8f35-f3deac835b12 -> ../../fa/38/fa38d55b-edd1-400d-8f25-04d766bd372a/dcba

Expected results:
Directory structure has to be consistent

Additional info:
Pasting Dev's analysis from https://bugzilla.redhat.com/show_bug.cgi?id=1118762#c19 and https://bugzilla.redhat.com/show_bug.cgi?id=1118762#c20

 Raghavendra G 2016-05-25 00:02:59 EDT

On two of the bricks, gfid handle <brick-export/.glusterfs/fa/38/fa38d55b-edd1-400d-8f25-04d766bd372a was not present (ls said ENOENT). One of those bricks was hashed-subvolume of directory /mnt/glusterfs/abcd/dcba. So, stat sent along with dentry "dcba" from hashed-subvol was invalid (zeroed out). This resulted in readdirp not listing the directory and hence rmdir never came on "dcba". When rm crawled up to parent directory "abcd", rmdir failed with ENOTEMPTY as "dcba" was present on all bricks.

I assume the reason for absence of gfid handle for fa38d55b-edd1-400d-8f25-04d766bd372a was due to a race in handling gfid handles during rename at brick. I confirm the hypothesis once I figure exact sequence of steps that might result in such a loss of handle.

 Raghavendra G 2016-05-25 01:23:12 EDT

From one of the bricks on which handle fa38d55b-edd1-400d-8f25-04d766bd372a is missing I could see following error messages:

[2016-05-24 08:26:22.422505] E [MSGID: 113071] [posix.c:2414:posix_rename] 0-yamaha-posix: rename of /bricks/brick0/yamaha/abcd to /bricks/brick0/yamaha/abcd/dcba/abcd failed [Invalid argument]
[2016-05-24 08:26:22.422670] W [MSGID: 113001] [posix.c:2464:posix_rename] 0-yamaha-posix: modification of parent gfid xattr failed (gfid:fa38d55b-edd1-400d-8f25-04d766bd372a)
[2016-05-24 08:26:22.422745] I [MSGID: 115061] [server-rpc-fops.c:1032:server_rename_cbk] 0-yamaha-server: 385: RENAME /abcd (00000000-0000-0000-0000-000000000000/abcd) -> /abcd/dcba/abcd (00000000-0000-0000-0000-000000000000/abcd) ==> (Invalid argument) [Invalid argument]

in posix_rename (storage/posix), for rename (src dst) we:

1. unset gfid handle of src.
2. call sys_rename.
3. If failure, unwind

As can be seen when sys_rename fails, gfid handle of src is never recreated. The rename logs above give an indication of sys_rename failure with fa38d55b-edd1-400d-8f25-04d766bd372a as src. So, incomplete failure handling in posix_rename is the root cause of this issue.

Please note that this issue can happen even on single brick non-dht setup. So, I assume we can file a new bug on this (also this is a failure path, unlike this bug which uncovers inconsistencies even in success path).


Note You need to log in before you can comment on or make changes to this bug.