+++ This bug was initially created as a clone of Bug #1336698 +++ +++ This bug was initially created as a clone of Bug #1118762 +++ Description of problem: ======================= try to rename Directories from multiple client at same time. e.g. from one mount give mv <src> <dest> and from another mv <src> <dest1> (where dest and dest1 is not hashing to same) or simply from one mount point mv <src> <dest> and from another mount point mv <src> <dest> . If they are not done one after another, then due to race after rename is finished :- - same gfid for different Directories (at same level and/or diffrent level) - files inside those directories are not listed on mount and sometime even Directory is not listed (so unable to access data) - sometimes ls output shows question mark in attribute filed. Version-Release number : ========================= 3.6.0.24-1.el6rhs.x86_64 How reproducible: ================= Intermittent Steps to Reproduce: ==================== 1. create and mount distributed volume. (mount on multiple client) 2. create few files and Directories on mount point. [root@OVM5 race]# mkdir src dest [root@OVM5 race]# touch src/f{1..5} [to reproduce race, we are putting breakpoint at dht_rename_hashed_dir_cbk for both rename operation] 3. from one mount point execute 'mv src dest' and from other mpount point execute 'mv dest src' Now continue execution from break point from both mount point. 4. verify data from another mount and bricks also mount:- [root@OVM5 race]# ls -l total 0 drwxr-xr-x 3 root root 69 Jul 10 12:32 src <---only src present [root@OVM5 race]# mkdir dest mkdir: cannot create directory `dest': File exists <-------- [root@OVM5 race]# ls -l <------- dest is not shown total 0 drwxr-xr-x 3 root root 69 Jul 10 12:32 src [root@OVM5 race]# touch src/f1 touch: cannot touch `src/f1': Input/output error [root@OVM5 race]# [root@OVM5 race]# ls -l src <-------f1, f2 and f5 are not listed total 0 drwxr-xr-x 3 root root 38 Jul 10 12:32 dest <--- directory having same name drwxr-xr-x 3 root root 38 Jul 10 12:32 dest -rw-r--r-- 1 root root 0 Jul 10 12:32 f3 -rw-r--r-- 1 root root 0 Jul 10 12:32 f4 another mount point:- [root@OVM5 race]# ls -lR .: total 0 drwxr-xr-x 3 root root 69 Jul 10 12:32 src ./src: total 0 drwxr-xr-x 3 root root 70 Jul 10 12:32 dest drwxr-xr-x 3 root root 70 Jul 10 12:32 dest -rw-r--r-- 1 root root 0 Jul 10 12:32 f3 -rw-r--r-- 1 root root 0 Jul 10 12:32 f4 ./src/dest: ls: cannot access ./src/dest/src: No such file or directory ls: cannot access ./src/dest/src: No such file or directory total 0 ?????????? ? ? ? ? ? src ?????????? ? ? ? ? ? src ./src/dest: ls: cannot access ./src/dest/src: No such file or directory ls: cannot access ./src/dest/src: No such file or directory total 0 ?????????? ? ? ? ? ? src <------------------ ?? in attribute ?????????? ? ? ? ? ? src bricks :- [root@OVM5 race]# tree /brick*/race/ /brick1/race/ ├── dest │ └── src │ ├── f1 │ ├── f2 │ └── f5 └── src └── dest /brick2/race/ └── src ├── dest │ └── src └── f4 /brick3/race/ ├── dest └── src ├── dest │ └── src └── f3 11 directories, 5 files Actual results: =============== - same gfid for different Directories (at same level and/or different level) - files inside those directories are not listed on mount and sometime even Directory is not listed (so unable to access data) - sometimes ls output shows question mark in attribute filed. Expected results: ================= - no two directory should have same gfid - if both renames are fail/successful or one succeed and other fail; regardless of all these - all files inside those Directories should be accessible from mount point --- Additional comment from Rachana Patel on 2014-07-11 09:27:34 EDT --- [root@OVM3 ~]# gluster v info race Volume Name: race Type: Distribute Volume ID: 30f7ff59-a90b-44e3-a991-223c81c15d67 Status: Started Snap Volume: no Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 10.70.35.172:/brick1/race Brick2: 10.70.35.172:/brick2/race Brick3: 10.70.35.172:/brick3/race Options Reconfigured: performance.readdir-ahead: on snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable --- Additional comment from Vijay Bellur on 2016-05-17 04:52:04 EDT --- REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent directory if destination exists) posted (#2) for review on master by Sakshi Bansal --- Additional comment from Vijay Bellur on 2016-05-17 05:32:25 EDT --- REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent directory if destination exists) posted (#3) for review on master by Sakshi Bansal --- Additional comment from Vijay Bellur on 2016-05-18 05:59:28 EDT --- COMMIT: http://review.gluster.org/14371 committed in master by Raghavendra G (rgowdapp) ------ commit 6a6b953d4f999b6db11cc6b62880e810aee9a4c7 Author: Sakshi Bansal <sabansal> Date: Tue May 17 13:23:28 2016 +0530 dht: rename takes lock on parent directory if destination exists For directory rename if destination exists the source directory is created as a child of the given destination directory. Since the new child directory does not exist take lock on parent of the child directory. Change-Id: I24a34605a2cd65984910643ff5462f35e8fc7e71 BUG: 1336698 Signed-off-by: Sakshi Bansal <sabansal> Reviewed-on: http://review.gluster.org/14371 Smoke: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com>
REVIEW: http://review.gluster.org/14417 (dht: rename takes lock on parent directory if destination exists) posted (#1) for review on release-3.8 by Sakshi Bansal
COMMIT: http://review.gluster.org/14417 committed in release-3.8 by Raghavendra G (rgowdapp) ------ commit b82fefed59cd3964c4d667b1645e70825be8e984 Author: Sakshi Bansal <sabansal> Date: Tue May 17 13:23:28 2016 +0530 dht: rename takes lock on parent directory if destination exists For directory rename if destination exists the source directory is created as a child of the given destination directory. Since the new child directory does not exist take lock on parent of the child directory. Backport of http://review.gluster.org/#/c/14371/ > Change-Id: I24a34605a2cd65984910643ff5462f35e8fc7e71 > BUG: 1336698 > Signed-off-by: Sakshi Bansal <sabansal> Change-Id: I24a34605a2cd65984910643ff5462f35e8fc7e71 BUG: 1337394 Signed-off-by: Sakshi Bansal <sabansal> Reviewed-on: http://review.gluster.org/14417 Smoke: Gluster Build System <jenkins.com> CentOS-regression: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user