Red Hat Bugzilla – Bug 1117172
DHT : - rename of files failed with 'No such File or Directory' when Source file was already present and all sub-volumes were up
Last modified: 2015-07-29 00:33:58 EDT
Description of problem: ======================= In a Distributed volume, Both source file and Destination file were present but rename of file from source to destination failed with error 'No such File or Directory'. Tried to rename from another mount also but it failed with same error mv: cannot move `c10' to `d10': No such file or directory Version-Release number : ========================= 3.6.0.24-1.el6rhs.x86_64 How reproducible: ================= Intermittent Steps to Reproduce: ==================== 1. create and mount distributed volume. (mount on multiple client) 2. create few files from mount point. 3. performed multiple rename. from a$i to b$i, b$i to c$i. while renaming from c$i to d$i it falied as below [root@OVM3 ren]# ls b2 c10 d10 d3 d5 d8 b3 d1 d2 d4 d6 d9 [root@OVM3 ren]# mv c10 d10 mv: overwrite `d10'? y mv: cannot move `c10' to `d10': No such file or directory - verified on bricks:- both files are present and those are stored on hashed sub-volume only [root@OVM3 ren]# ls -l /brick2/*/c10 -rw-r--r-- 2 root root 0 Jul 7 22:33 /brick2/r3/c10 [root@OVM3 ren]# ls -l /brick2/*/d10 -rw-r--r-- 2 root root 0 Jul 7 23:03 /brick2/r2/d10 4. tried to rename from another mount point and it failed again Actual results: =============== rename failed with error 'No such file or Directory', even though file exist and all sub-volumes were up Expected results: =============== If source file already exists, rename should not fail with error 'No such file or Directory' Additional Info:- [2014-07-07 17:34:23.071914] D [fuse-bridge.c:49:fuse_invalidate] 0-fuse: Invalidate inode id 140580139415140. [2014-07-07 17:34:23.073847] D [MSGID: 0] [dht-common.c:322:dht_discover_cbk] 0-ren-dht: lookup of /c10 on ren-client-0 returned error (Stale file handle) [2014-07-07 17:34:23.074049] D [MSGID: 0] [dht-common.c:322:dht_discover_cbk] 0-ren-dht: lookup of /c10 on ren-client-1 returned error (Stale file handle) [2014-07-07 17:34:42.647975] W [client-rpc-fops.c:2536:client3_3_rename_cbk] 0-ren-client-1: remote operation failed: No such file or directory [2014-07-07 17:34:42.648521] W [fuse-bridge.c:1727:fuse_rename_cbk] 0-glusterfs-fuse: 442: /c10 -> /d10 => -1 (No such file or directory)
As there is no data loss, this is not going to be considered a blocker based on the blocker criteria.
Performed 2 set of tests: 1. Created and renamed folders in a loop. [root@dht-rhs-23 1117172]# ls -ltrh total 0 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a1-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a2-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a3-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a4-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a5-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a6-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a7-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a8-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a9-1 drwxr-xr-x. 2 root root 18 Jun 12 21:29 a10-1 [root@dht-rhs-23 1117172]# [root@dht-rhs-23 1117172]# [root@dht-rhs-23 1117172]# rm -rf a* [root@dht-rhs-23 1117172]# ls -ltrh total 0 [root@dht-rhs-23 1117172]# [root@dht-rhs-23 1117172]# [root@dht-rhs-23 1117172]# for i in {1..10}; do mkdir a$i > done [root@dht-rhs-23 1117172]# ls -ltrh total 0 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a1 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a2 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a3 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a4 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a5 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a6 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a7 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a8 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a9 drwxr-xr-x. 2 root root 18 Jun 12 21:30 a10 [root@dht-rhs-23 1117172]# [root@dht-rhs-23 1117172]# ls a1 a10 a2 a3 a4 a5 a6 a7 a8 a9 [root@dht-rhs-23 1117172]# for i in `ls` > do > mv $i b$i > done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i c$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i d$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i e$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i f$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i g$i; done [root@dht-rhs-23 1117172]# ls -ltrh total 0 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba1 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba2 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba3 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba4 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba5 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba6 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba7 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba8 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba9 drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba10 [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i h$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i i$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i j$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i k$i; done [root@dht-rhs-23 1117172]# [root@dht-rhs-23 1117172]# ls -ltrh total 0 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba1 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba2 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba3 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba4 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba5 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba6 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba7 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba8 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba9 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba10 [root@dht-rhs-23 1117172]# [root@dht-rhs-23 1117172]# mkdir a [root@dht-rhs-23 1117172]# mv a b [root@dht-rhs-23 1117172]# mv b c [root@dht-rhs-23 1117172]# mv c d [root@dht-rhs-23 1117172]# mv d e [root@dht-rhs-23 1117172]# mv e f [root@dht-rhs-23 1117172]# mv f g [root@dht-rhs-23 1117172]# mv g h [root@dht-rhs-23 1117172]# mv h i [root@dht-rhs-23 1117172]# mv i j [root@dht-rhs-23 1117172]# mv j k [root@dht-rhs-23 1117172]# mv k l [root@dht-rhs-23 1117172]# mv l m [root@dht-rhs-23 1117172]# ls -ltrh total 0 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba1 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba2 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba3 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba4 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba5 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba6 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba7 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba8 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba9 drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba10 drwxr-xr-x. 2 root root 18 Jun 12 21:33 m [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i l$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i m$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i n$i; done [root@dht-rhs-23 1117172]# for i in `ls`; do mv $i i$i; done [root@dht-rhs-23 1117172]# [root@dht-rhs-23 1117172]# ls -ltrh total 0 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba1 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba2 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba3 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba4 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba5 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba6 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba7 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba8 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba9 drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba10 drwxr-xr-x. 2 root root 18 Jun 12 21:33 inmlm [root@dht-rhs-23 1117172]# ====== 2. Created, renamed, moved the link and read the file to ensure renames/readdirp/cat/lookup in a loop. No error seen in log files. while true ; do ln abc-ln abc; mv -f abc-ln abc; echo 3>/proc/sys/vm/drop_caches; cat abc; ln abc abc-ln; mv -f abc abc-ln; cho 3>/proc/sys/vm/drop_caches; while true ; do ln abc-ln abc; mv -f abc-ln abc; echo 3>/proc/sys/vm/drop_caches; cat abc; ln abc abc-ln; mv -f abc abc-ln; echo 3>/proc/sys/vm/drop_caches; cat abc-ln; done; done Volume is mounted direct-io to eliminate caches.. [root@dht-rhs-23 glusterfs]# mount | grep gluster 10.70.47.174:/testvol on /mnt/glusterfs type fuse.glusterfs (rw,direct-io-mode=yes) [root@dht-rhs-23 glusterfs]# Marking the bug verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html