Bug 1117172

Summary:	DHT : - rename of files failed with 'No such File or Directory' when Source file was already present and all sub-volumes were up
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rachana Patel <racpatel>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED ERRATA	QA Contact:	Amit Chaurasia <achauras>
Severity:	high	Docs Contact:
Priority:	high
Version:	rhgs-3.0	CC:	achauras, annair, mzywusko, nbalacha, nsathyan, vagarwal
Target Milestone:	---
Target Release:	RHGS 3.1.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.1-1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-07-29 04:33:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1136622
Bug Blocks:	1202842

Description Rachana Patel 2014-07-08 07:59:09 UTC

Description of problem:
=======================
In a Distributed volume, Both source file and Destination file were present but rename of file from source to destination failed with error 'No such File or Directory'.
Tried to rename from another mount also but it failed with same error

mv: cannot move `c10' to `d10': No such file or directory

Version-Release number :
=========================
3.6.0.24-1.el6rhs.x86_64


How reproducible:
=================
Intermittent 


Steps to Reproduce:
====================
1. create and mount distributed volume. (mount on multiple client)
2. create few files from mount point.
3. performed multiple rename. from a$i to b$i, b$i to c$i.
while renaming from c$i to d$i it falied as below

[root@OVM3 ren]# ls
b2  c10  d10  d3  d5  d8
b3  d1   d2   d4  d6  d9

[root@OVM3 ren]# mv c10 d10
mv: overwrite `d10'? y
mv: cannot move `c10' to `d10': No such file or directory


- verified on bricks:- both files are present and those are stored on hashed sub-volume only
[root@OVM3 ren]# ls -l /brick2/*/c10
-rw-r--r-- 2 root root 0 Jul  7 22:33 /brick2/r3/c10
[root@OVM3 ren]# ls -l /brick2/*/d10
-rw-r--r-- 2 root root 0 Jul  7 23:03 /brick2/r2/d10

4. tried to rename from another mount point and it failed again


Actual results:
===============
rename failed with error 'No such file or Directory', even though file exist and all sub-volumes were up

Expected results:
===============
If source file already exists, rename should not fail with error 'No such file or Directory'


Additional Info:-
[2014-07-07 17:34:23.071914] D [fuse-bridge.c:49:fuse_invalidate] 0-fuse: Invalidate inode id 140580139415140.
[2014-07-07 17:34:23.073847] D [MSGID: 0] [dht-common.c:322:dht_discover_cbk] 0-ren-dht: lookup of /c10 on ren-client-0 returned error (Stale file handle)
[2014-07-07 17:34:23.074049] D [MSGID: 0] [dht-common.c:322:dht_discover_cbk] 0-ren-dht: lookup of /c10 on ren-client-1 returned error (Stale file handle)


[2014-07-07 17:34:42.647975] W [client-rpc-fops.c:2536:client3_3_rename_cbk] 0-ren-client-1: remote operation failed: No such file or directory
[2014-07-07 17:34:42.648521] W [fuse-bridge.c:1727:fuse_rename_cbk] 0-glusterfs-fuse: 442: /c10 -> /d10 => -1 (No such file or directory)

Comment 4 Nithya Balachandran 2014-08-04 05:19:15 UTC

As there is no data loss, this is not going to be considered a blocker based on the blocker criteria.

Comment 12 Amit Chaurasia 2015-06-12 10:53:43 UTC

Performed 2 set of tests:

1. Created and renamed folders in a loop.


[root@dht-rhs-23 1117172]# ls -ltrh
total 0
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a1-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a2-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a3-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a4-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a5-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a6-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a7-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a8-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a9-1
drwxr-xr-x. 2 root root 18 Jun 12 21:29 a10-1
[root@dht-rhs-23 1117172]# 
[root@dht-rhs-23 1117172]# 
[root@dht-rhs-23 1117172]# rm -rf a*
[root@dht-rhs-23 1117172]# ls -ltrh
total 0
[root@dht-rhs-23 1117172]# 
[root@dht-rhs-23 1117172]# 
[root@dht-rhs-23 1117172]# for i in {1..10}; do mkdir a$i 
> done
[root@dht-rhs-23 1117172]# ls -ltrh
total 0
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a1
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a2
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a3
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a4
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a5
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a6
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a7
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a8
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a9
drwxr-xr-x. 2 root root 18 Jun 12 21:30 a10
[root@dht-rhs-23 1117172]# 
[root@dht-rhs-23 1117172]# ls
a1  a10  a2  a3  a4  a5  a6  a7  a8  a9
[root@dht-rhs-23 1117172]# for i in `ls`
> do
> mv $i b$i
> done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i c$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i d$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i e$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i f$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i g$i; done
[root@dht-rhs-23 1117172]# ls -ltrh
total 0
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba1
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba2
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba3
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba4
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba5
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba6
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba7
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba8
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba9
drwxr-xr-x. 2 root root 18 Jun 12 21:30 gfedcba10
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i h$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i i$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i j$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i k$i; done
[root@dht-rhs-23 1117172]# 
[root@dht-rhs-23 1117172]# ls -ltrh
total 0
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba1
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba2
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba3
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba4
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba5
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba6
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba7
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba8
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba9
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba10
[root@dht-rhs-23 1117172]# 
[root@dht-rhs-23 1117172]# mkdir a
[root@dht-rhs-23 1117172]# mv a b
[root@dht-rhs-23 1117172]# mv b c
[root@dht-rhs-23 1117172]# mv c d
[root@dht-rhs-23 1117172]# mv d e
[root@dht-rhs-23 1117172]# mv e f
[root@dht-rhs-23 1117172]# mv f g
[root@dht-rhs-23 1117172]# mv g h
[root@dht-rhs-23 1117172]# mv h i
[root@dht-rhs-23 1117172]# mv i j 
[root@dht-rhs-23 1117172]# mv j k 
[root@dht-rhs-23 1117172]# mv k l
[root@dht-rhs-23 1117172]# mv l m
[root@dht-rhs-23 1117172]# ls -ltrh
total 0
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba1
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba2
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba3
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba4
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba5
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba6
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba7
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba8
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba9
drwxr-xr-x. 2 root root 18 Jun 12 21:30 kjihgfedcba10
drwxr-xr-x. 2 root root 18 Jun 12 21:33 m
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i l$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i m$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i n$i; done
[root@dht-rhs-23 1117172]# for i in `ls`; do mv $i i$i; done
[root@dht-rhs-23 1117172]# 
[root@dht-rhs-23 1117172]# ls -ltrh
total 0
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba1
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba2
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba3
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba4
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba5
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba6
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba7
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba8
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba9
drwxr-xr-x. 2 root root 18 Jun 12 21:30 inmlkjihgfedcba10
drwxr-xr-x. 2 root root 18 Jun 12 21:33 inmlm
[root@dht-rhs-23 1117172]# 

======

2. Created, renamed, moved the link and read the file to ensure renames/readdirp/cat/lookup in a loop. No error seen in log files.

while true ; do ln  abc-ln abc; mv -f abc-ln abc; echo 3>/proc/sys/vm/drop_caches; cat abc; ln abc abc-ln; mv -f abc abc-ln; cho 3>/proc/sys/vm/drop_caches; while true ; do ln  abc-ln abc; mv -f abc-ln abc; echo 3>/proc/sys/vm/drop_caches; cat abc; ln abc abc-ln; mv -f abc abc-ln; echo 3>/proc/sys/vm/drop_caches; cat abc-ln; done; done

Volume is mounted direct-io to eliminate caches..

[root@dht-rhs-23 glusterfs]# mount | grep gluster
10.70.47.174:/testvol on /mnt/glusterfs type fuse.glusterfs (rw,direct-io-mode=yes)
[root@dht-rhs-23 glusterfs]# 


Marking the bug verified.

Comment 14 errata-xmlrpc 2015-07-29 04:33:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1495.html