Bug 1139676 - DHT + Rebalance + rename of files and Directory :- more than 80% files are missing from mount, not able to access them from mount
Summary: DHT + Rebalance + rename of files and Directory :- more than 80% files are mi...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: rhgs-3.0
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Raghavendra G
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard: dht-file-access
: 1140660 (view as bug list)
Depends On:
Blocks: 1087818
TreeView+ depends on / blocked
 
Reported: 2014-09-09 12:18 UTC by Rachana Patel
Modified: 2016-06-24 05:07 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Renaming a directory may cause both source and target directories to exist on the volume with the same GFID and make some files in these directories not visible from the mount point. The files will still be present on the bricks. Details on how to ensure that two directories do not have the same GFID is available at: https://access.redhat.com/solutions/1211133
Clone Of:
Environment:
Last Closed: 2016-06-24 05:07:42 UTC
Embargoed:


Attachments (Terms of Use)

Description Rachana Patel 2014-09-09 12:18:55 UTC
Description of problem:
========================
DHT + Rebalance + rename of files and Directory :- more than 80% files from that Directory are missing from mount. even though those are present on bricks,  not able to access them from mount at all and able to create new file with that name.

had 100 files(f{1..100}-*) in Directory but after multiple rename of that Directory and files inside that directory have only below files (rebalance was running at same time)

[root@dht16 .glusterfs]# ls /mnt/case1/dir1-35/
f30-101  f31-101  f36-101  f42-101  f43-101  f44-101  f45-101  f46-101  f47-101  f48-101  f49-101  f51-101  f53-101  

file f68-12 is present on backend, try to access that mount point :-

[root@dht16 nfs1]# ls dir1-35/f68-12
ls: cannot access dir1-35/f68-12: No such file or directory

now create file with same name from mount:-

[root@dht16 nfs1]# touch dir1-35/f68-12

[root@dht16 nfs1]# ls dir1-35/f68-12 -il
12144294061717970665 -rw-r--r-- 1 root root 0 Sep  9 12:31 dir1-35/f68-12



Version-Release number of selected component (if applicable):
=============================================================
3.6.0.28-1.el6rhs.x86_64



How reproducible:
=================
always


Steps to Reproduce:
===================
1. created distributed volume - 31 bricks - FUSE and nfs mount it and created data inside it.
2. one directory dir1-1 has 100 files inside it
3. added 2 bricks  and started rebalance
[root@dht16 dir1-1]# gluster volume add-brick case1 dht19.lab.eng.blr.redhat.com:/brick1/new13 dht18.lab.eng.blr.redhat.com:/brick1/new13
volume add-brick: success
[root@dht16 dir1-1]#  gluster volume add-brick case1 start force

4. start renaming directory dir1-1 from multiple mount and same time start renaming all files inside it

5. once rebalance and rename is completed check files .

- 2 directory has same gfid and both of them shows only few files

Mount:-

[root@dht16 nfs1]# ls -il
total 132
11485443800915317544 drwxr-xr-x   2 root root   1342 Sep  9 12:31 dir1-25
11485443800915317544 drwxr-xr-x   2 root root   2795 Sep  9 11:55 dir1-35
10914239757854688148 drwxr-xr-x   3 root root   1718 Sep  9 11:55 git
10959170655335662409 drwxr-xr-x 106 root root 135168 Sep  9 12:17 rename
13582860081623689502 drwxr-xr-x   2 root root    198 Sep  9 12:18 rename1

[root@dht16 nfs1]# ls -l dir*
dir1-25:
total 53248
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f30-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f31-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f36-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f42-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f43-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f44-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f45-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f46-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f47-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f48-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f49-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f51-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f53-101

dir1-35:
total 53248
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f30-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f31-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f36-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f42-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:46 f43-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f44-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f45-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f46-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f47-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f48-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f49-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f51-101
-rw-r--r-- 2 root root 4194304 Sep  9 11:47 f53-101

----> It shows files(13 files) only present in dir1-25, remaining 87 files are in dir1-35 and unable to access those files from mount point

----> able to create new files which has same name 

e.g. 
mount:-
[root@dht16 nfs1]# ls dir1-35/f68-12
ls: cannot access dir1-35/f68-12: No such file or directory

backend:-
134330444 ---------T 2 root root       0 Sep  9 11:55 /brick1/new5/dir1-35/f68-12
117442323 -rw-r--r-- 2 root root 4194304 Sep  9 11:47 /brick0/case17/dir1-35/f68-12

mount:-
[root@dht16 nfs1]# touch dir1-35/f68-12

[root@dht16 nfs1]# ls dir1-35/f68-12 -il
12144294061717970665 -rw-r--r-- 1 root root 0 Sep  9 12:31 dir1-35/f68-12


backend:-
[root@OVM1 ~]# grep f68- /result1
 16784513 -rw-r--r-- 2 root root       0 Sep  9 12:31 /brick1/new4/dir1-25/f68-12
134330444 ---------T 2 root root       0 Sep  9 11:55 /brick1/new5/dir1-35/f68-12
117442323 -rw-r--r-- 2 root root 4194304 Sep  9 11:47 /brick0/case17/dir1-35/f68-12



Actual results:
===============
87 files are missing from mount and no way to access those files from mount


Expected results:
================
all 100 files should be accessible from mount

Additional info:

Comment 7 Shalaka 2014-09-25 11:53:57 UTC
Updated the doc text.

Comment 8 Raghavendra G 2014-10-20 06:19:15 UTC
*** Bug 1140660 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.