Description of problem: ======================== DHT + Rebalance + rename of files and Directory :- more than 80% files from that Directory are missing from mount. even though those are present on bricks, not able to access them from mount at all and able to create new file with that name. had 100 files(f{1..100}-*) in Directory but after multiple rename of that Directory and files inside that directory have only below files (rebalance was running at same time) [root@dht16 .glusterfs]# ls /mnt/case1/dir1-35/ f30-101 f31-101 f36-101 f42-101 f43-101 f44-101 f45-101 f46-101 f47-101 f48-101 f49-101 f51-101 f53-101 file f68-12 is present on backend, try to access that mount point :- [root@dht16 nfs1]# ls dir1-35/f68-12 ls: cannot access dir1-35/f68-12: No such file or directory now create file with same name from mount:- [root@dht16 nfs1]# touch dir1-35/f68-12 [root@dht16 nfs1]# ls dir1-35/f68-12 -il 12144294061717970665 -rw-r--r-- 1 root root 0 Sep 9 12:31 dir1-35/f68-12 Version-Release number of selected component (if applicable): ============================================================= 3.6.0.28-1.el6rhs.x86_64 How reproducible: ================= always Steps to Reproduce: =================== 1. created distributed volume - 31 bricks - FUSE and nfs mount it and created data inside it. 2. one directory dir1-1 has 100 files inside it 3. added 2 bricks and started rebalance [root@dht16 dir1-1]# gluster volume add-brick case1 dht19.lab.eng.blr.redhat.com:/brick1/new13 dht18.lab.eng.blr.redhat.com:/brick1/new13 volume add-brick: success [root@dht16 dir1-1]# gluster volume add-brick case1 start force 4. start renaming directory dir1-1 from multiple mount and same time start renaming all files inside it 5. once rebalance and rename is completed check files . - 2 directory has same gfid and both of them shows only few files Mount:- [root@dht16 nfs1]# ls -il total 132 11485443800915317544 drwxr-xr-x 2 root root 1342 Sep 9 12:31 dir1-25 11485443800915317544 drwxr-xr-x 2 root root 2795 Sep 9 11:55 dir1-35 10914239757854688148 drwxr-xr-x 3 root root 1718 Sep 9 11:55 git 10959170655335662409 drwxr-xr-x 106 root root 135168 Sep 9 12:17 rename 13582860081623689502 drwxr-xr-x 2 root root 198 Sep 9 12:18 rename1 [root@dht16 nfs1]# ls -l dir* dir1-25: total 53248 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f30-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f31-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f36-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f42-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f43-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f44-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f45-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f46-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f47-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f48-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f49-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f51-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f53-101 dir1-35: total 53248 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f30-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f31-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f36-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f42-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:46 f43-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f44-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f45-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f46-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f47-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f48-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f49-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f51-101 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 f53-101 ----> It shows files(13 files) only present in dir1-25, remaining 87 files are in dir1-35 and unable to access those files from mount point ----> able to create new files which has same name e.g. mount:- [root@dht16 nfs1]# ls dir1-35/f68-12 ls: cannot access dir1-35/f68-12: No such file or directory backend:- 134330444 ---------T 2 root root 0 Sep 9 11:55 /brick1/new5/dir1-35/f68-12 117442323 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 /brick0/case17/dir1-35/f68-12 mount:- [root@dht16 nfs1]# touch dir1-35/f68-12 [root@dht16 nfs1]# ls dir1-35/f68-12 -il 12144294061717970665 -rw-r--r-- 1 root root 0 Sep 9 12:31 dir1-35/f68-12 backend:- [root@OVM1 ~]# grep f68- /result1 16784513 -rw-r--r-- 2 root root 0 Sep 9 12:31 /brick1/new4/dir1-25/f68-12 134330444 ---------T 2 root root 0 Sep 9 11:55 /brick1/new5/dir1-35/f68-12 117442323 -rw-r--r-- 2 root root 4194304 Sep 9 11:47 /brick0/case17/dir1-35/f68-12 Actual results: =============== 87 files are missing from mount and no way to access those files from mount Expected results: ================ all 100 files should be accessible from mount Additional info:
Updated the doc text.
*** Bug 1140660 has been marked as a duplicate of this bug. ***