Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 1118770 - DHT : If Directory creation is in progress and rename of that Directory comes from another mount point then after both operation few files are not accessible and not listed on mount and more than one Directory have same gfid [NEEDINFO]
DHT : If Directory creation is in progress and rename of that Directory comes...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
3.0
x86_64 Linux
unspecified Severity high
: ---
: RHGS 3.4.0
Assigned To: Raghavendra G
Prasad Desala
triaged,hotfix, dht-directory-consist...
: ZStream
Depends On: 1369312
Blocks: 1118762 1252244 1311843 1324381 1336698 1337022 1337394 1345828 1503134
  Show dependency treegraph
 
Reported: 2014-07-11 09:43 EDT by Rachana Patel
Modified: 2018-09-17 07:05 EDT (History)
13 users (show)

See Also:
Fixed In Version: glusterfs-3.12.2-1
Doc Type: Bug Fix
Doc Text:
Previously, rename and lookup option were not synchronized. The source gfid was not always associated with destination path once rename is complete and hence contents of the directories were not visible. This resulted in the creation of different directories with the same gfid.
Story Points: ---
Clone Of:
: 1252244 1328473 (view as bug list)
Environment:
Last Closed: 2018-09-04 02:26:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
srmukher: needinfo? (rgowdapp)


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 None None None 2018-09-04 02:28 EDT

  None (edit)
Description Rachana Patel 2014-07-11 09:43:26 EDT
Description of problem:
=======================
Create Directory from mountpoint and while creation is in progress (Directory created only on hashed sub-volume), execute rename of that Directory(destination Directory does not exist and both Source and Destination hash to same sub-volume here)

i.e. from one mout point :- mkfir dir1
from another mount point mv dir1 dir2

After both operation are finished:-
- same gfid for different Directories (at same level)
- sometimes few files inside those directories are not listed on mount and not accessible


Version-Release number :
=========================
3.6.0.24-1.el6rhs.x86_64


How reproducible:
=================
always


Steps to Reproduce:
====================
1. create and mount distributed volume. (mount on multiple client)
2. [to reproduce race, we are putting breakpoint at dht_mkdir_hashed_dir_cbk and dht_rename_hashed_dir_cbk]

3. from one mount point execute 

[root@OVM1 race]# mkdir inprogress

bricks:-
[root@OVM5 race]# tree /brick*/race/ 
/brick1/race/ 
/brick2/race/ 
└── inprogress 
/brick3/race/ 

1 directory, 0 files 

from another mount point:-
[root@OVM1 race1]# mv inprogress rename

bricks:-
[root@OVM5 race]# tree /brick*/race/ 
/brick1/race/ 
└── rename 
/brick2/race/ 
└── inprogress 
/brick3/race/ 
└── inprogress 

3 directories, 0 files 

4. now continue bothe operation

5. verify data from another mount and bricks also

mount:-
[root@OVM5 race]# ls -lR 
.: 
total 0 
drwxr-xr-x 2 root root 18 Jul 10 12:50 rename 

./rename: 
total 0 
[root@OVM5 race]# mkdir inprogress 
mkdir: cannot create directory `inprogress': File exists 
[root@OVM5 race]# ls -lR 
.: 
total 0 
drwxr-xr-x 2 root root 18 Jul 10 12:50 inprogress 
drwxr-xr-x 2 root root 18 Jul 10 12:50 rename 

./inprogress: 
total 0 

./rename: 
total 0 

bricks:-
same gfid:-
[root@OVM5 race]# getfattr -d -m . /brick3/race/* -e hex 
getfattr: Removing leading '/' from absolute path names 
# file: brick3/race/inprogress 
trusted.gfid=0x5b3c1a8ca4b84f27912880710a165fb7 
trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9 
 
# file: brick3/race/rename 
trusted.gfid=0x5b3c1a8ca4b84f27912880710a165fb7 
trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9 

[root@OVM5 race]# tree /brick*/race/ 
/brick1/race/ 
├── inprogress 
└── rename 
/brick2/race/ 
├── inprogress 
└── rename 
/brick3/race/ 
├── inprogress 
└── rename 

Actual results:
===============
- same gfid for different Directories 
- sometimes files inside those directories are not listed on mount and 

Expected results:
=================
- no two directory should have same gfid
- all files inside those Directories should be accessible from mount point
Comment 2 Rachana Patel 2014-07-11 09:47:33 EDT
In case if destination directory exist, output would be

rename1 already exist
and race :-
[root@OVM1 race]# mkdir rename
[root@OVM1 race1]# mv rename rename1


output on mount:-

[root@OVM5 race]# ls -lR 
.: 
total 0 
drwxr-xr-x 2 root root 18 Jul 10 15:00 rename 
drwxr-xr-x 3 root root 57 Jul 10 15:00 rename1 

./rename: 
total 0 

./rename1: 
total 0 
drwxr-xr-x 2 root root 18 Jul 10 15:00 rename 
 
./rename1/rename: 
total 0 


bricks:-
[root@OVM5 race]# tree /brick*/race/ 
/brick1/race/ 
├── rename 
└── rename1 
    └── rename 
/brick2/race/ 
├── rename 
└── rename1 
    └── rename 
/brick3/race/ 
├── rename 
└── rename1 
    └── rename 

9 directories, 0 files 

[root@OVM5 race]# getfattr -d -m . -e hex /brick3/race/* -R 
getfattr: Removing leading '/' from absolute path names 
# file: brick3/race/rename 
trusted.gfid=0xac6b95cb620c400d91a55f3ce66ee005 
trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff 

# file: brick3/race/rename1 
trusted.gfid=0x9482dd3bf0834596bb74d6ffeffa40d2 
trusted.glusterfs.dht=0x00000001000000000000000055555554 

# file: brick3/race/rename1/rename 
trusted.gfid=0xac6b95cb620c400d91a55f3ce66ee005 
trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff
Comment 7 Susant Kumar Palai 2015-12-28 01:25:45 EST
Triage Update: RCA is known. Design and fix need to be done.
Comment 12 Raghavendra G 2016-04-25 02:55:48 EDT
Sakshi,

Is there a reason, you moved the bug back to MODIFIED? The patch is already merged. Can you please revisit and move the bug to modified if there is no valid reason?

regards,
Raghavendra
Comment 16 krishnaram Karthick 2016-05-16 05:06:10 EDT
The issue reported in the bug is still seen in build - glusterfs-3.7.9-4.el7rhgs.x86_64

With parallel mkdir and rename operation, mismatching gfids is seen on the directory under test.

Moving this back to assigned.
Comment 22 Bhavana 2016-06-17 04:47:27 EDT
Thanks raghavendra,

I have updated the doc text further, let me know if this looks ok.
Comment 23 Raghavendra G 2016-06-23 01:20:01 EDT
Doc text seems fine too.
Comment 24 Raghavendra G 2017-08-22 05:56:41 EDT
upstream patch https://review.gluster.org/15472 fixes this issue
Comment 28 Prasad Desala 2018-04-20 09:15:50 EDT
Verified this BZ on glusterfs version 3.12.2-7.el7rhgs.x86_64.

Ran testcases mentioned in description, Comment2 and Comment16. I didn't see any gfid mismatch on the directory under test.

Moving this BZ to Verified.
Comment 29 Srijita Mukherjee 2018-09-03 09:38:37 EDT
Updated the doc text. Kindly review and confirm
Comment 31 errata-xmlrpc 2018-09-04 02:26:56 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607

Note You need to log in before you can comment on or make changes to this bug.