This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1030200 - DHT : file rename operation is successful but log has error 'key:trusted.glusterfs.dht.linkto error:File exists' , 'setting xattrs on <old_filename> failed (File exists)'
DHT : file rename operation is successful but log has error 'key:trusted.glus...
Status: CLOSED DEFERRED
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
2.1
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Nithya Balachandran
storage-qa-internal@redhat.com
:
Depends On:
Blocks: 1282318 1404905
  Show dependency treegraph
 
Reported: 2013-11-14 01:57 EST by Rachana Patel
Modified: 2016-12-14 22:46 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1282318 (view as bug list)
Environment:
Last Closed: 2015-11-27 07:32:22 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rachana Patel 2013-11-14 01:57:05 EST
Description of problem:
on Distributed volume, file rename operation is completed without fail but brick log has errors like

renaming form f11 to mv11

[2013-11-14 02:31:26.572912] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/mvt/def/f11: key:trusted.glusterfs.dht.linkto error:File exists
[2013-11-14 02:31:26.572962] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/mvt/def/f11 failed (File exists)

-- > when rename operation is successful and xattr is pointing to correct location why these Errors

verified on backend. file has trusted.glusterfs.dht.linkto xattr and it is pointing to correct sub-volume.

[root@7-VM3 ~]# ls -l /rhs/brick4/d*/mvt/def/mv11
-rw-rw-r-- 3 503 503 0 Nov 14 07:58 /rhs/brick4/d1/mvt/def/mv11
---------T 3 503 503 0 Nov 14 07:58 /rhs/brick4/d2/mvt/def/mv11


[root@7-VM3 ~]# getfattr -d -m . /rhs/brick4/d2/mvt/def/mv11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick4/d2/mvt/def/mv11
trusted.gfid=0s5VrPgZBWSc+7MAQOxLMnXQ==
trusted.glusterfs.dht.linkto="dht-client-0"



Version-Release number of selected component (if applicable):
3.4.0.44rhs-1.el6rhs.x86_64

How reproducible:
always reproducible but not giving error for all file rename operation

Steps to Reproduce:
1.rename 100 files(where destination file is not present and destination hash and cached sub-vol is different) and will get error in brick log for few files
2.
3.

Actual results:
brick log

[2013-11-14 02:43:38.966275] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f31: key:trusted.glusterfs.dht.linkto error:File exists
[2013-11-14 02:43:38.966329] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f31 failed (File exists)
[2013-11-14 02:43:39.158963] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f48: key:trusted.glusterfs.dht.linkto error:File exists
[2013-11-14 02:43:39.159024] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f48 failed (File exists)
[2013-11-14 02:43:39.369143] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f67: key:trusted.glusterfs.dht.linkto error:File exists
[2013-11-14 02:43:39.369205] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f67 failed (File exists)
[2013-11-14 02:43:39.510074] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f78: key:trusted.glusterfs.dht.linkto error:File exists
[2013-11-14 02:43:39.510125] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f78 failed (File exists)
[2013-11-14 02:43:39.577798] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f85: key:trusted.glusterfs.dht.linkto error:File exists
[2013-11-14 02:43:39.577914] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f85 failed (File exists)
[2013-11-14 02:43:39.644510] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f91: key:trusted.glusterfs.dht.linkto error:File exists
[2013-11-14 02:43:39.644571] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f91 failed (File exists)



Expected results:
log should not have errors; if rename is successful

Additional info:
Comment 5 Peter Auyeung 2014-08-25 18:13:26 EDT
Add hit this bug in 3.5.2 ubuntu xfs.

[2014-08-25 22:04:39.975617] E [posix-helpers.c:893:posix_handle_pair] 0-sas03-posix: /brick03/gfs/DevMordorHomeSata03//hcamara//custom_interim_reports//lumber_liquidators/weekly_report/lumber_liquidator_weekly_custom_pos.py: key:trusted.glusterfs.dht.linkto error:File exists
[2014-08-25 22:04:39.975634] E [posix.c:1177:posix_mknod] 0-sas03-posix: setting xattrs on /brick03/gfs/DevMordorHomeSata03//hcamara//custom_interim_reports//lumber_liquidators/weekly_report/lumber_liquidator_weekly_custom_pos.py failed (File exists)
Comment 6 Peter Auyeung 2014-08-27 18:49:08 EDT
Would like bug be fix in 3.5.3?

I keep getting random Error messages like these in 3.5.2:

2014-08-27 21:56:33.580521] E [posix-helpers.c:893:posix_handle_pair] 0-sas03-posix: /brick03/gfs/DevMordorHomeSata03//hcamara//custom_interim_reports//ospgroup/womanwithin/weekly_report/mtd/survey_questionstest.txt: key:trusted.glusterfs.dht.linkto error:File exists
&yellow .2014-08-27 21:56:33.580538] E [posix.c:1177:posix_mknod] 0-sas03-posix: setting xattrs on /brick03/gfs/DevMordorHomeSata03//hcamara//custom_interim_reports//ospgroup/womanwithin/weekly_report/mtd/survey_questionstest.txt failed (File exists)

Thanks
Peter
Comment 7 spandura 2014-09-18 02:52:39 EDT
Seeing this issue on "glusterfs 3.6.0.28 built on Sep  3 2014 10:13:12" . 

Following is the case executed:
===============================
1. Create 1 x 2 replicate volume. 

2. From 2 clients create 1 fuse and 1 nfs mount on each client with " -o direct-io-mode=yes" option

3. 
From 1st client fuse mount execute the following:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo abc > abc-ln

while true; do ln abc-ln abc; mv -f abc-ln abc; echo 3>/proc/sys/vm/drop_caches; cat abc; ln abc abc-ln; mv -f abc abc-ln; echo 3>/proc/sys/vm/drop_caches; cat abc-ln; done

From 1st client nfs mount execute the following:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
while true; do find . | xargs stat ; done

From 2nd client fuse mount execute the following:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
while true; do find . | xargs stat ; done

From 2nd client nfs mount execute the following:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
while true; do ls -liR ; done

4. add 2 more bricks to the volume to make it 2x2 dis-replicate volume

In the newly added brick logs saw the messages :
[2014-09-18 04:09:02.055008] E [posix-helpers.c:948:posix_handle_pair] 0-vol1-posix: /rhs/device0/b3/abc-ln: key:trusted.glusterfs.dht.linkto flags: 1 length:17 error:File exists
[2014-09-18 04:09:02.055035] E [posix.c:1181:posix_mknod] 0-vol1-posix: setting xattrs on /rhs/device0/b3/abc-ln failed (File exists)

root@rhs-client13 [Sep-18-2014- 6:51:05] >grep "posix_handle_pair" /var/log/glusterfs/bricks/rhs-device0-b3.log | wc
 420414 5044968 74623494
root@rhs-client13 [Sep-18-2014- 6:51:38] >grep "posix_mknod" /var/log/glusterfs/bricks/rhs-device0-b3.log | wc
 420414 5044968 55704864
root@rhs-client13 [Sep-18-2014- 6:51:58] >
Comment 9 Susant Kumar Palai 2015-11-27 07:32:22 EST
Cloning this to 3.1. To be fixed in future release.

Note You need to log in before you can comment on or make changes to this bug.