Description of problem: on Distributed volume, file rename operation is completed without fail but brick log has errors like renaming form f11 to mv11 [2013-11-14 02:31:26.572912] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/mvt/def/f11: key:trusted.glusterfs.dht.linkto error:File exists [2013-11-14 02:31:26.572962] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/mvt/def/f11 failed (File exists) -- > when rename operation is successful and xattr is pointing to correct location why these Errors verified on backend. file has trusted.glusterfs.dht.linkto xattr and it is pointing to correct sub-volume. [root@7-VM3 ~]# ls -l /rhs/brick4/d*/mvt/def/mv11 -rw-rw-r-- 3 503 503 0 Nov 14 07:58 /rhs/brick4/d1/mvt/def/mv11 ---------T 3 503 503 0 Nov 14 07:58 /rhs/brick4/d2/mvt/def/mv11 [root@7-VM3 ~]# getfattr -d -m . /rhs/brick4/d2/mvt/def/mv11 getfattr: Removing leading '/' from absolute path names # file: rhs/brick4/d2/mvt/def/mv11 trusted.gfid=0s5VrPgZBWSc+7MAQOxLMnXQ== trusted.glusterfs.dht.linkto="dht-client-0" Version-Release number of selected component (if applicable): 3.4.0.44rhs-1.el6rhs.x86_64 How reproducible: always reproducible but not giving error for all file rename operation Steps to Reproduce: 1.rename 100 files(where destination file is not present and destination hash and cached sub-vol is different) and will get error in brick log for few files 2. 3. Actual results: brick log [2013-11-14 02:43:38.966275] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f31: key:trusted.glusterfs.dht.linkto error:File exists [2013-11-14 02:43:38.966329] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f31 failed (File exists) [2013-11-14 02:43:39.158963] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f48: key:trusted.glusterfs.dht.linkto error:File exists [2013-11-14 02:43:39.159024] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f48 failed (File exists) [2013-11-14 02:43:39.369143] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f67: key:trusted.glusterfs.dht.linkto error:File exists [2013-11-14 02:43:39.369205] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f67 failed (File exists) [2013-11-14 02:43:39.510074] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f78: key:trusted.glusterfs.dht.linkto error:File exists [2013-11-14 02:43:39.510125] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f78 failed (File exists) [2013-11-14 02:43:39.577798] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f85: key:trusted.glusterfs.dht.linkto error:File exists [2013-11-14 02:43:39.577914] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f85 failed (File exists) [2013-11-14 02:43:39.644510] E [posix-helpers.c:809:posix_handle_pair] 0-dht-posix: /rhs/brick4/d2/ag100/f91: key:trusted.glusterfs.dht.linkto error:File exists [2013-11-14 02:43:39.644571] E [posix.c:915:posix_mknod] 0-dht-posix: setting xattrs on /rhs/brick4/d2/ag100/f91 failed (File exists) Expected results: log should not have errors; if rename is successful Additional info:
Add hit this bug in 3.5.2 ubuntu xfs. [2014-08-25 22:04:39.975617] E [posix-helpers.c:893:posix_handle_pair] 0-sas03-posix: /brick03/gfs/DevMordorHomeSata03//hcamara//custom_interim_reports//lumber_liquidators/weekly_report/lumber_liquidator_weekly_custom_pos.py: key:trusted.glusterfs.dht.linkto error:File exists [2014-08-25 22:04:39.975634] E [posix.c:1177:posix_mknod] 0-sas03-posix: setting xattrs on /brick03/gfs/DevMordorHomeSata03//hcamara//custom_interim_reports//lumber_liquidators/weekly_report/lumber_liquidator_weekly_custom_pos.py failed (File exists)
Would like bug be fix in 3.5.3? I keep getting random Error messages like these in 3.5.2: 2014-08-27 21:56:33.580521] E [posix-helpers.c:893:posix_handle_pair] 0-sas03-posix: /brick03/gfs/DevMordorHomeSata03//hcamara//custom_interim_reports//ospgroup/womanwithin/weekly_report/mtd/survey_questionstest.txt: key:trusted.glusterfs.dht.linkto error:File exists &yellow .2014-08-27 21:56:33.580538] E [posix.c:1177:posix_mknod] 0-sas03-posix: setting xattrs on /brick03/gfs/DevMordorHomeSata03//hcamara//custom_interim_reports//ospgroup/womanwithin/weekly_report/mtd/survey_questionstest.txt failed (File exists) Thanks Peter
Seeing this issue on "glusterfs 3.6.0.28 built on Sep 3 2014 10:13:12" . Following is the case executed: =============================== 1. Create 1 x 2 replicate volume. 2. From 2 clients create 1 fuse and 1 nfs mount on each client with " -o direct-io-mode=yes" option 3. From 1st client fuse mount execute the following: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ echo abc > abc-ln while true; do ln abc-ln abc; mv -f abc-ln abc; echo 3>/proc/sys/vm/drop_caches; cat abc; ln abc abc-ln; mv -f abc abc-ln; echo 3>/proc/sys/vm/drop_caches; cat abc-ln; done From 1st client nfs mount execute the following: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ while true; do find . | xargs stat ; done From 2nd client fuse mount execute the following: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ while true; do find . | xargs stat ; done From 2nd client nfs mount execute the following: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ while true; do ls -liR ; done 4. add 2 more bricks to the volume to make it 2x2 dis-replicate volume In the newly added brick logs saw the messages : [2014-09-18 04:09:02.055008] E [posix-helpers.c:948:posix_handle_pair] 0-vol1-posix: /rhs/device0/b3/abc-ln: key:trusted.glusterfs.dht.linkto flags: 1 length:17 error:File exists [2014-09-18 04:09:02.055035] E [posix.c:1181:posix_mknod] 0-vol1-posix: setting xattrs on /rhs/device0/b3/abc-ln failed (File exists) root@rhs-client13 [Sep-18-2014- 6:51:05] >grep "posix_handle_pair" /var/log/glusterfs/bricks/rhs-device0-b3.log | wc 420414 5044968 74623494 root@rhs-client13 [Sep-18-2014- 6:51:38] >grep "posix_mknod" /var/log/glusterfs/bricks/rhs-device0-b3.log | wc 420414 5044968 55704864 root@rhs-client13 [Sep-18-2014- 6:51:58] >
Cloning this to 3.1. To be fixed in future release.