Description of problem: ======================= was trying to reproduce bug 1110730. kept renaming all files from mountpoint while rebalance was in progress and while rebalance and rename got over found that for few files <old file> and <new file> both are present having same gfid (i.e. hard link of file). rather than renaming from old to new it has created link file for it. (or raname operation was not completed, it has created link file but unlink of source failed) Version-Release number of selected component (if applicable): ============================================================= 3.6.0.19-1.el6rhs.x86_64 How reproducible: ================= Intermittent Steps to Reproduce: ==================== 1. create and mount(NFS) distributed volume( 3 bricks) 2. add 2 more bricks. 3. start creating files and Dires on mount point. start rebalance. 4. from mount point rename files while rebalance is in progress (cd /mnt/nfs; for i in {1..100} ; do mv new$i $i ; cd $i; mv etcn$i etc$i; for j in {1..100}; do mv newf$j file$j; done ; done cd /mnt/nfs; for i in {1..100} ; do mv $i new$i ; cd new$i; mv etc$i etcn$i; for j in {1..100}; do mv file$j newf$j; done ; done ... ... ) after 2-3 times stop renaming in between 5. once rebalance is finish. check data from mount point. for few files source and destination both files exist e.g. [root@OVM1 47]# ls -l total 36 drwxr-xr-x 4 root root 1557 Jun 25 12:40 48 drwxr-xr-x 110 root root 36864 Jun 25 13:36 etc47 -rw-r--r-- 1 root root 0 Jun 17 14:01 file1 -rw-r--r-- 1 root root 0 Jun 17 14:01 file10 -rw-r--r-- 1 root root 0 Jun 17 14:01 file100 -rw-r--r-- 1 root root 0 Jun 17 14:01 file11 -rw-r--r-- 1 root root 0 Jun 17 14:01 file12 -rw-r--r-- 2 root root 0 Jun 17 14:01 file13 -rw-r--r-- 1 root root 0 Jun 17 14:01 file14 -rw-r--r-- 1 root root 0 Jun 17 14:01 file15 -rw-r--r-- 1 root root 0 Jun 17 14:01 file16 -rw-r--r-- 1 root root 0 Jun 17 14:01 file17 -rw-r--r-- 1 root root 0 Jun 17 14:01 file18 -rw-r--r-- 1 root root 0 Jun 17 14:01 file19 -rw-r--r-- 1 root root 0 Jun 17 14:01 file2 -rw-r--r-- 1 root root 0 Jun 17 14:01 file20 -rw-r--r-- 1 root root 0 Jun 17 14:01 file21 -rw-r--r-- 1 root root 0 Jun 17 14:01 file22 -rw-r--r-- 1 root root 0 Jun 17 14:01 file23 -rw-r--r-- 1 root root 0 Jun 17 14:01 file24 -rw-r--r-- 1 root root 0 Jun 17 14:01 file25 -rw-r--r-- 1 root root 0 Jun 17 14:01 file26 -rw-r--r-- 1 root root 0 Jun 17 14:01 file27 -rw-r--r-- 1 root root 0 Jun 17 14:01 file28 -rw-r--r-- 1 root root 0 Jun 17 14:01 file29 -rw-r--r-- 1 root root 0 Jun 17 14:01 file3 -rw-r--r-- 1 root root 0 Jun 17 14:01 file30 -rw-r--r-- 1 root root 0 Jun 17 14:01 file31 -rw-r--r-- 2 root root 0 Jun 17 14:01 file32 -rw-r--r-- 1 root root 0 Jun 17 14:01 file33 -rw-r--r-- 1 root root 0 Jun 17 14:01 file34 -rw-r--r-- 1 root root 0 Jun 17 14:01 file35 -rw-r--r-- 1 root root 0 Jun 17 14:01 file36 -rw-r--r-- 1 root root 0 Jun 17 14:01 file37 -rw-r--r-- 1 root root 0 Jun 17 14:01 file38 -rw-r--r-- 1 root root 0 Jun 17 14:01 file39 -rw-r--r-- 1 root root 0 Jun 17 14:01 file4 -rw-r--r-- 1 root root 0 Jun 17 14:01 file40 -rw-r--r-- 1 root root 0 Jun 17 14:01 file41 -rw-r--r-- 1 root root 0 Jun 17 14:01 file42 -rw-r--r-- 1 root root 0 Jun 17 14:01 file43 -rw-r--r-- 2 root root 0 Jun 17 14:01 file44 -rw-r--r-- 1 root root 0 Jun 17 14:01 file45 -rw-r--r-- 2 root root 0 Jun 17 14:01 file46 -rw-r--r-- 1 root root 0 Jun 17 14:01 file47 -rw-r--r-- 1 root root 0 Jun 17 14:01 file48 -rw-r--r-- 1 root root 0 Jun 17 14:01 file49 -rw-r--r-- 1 root root 0 Jun 17 14:01 file5 -rw-r--r-- 1 root root 0 Jun 17 14:01 file50 -rw-r--r-- 1 root root 0 Jun 17 14:01 file51 -rw-r--r-- 1 root root 0 Jun 17 14:01 file52 -rw-r--r-- 2 root root 0 Jun 17 14:01 file53 -rw-r--r-- 1 root root 0 Jun 17 14:01 file54 -rw-r--r-- 2 root root 0 Jun 30 11:33 file55 -rw-r--r-- 1 root root 0 Jun 17 14:01 file56 -rw-r--r-- 1 root root 0 Jun 17 14:01 file57 -rw-r--r-- 1 root root 0 Jun 17 14:01 file58 -rw-r--r-- 1 root root 0 Jun 17 14:01 file59 -rw-r--r-- 1 root root 0 Jun 17 14:01 file6 -rw-r--r-- 1 root root 0 Jun 17 14:01 file60 -rw-r--r-- 1 root root 0 Jun 17 14:01 file61 -rw-r--r-- 2 root root 0 Jun 17 14:01 file62 -rw-r--r-- 1 root root 0 Jun 17 14:01 file63 -rw-r--r-- 1 root root 0 Jun 17 14:01 file64 -rw-r--r-- 1 root root 0 Jun 17 14:01 file65 -rw-r--r-- 1 root root 0 Jun 17 14:01 file66 -rw-r--r-- 1 root root 0 Jun 17 14:01 file67 -rw-r--r-- 1 root root 0 Jun 17 14:01 file68 -rw-r--r-- 1 root root 0 Jun 17 14:01 file69 -rw-r--r-- 1 root root 0 Jun 17 14:01 file7 -rw-r--r-- 1 root root 0 Jun 17 14:01 file70 -rw-r--r-- 1 root root 0 Jun 17 14:01 file71 -rw-r--r-- 1 root root 0 Jun 17 14:01 file72 -rw-r--r-- 1 root root 0 Jun 17 14:01 file73 -rw-r--r-- 1 root root 0 Jun 17 14:01 file74 -rw-r--r-- 1 root root 0 Jun 17 14:01 file75 -rw-r--r-- 1 root root 0 Jun 17 14:01 file76 -rw-r--r-- 1 root root 0 Jun 17 14:01 file77 -rw-r--r-- 1 root root 0 Jun 17 14:01 file78 -rw-r--r-- 1 root root 0 Jun 17 14:01 file79 -rw-r--r-- 1 root root 0 Jun 17 14:01 file8 -rw-r--r-- 1 root root 0 Jun 17 14:01 file80 -rw-r--r-- 1 root root 0 Jun 17 14:01 file81 -rw-r--r-- 1 root root 0 Jun 17 14:01 file82 -rw-r--r-- 1 root root 0 Jun 17 14:01 file83 -rw-r--r-- 1 root root 0 Jun 17 14:01 file84 -rw-r--r-- 1 root root 0 Jun 17 14:01 file85 -rw-r--r-- 1 root root 0 Jun 17 14:01 file86 -rw-r--r-- 1 root root 0 Jun 17 14:01 file87 -rw-r--r-- 1 root root 0 Jun 17 14:01 file88 -rw-r--r-- 1 root root 0 Jun 17 14:01 file89 -rw-r--r-- 1 root root 0 Jun 17 14:01 file9 -rw-r--r-- 1 root root 0 Jun 17 14:01 file90 -rw-r--r-- 1 root root 0 Jun 17 14:01 file91 -rw-r--r-- 1 root root 0 Jun 17 14:01 file92 -rw-r--r-- 1 root root 0 Jun 17 14:01 file93 -rw-r--r-- 1 root root 0 Jun 17 14:01 file94 -rw-r--r-- 1 root root 0 Jun 17 14:01 file95 -rw-r--r-- 1 root root 0 Jun 17 14:01 file96 -rw-r--r-- 1 root root 0 Jun 17 14:01 file97 -rw-r--r-- 1 root root 0 Jun 17 14:01 file98 -rw-r--r-- 1 root root 0 Jun 17 14:01 file99 -rw-r--r-- 2 root root 0 Jun 17 14:01 newf13 <----------- -rw-r--r-- 2 root root 0 Jun 17 14:01 newf32 -rw-r--r-- 2 root root 0 Jun 17 14:01 newf44 -rw-r--r-- 2 root root 0 Jun 17 14:01 newf46 -rw-r--r-- 2 root root 0 Jun 17 14:01 newf53 -rw-r--r-- 2 root root 0 Jun 30 11:33 newf55 -rw-r--r-- 2 root root 0 Jun 17 14:01 newf62 verify same on backend # file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46 trusted.gfid=0xdba5f967b2334725905bab16cbc39f88 -- # file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf46 trusted.gfid=0xdba5f967b2334725905bab16cbc39f88 ---------------------- # file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32 trusted.gfid=0xe5cf1c5ee07a413ca0f5c27660871a1a -- # file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf32 trusted.gfid=0xe5cf1c5ee07a413ca0f5c27660871a1a ---------------------- # file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file55 trusted.gfid=0x81ca84574f594e25ad9f2de73d218f75 -- # file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf55 trusted.gfid=0x81ca84574f594e25ad9f2de73d218f75 ---------------------- # file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13 trusted.gfid=0x3e7dbc6d66844289b734d713e6212ec6 -- # file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf13 trusted.gfid=0x3e7dbc6d66844289b734d713e6212ec6 ########### [root@OVM3 tmp]# stat /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13 File: `/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fd06h/64774d Inode: 8613923 Links: 3 Access: (1000/---------T) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2014-06-25 12:38:53.427004085 +0530 Modify: 2014-06-25 12:38:53.427004085 +0530 Change: 2014-06-25 12:38:53.484004085 +0530 log snippet :- [root@OVM3 ~]# grep '/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32' /var/log/glusterfs/* -R /var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.487798] E [posix-helpers.c:930:posix_handle_pair] 0-test1-posix: /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32: key:trusted.glusterfs.dht.linkto flags: 1 length:15 error:File exists /var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.487841] E [posix.c:1169:posix_mknod] 0-test1-posix: setting xattrs on /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32 failed (File exists) [root@OVM3 ~]# grep '/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46' /var/log/glusterfs/* -R /var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.491410] E [posix-helpers.c:930:posix_handle_pair] 0-test1-posix: /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46: key:trusted.glusterfs.dht.linkto flags: 1 length:15 error:File exists /var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.491442] E [posix.c:1169:posix_mknod] 0-test1-posix: setting xattrs on /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46 failed (File exists) Actual results: =============== (Due to some race condition or self healing or because of some other reason)even after rename operation is completed, source and destination file exist with same gfid Expected results: =============== rename operation should be completed successfully and it should unlink source file
Hit the same issue while doing rename and rebalance on samba share from windows client. Steps to reproduce: create a volume and mount it via cifs create a doc Set acl for it Do add-brick, when add-brick is in progress do multiple renames of the same file. Run rebalance After rebalance and rename is finished ,on the mount point both the files exists. On the back-end it exists with the same gfid.
*** Bug 1286098 has been marked as a duplicate of this bug. ***