Bug 1114710 - DHT + Rebalance + rename :- while rebalance is in progress, for few files - renaming of files creates hard link for file rather than renaming
Summary: DHT + Rebalance + rename :- while rebalance is in progress, for few files - r...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: rhgs-3.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Raghavendra G
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard: dht-rename-file
: 1286098 (view as bug list)
Depends On:
Blocks: 1155463 1286098
TreeView+ depends on / blocked
 
Reported: 2014-06-30 18:02 UTC by Rachana Patel
Modified: 2017-02-13 08:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1155463 (view as bug list)
Environment:
Last Closed: 2017-02-13 08:03:32 UTC
Embargoed:


Attachments (Terms of Use)

Description Rachana Patel 2014-06-30 18:02:25 UTC
Description of problem:
=======================
was trying to reproduce  bug 1110730. kept renaming all files from mountpoint while rebalance was in progress and while rebalance and rename got over found that for few files <old file> and <new file> both are present having same gfid (i.e. hard link of file). 
rather than renaming from old to new it has created link file for it. (or raname operation was not completed, it has created link file but unlink of source failed)



Version-Release number of selected component (if applicable):
=============================================================
3.6.0.19-1.el6rhs.x86_64

How reproducible:
=================
Intermittent

Steps to Reproduce:
====================
1. create and mount(NFS) distributed volume( 3 bricks)
2. add 2 more bricks.
3. start creating files and Dires on mount point. start rebalance.
4. from mount point rename files while rebalance is in progress
(cd /mnt/nfs; for i in {1..100} ; do mv new$i $i ; cd $i; mv etcn$i etc$i; for j in {1..100}; do mv  newf$j file$j; done ; done
cd /mnt/nfs; for i in {1..100} ; do mv $i new$i ; cd new$i; mv etc$i etcn$i; for j in {1..100}; do mv  file$j newf$j; done ; done
...
...
)
after 2-3 times stop renaming in between
5. once rebalance is finish. check data from mount point. for few files source and destination both files exist

e.g. 
[root@OVM1 47]# ls -l
total 36
drwxr-xr-x   4 root root  1557 Jun 25 12:40 48
drwxr-xr-x 110 root root 36864 Jun 25 13:36 etc47
-rw-r--r--   1 root root     0 Jun 17 14:01 file1
-rw-r--r--   1 root root     0 Jun 17 14:01 file10
-rw-r--r--   1 root root     0 Jun 17 14:01 file100
-rw-r--r--   1 root root     0 Jun 17 14:01 file11
-rw-r--r--   1 root root     0 Jun 17 14:01 file12
-rw-r--r--   2 root root     0 Jun 17 14:01 file13
-rw-r--r--   1 root root     0 Jun 17 14:01 file14
-rw-r--r--   1 root root     0 Jun 17 14:01 file15
-rw-r--r--   1 root root     0 Jun 17 14:01 file16
-rw-r--r--   1 root root     0 Jun 17 14:01 file17
-rw-r--r--   1 root root     0 Jun 17 14:01 file18
-rw-r--r--   1 root root     0 Jun 17 14:01 file19
-rw-r--r--   1 root root     0 Jun 17 14:01 file2
-rw-r--r--   1 root root     0 Jun 17 14:01 file20
-rw-r--r--   1 root root     0 Jun 17 14:01 file21
-rw-r--r--   1 root root     0 Jun 17 14:01 file22
-rw-r--r--   1 root root     0 Jun 17 14:01 file23
-rw-r--r--   1 root root     0 Jun 17 14:01 file24
-rw-r--r--   1 root root     0 Jun 17 14:01 file25
-rw-r--r--   1 root root     0 Jun 17 14:01 file26
-rw-r--r--   1 root root     0 Jun 17 14:01 file27
-rw-r--r--   1 root root     0 Jun 17 14:01 file28
-rw-r--r--   1 root root     0 Jun 17 14:01 file29
-rw-r--r--   1 root root     0 Jun 17 14:01 file3
-rw-r--r--   1 root root     0 Jun 17 14:01 file30
-rw-r--r--   1 root root     0 Jun 17 14:01 file31
-rw-r--r--   2 root root     0 Jun 17 14:01 file32
-rw-r--r--   1 root root     0 Jun 17 14:01 file33
-rw-r--r--   1 root root     0 Jun 17 14:01 file34
-rw-r--r--   1 root root     0 Jun 17 14:01 file35
-rw-r--r--   1 root root     0 Jun 17 14:01 file36
-rw-r--r--   1 root root     0 Jun 17 14:01 file37
-rw-r--r--   1 root root     0 Jun 17 14:01 file38
-rw-r--r--   1 root root     0 Jun 17 14:01 file39
-rw-r--r--   1 root root     0 Jun 17 14:01 file4
-rw-r--r--   1 root root     0 Jun 17 14:01 file40
-rw-r--r--   1 root root     0 Jun 17 14:01 file41
-rw-r--r--   1 root root     0 Jun 17 14:01 file42
-rw-r--r--   1 root root     0 Jun 17 14:01 file43
-rw-r--r--   2 root root     0 Jun 17 14:01 file44
-rw-r--r--   1 root root     0 Jun 17 14:01 file45
-rw-r--r--   2 root root     0 Jun 17 14:01 file46
-rw-r--r--   1 root root     0 Jun 17 14:01 file47
-rw-r--r--   1 root root     0 Jun 17 14:01 file48
-rw-r--r--   1 root root     0 Jun 17 14:01 file49
-rw-r--r--   1 root root     0 Jun 17 14:01 file5
-rw-r--r--   1 root root     0 Jun 17 14:01 file50
-rw-r--r--   1 root root     0 Jun 17 14:01 file51
-rw-r--r--   1 root root     0 Jun 17 14:01 file52
-rw-r--r--   2 root root     0 Jun 17 14:01 file53
-rw-r--r--   1 root root     0 Jun 17 14:01 file54
-rw-r--r--   2 root root     0 Jun 30 11:33 file55
-rw-r--r--   1 root root     0 Jun 17 14:01 file56
-rw-r--r--   1 root root     0 Jun 17 14:01 file57
-rw-r--r--   1 root root     0 Jun 17 14:01 file58
-rw-r--r--   1 root root     0 Jun 17 14:01 file59
-rw-r--r--   1 root root     0 Jun 17 14:01 file6
-rw-r--r--   1 root root     0 Jun 17 14:01 file60
-rw-r--r--   1 root root     0 Jun 17 14:01 file61
-rw-r--r--   2 root root     0 Jun 17 14:01 file62
-rw-r--r--   1 root root     0 Jun 17 14:01 file63
-rw-r--r--   1 root root     0 Jun 17 14:01 file64
-rw-r--r--   1 root root     0 Jun 17 14:01 file65
-rw-r--r--   1 root root     0 Jun 17 14:01 file66
-rw-r--r--   1 root root     0 Jun 17 14:01 file67
-rw-r--r--   1 root root     0 Jun 17 14:01 file68
-rw-r--r--   1 root root     0 Jun 17 14:01 file69
-rw-r--r--   1 root root     0 Jun 17 14:01 file7
-rw-r--r--   1 root root     0 Jun 17 14:01 file70
-rw-r--r--   1 root root     0 Jun 17 14:01 file71
-rw-r--r--   1 root root     0 Jun 17 14:01 file72
-rw-r--r--   1 root root     0 Jun 17 14:01 file73
-rw-r--r--   1 root root     0 Jun 17 14:01 file74
-rw-r--r--   1 root root     0 Jun 17 14:01 file75
-rw-r--r--   1 root root     0 Jun 17 14:01 file76
-rw-r--r--   1 root root     0 Jun 17 14:01 file77
-rw-r--r--   1 root root     0 Jun 17 14:01 file78
-rw-r--r--   1 root root     0 Jun 17 14:01 file79
-rw-r--r--   1 root root     0 Jun 17 14:01 file8
-rw-r--r--   1 root root     0 Jun 17 14:01 file80
-rw-r--r--   1 root root     0 Jun 17 14:01 file81
-rw-r--r--   1 root root     0 Jun 17 14:01 file82
-rw-r--r--   1 root root     0 Jun 17 14:01 file83
-rw-r--r--   1 root root     0 Jun 17 14:01 file84
-rw-r--r--   1 root root     0 Jun 17 14:01 file85
-rw-r--r--   1 root root     0 Jun 17 14:01 file86
-rw-r--r--   1 root root     0 Jun 17 14:01 file87
-rw-r--r--   1 root root     0 Jun 17 14:01 file88
-rw-r--r--   1 root root     0 Jun 17 14:01 file89
-rw-r--r--   1 root root     0 Jun 17 14:01 file9
-rw-r--r--   1 root root     0 Jun 17 14:01 file90
-rw-r--r--   1 root root     0 Jun 17 14:01 file91
-rw-r--r--   1 root root     0 Jun 17 14:01 file92
-rw-r--r--   1 root root     0 Jun 17 14:01 file93
-rw-r--r--   1 root root     0 Jun 17 14:01 file94
-rw-r--r--   1 root root     0 Jun 17 14:01 file95
-rw-r--r--   1 root root     0 Jun 17 14:01 file96
-rw-r--r--   1 root root     0 Jun 17 14:01 file97
-rw-r--r--   1 root root     0 Jun 17 14:01 file98
-rw-r--r--   1 root root     0 Jun 17 14:01 file99
-rw-r--r--   2 root root     0 Jun 17 14:01 newf13    <-----------
-rw-r--r--   2 root root     0 Jun 17 14:01 newf32
-rw-r--r--   2 root root     0 Jun 17 14:01 newf44
-rw-r--r--   2 root root     0 Jun 17 14:01 newf46
-rw-r--r--   2 root root     0 Jun 17 14:01 newf53
-rw-r--r--   2 root root     0 Jun 30 11:33 newf55
-rw-r--r--   2 root root     0 Jun 17 14:01 newf62


verify same on backend


# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46
trusted.gfid=0xdba5f967b2334725905bab16cbc39f88
--
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf46
trusted.gfid=0xdba5f967b2334725905bab16cbc39f88

----------------------
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32
trusted.gfid=0xe5cf1c5ee07a413ca0f5c27660871a1a
--
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf32
trusted.gfid=0xe5cf1c5ee07a413ca0f5c27660871a1a

----------------------
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file55
trusted.gfid=0x81ca84574f594e25ad9f2de73d218f75
--
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf55
trusted.gfid=0x81ca84574f594e25ad9f2de73d218f75

----------------------
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13
trusted.gfid=0x3e7dbc6d66844289b734d713e6212ec6
--
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf13
trusted.gfid=0x3e7dbc6d66844289b734d713e6212ec6

###########
[root@OVM3 tmp]# stat /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13
  File: `/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: fd06h/64774d    Inode: 8613923     Links: 3
Access: (1000/---------T)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-06-25 12:38:53.427004085 +0530
Modify: 2014-06-25 12:38:53.427004085 +0530
Change: 2014-06-25 12:38:53.484004085 +0530



log snippet :-
[root@OVM3 ~]# grep '/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32'  /var/log/glusterfs/* -R  
/var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.487798] E [posix-helpers.c:930:posix_handle_pair] 0-test1-posix: /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32: key:trusted.glusterfs.dht.linkto flags: 1 length:15 error:File exists
/var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.487841] E [posix.c:1169:posix_mknod] 0-test1-posix: setting xattrs on /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32 failed (File exists)




[root@OVM3 ~]# grep '/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46'  /var/log/glusterfs/* -R  
/var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.491410] E [posix-helpers.c:930:posix_handle_pair] 0-test1-posix: /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46: key:trusted.glusterfs.dht.linkto flags: 1 length:15 error:File exists
/var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.491442] E [posix.c:1169:posix_mknod] 0-test1-posix: setting xattrs on /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46 failed (File exists)




Actual results:
===============
(Due to some race condition or self healing or because of some other reason)even after rename operation is completed,  source and destination file exist with same gfid



Expected results:
===============
rename operation should be completed successfully and it should unlink source file

Comment 4 surabhi 2014-09-13 11:21:19 UTC
Hit the same issue while doing rename and rebalance on samba share from windows client.
Steps to reproduce:
create a volume and mount it via cifs
create a doc
Set acl for it
Do add-brick, when add-brick is in progress do multiple renames of the same file.
Run rebalance
After rebalance and rename is finished ,on the mount point both the files exists.
On the back-end it exists with the same gfid.

Comment 7 Nithya Balachandran 2016-01-11 10:12:18 UTC
*** Bug 1286098 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.