Bug 1114710

Summary: DHT + Rebalance + rename :- while rebalance is in progress, for few files - renaming of files creates hard link for file rather than renaming
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rachana Patel <racpatel>
Component: distributeAssignee: Raghavendra G <rgowdapp>
Status: CLOSED WORKSFORME QA Contact: storage-qa-internal <storage-qa-internal>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.0CC: mzywusko, nbalacha, sbhaloth, smohan, spalai
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: dht-rename-file
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1155463 (view as bug list) Environment:
Last Closed: 2017-02-13 08:03:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1155463, 1286098    

Description Rachana Patel 2014-06-30 18:02:25 UTC
Description of problem:
=======================
was trying to reproduce  bug 1110730. kept renaming all files from mountpoint while rebalance was in progress and while rebalance and rename got over found that for few files <old file> and <new file> both are present having same gfid (i.e. hard link of file). 
rather than renaming from old to new it has created link file for it. (or raname operation was not completed, it has created link file but unlink of source failed)



Version-Release number of selected component (if applicable):
=============================================================
3.6.0.19-1.el6rhs.x86_64

How reproducible:
=================
Intermittent

Steps to Reproduce:
====================
1. create and mount(NFS) distributed volume( 3 bricks)
2. add 2 more bricks.
3. start creating files and Dires on mount point. start rebalance.
4. from mount point rename files while rebalance is in progress
(cd /mnt/nfs; for i in {1..100} ; do mv new$i $i ; cd $i; mv etcn$i etc$i; for j in {1..100}; do mv  newf$j file$j; done ; done
cd /mnt/nfs; for i in {1..100} ; do mv $i new$i ; cd new$i; mv etc$i etcn$i; for j in {1..100}; do mv  file$j newf$j; done ; done
...
...
)
after 2-3 times stop renaming in between
5. once rebalance is finish. check data from mount point. for few files source and destination both files exist

e.g. 
[root@OVM1 47]# ls -l
total 36
drwxr-xr-x   4 root root  1557 Jun 25 12:40 48
drwxr-xr-x 110 root root 36864 Jun 25 13:36 etc47
-rw-r--r--   1 root root     0 Jun 17 14:01 file1
-rw-r--r--   1 root root     0 Jun 17 14:01 file10
-rw-r--r--   1 root root     0 Jun 17 14:01 file100
-rw-r--r--   1 root root     0 Jun 17 14:01 file11
-rw-r--r--   1 root root     0 Jun 17 14:01 file12
-rw-r--r--   2 root root     0 Jun 17 14:01 file13
-rw-r--r--   1 root root     0 Jun 17 14:01 file14
-rw-r--r--   1 root root     0 Jun 17 14:01 file15
-rw-r--r--   1 root root     0 Jun 17 14:01 file16
-rw-r--r--   1 root root     0 Jun 17 14:01 file17
-rw-r--r--   1 root root     0 Jun 17 14:01 file18
-rw-r--r--   1 root root     0 Jun 17 14:01 file19
-rw-r--r--   1 root root     0 Jun 17 14:01 file2
-rw-r--r--   1 root root     0 Jun 17 14:01 file20
-rw-r--r--   1 root root     0 Jun 17 14:01 file21
-rw-r--r--   1 root root     0 Jun 17 14:01 file22
-rw-r--r--   1 root root     0 Jun 17 14:01 file23
-rw-r--r--   1 root root     0 Jun 17 14:01 file24
-rw-r--r--   1 root root     0 Jun 17 14:01 file25
-rw-r--r--   1 root root     0 Jun 17 14:01 file26
-rw-r--r--   1 root root     0 Jun 17 14:01 file27
-rw-r--r--   1 root root     0 Jun 17 14:01 file28
-rw-r--r--   1 root root     0 Jun 17 14:01 file29
-rw-r--r--   1 root root     0 Jun 17 14:01 file3
-rw-r--r--   1 root root     0 Jun 17 14:01 file30
-rw-r--r--   1 root root     0 Jun 17 14:01 file31
-rw-r--r--   2 root root     0 Jun 17 14:01 file32
-rw-r--r--   1 root root     0 Jun 17 14:01 file33
-rw-r--r--   1 root root     0 Jun 17 14:01 file34
-rw-r--r--   1 root root     0 Jun 17 14:01 file35
-rw-r--r--   1 root root     0 Jun 17 14:01 file36
-rw-r--r--   1 root root     0 Jun 17 14:01 file37
-rw-r--r--   1 root root     0 Jun 17 14:01 file38
-rw-r--r--   1 root root     0 Jun 17 14:01 file39
-rw-r--r--   1 root root     0 Jun 17 14:01 file4
-rw-r--r--   1 root root     0 Jun 17 14:01 file40
-rw-r--r--   1 root root     0 Jun 17 14:01 file41
-rw-r--r--   1 root root     0 Jun 17 14:01 file42
-rw-r--r--   1 root root     0 Jun 17 14:01 file43
-rw-r--r--   2 root root     0 Jun 17 14:01 file44
-rw-r--r--   1 root root     0 Jun 17 14:01 file45
-rw-r--r--   2 root root     0 Jun 17 14:01 file46
-rw-r--r--   1 root root     0 Jun 17 14:01 file47
-rw-r--r--   1 root root     0 Jun 17 14:01 file48
-rw-r--r--   1 root root     0 Jun 17 14:01 file49
-rw-r--r--   1 root root     0 Jun 17 14:01 file5
-rw-r--r--   1 root root     0 Jun 17 14:01 file50
-rw-r--r--   1 root root     0 Jun 17 14:01 file51
-rw-r--r--   1 root root     0 Jun 17 14:01 file52
-rw-r--r--   2 root root     0 Jun 17 14:01 file53
-rw-r--r--   1 root root     0 Jun 17 14:01 file54
-rw-r--r--   2 root root     0 Jun 30 11:33 file55
-rw-r--r--   1 root root     0 Jun 17 14:01 file56
-rw-r--r--   1 root root     0 Jun 17 14:01 file57
-rw-r--r--   1 root root     0 Jun 17 14:01 file58
-rw-r--r--   1 root root     0 Jun 17 14:01 file59
-rw-r--r--   1 root root     0 Jun 17 14:01 file6
-rw-r--r--   1 root root     0 Jun 17 14:01 file60
-rw-r--r--   1 root root     0 Jun 17 14:01 file61
-rw-r--r--   2 root root     0 Jun 17 14:01 file62
-rw-r--r--   1 root root     0 Jun 17 14:01 file63
-rw-r--r--   1 root root     0 Jun 17 14:01 file64
-rw-r--r--   1 root root     0 Jun 17 14:01 file65
-rw-r--r--   1 root root     0 Jun 17 14:01 file66
-rw-r--r--   1 root root     0 Jun 17 14:01 file67
-rw-r--r--   1 root root     0 Jun 17 14:01 file68
-rw-r--r--   1 root root     0 Jun 17 14:01 file69
-rw-r--r--   1 root root     0 Jun 17 14:01 file7
-rw-r--r--   1 root root     0 Jun 17 14:01 file70
-rw-r--r--   1 root root     0 Jun 17 14:01 file71
-rw-r--r--   1 root root     0 Jun 17 14:01 file72
-rw-r--r--   1 root root     0 Jun 17 14:01 file73
-rw-r--r--   1 root root     0 Jun 17 14:01 file74
-rw-r--r--   1 root root     0 Jun 17 14:01 file75
-rw-r--r--   1 root root     0 Jun 17 14:01 file76
-rw-r--r--   1 root root     0 Jun 17 14:01 file77
-rw-r--r--   1 root root     0 Jun 17 14:01 file78
-rw-r--r--   1 root root     0 Jun 17 14:01 file79
-rw-r--r--   1 root root     0 Jun 17 14:01 file8
-rw-r--r--   1 root root     0 Jun 17 14:01 file80
-rw-r--r--   1 root root     0 Jun 17 14:01 file81
-rw-r--r--   1 root root     0 Jun 17 14:01 file82
-rw-r--r--   1 root root     0 Jun 17 14:01 file83
-rw-r--r--   1 root root     0 Jun 17 14:01 file84
-rw-r--r--   1 root root     0 Jun 17 14:01 file85
-rw-r--r--   1 root root     0 Jun 17 14:01 file86
-rw-r--r--   1 root root     0 Jun 17 14:01 file87
-rw-r--r--   1 root root     0 Jun 17 14:01 file88
-rw-r--r--   1 root root     0 Jun 17 14:01 file89
-rw-r--r--   1 root root     0 Jun 17 14:01 file9
-rw-r--r--   1 root root     0 Jun 17 14:01 file90
-rw-r--r--   1 root root     0 Jun 17 14:01 file91
-rw-r--r--   1 root root     0 Jun 17 14:01 file92
-rw-r--r--   1 root root     0 Jun 17 14:01 file93
-rw-r--r--   1 root root     0 Jun 17 14:01 file94
-rw-r--r--   1 root root     0 Jun 17 14:01 file95
-rw-r--r--   1 root root     0 Jun 17 14:01 file96
-rw-r--r--   1 root root     0 Jun 17 14:01 file97
-rw-r--r--   1 root root     0 Jun 17 14:01 file98
-rw-r--r--   1 root root     0 Jun 17 14:01 file99
-rw-r--r--   2 root root     0 Jun 17 14:01 newf13    <-----------
-rw-r--r--   2 root root     0 Jun 17 14:01 newf32
-rw-r--r--   2 root root     0 Jun 17 14:01 newf44
-rw-r--r--   2 root root     0 Jun 17 14:01 newf46
-rw-r--r--   2 root root     0 Jun 17 14:01 newf53
-rw-r--r--   2 root root     0 Jun 30 11:33 newf55
-rw-r--r--   2 root root     0 Jun 17 14:01 newf62


verify same on backend


# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46
trusted.gfid=0xdba5f967b2334725905bab16cbc39f88
--
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf46
trusted.gfid=0xdba5f967b2334725905bab16cbc39f88

----------------------
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32
trusted.gfid=0xe5cf1c5ee07a413ca0f5c27660871a1a
--
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf32
trusted.gfid=0xe5cf1c5ee07a413ca0f5c27660871a1a

----------------------
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file55
trusted.gfid=0x81ca84574f594e25ad9f2de73d218f75
--
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf55
trusted.gfid=0x81ca84574f594e25ad9f2de73d218f75

----------------------
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13
trusted.gfid=0x3e7dbc6d66844289b734d713e6212ec6
--
# file: brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/newf13
trusted.gfid=0x3e7dbc6d66844289b734d713e6212ec6

###########
[root@OVM3 tmp]# stat /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13
  File: `/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file13'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: fd06h/64774d    Inode: 8613923     Links: 3
Access: (1000/---------T)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-06-25 12:38:53.427004085 +0530
Modify: 2014-06-25 12:38:53.427004085 +0530
Change: 2014-06-25 12:38:53.484004085 +0530



log snippet :-
[root@OVM3 ~]# grep '/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32'  /var/log/glusterfs/* -R  
/var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.487798] E [posix-helpers.c:930:posix_handle_pair] 0-test1-posix: /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32: key:trusted.glusterfs.dht.linkto flags: 1 length:15 error:File exists
/var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.487841] E [posix.c:1169:posix_mknod] 0-test1-posix: setting xattrs on /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file32 failed (File exists)




[root@OVM3 ~]# grep '/brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46'  /var/log/glusterfs/* -R  
/var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.491410] E [posix-helpers.c:930:posix_handle_pair] 0-test1-posix: /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46: key:trusted.glusterfs.dht.linkto flags: 1 length:15 error:File exists
/var/log/glusterfs/bricks/brick0.log-20140629:[2014-06-25 07:08:53.491442] E [posix.c:1169:posix_mknod] 0-test1-posix: setting xattrs on /brick0/1/new1/new2/new3/new4/new5/new6/new7/new8/new9/new10/new11/new12/new13/new14/new15/new16/new17/new18/new19/new20/new21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/file46 failed (File exists)




Actual results:
===============
(Due to some race condition or self healing or because of some other reason)even after rename operation is completed,  source and destination file exist with same gfid



Expected results:
===============
rename operation should be completed successfully and it should unlink source file

Comment 4 surabhi 2014-09-13 11:21:19 UTC
Hit the same issue while doing rename and rebalance on samba share from windows client.
Steps to reproduce:
create a volume and mount it via cifs
create a doc
Set acl for it
Do add-brick, when add-brick is in progress do multiple renames of the same file.
Run rebalance
After rebalance and rename is finished ,on the mount point both the files exists.
On the back-end it exists with the same gfid.

Comment 7 Nithya Balachandran 2016-01-11 10:12:18 UTC
*** Bug 1286098 has been marked as a duplicate of this bug. ***