Bug 1171681

Summary: rename operation failed on disperse volume with glusterfs 3.6.1
Product: [Community] GlusterFS Reporter: Panpan Feng <jiaowopan>
Component: disperseAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.1CC: bugs, jiaowopan, kaushal, mingfan.lu, pkarampu
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: 3.6.3 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1184587 (view as bug list) Environment:
Last Closed: 2016-08-16 13:02:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1184587    
Attachments:
Description Flags
client mounting log none

Description Panpan Feng 2014-12-08 11:43:02 UTC 
Created attachment 965795 [details]
client mounting log

Description of problem:

rename operation will failed on disperse volume  servered by Glusterfs 3.6.1

Version-Release number of selected component (if applicable):

glusterfs 3.6.1

How reproducible:


Steps to Reproduce:
1. create a disperse volume with GlusterFs 3.6.1
2. create a file1
3. mv file1  file2  (this operation will fail)

Actual results:

file1 cannot be renamed to file2

Expected results:

file1 will be renamed to file2

Additional info:

------------------about glusterfs version-------------------------------

># /usr/sbin/glusterd -V
glusterfs 3.6.1 built on Nov  7 2014 15:15:48
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.

-----------------command to create volume -----------------------------
gluster v create ec-6-2  disperse  6  redundancy  2   10.23.72.241:/mnt/xfsd/ec-6-2-{0..11}

----------------- volume info  -----------------------------------

Volume Name: ec-6-2
Type: Distributed-Disperse
Volume ID: 7bda8c2f-2a98-4b5f-aab7-b3d652fae622
Status: Started
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: 10.23.72.241:/mnt/xfsd/ec-6-2-0
Brick2: 10.23.72.241:/mnt/xfsd/ec-6-2-1
Brick3: 10.23.72.241:/mnt/xfsd/ec-6-2-2
Brick4: 10.23.72.241:/mnt/xfsd/ec-6-2-3
Brick5: 10.23.72.241:/mnt/xfsd/ec-6-2-4
Brick6: 10.23.72.241:/mnt/xfsd/ec-6-2-5
Brick7: 10.23.72.241:/mnt/xfsd/ec-6-2-6
Brick8: 10.23.72.241:/mnt/xfsd/ec-6-2-7
Brick9: 10.23.72.241:/mnt/xfsd/ec-6-2-8
Brick10: 10.23.72.241:/mnt/xfsd/ec-6-2-9
Brick11: 10.23.72.241:/mnt/xfsd/ec-6-2-10
Brick12: 10.23.72.241:/mnt/xfsd/ec-6-2-11

-----------  some log  (entire log can be found in Attachment ------------

[2014-12-08 11:38:16.799225] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /file2 missing on subvol ec-6-2-disperse-1
[2014-12-08 11:38:16.800420] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /file2 missing on subvol ec-6-2-disperse-1
[2014-12-08 11:38:16.803667] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /file2 missing on subvol ec-6-2-disperse-1
[2014-12-08 11:38:16.804587] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /file2 missing on subvol ec-6-2-disperse-1
[2014-12-08 11:38:16.805538] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /file2 missing on subvol ec-6-2-disperse-1
[2014-12-08 11:38:16.806351] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /file2 missing on subvol ec-6-2-disperse-1
[2014-12-08 11:38:16.807289] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /file2 missing on subvol ec-6-2-disperse-1
[2014-12-08 11:38:16.807779] I [dht-rename.c:1344:dht_rename] 0-ec-6-2-dht: renaming /file1 (hash=ec-6-2-disperse-1/cache=ec-6-2-disperse-1) => /file2 (hash=ec-6-2-disperse-1/cache=<nul>)
[2014-12-08 11:38:16.808351] E [ec-helpers.c:346:ec_loc_parent] 0-ec-6-2-disperse-1: Invalid loc
[2014-12-08 11:38:16.808578] I [MSGID: 109030] [dht-rename.c:665:dht_rename_cbk] 0-ec-6-2-dht: /file1: Rename (linkto file) on ec-6-2-disperse-1 failed, (gfid = 210757b2-0cb1-4a8c-8f19-aeecd7d1a216)  [Invalid argument]
[2014-12-08 11:38:28.445397] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /cscope.out missing on subvol ec-6-2-disperse-0
[2014-12-08 11:38:28.446342] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /cscope.out missing on subvol ec-6-2-disperse-0
[2014-12-08 11:38:28.447335] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /cscope.out missing on subvol ec-6-2-disperse-0
[2014-12-08 11:38:28.448163] I [dht-common.c:1822:dht_lookup_cbk] 0-ec-6-2-dht: Entry /cscope.out missing on subvol ec-6-2-disperse-0
Comment 1 Pranith Kumar K 2014-12-09 07:23:49 UTC 
Rename on a non-existing file comes with no-inode because the destination path doesn't exist. This case is handled as error in ec_loc_parent. That is the reason for the failure in rename.
Comment 2 Pranith Kumar K 2014-12-09 09:37:58 UTC 
I see that this issue is already fixed in latest master. It doesn't check for loc->inode presence.
Comment 3 Frank Lu 2014-12-10 07:51:40 UTC 
which commit? I could backport it to my local build.
Thanks Pranith.
(In reply to Pranith Kumar K from comment #2)
> I see that this issue is already fixed in latest master. It doesn't check
> for loc->inode presence.
Comment 4 Panpan Feng 2015-01-21 07:47:49 UTC 
 I have tried  patch  9072, and found that the rename operation will hang !  And  the nightly build  glusterfs-3.6.2 beta1-0.55-git443cda3  still face the same problem.

Volume  info as following:

Volume Name: ec-test
Type: Distributed-Disperse
Volume ID: a77d8bfa-c89a-49bc-a6c7-1690984c6555
Status: Started
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: 10.23.72.229:/mnt/xfsd/ec-test-0
Brick2: 10.23.72.229:/mnt/xfsd/ec-test-1
Brick3: 10.23.72.229:/mnt/xfsd/ec-test-2
Brick4: 10.23.72.229:/mnt/xfsd/ec-test-3
Brick5: 10.23.72.229:/mnt/xfsd/ec-test-4
Brick6: 10.23.72.229:/mnt/xfsd/ec-test-5
Brick7: 10.23.72.229:/mnt/xfsd/ec-test-6
Brick8: 10.23.72.229:/mnt/xfsd/ec-test-7
Brick9: 10.23.72.229:/mnt/xfsd/ec-test-8
Brick10: 10.23.72.229:/mnt/xfsd/ec-test-9
Brick11: 10.23.72.229:/mnt/xfsd/ec-test-10
Brick12: 10.23.72.229:/mnt/xfsd/ec-test-11
Options Reconfigured:
server.allow-insecure: on
features.quota-timeout: 5
features.quota: on

------------------------------------------------
Steps to reproduce:
1. mount -t glusterfs  localhost:ec-test  /mnt/glusterfs/ec-test 
2. cd /mnt/gluster/ec-test
3.  echo "hello" > file1
4.   mv file1  file2


mount log snap view:

[2015-01-16 13:43:46.583860] I [ec.c:195:ec_up] 0-ec-test-disperse-1: Going UP
[2015-01-16 13:43:46.588176] I [fuse-bridge.c:5080:fuse_graph_setup] 0-fuse: switched to graph 0
[2015-01-16 13:43:46.588303] I [client-handshake.c:188:client_set_lk_version_cbk] 0-ec-test-client-11: Server lk version = 1
[2015-01-16 13:43:46.588336] I [client-handshake.c:188:client_set_lk_version_cbk] 0-ec-test-client-10: Server lk version = 1
[2015-01-16 13:43:46.588359] I [client-handshake.c:188:client_set_lk_version_cbk] 0-ec-test-client-7: Server lk version = 1
[2015-01-16 13:43:46.588374] I [client-handshake.c:188:client_set_lk_version_cbk] 0-ec-test-client-6: Server lk version = 1
[2015-01-16 13:43:46.588387] I [client-handshake.c:188:client_set_lk_version_cbk] 0-ec-test-client-8: Server lk version = 1
[2015-01-16 13:43:46.588392] I [fuse-bridge.c:4009:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.13
[2015-01-16 13:44:03.954295] I [dht-rename.c:1344:dht_rename] 0-ec-test-dht: renaming /file1 (hash=ec-test-disperse-1/cache=ec-test-disperse-1) => /file2 (hash=ec-test-disperse-1/cache=<nul>)
Comment 5 Kaushal 2016-08-16 13:02:21 UTC 
GlusterFS-3.6.3 was released in April 2015, with a fix for this bug. Closing this bug.