Bug 1231166

Summary: Disperse volume : fuse mount hung on renames on a distributed disperse volume
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Bhaskarakiran <byarlaga>
Component: disperseAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED ERRATA QA Contact: Bhaskarakiran <byarlaga>
Severity: unspecified Docs Contact:
Priority: high    
Version: rhgs-3.1CC: annair, mzywusko, rhs-bugs, storage-qa-internal, vagarwal
Target Milestone: ---   
Target Release: RHGS 3.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.1-4 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1232420 (view as bug list) Environment:
Last Closed: 2015-07-29 05:02:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1202842, 1232420, 1232983    
Attachments:
Description Flags
client fuse log none

Description Bhaskarakiran 2015-06-12 10:15:24 UTC 
Description of problem:
=======================

fuse mount hung while doing renames from / to a directory inside. created around 1000 files and moved them to a directory with 'mv' command.

volume info:
============
[root@interstellar ~]# gluster v info vol1
 
Volume Name: vol1
Type: Distributed-Disperse
Volume ID: 8136d188-7ab5-4cb5-97c2-5138431c3dec
Status: Started
Number of Bricks: 2 x (8 + 4) = 24
Transport-type: tcp
Bricks:
Brick1: transformers:/rhs/brick1/b1
Brick2: interstellar:/rhs/brick1/b2
Brick3: ninja:/rhs/brick1/b3
Brick4: transformers:/rhs/brick2/b4
Brick5: interstellar:/rhs/brick2/b5
Brick6: ninja:/rhs/brick2/b6
Brick7: transformers:/rhs/brick3/b7
Brick8: interstellar:/rhs/brick3/b8
Brick9: ninja:/rhs/brick3/b9
Brick10: transformers:/rhs/brick4/b10
Brick11: interstellar:/rhs/brick4/b11
Brick12: ninja:/rhs/brick4/b12
Brick13: transformers:/rhs/brick1/b3
Brick14: interstellar:/rhs/brick1/b14
Brick15: ninja:/rhs/brick1/b15
Brick16: transformers:/rhs/brick2/b16
Brick17: interstellar:/rhs/brick2/b17
Brick18: ninja:/rhs/brick2/b18
Brick19: transformers:/rhs/brick3/b19
Brick20: interstellar:/rhs/brick3/b20
Brick21: ninja:/rhs/brick3/b21
Brick22: transformers:/rhs/brick4/b22
Brick23: interstellar:/rhs/brick4/b23
Brick24: ninja:/rhs/brick4/b24
Options Reconfigured:
features.barrier: disable
cluster.disperse-self-heal-daemon: enable
features.inode-quota: on
features.quota: on
features.uss: on
server.event-threads: 2
client.event-threads: 2
performance.readdir-ahead: on
features.quota-deem-statfs: on
[root@interstellar ~]# 


log snippet:
============
[2015-06-12 09:09:03.509960] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-13: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510033] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-16: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510111] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-12: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510237] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-19: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510312] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-15: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510361] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-14: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510500] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-22: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510572] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-18: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510673] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-23: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510728] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-17: remote operation failed: Invalid argument
[2015-06-12 09:09:03.510809] W [ec-combine.c:860:ec_combine_check] 0-vol1-disperse-0: Mismatching xdata in answers of 'LOOKUP'
[2015-06-12 09:09:03.510886] W [ec-combine.c:860:ec_combine_check] 0-vol1-disperse-0: Mismatching xdata in answers of 'LOOKUP'
[2015-06-12 09:09:03.510941] W [ec-combine.c:860:ec_combine_check] 0-vol1-disperse-0: Mismatching xdata in answers of 'LOOKUP'
[2015-06-12 09:09:03.511040] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-21: remote operation failed: Invalid argument
[2015-06-12 09:09:03.511255] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vol1-client-20: remote operation failed: Invalid argument
[2015-06-12 09:09:03.511287] W [ec-common.c:1397:ec_unlocked] 0-vol1-disperse-1: entry/inode unlocking failed (RENAME)
[2015-06-12 09:09:03.511311] W [ec-combine.c:860:ec_combine_check] 0-vol1-disperse-1: Mismatching xdata in answers of 'LOOKUP'
[2015-06-12 09:09:03.512823] W [ec-combine.c:860:ec_combine_check] 0-vol1-disperse-0: Mismatching xdata in answers of 'LOOKUP'

Version-Release number of selected component (if applicable):
==============================================================
[root@transformers ~]# gluster --version
glusterfs 3.7.1 built on Jun 12 2015 00:21:18
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@transformers ~]# 

How reproducible:
=================
100%

Steps to Reproduce:
As in description

Actual results:


Expected results:


Additional info:

Attaching the clinet fuse log.
Comment 2 Bhaskarakiran 2015-06-16 10:42:29 UTC 
Created attachment 1039366 [details]
client fuse log
Comment 6 Bhaskarakiran 2015-07-07 10:08:20 UTC 
verified this on 3.7.1-7 build and didn't see the issue. Marking this as fixed.
Comment 7 errata-xmlrpc 2015-07-29 05:02:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1495.html