Bug 1478730

Summary:	[Scale] : Input/Output Error in brick logs during rebalance.
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Ambarish <asoman>
Component:	disperse	Assignee:	Ashish Pandey <aspandey>
Status:	CLOSED NOTABUG	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	medium	Docs Contact:
Priority:	low
Version:	rhgs-3.3	CC:	aspandey, nbalacha, pkarampu, rhs-bugs, sheggodu, storage-qa-internal
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-11-15 11:12:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ambarish 2017-08-06 16:28:53 UTC

Description of problem:
-----------------------

I expanded a 3*(4+2) to 4*(4+2) with lots of lookups and kernel untars from multiple Ganesha v3/v4 mounts.

During rebalance I could see some EIOs in the brick logs :

<snip>

bricks/bricks4-brick.log:[2017-08-06 16:03:02.551988] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks4/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error]
bricks/bricks4-brick.log:[2017-08-06 16:03:02.552070] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893688: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-6-2-0 [Input/output error]
bricks/bricks5-brick.log:[2017-08-06 16:03:02.552035] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks5/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error]
bricks/bricks5-brick.log:[2017-08-06 16:03:02.552111] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893642: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-8-2-0 [Input/output error]
bricks/bricks6-brick.log:[2017-08-06 16:03:02.552089] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks6/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error]
bricks/bricks6-brick.log:[2017-08-06 16:03:02.552173] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893670: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-10-2-0 [Input/output error]

</snip>

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

3.8.4-38

How reproducible:
-----------------

1/1



Actual results:
---------------

EIO in logs.


Expected results:
-----------------

No  EIO in logs :)


Additional info:
-----------------

Volume Name: butcher
Type: Distributed-Disperse
Volume ID: 8e373be1-81ee-497a-8aa8-44fa9d98a89c
Status: Started
Snapshot Count: 0
Number of Bricks: 4 x (4 + 2) = 24
Transport-type: tcp
Bricks:
Brick1: gqas009.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick2: gqas016.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick3: gqas009.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick4: gqas016.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick5: gqas009.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick6: gqas016.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick7: gqas009.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick8: gqas016.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick9: gqas009.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick10: gqas016.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick11: gqas009.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick12: gqas016.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick13: gqas009.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick14: gqas016.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick15: gqas009.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick16: gqas016.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick17: gqas009.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick18: gqas016.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick19: gqas009.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick20: gqas016.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick21: gqas009.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick22: gqas016.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick23: gqas009.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick24: gqas016.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Options Reconfigured:
ganesha.enable: on
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
features.uss: enable
client.event-threads: 4
server.event-threads: 4
network.inode-lru-limit: 50000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
transport.address-family: inet
nfs.disable: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable