Bug 1478730 - [Scale] : Input/Output Error in brick logs during rebalance.
[Scale] : Input/Output Error in brick logs during rebalance.
Status: NEW
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: disperse (Show other bugs)
3.3
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Ashish Pandey
Ambarish
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-06 12:28 EDT by Ambarish
Modified: 2017-08-09 07:02 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ambarish 2017-08-06 12:28:53 EDT
Description of problem:
-----------------------

I expanded a 3*(4+2) to 4*(4+2) with lots of lookups and kernel untars from multiple Ganesha v3/v4 mounts.

During rebalance I could see some EIOs in the brick logs :

<snip>

bricks/bricks4-brick.log:[2017-08-06 16:03:02.551988] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks4/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error]
bricks/bricks4-brick.log:[2017-08-06 16:03:02.552070] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893688: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-6-2-0 [Input/output error]
bricks/bricks5-brick.log:[2017-08-06 16:03:02.552035] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks5/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error]
bricks/bricks5-brick.log:[2017-08-06 16:03:02.552111] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893642: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-8-2-0 [Input/output error]
bricks/bricks6-brick.log:[2017-08-06 16:03:02.552089] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks6/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error]
bricks/bricks6-brick.log:[2017-08-06 16:03:02.552173] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893670: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-10-2-0 [Input/output error]

</snip>

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

3.8.4-38

How reproducible:
-----------------

1/1



Actual results:
---------------

EIO in logs.


Expected results:
-----------------

No  EIO in logs :)


Additional info:
-----------------

Volume Name: butcher
Type: Distributed-Disperse
Volume ID: 8e373be1-81ee-497a-8aa8-44fa9d98a89c
Status: Started
Snapshot Count: 0
Number of Bricks: 4 x (4 + 2) = 24
Transport-type: tcp
Bricks:
Brick1: gqas009.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick2: gqas016.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick3: gqas009.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick4: gqas016.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick5: gqas009.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick6: gqas016.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick7: gqas009.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick8: gqas016.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick9: gqas009.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick10: gqas016.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick11: gqas009.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick12: gqas016.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick13: gqas009.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick14: gqas016.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick15: gqas009.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick16: gqas016.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick17: gqas009.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick18: gqas016.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick19: gqas009.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick20: gqas016.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick21: gqas009.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick22: gqas016.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick23: gqas009.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick24: gqas016.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Options Reconfigured:
ganesha.enable: on
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
features.uss: enable
client.event-threads: 4
server.event-threads: 4
network.inode-lru-limit: 50000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
transport.address-family: inet
nfs.disable: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable

Note You need to log in before you can comment on or make changes to this bug.