Description of problem: ----------------------- I expanded a 3*(4+2) to 4*(4+2) with lots of lookups and kernel untars from multiple Ganesha v3/v4 mounts. During rebalance I could see some EIOs in the brick logs : <snip> bricks/bricks4-brick.log:[2017-08-06 16:03:02.551988] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks4/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error] bricks/bricks4-brick.log:[2017-08-06 16:03:02.552070] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893688: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-6-2-0 [Input/output error] bricks/bricks5-brick.log:[2017-08-06 16:03:02.552035] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks5/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error] bricks/bricks5-brick.log:[2017-08-06 16:03:02.552111] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893642: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-8-2-0 [Input/output error] bricks/bricks6-brick.log:[2017-08-06 16:03:02.552089] I [MSGID: 113109] [posix.c:1608:posix_mkdir] 0-butcher-posix: mkdir (00000000-0000-0000-0000-000000000001/dir5): failing preop of mkdir (/bricks6/brick/dir5) as on-disk xattr value differs from argument value for key trusted.glusterfs.dht [Input/output error] bricks/bricks6-brick.log:[2017-08-06 16:03:02.552173] E [MSGID: 115056] [server-rpc-fops.c:504:server_mkdir_cbk] 0-butcher-server: 893670: MKDIR /dir5 (00000000-0000-0000-0000-000000000001/dir5) client: gqas009.sbu.lab.eng.bos.redhat.com-30438-2017/08/06-14:36:07:238010-butcher-client-10-2-0 [Input/output error] </snip> Version-Release number of selected component (if applicable): ------------------------------------------------------------- 3.8.4-38 How reproducible: ----------------- 1/1 Actual results: --------------- EIO in logs. Expected results: ----------------- No EIO in logs :) Additional info: ----------------- Volume Name: butcher Type: Distributed-Disperse Volume ID: 8e373be1-81ee-497a-8aa8-44fa9d98a89c Status: Started Snapshot Count: 0 Number of Bricks: 4 x (4 + 2) = 24 Transport-type: tcp Bricks: Brick1: gqas009.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick2: gqas016.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick3: gqas009.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick4: gqas016.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick5: gqas009.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick6: gqas016.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick7: gqas009.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick8: gqas016.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick9: gqas009.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick10: gqas016.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick11: gqas009.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick12: gqas016.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick13: gqas009.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick14: gqas016.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick15: gqas009.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick16: gqas016.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick17: gqas009.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick18: gqas016.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick19: gqas009.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick20: gqas016.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick21: gqas009.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick22: gqas016.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick23: gqas009.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick24: gqas016.sbu.lab.eng.bos.redhat.com:/bricks12/brick Options Reconfigured: ganesha.enable: on features.quota-deem-statfs: on features.inode-quota: on features.quota: on features.uss: enable client.event-threads: 4 server.event-threads: 4 network.inode-lru-limit: 50000 performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.stat-prefetch: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on transport.address-family: inet nfs.disable: on nfs-ganesha: enable cluster.enable-shared-storage: enable