Description of problem: ----------------------- EC (4+2),mounted via FUSE on 6 clients. Running rm from 6 clients to clean up the mount point. rm abruptly ends on all of my clients with the following error : <snip> /pistachio-marduk.txt’ removed ‘linux-4.11-rc5/Documentation/devicetree/bindings/mips/img/pistachio.txt’ removed ‘linux-4.11-rc5/Documentation/devicetree/bindings/mips/img/xilfpga.txt’ removed directory: ‘linux-4.11-rc5/Documentation/devicetree/bindings/mips/img’ removed ‘linux-4.11-rc5/Documentation/devicetree/bindings/mips/ath79-soc.txt’ rm: fts_read failed: No such file or directory [root@gqac015 gluster-mount]# </snip> There were a bunch of EIOs too,but that already known to us : https://bugzilla.redhat.com/show_bug.cgi?id=1395161 I could not reproduce this on *2 though. Version-Release number of selected component (if applicable): ------------------------------------------------------------- How reproducible: ----------------- 2/2,on a large data set with a lot of small files. Actual results: --------------- rm fails with EIO and fts_read failure messages. Expected results: ----------------- Successful rm completion. Additional info: ---------------- [root@gqas013 ~]# gluster v info Volume Name: butcher Type: Distributed-Disperse Volume ID: 55902003-7ea9-4f58-987d-63c6c759a385 Status: Started Snapshot Count: 0 Number of Bricks: 12 x (4 + 2) = 72 Transport-type: tcp Bricks: Brick1: gqas013.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick2: gqas005.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick3: gqas006.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick5: gqas014.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick6: gqas015.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick7: gqas013.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick8: gqas005.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick9: gqas006.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick10: gqas008.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick11: gqas014.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick12: gqas015.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick13: gqas013.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick14: gqas005.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick15: gqas006.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick16: gqas008.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick17: gqas014.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick18: gqas015.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick19: gqas013.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick20: gqas005.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick21: gqas006.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick22: gqas008.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick23: gqas014.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick24: gqas015.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick25: gqas013.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick26: gqas005.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick27: gqas006.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick28: gqas008.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick29: gqas014.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick30: gqas015.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick31: gqas013.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick32: gqas005.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick33: gqas006.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick34: gqas008.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick35: gqas014.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick36: gqas015.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick37: gqas013.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick38: gqas005.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick39: gqas006.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick40: gqas008.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick41: gqas014.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick42: gqas015.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick43: gqas013.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick44: gqas005.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick45: gqas006.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick46: gqas008.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick47: gqas014.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick48: gqas015.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick49: gqas013.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick50: gqas005.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick51: gqas006.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick52: gqas008.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick53: gqas014.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick54: gqas015.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick55: gqas013.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick56: gqas005.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick57: gqas006.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick58: gqas008.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick59: gqas014.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick60: gqas015.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick61: gqas013.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick62: gqas005.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick63: gqas006.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick64: gqas008.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick65: gqas014.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick66: gqas015.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick67: gqas013.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick68: gqas005.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick69: gqas006.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick70: gqas008.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick71: gqas014.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick72: gqas015.sbu.lab.eng.bos.redhat.com:/bricks12/brick Options Reconfigured: performance.quick-read: on performance.io-cache: on performance.parallel-readdir: off transport.address-family: inet nfs.disable: on features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.stat-prefetch: on performance.cache-invalidation: on performance.md-cache-timeout: 600 network.inode-lru-limit: 50000 cluster.lookup-optimize: on server.event-threads: 4 client.event-threads: 4 [root@gqas013 ~]#