Bug 1439599

Summary: [DHT] : rm fails/ends abruptly with "rm: fts_read failed: No such file or directory"
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Ambarish <asoman>
Component: distributeAssignee: Susant Kumar Palai <spalai>
Status: CLOSED WONTFIX QA Contact: Prasad Desala <tdesala>
Severity: low Docs Contact:
Priority: low    
Version: rhgs-3.3CC: amukherj, bturner, nbalacha, pkarampu, rgowdapp, rhinduja, rhs-bugs, saraut, storage-qa-internal, tdesala
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: dht-rm-rf
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-08 06:13:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1678183    

Description Ambarish 2017-04-06 10:03:32 UTC 
Description of problem:
-----------------------

EC (4+2),mounted via FUSE on 6 clients.

Running rm from 6 clients to clean up the mount point.


rm abruptly ends on all of my clients with the following error :

<snip>
/pistachio-marduk.txt’
removed ‘linux-4.11-rc5/Documentation/devicetree/bindings/mips/img/pistachio.txt’
removed ‘linux-4.11-rc5/Documentation/devicetree/bindings/mips/img/xilfpga.txt’
removed directory: ‘linux-4.11-rc5/Documentation/devicetree/bindings/mips/img’
removed ‘linux-4.11-rc5/Documentation/devicetree/bindings/mips/ath79-soc.txt’
rm: fts_read failed: No such file or directory
[root@gqac015 gluster-mount]# 
</snip>


There were a bunch of EIOs too,but that already known to us : 

https://bugzilla.redhat.com/show_bug.cgi?id=1395161

I could not reproduce this on *2 though.


Version-Release number of selected component (if applicable):
-------------------------------------------------------------



How reproducible:
-----------------

2/2,on a large data set with a  lot of small files.

Actual results:
---------------

rm fails with EIO and fts_read failure messages.

Expected results:
-----------------

Successful rm completion.

Additional info:
----------------


[root@gqas013 ~]# gluster v info
 
Volume Name: butcher
Type: Distributed-Disperse
Volume ID: 55902003-7ea9-4f58-987d-63c6c759a385
Status: Started
Snapshot Count: 0
Number of Bricks: 12 x (4 + 2) = 72
Transport-type: tcp
Bricks:
Brick1: gqas013.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick2: gqas005.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick3: gqas006.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick5: gqas014.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick6: gqas015.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick7: gqas013.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick8: gqas005.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick9: gqas006.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick10: gqas008.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick11: gqas014.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick12: gqas015.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick13: gqas013.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick14: gqas005.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick15: gqas006.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick16: gqas008.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick17: gqas014.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick18: gqas015.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick19: gqas013.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick20: gqas005.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick21: gqas006.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick22: gqas008.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick23: gqas014.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick24: gqas015.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick25: gqas013.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick26: gqas005.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick27: gqas006.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick28: gqas008.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick29: gqas014.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick30: gqas015.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick31: gqas013.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick32: gqas005.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick33: gqas006.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick34: gqas008.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick35: gqas014.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick36: gqas015.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick37: gqas013.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick38: gqas005.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick39: gqas006.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick40: gqas008.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick41: gqas014.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick42: gqas015.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick43: gqas013.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick44: gqas005.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick45: gqas006.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick46: gqas008.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick47: gqas014.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick48: gqas015.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick49: gqas013.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick50: gqas005.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick51: gqas006.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick52: gqas008.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick53: gqas014.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick54: gqas015.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick55: gqas013.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick56: gqas005.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick57: gqas006.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick58: gqas008.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick59: gqas014.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick60: gqas015.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick61: gqas013.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick62: gqas005.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick63: gqas006.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick64: gqas008.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick65: gqas014.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick66: gqas015.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick67: gqas013.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick68: gqas005.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick69: gqas006.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick70: gqas008.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick71: gqas014.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick72: gqas015.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Options Reconfigured:
performance.quick-read: on
performance.io-cache: on
performance.parallel-readdir: off
transport.address-family: inet
nfs.disable: on
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 50000
cluster.lookup-optimize: on
server.event-threads: 4
client.event-threads: 4
[root@gqas013 ~]#