Bug 1719321 - getting Input/Output when triggering rm -rf from multiple clients
Summary: getting Input/Output when triggering rm -rf from multiple clients
Keywords:
Status: CLOSED DUPLICATE of bug 1395161
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: disperse
Version: rhgs-3.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Ashish Pandey
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-11 13:59 UTC by Nag Pavan Chilakam
Modified: 2019-10-31 10:38 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-13 05:08:30 UTC
Embargoed:


Attachments (Terms of Use)

Description Nag Pavan Chilakam 2019-06-11 13:59:35 UTC
Description of problem:
======================
on my test cluster, I am seeing Input/Output error while issuing rm -rf simultaneously from 3 clients.

I have retried on just 2 clients and the issue still exists

However from single client I don't see the issue.



Version-Release number of selected component (if applicable):
===================
6.0.3 on rhel7.7 beta

How reproducible:
==================
always on my cluster

Steps to Reproduce:
1.created a 3 node brickmux enabled setup
2. created 1 1x3 volume
3. created 1 ec volume 22x(4+2) with other-eagerlock off
4. running IOs
5. under a directory I had created multiple directories and each hosting one file.
6. did rm -rf * from 2 or 3 clients 

Actual results:
==============
seeing EIO error

Volume Name: cvlt-ecv
Type: Distributed-Disperse
Volume ID: c500e86b-f505-48c8-8141-cb3de5956c24
Status: Started
Snapshot Count: 0
Number of Bricks: 12 x (4 + 2) = 72
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1
Brick4: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1
Brick5: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1
Brick6: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1
Brick7: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2
Brick8: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2
Brick9: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2
Brick10: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2
Brick11: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2
Brick12: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2
Brick13: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3
Brick14: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3
Brick15: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3
Brick16: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3
Brick17: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3
Brick18: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3
Brick19: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4
Brick20: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4
Brick21: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4
Brick22: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4
Brick23: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4
Brick24: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4
Brick25: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5
Brick26: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5
Brick27: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5
Brick28: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5
Brick29: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5
Brick30: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5
Brick31: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6
Brick32: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6
Brick33: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6
Brick34: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6
Brick35: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6
Brick36: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6
Brick37: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7
Brick38: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7
Brick39: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7
Brick40: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7
Brick41: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7
Brick42: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7
Brick43: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8
Brick44: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8
Brick45: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8
Brick46: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8
Brick47: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8
Brick48: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8
Brick49: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9
Brick50: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9
Brick51: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9
Brick52: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9
Brick53: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9
Brick54: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9
Brick55: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10
Brick56: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10
Brick57: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10
Brick58: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10
Brick59: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10
Brick60: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10
Brick61: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11
Brick62: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11
Brick63: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11
Brick64: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11
Brick65: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11
Brick66: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11
Brick67: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12
Brick68: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12
Brick69: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12
Brick70: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12
Brick71: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12
Brick72: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12
Options Reconfigured:
diagnostics.client-log-level: INFO
cluster.use-compound-fops: off
disperse.other-eager-lock: off
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: enable
 
Volume Name: logvol
Type: Replicate
Volume ID: dd1a76f5-6d16-40bd-90c1-215cf031c2ae
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/logvol
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/logvol
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/logvol
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
cluster.brick-multiplex: enable
 
Volume Name: testvol
Type: Replicate
Volume ID: 2b9ae025-6331-420c-8a29-a6345085d853
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/testvol
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/testvol
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/testvol
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
cluster.brick-multiplex: enable

Comment 3 Nag Pavan Chilakam 2019-06-11 14:07:21 UTC
Hi Ashish,
while I will be attaching the sosreports and logs, I have mailed you the setup details.
You can look into it and help expedite the issue faster and hence help unblock my testing.

~nag

Comment 6 Nag Pavan Chilakam 2019-06-12 05:42:27 UTC
sosreports and logs available @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1719321/


Note You need to log in before you can comment on or make changes to this bug.