Bug 1421581 - [Stress] : Mismatching iatt in glustershd logs during MTSH and continous IO from Ganesha mounts
Summary: [Stress] : Mismatching iatt in glustershd logs during MTSH and continous IO f...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: disperse
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Ashish Pandey
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On:
Blocks: 1640066 1644622
TreeView+ depends on / blocked
 
Reported: 2017-02-13 07:40 UTC by Ambarish
Modified: 2019-11-18 15:18 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1640066 (view as bug list)
Environment:
Last Closed: 2019-11-18 15:18:19 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1414287 0 unspecified CLOSED repeated operation failed warnings in gluster mount logs with disperse volume 2021-02-22 00:41:40 UTC

Internal Links: 1414287

Description Ambarish 2017-02-13 07:40:17 UTC
Description of problem:
-----------------------

6 Node cluster,one 12*(4+2) volume,exported via Ganesha.

Replaced 12 bricks,which triggered a self heal.

I could see the below Notice messages in glustershd logs,there was no fxnal impact :

[2017-02-13 06:23:49.028223] N [MSGID: 122029] [ec-generic.c:683:ec_combine_lookup] 0-butcher-disperse-5: Mismatching iatt in answers of 'GF_FOP_LOOKUP'
[2017-02-13 06:49:13.043815] N [MSGID: 122029] [ec-generic.c:683:ec_combine_lookup] 0-butcher-disperse-3: Mismatching iatt in answers of 'GF_FOP_LOOKUP'
[2017-02-13 06:49:13.043891] N [MSGID: 122029] [ec-generic.c:683:ec_combine_lookup] 0-butcher-disperse-3: Mismatching iatt in answers of 'GF_FOP_LOOKUP'
[2017-02-13 06:55:14.024356] N [MSGID: 122029] [ec-generic.c:683:ec_combine_lookup] 0-butcher-disperse-6: Mismatching iatt in answers of 'GF_FOP_LOOKUP'


Version-Release number of selected component (if applicable):
-------------------------------------------------------------

glusterfs-3.8.4-14.el7rhgs.x86_64
nfs-ganesha-2.4.1-7.el7rhgs.x86_64

How reproducible:

---------------

1/1


Additional info:
-----------------


Volume Name: butcher
Type: Distributed-Disperse
Volume ID: b3b2211e-9e30-4227-9785-ad0f325f8d21
Status: Started
Snapshot Count: 0
Number of Bricks: 12 x (4 + 2) = 72
Transport-type: tcp
Bricks:
Brick1: gqas003.sbu.lab.eng.bos.redhat.com:/bricks1/brick1
Brick2: gqas009.sbu.lab.eng.bos.redhat.com:/bricks1/brick1
Brick3: gqas010.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick4: gqas012.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick5: gqas014.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick6: gqas015.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick7: gqas003.sbu.lab.eng.bos.redhat.com:/bricks2/brick1
Brick8: gqas009.sbu.lab.eng.bos.redhat.com:/bricks2/brick1
Brick9: gqas010.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick10: gqas012.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick11: gqas014.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick12: gqas015.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick13: gqas003.sbu.lab.eng.bos.redhat.com:/bricks3/brick1
Brick14: gqas009.sbu.lab.eng.bos.redhat.com:/bricks3/brick1
Brick15: gqas010.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick16: gqas012.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick17: gqas014.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick18: gqas015.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick19: gqas003.sbu.lab.eng.bos.redhat.com:/bricks4/brick1
Brick20: gqas009.sbu.lab.eng.bos.redhat.com:/bricks4/brick1
Brick21: gqas010.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick22: gqas012.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick23: gqas014.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick24: gqas015.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick25: gqas003.sbu.lab.eng.bos.redhat.com:/bricks5/brick1
Brick26: gqas009.sbu.lab.eng.bos.redhat.com:/bricks5/brick1
Brick27: gqas010.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick28: gqas012.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick29: gqas014.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick30: gqas015.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick31: gqas003.sbu.lab.eng.bos.redhat.com:/bricks6/brick1
Brick32: gqas009.sbu.lab.eng.bos.redhat.com:/bricks6/brick1
Brick33: gqas010.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick34: gqas012.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick35: gqas014.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick36: gqas015.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick37: gqas003.sbu.lab.eng.bos.redhat.com:/bricks7/brick1
Brick38: gqas009.sbu.lab.eng.bos.redhat.com:/bricks7/brick1
Brick39: gqas010.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick40: gqas012.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick41: gqas014.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick42: gqas015.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick43: gqas003.sbu.lab.eng.bos.redhat.com:/bricks8/brick1
Brick44: gqas009.sbu.lab.eng.bos.redhat.com:/bricks8/brick1
Brick45: gqas010.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick46: gqas012.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick47: gqas014.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick48: gqas015.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick49: gqas003.sbu.lab.eng.bos.redhat.com:/bricks9/brick1
Brick50: gqas009.sbu.lab.eng.bos.redhat.com:/bricks9/brick1
Brick51: gqas010.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick52: gqas012.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick53: gqas014.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick54: gqas015.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick55: gqas003.sbu.lab.eng.bos.redhat.com:/bricks10/brick1
Brick56: gqas009.sbu.lab.eng.bos.redhat.com:/bricks10/brick1
Brick57: gqas010.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick58: gqas012.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick59: gqas014.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick60: gqas015.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick61: gqas003.sbu.lab.eng.bos.redhat.com:/bricks11/brick1
Brick62: gqas009.sbu.lab.eng.bos.redhat.com:/bricks11/brick1
Brick63: gqas010.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick64: gqas012.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick65: gqas014.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick66: gqas015.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick67: gqas003.sbu.lab.eng.bos.redhat.com:/bricks12/brick1
Brick68: gqas009.sbu.lab.eng.bos.redhat.com:/bricks12/brick1
Brick69: gqas010.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick70: gqas012.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick71: gqas014.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick72: gqas015.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Options Reconfigured:
ganesha.enable: on
features.cache-invalidation: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
disperse.shd-max-threads: 64
disperse.shd-wait-qlength: 655536
cluster.enable-shared-storage: enable
nfs-ganesha: enable


Note You need to log in before you can comment on or make changes to this bug.