Bug 1731826 - bricks gone down unexpectedly
Summary: bricks gone down unexpectedly
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: posix
Version: rhgs-3.5
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: RHGS 3.5.0
Assignee: Mohit Agrawal
QA Contact: nchilaka
URL:
Whiteboard:
Depends On:
Blocks: 1696809 1751907
TreeView+ depends on / blocked
 
Reported: 2019-07-22 07:07 UTC by nchilaka
Modified: 2019-10-31 10:38 UTC (History)
9 users (show)

Fixed In Version: glusterfs-6.0-15
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1751907 (view as bug list)
Environment:
Last Closed: 2019-10-30 12:22:15 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:3249 None None None 2019-10-30 12:22:51 UTC

Description nchilaka 2019-07-22 07:07:18 UTC
Description of problem:
========================
On my non-functional testbed, I see that some of the bricks have gone down unexpectedly.



Version-Release number of selected component (if applicable):
====================
6.0.8

Steps to Reproduce:
====================
1.created a 3 node cluster , enabled  brickmux
2. created a 1x3 volume "repvol"and 21x(4+2) ecvolume "ecv" {each server hosts 2 bricks of each ec set)
3. set below options to ecvol
server.event-threads: 8
client.event-threads: 8
disperse.shd-max-threads: 24
4. mounted both ecv and repvol on 6 different clients
5. top o/p being continously captured for each client in repvol
6. IOs triggered on ecv in below pattern
    a). linux untar from 2 clients for 50times
    b). crefi from 2 clients "#for j in {1..20};do for i in {create,chmod,chown,chgrp,symlink,truncate,rename,hardlink}; do crefi --multi -n 5 -b 20 -d 10 --max=1K --min=50 --random -T 2 -t text --fop=$i /mnt/cvlt-ecv/IOs/crefi/$HOSTNAME/ ; sleep 10 ; done;rm -rf /mnt/cvlt-ecv/IOs/crefi/$HOSTNAME/*;done"
    c). lookups (find *|xargs stat) from all 6 clients

7. after 2.5 days, both a) and b) completed successfully
8. issued a du -sh on the directory hosting linux untar directories from client1
9. stopped lookups on couple of clients
10. reissued crefi from the 2 client , but this time without rm -rf
11. in about 15 min, some of the bricks went offline as below

[root@rhs-gp-srv1 glusterfs]# gluster v info
 
Volume Name: ecv
Type: Distributed-Disperse
Volume ID: 4d6d0f89-cd0b-4c27-bfb9-87a93dce21b2
Status: Started
Snapshot Count: 0
Number of Bricks: 21 x (4 + 2) = 126
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1
Brick4: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1
Brick5: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1
Brick6: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1
Brick7: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2
Brick8: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2
Brick9: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2
Brick10: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2
Brick11: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2
Brick12: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2
Brick13: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3
Brick14: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3
Brick15: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3
Brick16: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3
Brick17: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3
Brick18: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3
Brick19: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4
Brick20: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4
Brick21: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4
Brick22: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4
Brick23: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4
Brick24: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4
Brick25: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5
Brick26: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5
Brick27: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5
Brick28: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5
Brick29: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5
Brick30: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5
Brick31: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6
Brick32: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6
Brick33: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6
Brick34: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6
Brick35: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6
Brick36: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6
Brick37: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7
Brick38: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7
Brick39: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7
Brick40: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7
Brick41: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7
Brick42: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7
Brick43: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8
Brick44: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8
Brick45: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8
Brick46: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8
Brick47: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8
Brick48: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8
Brick49: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9
Brick50: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9
Brick51: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9
Brick52: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9
Brick53: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9
Brick54: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9
Brick55: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10
Brick56: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10
Brick57: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10
Brick58: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10
Brick59: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10
Brick60: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10
Brick61: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11
Brick62: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11
Brick63: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11
Brick64: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11
Brick65: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11
Brick66: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11
Brick67: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12
Brick68: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12
Brick69: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12
Brick70: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12
Brick71: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12
Brick72: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12
Brick73: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13
Brick74: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13
Brick75: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13
Brick76: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13
Brick77: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13
Brick78: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13
Brick79: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14
Brick80: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14
Brick81: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14
Brick82: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14
Brick83: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14
Brick84: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14
Brick85: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15
Brick86: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15
Brick87: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15
Brick88: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15
Brick89: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15
Brick90: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15
Brick91: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16
Brick92: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16
Brick93: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16
Brick94: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16
Brick95: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16
Brick96: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16
Brick97: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17
Brick98: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17
Brick99: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17
Brick100: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17
Brick101: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17
Brick102: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17
Brick103: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18
Brick104: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18
Brick105: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18
Brick106: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18
Brick107: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18
Brick108: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18
Brick109: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19
Brick110: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19
Brick111: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19
Brick112: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19
Brick113: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19
Brick114: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19
Brick115: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20
Brick116: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20
Brick117: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20
Brick118: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20
Brick119: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20
Brick120: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20
Brick121: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21
Brick122: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21
Brick123: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21
Brick124: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21
Brick125: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21
Brick126: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
server.event-threads: 8
client.event-threads: 8
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
disperse.shd-max-threads: 24
cluster.brick-multiplex: enable
 
Volume Name: repvol
Type: Replicate
Volume ID: e8dca4b7-5cfb-4cb7-84b6-7db490854d59
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick43/repvol
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick43/repvol
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick43/repvol
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
cluster.brick-multiplex: enable
[root@rhs-gp-srv1 glusterfs]# 
[root@rhs-gp-srv1 glusterfs]# 
[root@rhs-gp-srv1 glusterfs]# gluster v status
Status of volume: ecv
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       N/A       N/A        N       N/A  
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      N/A       N/A        N       N/A  
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     N/A       N/A        N       N/A  
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     N/A       N/A        N       N/A  
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     N/A       N/A        N       N/A  
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     N/A       N/A        N       N/A  
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     N/A       N/A        N       N/A  
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     N/A       N/A        N       N/A  
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49152     0          Y       6852 
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49152     0          Y       6535 
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49152     0          Y       6907 
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49152     0          Y       6852 
Self-heal Daemon on localhost               N/A       N/A        Y       11531
Self-heal Daemon on rhs-gp-srv4.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       12654
Self-heal Daemon on rhs-gp-srv2.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       11547
 
Task Status of Volume ecv
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: repvol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49153     0          Y       11502
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49153     0          Y       11519
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49153     0          Y       12615
Self-heal Daemon on localhost               N/A       N/A        Y       11531
Self-heal Daemon on rhs-gp-srv4.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       12654
Self-heal Daemon on rhs-gp-srv2.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       11547
 
Task Status of Volume repvol
------------------------------------------------------------------------------
There are no active volume tasks

Comment 8 nchilaka 2019-07-22 13:38:58 UTC
sosreports and logs @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1731826/

Comment 30 nchilaka 2019-08-21 10:06:29 UTC
again yet another brick on a different node went down
below is the snippet of brick log

[2019-08-21 05:55:57.496917] W [MSGID: 113096] [posix-handle.c:837:posix_handle_soft] 0-repvol-posix: symlink ../../54/93/54931363-cfa4-428b-83d2-3f5a3dcb1c97/level5.38 -> /gluster/brick43/repvol/.glusterfs/62/67/62676845-eaa3-4dac-8b1b-18c6b1596c9f failed [File exists]
[2019-08-21 05:56:58.053640] E [MSGID: 113118] [posix-helpers.c:1728:posix_gfid_heal] 0-repvol-posix: Fresh file: /gluster/brick43/repvol/IOs/deep-dir-creates/level1.1/level2.1/level3.23/level4.67/level5.62 [No such file or directory]
[2019-08-21 05:57:14.887230] E [MSGID: 113118] [posix-helpers.c:1728:posix_gfid_heal] 0-repvol-posix: Fresh file: /gluster/brick43/repvol/IOs/deep-dir-creates/level1.1/level2.1/level3.23/level4.68/level5.83 [No such file or directory]
[2019-08-21 05:57:29.197112] E [MSGID: 113118] [posix-helpers.c:1728:posix_gfid_heal] 0-repvol-posix: Fresh file: /gluster/brick43/repvol/IOs/deep-dir-creates/level1.1/level2.1/level3.23/level4.69/level5.13 [No such file or directory]
[2019-08-21 05:58:37.079716] W [MSGID: 113075] [posix-helpers.c:2076:posix_fs_health_check] 0-repvol-posix: aio_write_error() on /gluster/brick43/repvol/.glusterfs/health_check returned
[2019-08-21 05:58:37.082198] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down
[2019-08-21 05:58:37.082882] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM
[2019-08-21 05:57:29.198638] E [MSGID: 113118] [posix-helpers.c:1728:posix_gfid_heal] 0-repvol-posix: Fresh file: /gluster/brick43/repvol/IOs/deep-dir-creates/level1.1/level2.1/level3.23/level4.69/level5.13 [No such file or directory]
[2019-08-21 05:59:07.087199] W [glusterfsd.c:1560:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7ea5) [0x7f291e545ea5] -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x5633873a61f5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x5633873a605b] ) 0-: received signum (15), shutting down
(END)

Comment 35 nchilaka 2019-08-22 06:38:57 UTC
also, if you are noting, again bricks went down in the same machine


[root@rhs-gp-srv2 bricks]# 
Broadcast message from systemd-journald@rhs-gp-srv2.lab.eng.blr.redhat.com (Wed 2019-08-21 20:32:56 IST):

gluster-brick41-repvol[748]: [2019-08-21 15:02:56.632219] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down


Broadcast message from systemd-journald@rhs-gp-srv2.lab.eng.blr.redhat.com (Wed 2019-08-21 20:32:56 IST):

gluster-brick41-repvol[748]: [2019-08-21 15:02:56.632982] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM


Message from syslogd@rhs-gp-srv2 at Aug 21 20:32:56 ...
 gluster-brick41-repvol[748]:[2019-08-21 15:02:56.632219] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down

Message from syslogd@rhs-gp-srv2 at Aug 21 20:32:56 ...
 gluster-brick41-repvol[748]:[2019-08-21 15:02:56.632982] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM

Broadcast message from systemd-journald@rhs-gp-srv2.lab.eng.blr.redhat.com (Wed 2019-08-21 20:38:37 IST):

gluster-brick42-repvol[722]: [2019-08-21 15:08:37.575799] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down


Broadcast message from systemd-journald@rhs-gp-srv2.lab.eng.blr.redhat.com (Wed 2019-08-21 20:38:37 IST):

gluster-brick42-repvol[722]: [2019-08-21 15:08:37.576113] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM


Message from syslogd@rhs-gp-srv2 at Aug 21 20:38:37 ...
 gluster-brick42-repvol[722]:[2019-08-21 15:08:37.575799] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down

Message from syslogd@rhs-gp-srv2 at Aug 21 20:38:37 ...
 gluster-brick42-repvol[722]:[2019-08-21 15:08:37.576113] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM

Comment 51 nchilaka 2019-10-10 07:48:45 UTC
verified the steps as mentioned in description on 6.0.15 and after 5 days of the tests run, didn't hit brick down issue.
hence moving to verified. Note that, the test bed details and configurations were the same as what was reported during the issue.

Comment 53 errata-xmlrpc 2019-10-30 12:22:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249


Note You need to log in before you can comment on or make changes to this bug.