Hide Forgot
Description of problem: ======================== On my non-functional testbed, I see that some of the bricks have gone down unexpectedly. Version-Release number of selected component (if applicable): ==================== 6.0.8 Steps to Reproduce: ==================== 1.created a 3 node cluster , enabled brickmux 2. created a 1x3 volume "repvol"and 21x(4+2) ecvolume "ecv" {each server hosts 2 bricks of each ec set) 3. set below options to ecvol server.event-threads: 8 client.event-threads: 8 disperse.shd-max-threads: 24 4. mounted both ecv and repvol on 6 different clients 5. top o/p being continously captured for each client in repvol 6. IOs triggered on ecv in below pattern a). linux untar from 2 clients for 50times b). crefi from 2 clients "#for j in {1..20};do for i in {create,chmod,chown,chgrp,symlink,truncate,rename,hardlink}; do crefi --multi -n 5 -b 20 -d 10 --max=1K --min=50 --random -T 2 -t text --fop=$i /mnt/cvlt-ecv/IOs/crefi/$HOSTNAME/ ; sleep 10 ; done;rm -rf /mnt/cvlt-ecv/IOs/crefi/$HOSTNAME/*;done" c). lookups (find *|xargs stat) from all 6 clients 7. after 2.5 days, both a) and b) completed successfully 8. issued a du -sh on the directory hosting linux untar directories from client1 9. stopped lookups on couple of clients 10. reissued crefi from the 2 client , but this time without rm -rf 11. in about 15 min, some of the bricks went offline as below [root@rhs-gp-srv1 glusterfs]# gluster v info Volume Name: ecv Type: Distributed-Disperse Volume ID: 4d6d0f89-cd0b-4c27-bfb9-87a93dce21b2 Status: Started Snapshot Count: 0 Number of Bricks: 21 x (4 + 2) = 126 Transport-type: tcp Bricks: Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1 Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1 Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1 Brick4: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1 Brick5: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1 Brick6: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1 Brick7: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2 Brick8: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2 Brick9: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2 Brick10: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2 Brick11: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2 Brick12: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2 Brick13: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3 Brick14: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3 Brick15: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3 Brick16: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3 Brick17: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3 Brick18: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3 Brick19: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4 Brick20: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4 Brick21: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4 Brick22: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4 Brick23: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4 Brick24: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4 Brick25: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5 Brick26: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5 Brick27: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5 Brick28: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5 Brick29: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5 Brick30: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5 Brick31: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6 Brick32: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6 Brick33: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6 Brick34: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6 Brick35: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6 Brick36: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6 Brick37: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7 Brick38: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7 Brick39: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7 Brick40: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7 Brick41: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7 Brick42: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7 Brick43: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8 Brick44: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8 Brick45: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8 Brick46: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8 Brick47: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8 Brick48: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8 Brick49: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9 Brick50: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9 Brick51: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9 Brick52: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9 Brick53: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9 Brick54: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9 Brick55: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10 Brick56: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10 Brick57: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10 Brick58: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10 Brick59: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10 Brick60: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10 Brick61: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11 Brick62: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11 Brick63: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11 Brick64: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11 Brick65: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11 Brick66: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11 Brick67: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12 Brick68: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12 Brick69: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12 Brick70: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12 Brick71: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12 Brick72: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12 Brick73: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13 Brick74: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13 Brick75: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13 Brick76: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13 Brick77: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13 Brick78: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13 Brick79: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14 Brick80: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14 Brick81: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14 Brick82: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14 Brick83: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14 Brick84: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14 Brick85: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15 Brick86: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15 Brick87: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15 Brick88: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15 Brick89: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15 Brick90: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15 Brick91: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16 Brick92: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16 Brick93: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16 Brick94: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16 Brick95: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16 Brick96: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16 Brick97: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17 Brick98: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17 Brick99: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17 Brick100: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17 Brick101: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17 Brick102: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17 Brick103: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18 Brick104: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18 Brick105: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18 Brick106: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18 Brick107: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18 Brick108: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18 Brick109: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19 Brick110: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19 Brick111: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19 Brick112: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19 Brick113: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19 Brick114: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19 Brick115: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20 Brick116: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20 Brick117: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20 Brick118: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20 Brick119: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20 Brick120: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20 Brick121: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21 Brick122: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21 Brick123: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21 Brick124: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21 Brick125: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21 Brick126: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21 Options Reconfigured: diagnostics.count-fop-hits: on diagnostics.latency-measurement: on server.event-threads: 8 client.event-threads: 8 nfs.disable: on storage.fips-mode-rchecksum: on transport.address-family: inet disperse.shd-max-threads: 24 cluster.brick-multiplex: enable Volume Name: repvol Type: Replicate Volume ID: e8dca4b7-5cfb-4cb7-84b6-7db490854d59 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick43/repvol Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick43/repvol Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick43/repvol Options Reconfigured: performance.client-io-threads: off nfs.disable: on storage.fips-mode-rchecksum: on transport.address-family: inet cluster.brick-multiplex: enable [root@rhs-gp-srv1 glusterfs]# [root@rhs-gp-srv1 glusterfs]# [root@rhs-gp-srv1 glusterfs]# gluster v status Status of volume: ecv Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick1/ecv-sv1 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick1/ecv-sv1 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick1/ecv-sv1 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick2/ecv-sv1 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick2/ecv-sv1 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick2/ecv-sv1 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick3/ecv-sv2 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick3/ecv-sv2 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick3/ecv-sv2 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick4/ecv-sv2 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick4/ecv-sv2 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick4/ecv-sv2 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick5/ecv-sv3 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick5/ecv-sv3 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick5/ecv-sv3 N/A N/A N N/A Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick6/ecv-sv3 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick6/ecv-sv3 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick6/ecv-sv3 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick7/ecv-sv4 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick7/ecv-sv4 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick7/ecv-sv4 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick8/ecv-sv4 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick8/ecv-sv4 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick8/ecv-sv4 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick9/ecv-sv5 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick9/ecv-sv5 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick9/ecv-sv5 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick10/ecv-sv5 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick10/ecv-sv5 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick10/ecv-sv5 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick11/ecv-sv6 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick11/ecv-sv6 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick11/ecv-sv6 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick12/ecv-sv6 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick12/ecv-sv6 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick12/ecv-sv6 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick13/ecv-sv7 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick13/ecv-sv7 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick13/ecv-sv7 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick14/ecv-sv7 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick14/ecv-sv7 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick14/ecv-sv7 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick15/ecv-sv8 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick15/ecv-sv8 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick15/ecv-sv8 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick16/ecv-sv8 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick16/ecv-sv8 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick16/ecv-sv8 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick17/ecv-sv9 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick17/ecv-sv9 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick17/ecv-sv9 N/A N/A N N/A Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick18/ecv-sv9 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick18/ecv-sv9 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick18/ecv-sv9 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick19/ecv-sv10 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick19/ecv-sv10 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick19/ecv-sv10 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick20/ecv-sv10 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick20/ecv-sv10 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick20/ecv-sv10 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick21/ecv-sv11 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick21/ecv-sv11 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick21/ecv-sv11 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick22/ecv-sv11 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick22/ecv-sv11 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick22/ecv-sv11 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick23/ecv-sv12 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick23/ecv-sv12 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick23/ecv-sv12 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick24/ecv-sv12 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick24/ecv-sv12 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick24/ecv-sv12 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick25/ecv-sv13 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick25/ecv-sv13 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick25/ecv-sv13 N/A N/A N N/A Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick26/ecv-sv13 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick26/ecv-sv13 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick26/ecv-sv13 N/A N/A N N/A Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick27/ecv-sv14 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick27/ecv-sv14 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick27/ecv-sv14 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick28/ecv-sv14 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick28/ecv-sv14 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick28/ecv-sv14 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick29/ecv-sv15 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick29/ecv-sv15 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick29/ecv-sv15 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick30/ecv-sv15 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick30/ecv-sv15 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick30/ecv-sv15 N/A N/A N N/A Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick31/ecv-sv16 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick31/ecv-sv16 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick31/ecv-sv16 N/A N/A N N/A Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick32/ecv-sv16 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick32/ecv-sv16 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick32/ecv-sv16 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick33/ecv-sv17 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick33/ecv-sv17 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick33/ecv-sv17 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick34/ecv-sv17 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick34/ecv-sv17 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick34/ecv-sv17 N/A N/A N N/A Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick35/ecv-sv18 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick35/ecv-sv18 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick35/ecv-sv18 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick36/ecv-sv18 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick36/ecv-sv18 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick36/ecv-sv18 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick37/ecv-sv19 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick37/ecv-sv19 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick37/ecv-sv19 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick38/ecv-sv19 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick38/ecv-sv19 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick38/ecv-sv19 N/A N/A N N/A Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick39/ecv-sv20 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick39/ecv-sv20 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick39/ecv-sv20 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick40/ecv-sv20 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick40/ecv-sv20 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick40/ecv-sv20 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick41/ecv-sv21 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick41/ecv-sv21 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick41/ecv-sv21 49152 0 Y 6852 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick42/ecv-sv21 49152 0 Y 6535 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick42/ecv-sv21 49152 0 Y 6907 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick42/ecv-sv21 49152 0 Y 6852 Self-heal Daemon on localhost N/A N/A Y 11531 Self-heal Daemon on rhs-gp-srv4.lab.eng.blr .redhat.com N/A N/A Y 12654 Self-heal Daemon on rhs-gp-srv2.lab.eng.blr .redhat.com N/A N/A Y 11547 Task Status of Volume ecv ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: repvol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g luster/brick43/repvol 49153 0 Y 11502 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g luster/brick43/repvol 49153 0 Y 11519 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g luster/brick43/repvol 49153 0 Y 12615 Self-heal Daemon on localhost N/A N/A Y 11531 Self-heal Daemon on rhs-gp-srv4.lab.eng.blr .redhat.com N/A N/A Y 12654 Self-heal Daemon on rhs-gp-srv2.lab.eng.blr .redhat.com N/A N/A Y 11547 Task Status of Volume repvol ------------------------------------------------------------------------------ There are no active volume tasks
sosreports and logs @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1731826/
again yet another brick on a different node went down below is the snippet of brick log [2019-08-21 05:55:57.496917] W [MSGID: 113096] [posix-handle.c:837:posix_handle_soft] 0-repvol-posix: symlink ../../54/93/54931363-cfa4-428b-83d2-3f5a3dcb1c97/level5.38 -> /gluster/brick43/repvol/.glusterfs/62/67/62676845-eaa3-4dac-8b1b-18c6b1596c9f failed [File exists] [2019-08-21 05:56:58.053640] E [MSGID: 113118] [posix-helpers.c:1728:posix_gfid_heal] 0-repvol-posix: Fresh file: /gluster/brick43/repvol/IOs/deep-dir-creates/level1.1/level2.1/level3.23/level4.67/level5.62 [No such file or directory] [2019-08-21 05:57:14.887230] E [MSGID: 113118] [posix-helpers.c:1728:posix_gfid_heal] 0-repvol-posix: Fresh file: /gluster/brick43/repvol/IOs/deep-dir-creates/level1.1/level2.1/level3.23/level4.68/level5.83 [No such file or directory] [2019-08-21 05:57:29.197112] E [MSGID: 113118] [posix-helpers.c:1728:posix_gfid_heal] 0-repvol-posix: Fresh file: /gluster/brick43/repvol/IOs/deep-dir-creates/level1.1/level2.1/level3.23/level4.69/level5.13 [No such file or directory] [2019-08-21 05:58:37.079716] W [MSGID: 113075] [posix-helpers.c:2076:posix_fs_health_check] 0-repvol-posix: aio_write_error() on /gluster/brick43/repvol/.glusterfs/health_check returned [2019-08-21 05:58:37.082198] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down [2019-08-21 05:58:37.082882] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM [2019-08-21 05:57:29.198638] E [MSGID: 113118] [posix-helpers.c:1728:posix_gfid_heal] 0-repvol-posix: Fresh file: /gluster/brick43/repvol/IOs/deep-dir-creates/level1.1/level2.1/level3.23/level4.69/level5.13 [No such file or directory] [2019-08-21 05:59:07.087199] W [glusterfsd.c:1560:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7ea5) [0x7f291e545ea5] -->/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xe5) [0x5633873a61f5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x5633873a605b] ) 0-: received signum (15), shutting down (END)
also, if you are noting, again bricks went down in the same machine [root@rhs-gp-srv2 bricks]# Broadcast message from systemd-journald@rhs-gp-srv2.lab.eng.blr.redhat.com (Wed 2019-08-21 20:32:56 IST): gluster-brick41-repvol[748]: [2019-08-21 15:02:56.632219] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down Broadcast message from systemd-journald@rhs-gp-srv2.lab.eng.blr.redhat.com (Wed 2019-08-21 20:32:56 IST): gluster-brick41-repvol[748]: [2019-08-21 15:02:56.632982] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM Message from syslogd@rhs-gp-srv2 at Aug 21 20:32:56 ... gluster-brick41-repvol[748]:[2019-08-21 15:02:56.632219] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down Message from syslogd@rhs-gp-srv2 at Aug 21 20:32:56 ... gluster-brick41-repvol[748]:[2019-08-21 15:02:56.632982] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM Broadcast message from systemd-journald@rhs-gp-srv2.lab.eng.blr.redhat.com (Wed 2019-08-21 20:38:37 IST): gluster-brick42-repvol[722]: [2019-08-21 15:08:37.575799] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down Broadcast message from systemd-journald@rhs-gp-srv2.lab.eng.blr.redhat.com (Wed 2019-08-21 20:38:37 IST): gluster-brick42-repvol[722]: [2019-08-21 15:08:37.576113] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM Message from syslogd@rhs-gp-srv2 at Aug 21 20:38:37 ... gluster-brick42-repvol[722]:[2019-08-21 15:08:37.575799] M [MSGID: 113075] [posix-helpers.c:2150:posix_health_check_thread_proc] 0-repvol-posix: health-check failed, going down Message from syslogd@rhs-gp-srv2 at Aug 21 20:38:37 ... gluster-brick42-repvol[722]:[2019-08-21 15:08:37.576113] M [MSGID: 113075] [posix-helpers.c:2168:posix_health_check_thread_proc] 0-repvol-posix: still alive! -> SIGTERM
verified the steps as mentioned in description on 6.0.15 and after 5 days of the tests run, didn't hit brick down issue. hence moving to verified. Note that, the test bed details and configurations were the same as what was reported during the issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:3249