Bug 1619158

Summary: [Tracker-RHGS-BZ#1612098] - Bricks are not coming up after rebooting whole setup
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nitin Goyal <nigoyal>
Component: rhgs-server-containerAssignee: Saravanakumar <sarumuga>
Status: CLOSED ERRATA QA Contact: Nitin Goyal <nigoyal>
Severity: urgent Docs Contact:
Priority: urgent    
Version: cns-3.10CC: aclewett, hbustam, hchiramm, kramdoss, madam, nchilaka, rhs-bugs, sankarshan
Target Milestone: ---   
Target Release: CNS 3.10   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhgs-server-rhel7:3.4.0-2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-12 10:54:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1612098    
Bug Blocks: 1568862    

Description Nitin Goyal 2018-08-20 08:37:29 UTC 
Description of problem:
I have ocp 3.10 + cns 3.10 setup. 9 gluster volumes were there (2 arbiter, & 7 replica). After rebooting the setup bricks are not coming up.

Version-Release number of selected component (if applicable):
rhgs-server-rhel7:3.4.0-1
glusterfs-cli-3.12.2-15.el7rhgs.x86_64

How reproducible:

Steps to Reproduce:
1. Create setup with ocp 3.10 + cns 3.10
2. Create some arbiter volumes and some replica 3 volumes.
3. Reboot whole cluster

Actual results:
Bricks are not coming up.

Status of volume: heketidbstorage
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.47.196:/var/lib/heketi/mounts/v
g_8028649c3a72a923844d674fe0ffa9ea/brick_88
06e3c51cc4483cd38767ee4c08a7dd/brick        N/A       N/A        N       N/A  
Brick 10.70.47.110:/var/lib/heketi/mounts/v
g_938d864334e0098ab7eb5a8543cb366f/brick_45
2a51ac248a76169dfc776f9855cf60/brick        N/A       N/A        N       N/A  
Brick 10.70.46.144:/var/lib/heketi/mounts/v
g_93d3464978dc46771cf773e4cdbc9270/brick_ad
a9b96850b5d851240e62f5a421be24/brick        N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       521  
Self-heal Daemon on 10.70.47.110            N/A       N/A        Y       582  
Self-heal Daemon on dhcp46-144.lab.eng.blr.
redhat.com                                  N/A       N/A        Y       447  
 
Task Status of Volume heketidbstorage
------------------------------------------------------------------------------
There are no active volume tasks



Expected results:
Bricks should be up.

Additional info:
sh-4.2# gluster v heal heketidbstorage info
heketidbstorage: Not able to fetch volfile from glusterd
Volume heal failed.
Comment 2 Nitin Goyal 2018-08-20 08:45:18 UTC 
Sosreports

http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1619158/
Comment 9 Nitin Goyal 2018-08-24 14:29:29 UTC 
Status of volume: vol020
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.47.110:/var/lib/heketi/mounts/v
g_f23418d7399662f977e3988cde0af84a/brick_ab
63dc73c73bc6bdb50d1c25f135183f/brick        49153     0          Y       670  
Brick 10.70.46.144:/var/lib/heketi/mounts/v
g_7a53bad66acec762214f7bf885ad8d13/brick_33
7ac1f1588622f17e0404ca3f611038/brick        49154     0          Y       664  
Brick 10.70.46.203:/var/lib/heketi/mounts/v
g_05b15c2b75716e57fa262a0ffc33cf90/brick_90
6d02067aa98fb5adea46257748396f/brick        49155     0          Y       681  
Self-heal Daemon on localhost               N/A       N/A        Y       652  
Self-heal Daemon on 10.70.46.144            N/A       N/A        Y       646  
Self-heal Daemon on dhcp46-203.lab.eng.blr.
redhat.com                                  N/A       N/A        Y       642  
Self-heal Daemon on 10.70.47.196            N/A       N/A        Y       531  
 
Task Status of Volume vol020
------------------------------------------------------------------------------
There are no active volume tasks


gluster package

glusterfs-3.12.2-16.el7rhgs.x86_64


Marking this as verified because in new build bricks are coming properly
Comment 11 errata-xmlrpc 2018-09-12 10:54:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2688