Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1594132

Summary: glusterd not surviving reboot
Product: [oVirt] ovirt-host-deploy Reporter: femi
Component: Plugins.GlusterAssignee: Sahina Bose <sabose>
Status: CLOSED CANTFIX QA Contact: SATHEESARAN <sasundar>
Severity: high Docs Contact:
Priority: high    
Version: 1.7.3CC: bugs, femi, sabose, sasundar
Target Milestone: ovirt-4.2.7Flags: rule-engine: ovirt-4.2+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-05 11:21:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Gluster RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs
none
glusterd logs none

Description femi 2018-06-22 08:09:48 UTC 
Description of problem:
After upgrading & rebooting host, glusterd does not survive a reboot.
This is a single node install following this procedure: https://www.ovirt.org/documentation/gluster-hyperconverged/chap-Single_node_hyperconverged/

Version-Release number of selected component (if applicable):
oVirt 4.2.3.1

How reproducible:


Steps to Reproduce:
1. put host in global maintenance

2. run yum update on hosted engine

3. run engine-setup on hosted engine

4. shutdown all VMs and hosted engine

5. run yum update on physical host

6. reboot

7. hosted-engine --vm-status ->see results in Additional info section:
8. systemctl status ovirt-ha-agent ->see results in Additional info section:

9. systemctl start glusterd

10. start hosted-engine

11. disable global maintenance

Actual results:
glusterd service failed after reboot (step # 6)

Expected results:
glusterd should survive the reboot

Additional info:
after rebooting physical host, I had to "systemctl start glusterd", otherwise. hosted-engine --vm-status was giving the following error "The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable"
Comment 2 Sahina Bose 2018-06-28 06:37:32 UTC 
Also please provide the version of glusterfs being used
Comment 3 femi 2018-06-28 14:10:55 UTC 
GlusterFS Version: glusterfs-3.12.9-1.el7
Comment 5 femi 2018-06-28 17:12:43 UTC 
Created attachment 1455386 [details]
logs
Comment 7 SATHEESARAN 2018-07-03 08:47:59 UTC 
Hi Femi,

glusterd is always configured to start post reboot.
could you please check the same with the following:
# systemctl is-enabled glusterd

which distro is used here ?

Could you also attach the glusterd log file ?
glusterd log file is located at /var/log/glusterfs/glusterd.log. By now, I think the glusterd log should have been logrotated. If so could attach all the relevant glusterd logs - /var/log/glusterfs/glusterd*
Comment 9 femi 2018-07-05 17:00:39 UTC 
Created attachment 1456803 [details]
glusterd logs
Comment 10 femi 2018-07-05 17:02:23 UTC 
(In reply to SATHEESARAN from comment #7)
> Hi Femi,
> 
> glusterd is always configured to start post reboot.
> could you please check the same with the following:
> # systemctl is-enabled glusterd
"disabled"

 
> which distro is used here ?
ovirt node
 
> Could you also attach the glusterd log file ?
> glusterd log file is located at /var/log/glusterfs/glusterd.log. By now, I
> think the glusterd log should have been logrotated. If so could attach all
> the relevant glusterd logs - /var/log/glusterfs/glusterd*
attached
Comment 11 Sahina Bose 2018-08-17 08:00:52 UTC 
(In reply to femi from comment #10)
> (In reply to SATHEESARAN from comment #7)
> > Hi Femi,
> > 
> > glusterd is always configured to start post reboot.
> > could you please check the same with the following:
> > # systemctl is-enabled glusterd
> "disabled"

As per patch https://gerrit.ovirt.org/#/c/87192/, glusterd service should be enabled once the host is added to the cluster.

Can you check that your cluster had "Gluster service" enabled when the host was added? Otherwise this could explain the behaviour you're seeing
> 
>  
> > which distro is used here ?
> ovirt node
>  
> > Could you also attach the glusterd log file ?
> > glusterd log file is located at /var/log/glusterfs/glusterd.log. By now, I
> > think the glusterd log should have been logrotated. If so could attach all
> > the relevant glusterd logs - /var/log/glusterfs/glusterd*
> attached
Comment 12 femi 2018-08-17 14:52:44 UTC 
(In reply to Sahina Bose from comment #11)
> (In reply to femi from comment #10)
> > (In reply to SATHEESARAN from comment #7)
> > > Hi Femi,
> > > 
> > > glusterd is always configured to start post reboot.
> > > could you please check the same with the following:
> > > # systemctl is-enabled glusterd
> > "disabled"
> 
> As per patch https://gerrit.ovirt.org/#/c/87192/, glusterd service should be
> enabled once the host is added to the cluster.
> 
> Can you check that your cluster had "Gluster service" enabled when the host
> was added? Otherwise this could explain the behaviour you're seeing
> > 
> >  
> > > which distro is used here ?
> > ovirt node
> >  
> > > Could you also attach the glusterd log file ?
> > > glusterd log file is located at /var/log/glusterfs/glusterd.log. By now, I
> > > think the glusterd log should have been logrotated. If so could attach all
> > > the relevant glusterd logs - /var/log/glusterfs/glusterd*
> > attached

Gluster service was not enabled (post install).

Keep in mind, this was on a single node gluster install per https://www.ovirt.org/documentation/gluster-hyperconverged/chap-Single_node_hyperconverged/

I have since enable the gluster service & it now works as intended.

This problem does not exist on a typical 3 node HE type install.
Comment 13 Sahina Bose 2018-09-05 11:21:09 UTC 
Closing this, as the gluster service being enabled on a single node install is handled in a separate bug -  bug 1608660