Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1286562

Summary: [HC] Unable to move the host running hosted-engine to maintenance state
Product: [oVirt] ovirt-hosted-engine-ha Reporter: SATHEESARAN <sasundar>
Component: AgentAssignee: Martin Sivák <msivak>
Status: CLOSED CURRENTRELEASE QA Contact: Ilanit Stein <istein>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 1.3.2.1CC: bugs, dfediuck, lsurette, sabose, sasundar, sbonazzo, stirabos, ykaul
Target Milestone: ovirt-4.0.0-alphaFlags: dfediuck: ovirt-4.0.0?
rule-engine: planning_ack?
sasundar: devel_ack?
rule-engine: testing_ack?
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
RHEV+RHGS Hyperconvergence RHEL 7.2 Nodes as hypervisors
Last Closed: 2016-02-04 08:15:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1258386    
Attachments:
Description Flags
engine.log from hosted-engine
none
vdsm.log from host-3
none
agent.log from host-3
none
hosted-engine-setup log
none
screenshot showing host-3 was still in 'preparing for maintenance' state none

Description SATHEESARAN 2015-11-30 09:14:04 UTC
Description of problem:
------------------------
Hosted-engine was configured to use gluster domain. 2 Additional hosts are also added to the cluster. In this usecase, when a host running hosted-engine was moved to maintenance state, the host still remains in "Preparing for Maintenance", with the hosted engine running on it.

But the events tab indicate the host is switched to maintenance mode

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
RHEVM 3.6 Beta1 - ( 3.6.0.3-0.1.el6 )
qemu-kvm-rhev-2.3.0-31.el7_2.3.x86_64
gluster-3.8 dev rpms

How reproducible:
-----------------
Never tried to reproduce. But seen the same issue in another setup also

Steps to Reproduce:
-------------------
1. Setup the hosted engine with glusterfs volume as backend
2. Add 2 more additional hosts using hosted-engine setup
3. When the cluster is up, move the host running to hosted-engine to maintenance

Actual results:
---------------
Hosts status continues to stay in "Preparing for maintenance" for long time
Host's events tab says, the host switched to maintenance state, though hosted-engine is still running on that host

Expected results:
------------------
Hosted-engine should move to another host and host should move to maintenance state


Additional info:

Comment 1 Red Hat Bugzilla Rules Engine 2015-11-30 15:34:34 UTC
This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.

Comment 2 Martin Sivák 2015-11-30 15:40:30 UTC
Can you please attach the engine.log from the hosted engine VM and vdsm and hosted engine logs from the host?

Comment 3 Sahina Bose 2015-12-03 07:06:00 UTC
Related but maybe not the same - I was unable to move a host (host2) to maintenance mode even when hosted-engine was not running on it. 
hosted-engine --vm-status --showed that hosted-engine VM was running on host3.

Migrated all running VMs from host2 to other hosts. Engine continued to show that there was a VM running on host2.

vdsClient -s 0 list --> showed hosted-engine VM with state "Down"

Once I destroyed this VM (vdsClient destroy <vmid>), host2 moved to maintenance.

Comment 4 SATHEESARAN 2015-12-03 08:22:15 UTC
# hosted-engine --vm-status


--== Host 1 status ==--

Status up-to-date                  : True
Hostname                           : rhs-client10.lab.eng.blr.redhat.com
Host ID                            : 1
Engine status                      : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : f4d92048
Host timestamp                     : 175330


--== Host 2 status ==--

Status up-to-date                  : True
Hostname                           : rhs-gp-srv1.lab.eng.blr.redhat.com
Host ID                            : 2
Engine status                      : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score                              : 0
stopped                            : False
Local maintenance                  : True
crc32                              : c1e61847
Host timestamp                     : 158830


--== Host 3 status ==--

Status up-to-date                  : True
Hostname                           : rhs-hpc-srv3.lab.eng.blr.redhat.com
Host ID                            : 3
Engine status                      : {"health": "good", "vm": "up", "detail": "up"}
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : 5285de0a
Host timestamp                     : 137108

hosted_engine was running in hosted_engine_3

After moving hosted_engine_3 to maintenance state, 

# hosted-engine --vm-status


--== Host 1 status ==--

Status up-to-date                  : True
Hostname                           : rhs-client10.lab.eng.blr.redhat.com
Host ID                            : 1
Engine status                      : {"health": "good", "vm": "up", "detail": "up"}
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : c1e6f23e
Host timestamp                     : 178291


--== Host 2 status ==--

Status up-to-date                  : True
Hostname                           : rhs-gp-srv1.lab.eng.blr.redhat.com
Host ID                            : 2
Engine status                      : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : 122a0c92
Host timestamp                     : 161493


--== Host 3 status ==--

Status up-to-date                  : True
Hostname                           : rhs-hpc-srv3.lab.eng.blr.redhat.com
Host ID                            : 3
Engine status                      : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"}
Score                              : 0
stopped                            : False
Local maintenance                  : True
crc32                              : a854c225
Host timestamp                     : 140084

The UI still status of hosted_engine_3 as "Preparing for Maintenance" and event tabs states that - host is switched to maintenance

I will be attaching the engine.log from hosted-engine, vdsm logs post this comment

Comment 5 SATHEESARAN 2015-12-03 08:23:08 UTC
Created attachment 1101700 [details]
engine.log from hosted-engine

Comment 6 SATHEESARAN 2015-12-03 08:26:18 UTC
Created attachment 1101701 [details]
vdsm.log from host-3

Comment 7 SATHEESARAN 2015-12-03 08:30:56 UTC
Created attachment 1101702 [details]
agent.log from host-3

Comment 8 SATHEESARAN 2015-12-03 08:31:41 UTC
Created attachment 1101703 [details]
hosted-engine-setup log

Comment 9 SATHEESARAN 2015-12-03 08:33:11 UTC
Created attachment 1101704 [details]
screenshot showing host-3 was still in 'preparing for maintenance' state

Comment 10 SATHEESARAN 2016-02-04 08:15:06 UTC
Tested with RHEV 3.6 beta3 ( RHEVM 3.6.3 ), with hosted engine ( ovirt-hosted-engine-setup-1.3.2.3-1.el7ev.noarch )

This works now.
I could move the host to maintenance, once master storage domain is created ( which imports hosted-engine storage domain too ).