Bug 1419195 - Hosted Engine Migration fails when putting host in maintenance mode
Summary: Hosted Engine Migration fails when putting host in maintenance mode
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.0.4
Hardware: x86_64
OS: Linux
Target Milestone: ---
: ---
Assignee: Nobody
QA Contact: Nikolai Sednev
Depends On:
TreeView+ depends on / blocked
Reported: 2017-02-03 21:04 UTC by Tom Gamull
Modified: 2017-02-15 10:30 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-02-15 10:30:48 UTC
oVirt Team: SLA
Target Upstream Version:

Attachments (Terms of Use)
What hosted-engine --vm-status shows (2.18 KB, text/plain)
2017-02-03 21:04 UTC, Tom Gamull
no flags Details
GUI showing hung migration of HE (300.45 KB, image/jpeg)
2017-02-03 21:09 UTC, Tom Gamull
no flags Details
Host preparing for migration (129.18 KB, image/jpeg)
2017-02-03 21:10 UTC, Tom Gamull
no flags Details
source vdsm (15.29 MB, text/plain)
2017-02-03 21:20 UTC, Tom Gamull
no flags Details
target vdsm.log (17.07 MB, text/plain)
2017-02-03 21:21 UTC, Tom Gamull
no flags Details

Description Tom Gamull 2017-02-03 21:04:04 UTC
Created attachment 1247576 [details]
What hosted-engine --vm-status shows

Description of problem:
When putting the host running HE in maintenance mode through the RHV4 admin console GUI, the host prepares, moves VMs but appears stuck moving the HE.  The HE shows 82% completed but never shows as compelted (preventing the maintenance mode in GUI). I can activate from maintenance mode on the host and retry but HE migration status shows as stuck on 82%.

Version-Release number of selected component (if applicable):

How reproducible:
Often when migrating the host running the HE

Steps to Reproduce:
1.Select Host running SPM and HE and click maintenance mode, accept default (don't check take gluster offline)
2. Wait for a long time
3. Check CLI to ensure migration finished and HE is on new host.

Actual results:
Migration of HE shows 82% and doesn't finish. Shows running on old host in GUI yet hosted-engine --vm-status shows it running on new host

Expected results:
Maintenance mode entered and HE migration complete

Additional info:
Same issue here I beleive - https://bugzilla.redhat.com/show_bug.cgi?id=1377994
I tried closing and reopneing the Admin GUI
I set global maintenance afterwards
none of this helped
I am using gluster underneath as storage

Comment 1 Tom Gamull 2017-02-03 21:09:59 UTC
Created attachment 1247577 [details]
GUI showing hung migration of HE

Comment 2 Tom Gamull 2017-02-03 21:10:27 UTC
Created attachment 1247578 [details]
Host preparing for migration

Comment 3 Tom Gamull 2017-02-03 21:20:49 UTC
Created attachment 1247579 [details]
source vdsm

Comment 4 Tom Gamull 2017-02-03 21:21:58 UTC
Created attachment 1247580 [details]
target vdsm.log

Comment 5 Tom Gamull 2017-02-04 19:10:48 UTC
I just updated the hosts and engine and it seemed to work when I did the maintenance mode on virt05 (instead of virt04).  I will retry on virt04 and report results.  I did the appliance upgrade (from virt05 using the hosted-engine command (instead of yum update)).  I'm going to save the VDSM archives now in case they are useful, I also have a backup of the old engine. Let me know if anything else needs to be saved.

Comment 6 Tom Gamull 2017-02-04 19:40:01 UTC
Seems to be handling migrations now after upgrading to ovirt-engine-

Comment 7 Nikolai Sednev 2017-02-08 12:00:07 UTC
Hi Tom,
What did you tried to do, to upgrade using "hosted-engine --upgrade-appliance" tool from 3.6->4.0?

Please be aware that this tool was designed only for 3.6->4.0 upgrades.
For 4.0->4.1 you should use regular upgrade flow from the engine (repos, yum update rhevm*, engine-setup, etc.). Same for hosts.

Comment 8 Tom Gamull 2017-02-08 16:17:26 UTC
I think I realized that and did the rollback.  I will retest.  Thanks for the clarification, not idea why I tried it that other way.

Comment 9 Doron Fediuck 2017-02-15 10:30:48 UTC
Closing due to the upgrade flow.
If this is reproducible on a clean supported upgrade flow please reopen with the reproduction steps.

Note You need to log in before you can comment on or make changes to this bug.