Bug 1419195

Summary: Hosted Engine Migration fails when putting host in maintenance mode
Product: Red Hat Enterprise Virtualization Manager Reporter: Tom Gamull <tomg>
Component: ovirt-engineAssignee: Nobody <nobody>
Status: CLOSED NOTABUG QA Contact: Nikolai Sednev <nsednev>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.0.4CC: dfediuck, gklein, lsurette, oourfali, rbalakri, Rhev-m-bugs, srevivo, tomg, ykaul
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-15 10:30:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
What hosted-engine --vm-status shows
none
GUI showing hung migration of HE
none
Host preparing for migration
none
source vdsm
none
target vdsm.log none

Description Tom Gamull 2017-02-03 21:04:04 UTC 
Created attachment 1247576 [details]
What hosted-engine --vm-status shows

Description of problem:
When putting the host running HE in maintenance mode through the RHV4 admin console GUI, the host prepares, moves VMs but appears stuck moving the HE.  The HE shows 82% completed but never shows as compelted (preventing the maintenance mode in GUI). I can activate from maintenance mode on the host and retry but HE migration status shows as stuck on 82%.

Version-Release number of selected component (if applicable):
ovirt-engine-4.0.4.4-0.1.el7ev.noarch

How reproducible:
Often when migrating the host running the HE

Steps to Reproduce:
1.Select Host running SPM and HE and click maintenance mode, accept default (don't check take gluster offline)
2. Wait for a long time
3. Check CLI to ensure migration finished and HE is on new host.

Actual results:
Migration of HE shows 82% and doesn't finish. Shows running on old host in GUI yet hosted-engine --vm-status shows it running on new host

Expected results:
Maintenance mode entered and HE migration complete

Additional info:
Same issue here I beleive - https://bugzilla.redhat.com/show_bug.cgi?id=1377994
I tried closing and reopneing the Admin GUI
I set global maintenance afterwards
none of this helped
I am using gluster underneath as storage
Comment 1 Tom Gamull 2017-02-03 21:09:59 UTC 
Created attachment 1247577 [details]
GUI showing hung migration of HE
Comment 2 Tom Gamull 2017-02-03 21:10:27 UTC 
Created attachment 1247578 [details]
Host preparing for migration
Comment 3 Tom Gamull 2017-02-03 21:20:49 UTC 
Created attachment 1247579 [details]
source vdsm
Comment 4 Tom Gamull 2017-02-03 21:21:58 UTC 
Created attachment 1247580 [details]
target vdsm.log
Comment 5 Tom Gamull 2017-02-04 19:10:48 UTC 
I just updated the hosts and engine and it seemed to work when I did the maintenance mode on virt05 (instead of virt04).  I will retry on virt04 and report results.  I did the appliance upgrade (from virt05 using the hosted-engine command (instead of yum update)).  I'm going to save the VDSM archives now in case they are useful, I also have a backup of the old engine. Let me know if anything else needs to be saved.
Comment 6 Tom Gamull 2017-02-04 19:40:01 UTC 
Seems to be handling migrations now after upgrading to ovirt-engine-4.0.6.3-0.1.el7ev.noarch
Comment 7 Nikolai Sednev 2017-02-08 12:00:07 UTC 
Hi Tom,
What did you tried to do, to upgrade using "hosted-engine --upgrade-appliance" tool from 3.6->4.0?

Please be aware that this tool was designed only for 3.6->4.0 upgrades.
For 4.0->4.1 you should use regular upgrade flow from the engine (repos, yum update rhevm*, engine-setup, etc.). Same for hosts.
Comment 8 Tom Gamull 2017-02-08 16:17:26 UTC 
I think I realized that and did the rollback.  I will retest.  Thanks for the clarification, not idea why I tried it that other way.
Comment 9 Doron Fediuck 2017-02-15 10:30:48 UTC 
Closing due to the upgrade flow.
If this is reproducible on a clean supported upgrade flow please reopen with the reproduction steps.