Bug 1020858

Summary:	[RFE] rescue mode for hosted engine
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Pablo Iranzo Gómez <pablo.iranzo>
Component:	ovirt-hosted-engine-setup	Assignee:	Sandro Bonazzola <sbonazzo>
Status:	CLOSED ERRATA	QA Contact:	Leonid Natapov <lnatapov>
Severity:	medium	Docs Contact:
Priority:	high
Version:	3.3.0	CC:	acathrow, dfediuck, didi, gpadgett, iheim, lpeer, mkalinin, oschreib, pablo.iranzo, pstehlik, sbonazzo, yeylon
Target Milestone:	---	Keywords:	FutureFeature, Improvement, Triaged
Target Release:	3.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	integration
Fixed In Version:	ovirt-hosted-engine-setup-1.0.0-0.7.beta2.el6ev	Doc Type:	Enhancement
Doc Text:	Feature: Hosted engine maintenance mode Reason: Provide a method to stop hosted engine automation in order to maintain the hosted engine VM or hosts on which it runs. Result (if any): The hosted-engine tool has 2 new options added for this feature: --set-maintenance=<local\|global\|none> - This sets the maintenance mode of the hosted engine agent: - local: shut the engine VM down on the local host (if running), and lower the host's HA score to 0. This allows for maintenance of the local host, including putting the host into maintenance mode in the engine. - global: pause all activity of all HA agents, leaving the engine VM running on its current host. The VM can now be start, shut down, and/or changed as desired without the HA agent taking any action to ensure it is up and running. - none: un-set maintenance mode and resume normal operation on HA agents. Note that this option affects only the scope of maintenance in force at the time; that is, turning off local maintenance will turn it off only on the local host, while turning off global maintenance will turn it off for all hosts of the cluster. --vm-start-paused - This option will create the VM in paused state, allowing the VM's domain XML to be changed before the VM runs. One useful application of this is to add/remove devices, particularly boot media.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-01-21 16:54:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Pablo Iranzo Gómez 2013-10-18 11:45:32 UTC

Description of problem:

I did an upgrade of the hosted-engine VM to EL 6.5 beta (as I did with hypervisors), and after that, the VM no longer booted.

I was able to use the --console to see the kernel panic but I was not able to reboot it using keystrokes.

I finally needed to shutdown both hypervisors to ensure that VM was left shutdown, and use qemu from my desktop to boot the hardrive, use grub in the VM to select the working entry, and try to do the 'troubleshooting' from there.

Hosted should have some sort of 'emergency' or 'rescue' mode for engine, allowing to start the vm paused, maybe pressing for a key or whatever (even a custom iso that just reboots until yo press a key in bios or select another boot entry), or that even allows you to put a custom iso (or rhel one) to do the rescue itself.

During that period, no hypervisor should try to start it until the rescue is marked as finished.


Tested on IS19

Comment 1 Itamar Heim 2013-10-20 16:30:09 UTC

greg - we discussed some central "maint" flag which would cause HA activity to stop?

Comment 2 Greg Padgett 2013-10-20 20:23:52 UTC

(In reply to Itamar Heim from comment #1)
> greg - we discussed some central "maint" flag which would cause HA activity
> to stop?

Yes, bug 1015724 covers this part of the rfe.  There's a patch sent for it as well.

The remainder of the rfe seems to fall on the setup/cli side of things, adding Sandro as cc for this.

Comment 3 Sandro Bonazzola 2013-10-21 06:38:09 UTC

So the request is for adding --set-mainteinance=True/False and --vm-start-paused to hosted-engine tool?

Comment 4 Pablo Iranzo Gómez 2013-10-21 07:52:29 UTC

Sandro,(In reply to Sandro Bonazzola from comment #3)
> So the request is for adding --set-mainteinance=True/False and
> --vm-start-paused to hosted-engine tool?

I think that both of them could be helpful, adding the ability to start it with an ISO would be useful too, just to use it as a 'rescue' boot.

Thanks,
Pablo

Comment 5 Greg Padgett 2013-10-21 13:33:55 UTC

(In reply to Sandro Bonazzola from comment #3)
> So the request is for adding --set-mainteinance=True/False and

Probably global/local/none options, see agent api patch at http://gerrit.ovirt.org/#/c/20278/

Comment 6 Pablo Iranzo Gómez 2013-10-21 13:47:32 UTC

(In reply to Greg Padgett from comment #5)
> (In reply to Sandro Bonazzola from comment #3)
> > So the request is for adding --set-mainteinance=True/False and
> 
> Probably global/local/none options, see agent api patch at
> http://gerrit.ovirt.org/#/c/20278/

Looks good for me to use that flag, it can also be used by yum or third-party tools to avoid messing the system ;-)

Comment 7 Sandro Bonazzola 2013-10-23 14:23:35 UTC

(In reply to Pablo Iranzo Gómez from comment #4)
> Sandro,(In reply to Sandro Bonazzola from comment #3)
> > So the request is for adding --set-mainteinance=True/False and
> > --vm-start-paused to hosted-engine tool?
> 
> I think that both of them could be helpful, adding the ability to start it
> with an ISO would be useful too, just to use it as a 'rescue' boot.

once you start with --vm-start-paused you can use vdsClient hotplugDisk to attach a cd-rom image. I would like to avoid to replicate vdsClient commands in hosted-engine command if not really needed.

> 
> Thanks,
> Pablo

Comment 9 Sandro Bonazzola 2013-10-29 14:35:52 UTC

Patches merged on upstream master and 1.0 branches.

Comment 14 errata-xmlrpc 2014-01-21 16:54:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0083.html