Bug 1455119
| Summary: | [HE] when run reboot on the HE vm the vm stopped but not started. | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-hosted-engine-ha | Reporter: | Kobi Hakimi <khakimi> | ||||||||||||||||||
| Component: | General | Assignee: | bugs <bugs> | ||||||||||||||||||
| Status: | CLOSED NOTABUG | QA Contact: | Nikolai Sednev <nsednev> | ||||||||||||||||||
| Severity: | unspecified | Docs Contact: | |||||||||||||||||||
| Priority: | unspecified | ||||||||||||||||||||
| Version: | 2.1.0.5 | CC: | bugs, dfediuck, didi, khakimi, michal.skrivanek, msivak, nsednev, sbonazzo, stirabos | ||||||||||||||||||
| Target Milestone: | --- | Flags: | nsednev:
planning_ack?
nsednev: devel_ack? nsednev: testing_ack? |
||||||||||||||||||
| Target Release: | --- | ||||||||||||||||||||
| Hardware: | Unspecified | ||||||||||||||||||||
| OS: | Unspecified | ||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||||||
| Last Closed: | 2017-07-17 12:23:58 UTC | Type: | Bug | ||||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||||
| oVirt Team: | SLA | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||
| Embargoed: | |||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||
Hi Martin, May you please provide your input on this issue? AFAIK its by design and depends on ha-agent. In global maintenance the HA agent will not interfere with the HE VM lifecycle, so it won't be started externally. If you used reboot (as opposed to shut down), please provide the relevant logs (qemu, libvirt, vdsm) so we'll try to understand why the guest refused to restart. Created attachment 1294558 [details]
vdsm log
Created attachment 1294561 [details]
HostedEngine.log
Created attachment 1294562 [details]
libvirtd.log
Please let me know if you need more info or online reproduction Guest logs? We need to understand why the guest didn't reboot. Also, where in the attached logs did it happen? I cannot see the VM creation in VDSM logs. (In reply to Doron Fediuck from comment #7) > Guest logs? > We need to understand why the guest didn't reboot. > Also, where in the attached logs did it happen? I cannot see the VM creation > in VDSM logs. Attaching screencast of reproduction. Created attachment 1295615 [details]
screencast
Sosreport from alma03 which ran the engine before it was rebooted by me is here: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88UFJMUDU1SWEzeDg/view?usp=sharing Sosreport from second host alma04 available from here: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88Ync2QXphRDRnSG8/view?usp=sharing Both hosts remained in state=GlobalMaintenance, while engine was rebooted and did not returned to up. In case of disabling global maintenance, then after several minutes HE-VM being started by ha-agent on one of the ha-hosts, in my case it was started on second ha-host (alma04). Sosreport from the engine being attached. Created attachment 1295617 [details]
sosreport from the engine
Looking at the alma03 libvirt logs shows libvirt thinks the guest was shut down: 2017-07-09 15:13:26,202+0300 INFO (libvirt/events) [virt.vm] (vmId='1a23e4ae-e632-4654-8f53-24d0f92cdb5e') underlying process disconnected (vm:694) ... 2017-07-09 15:13:26,209+0300 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/1a23e4ae-e632-4654-8f53-24d0f92cdb5e.org.qemu.guest_agent.0 already removed (utils:120) ... 2017-07-09 15:13:26,212+0300 INFO (libvirt/events) [virt.vm] (vmId='1a23e4ae-e632-4654-8f53-24d0f92cdb5e') Changed state to Down: User shut down from within the guest (code=7) (vm:1221) Michal, any idea why? QEMU guest crashed: Jul 9 15:13:25 alma03 libvirtd: 2017-07-09 12:13:25.980+0000: 42987: error : qemuMonitorIO:697 : internal error: End of file from qemu monitor But apparently it was detected as a clean shutdown by vdsm. There is no error reported in qemu log and there are no further libvirt logs to be able to tell whether it is a libvirt or vdsm problem Need a reproduction with libvirt debug log level One thing worth noting is that the system was not rebooted after system packages installation. That might cause such issues (In reply to Michal Skrivanek from comment #14) > QEMU guest crashed: > Jul 9 15:13:25 alma03 libvirtd: 2017-07-09 12:13:25.980+0000: 42987: error > : qemuMonitorIO:697 : internal error: End of file from qemu monitor > But apparently it was detected as a clean shutdown by vdsm. There is no > error reported in qemu log and there are no further libvirt logs to be able > to tell whether it is a libvirt or vdsm problem > Need a reproduction with libvirt debug log level Attaching screencast of reproduction, libvirt logs with enabled debugging now, together with sosreport from host alma04, on which engine was running and sosreport from the engine. > > > One thing worth noting is that the system was not rebooted after system > packages installation. That might cause such issues I did not rebooted HE-VM after updating its components last time I've reproduced this bug. Now I did rebooted the engine before the reproduction and the result is the same. Created attachment 1296254 [details]
sosreport from alma04
Created attachment 1296255 [details]
libvirt log from alma04
Created attachment 1296256 [details]
sosreport from the engine
Screencast available from here: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88WmpOQ0ZyRlZRWWs/view?usp=sharing All hosts were rebooted prior to reproduction, engine was also powered-down and started before the reproduction. ah, finally....
HE setup creates a custom VM definition destroying the VM on reboots
Obligatory note: virt team cannot support any non-standard VMs, the HE code should be rewritten to use proper APIs instead of vdsm hooks.
commit efbfbf1cd9bebc526285da26f8f540b073fde997
Author: Sandro Bonazzola <sbonazzo>
Date: Thu Jul 4 09:39:12 2013 +0200
vdsm hostedengine hook
Add destroy on reboot, crash, poweroff to hosted engine vm.
Change-Id: I0313cd20b07f2c879462d1ce6c625590e8c00433
Signed-off-by: Sandro Bonazzola <sbonazzo>
Either way, the VM definition causes all in-guests reboots to result in a shutdown. Seems to me it works as designed since the HE shouldn't be managed in case of global maintenance.
Closing NOTABUG?
Could you please refer me to the doc which state that this behavior is not permitted - in other words "by design". I confirm that the VM destroy on crash, shutdown and reboot is by design. The hook destroying the Hosted Engine VM on those events exists since oVirt 3.3 when hosted engine has been introduced. Back on that time, the only supported way to get HE VM started was decided to be the hosted engine agent. Once the VM is down (due to reboot, crash or shutdown) the agent takes care to start it on the best available host. In global maintenance, hosted-engine --start-vm has been provided for doing maintenance reboots. Closing as not a bug being this by design. (In reply to Sandro Bonazzola from comment #24) > I confirm that the VM destroy on crash, shutdown and reboot is by design. > The hook destroying the Hosted Engine VM on those events exists since oVirt > 3.3 when hosted engine has been introduced. > > Back on that time, the only supported way to get HE VM started was decided > to be the hosted engine agent. Once the VM is down (due to reboot, crash or > shutdown) the agent takes care to start it on the best available host. > In global maintenance, hosted-engine --start-vm has been provided for doing > maintenance reboots. > > Closing as not a bug being this by design. Sandro, I'd like to reopen. I think it's an annoying bug and not sure we can't change the design. If the engine vm is rebooted with the 'reboot' command: 1. If in global maintenance, why not let it simply reboot, on same host? 2. If not in maintenance, why not let it reboot, and if it times out the logic of HA agent, it will handle this as a crash? I think one of the main reasons for 'reboot should shutdown' was for the initial deploy, where we had two different phases - 'install OS' (implicitly finishes by a reboot), 'install/setup engine' - and we wanted to guide the user through them as separate steps. If we did not know that the machine was rebooted, we couldn't separate them. This will be very beneficial option, due to the need of OS kernel upgrades, which require VM's reboot, thus making such functionality an essential for such use cases. As for now, customer needs to reboot the VM from ssh, then go to host's CLI and from there to cast "hosted-egnine --vm-start", this whole flow might be simplified by only casting "reboot" from SHE-VM. I've no specific objection about reopening and allow to reboot on the same host, Martin? Simone? The goal is noble, but it is a can of worms. As soon as the VM is down, other nodes will start competing for the privilege to start it. Also we need a way to distinguish a valid reboot from a crash (I think this is the primary reason we have it like it is now). We also have https://bugzilla.redhat.com/show_bug.cgi?id=1547390 that it will lead to this. (In reply to Simone Tiraboschi from comment #33) > We also have https://bugzilla.redhat.com/show_bug.cgi?id=1547390 that it > will lead to this. Sounds like a duplicate of current bug to me. Martin, if you object, you might comment there... Yeah, I think the bug Simone mentions is actually the action that will do what is requested here. I do not remember the reason for the hook being introduced in the first place. Sandro, do you remember why it was configured like it is today? What were we trying to solve? (In reply to Martin Sivák from comment #35) > Yeah, I think the bug Simone mentions is actually the action that will do > what is requested here. > > I do not remember the reason for the hook being introduced in the first > place. Sandro, do you remember why it was configured like it is today? What > were we trying to solve? I can't remember the failing flow and I can't find initial design docs, lost probably in website migration to github. I think it had something to do with the policy of starting the hosted engine on the host with highest score instead of rebooting on the same host. Hmm but we might be able to revisit that now as vdsm should be much better at detecting proper shutdown vs. reboot vs. crash. So if we can distinguish those then we can easily say that reboot is not a crash. On the other hand, long reboot causes the engine to become unavailable and the agent will notice it, so global maintenance is still needed when performing upgrades like this. |
Description of problem: [HE] when run reboot on the HE vm the vm stopped but not started. Version-Release number of selected component (if applicable): Red Hat Virtualization Manager Version: 4.1.2.2-0.1.el7 rhvm-appliance-4.1.20170221.0-1.el7ev.noarch How reproducible: 100% Steps to Reproduce: 1. In my case I run in the host: - hosted-engine --set-maintenance --mode=global 2. login into the HE vm and run reboot. Actual results: The HE vm stopped and not started. when I try to check in the console: hosted-engine --console The engine VM is on this host but its status is Down Expected results: Restart the HE vm - stop and start it. Additional info: