Bug 1401054

Summary: [RFE] Gracefully shutdown Virtual Machines on Host reboot/shutdown.
Product: Red Hat Enterprise Linux 7 Reporter: Michal Skrivanek <michal.skrivanek>
Component: libvirtAssignee: Martin Kletzander <mkletzan>
Status: CLOSED ERRATA QA Contact: Fangge Jin <fjin>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: dfediuck, dyuan, fjin, gordon.stocks, gveitmic, jdenemar, lsurette, mavital, michal.skrivanek, mtessun, nobody, rbalakri, Rhev-m-bugs, sbonazzo, srevivo, xuzhang, ykaul, ylavi
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-2.5.0-1.el7 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 1334982 Environment:
Last Closed: 2017-08-01 17:19:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1334982    

Description Michal Skrivanek 2016-12-02 16:13:00 UTC
We need to be able to override the service registered to be run before the guest is shut down by systemd
Just making it a configurable system-wide parameter would be enough

Thanks

+++ This bug was initially created as a clone of Bug #1334982 +++

[...]

In RHEV-M GUI, have options to manage shutdown (and maybe start) options for hosts.

-> Gracefully shutdown all VMs that are running, updating RHEV-M DB
-> Switch to maintenance mode, updating RHEV-M DB

[...]

Other information:

Well, I tried to achieve this by using API calls from the host to RHEV-M. The idea was:

A) systemd service that on shutdown would call the RHEV API to get a list of VMs it has running on it
B) Issue API calls to shutdown all these VMs
C) Put the Host into Maintenance mode
D) Continue shutdown process

Problems:

1) Scopes

My systemd service was always called AFTER the VMs were destroyed, even if it was 'After' vdsmd+libvirt and tons of targets.

This is due to libvirt. When it creates a VM, it tells systemd to assign a scope to it. To make the destruction of the scope dependent on another service, libvirt needs to specifically tell it to systemd. Current implementation is hardcoded in libvirt as 'libvirt-guests.sevice".

So in RHEV, these scopes do not depend on anything, so systemd shuts them down quite early in the shutdown process, killing all the VMs.

[  175.376302] systemd-logind[489]: Power key pressed.
[  175.376312] systemd-logind[489]: Powering Off...
...
[  175.388845] systemd[1]: systemd-machined.service changed running -> stop-sigterm
[  175.388854] systemd[1]: Stopping Virtual Machine and Container Registration Service...
[  175.389604] systemd[1]: machine-qemu\x2d2\x2dCentOS7.scope changed running -> stop-sigterm
[  175.389613] systemd[1]: Stopping Virtual Machine qemu-2-CentOS7.

Looking at libvirt code, it is hard-coded to make this scopes depend only on libvirt-guests.service.

int virSystemdCreateMachine(const char *name, ...
{
....
        if (virDBusCallMethod(conn,
                              NULL,
                              &error,
                              "org.freedesktop.machine1",
                              "/org/freedesktop/machine1",
                              "org.freedesktop.machine1.Manager",
                              "CreateMachineWithNetwork",
                              "sayssusa&ia(sv)",
                              name,
                              16,
                              uuid[0], uuid[1], uuid[2], uuid[3],
                              uuid[4], uuid[5], uuid[6], uuid[7],
                              uuid[8], uuid[9], uuid[10], uuid[11],
                              uuid[12], uuid[13], uuid[14], uuid[15],
                              creatorname,
                              iscontainer ? "container" : "vm",
                              (unsigned int)pidleader,
                              rootdir ? rootdir : "",
                              nnicindexes, nicindexes,
                              3,
                              "Slice", "s", slicename,
                              "After", "as", 1, "libvirtd.service",                  
                              "Before", "as", 1, "libvirt-guests.service") < 0)
...
}

So systemd won't care about my service depending on vdsm because the VMs are assigned to these scopes and not the vdsm service. Result: only the libvirt-guests.service is able to do something here.

Comment 1 Jiri Denemark 2016-12-02 17:26:55 UTC
I believe this is already implemented upstream by

commit 01079727fe29dfeafac751a07e564e0d6bb53389
Refs: v2.4.0-186-g01079727f
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Fri Oct 14 10:13:48 2016 +0300
Commit:     Maxim Nestratov <mnestratov>
CommitDate: Wed Nov 23 11:13:53 2016 +0300

    libvirtd: systemd: add special target for system shutdown

    It is already discussed in "[RFC] daemon: remove hardcode dep on libvirt-guests" [1].

    Mgmt can use means to save/restore domains on system shutdown/boot other than
    libvirt-guests.service. Thus we need to specify appropriate ordering dependency between
    libvirtd, domains and save/restore service. This patch takes approach suggested
    in RFC and introduces a systemd target, so that ordering can be built next way:

    libvirtd -> domain -> virt-guest-shutdown.target -> save-restore.service.

    This way domains are decoupled from specific shutdown service via intermediate
    target.

    [1] https://www.redhat.com/archives/libvir-list/2016-September/msg01353.html


In other words, you can just remove libvirt-guests from virt-guest-shutdown.target and put your custom service which will do it your way.

Comment 3 Fangge Jin 2017-03-15 09:18:40 UTC
Verify PASS libvirt-3.1.0-2.el7.x86_64

0.# systemctl cat libvirt-guests|grep virt-guest-shutdown
Requires=virt-guest-shutdown.target
After=virt-guest-shutdown.target

1.virt-guest-shutdown.target is inactive at first.
# systemctl status virt-guest-shutdown.target
● virt-guest-shutdown.target - Libvirt guests shutdown
   Loaded: loaded (/usr/lib/systemd/system/virt-guest-shutdown.target; static; vendor preset: disabled)
   Active: inactive (dead) since Wed 2017-03-15 05:15:14 EDT; 2s ago

2.# systemctl start libvirt-guests

3.virt-guest-shutdown.target is started along with libvirt-guests.
# systemctl status virt-guest-shutdown.target
● virt-guest-shutdown.target - Libvirt guests shutdown
   Loaded: loaded (/usr/lib/systemd/system/virt-guest-shutdown.target; static; vendor preset: disabled)
   Active: active since Wed 2017-03-15 05:16:02 EDT; 22s ago

4.# virsh start guest

5.# reboot

6. After host boots up:
# ll /var/lib/libvirt/qemu/save/
total 3616
-rw-------. 1 root root 3702094 Mar 15 05:11 guest.save
 
Guest is saved on host shutdown.

Comment 4 errata-xmlrpc 2017-08-01 17:19:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 5 errata-xmlrpc 2017-08-01 23:59:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846