Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1401054

Summary: [RFE] Gracefully shutdown Virtual Machines on Host reboot/shutdown.
Product: Red Hat Enterprise Linux 7 Reporter: Michal Skrivanek <michal.skrivanek>
Component: libvirtAssignee: Martin Kletzander <mkletzan>
Status: CLOSED ERRATA QA Contact: Fangge Jin <fjin>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: dfediuck, dyuan, fjin, gordon.stocks, gveitmic, jdenemar, lsurette, mavital, michal.skrivanek, mtessun, nobody, rbalakri, Rhev-m-bugs, sbonazzo, srevivo, xuzhang, ykaul, ylavi
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-2.5.0-1.el7 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 1334982 Environment:
Last Closed: 2017-08-01 17:19:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1334982    

Description Michal Skrivanek 2016-12-02 16:13:00 UTC
We need to be able to override the service registered to be run before the guest is shut down by systemd
Just making it a configurable system-wide parameter would be enough

Thanks

+++ This bug was initially created as a clone of Bug #1334982 +++

[...]

In RHEV-M GUI, have options to manage shutdown (and maybe start) options for hosts.

-> Gracefully shutdown all VMs that are running, updating RHEV-M DB
-> Switch to maintenance mode, updating RHEV-M DB

[...]

Other information:

Well, I tried to achieve this by using API calls from the host to RHEV-M. The idea was:

A) systemd service that on shutdown would call the RHEV API to get a list of VMs it has running on it
B) Issue API calls to shutdown all these VMs
C) Put the Host into Maintenance mode
D) Continue shutdown process

Problems:

1) Scopes

My systemd service was always called AFTER the VMs were destroyed, even if it was 'After' vdsmd+libvirt and tons of targets.

This is due to libvirt. When it creates a VM, it tells systemd to assign a scope to it. To make the destruction of the scope dependent on another service, libvirt needs to specifically tell it to systemd. Current implementation is hardcoded in libvirt as 'libvirt-guests.sevice".

So in RHEV, these scopes do not depend on anything, so systemd shuts them down quite early in the shutdown process, killing all the VMs.

[  175.376302] systemd-logind[489]: Power key pressed.
[  175.376312] systemd-logind[489]: Powering Off...
...
[  175.388845] systemd[1]: systemd-machined.service changed running -> stop-sigterm
[  175.388854] systemd[1]: Stopping Virtual Machine and Container Registration Service...
[  175.389604] systemd[1]: machine-qemu\x2d2\x2dCentOS7.scope changed running -> stop-sigterm
[  175.389613] systemd[1]: Stopping Virtual Machine qemu-2-CentOS7.

Looking at libvirt code, it is hard-coded to make this scopes depend only on libvirt-guests.service.

int virSystemdCreateMachine(const char *name, ...
{
....
        if (virDBusCallMethod(conn,
                              NULL,
                              &error,
                              "org.freedesktop.machine1",
                              "/org/freedesktop/machine1",
                              "org.freedesktop.machine1.Manager",
                              "CreateMachineWithNetwork",
                              "sayssusa&ia(sv)",
                              name,
                              16,
                              uuid[0], uuid[1], uuid[2], uuid[3],
                              uuid[4], uuid[5], uuid[6], uuid[7],
                              uuid[8], uuid[9], uuid[10], uuid[11],
                              uuid[12], uuid[13], uuid[14], uuid[15],
                              creatorname,
                              iscontainer ? "container" : "vm",
                              (unsigned int)pidleader,
                              rootdir ? rootdir : "",
                              nnicindexes, nicindexes,
                              3,
                              "Slice", "s", slicename,
                              "After", "as", 1, "libvirtd.service",                  
                              "Before", "as", 1, "libvirt-guests.service") < 0)
...
}

So systemd won't care about my service depending on vdsm because the VMs are assigned to these scopes and not the vdsm service. Result: only the libvirt-guests.service is able to do something here.

Comment 1 Jiri Denemark 2016-12-02 17:26:55 UTC
I believe this is already implemented upstream by

commit 01079727fe29dfeafac751a07e564e0d6bb53389
Refs: v2.4.0-186-g01079727f
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Fri Oct 14 10:13:48 2016 +0300
Commit:     Maxim Nestratov <mnestratov>
CommitDate: Wed Nov 23 11:13:53 2016 +0300

    libvirtd: systemd: add special target for system shutdown

    It is already discussed in "[RFC] daemon: remove hardcode dep on libvirt-guests" [1].

    Mgmt can use means to save/restore domains on system shutdown/boot other than
    libvirt-guests.service. Thus we need to specify appropriate ordering dependency between
    libvirtd, domains and save/restore service. This patch takes approach suggested
    in RFC and introduces a systemd target, so that ordering can be built next way:

    libvirtd -> domain -> virt-guest-shutdown.target -> save-restore.service.

    This way domains are decoupled from specific shutdown service via intermediate
    target.

    [1] https://www.redhat.com/archives/libvir-list/2016-September/msg01353.html


In other words, you can just remove libvirt-guests from virt-guest-shutdown.target and put your custom service which will do it your way.

Comment 3 Fangge Jin 2017-03-15 09:18:40 UTC
Verify PASS libvirt-3.1.0-2.el7.x86_64

0.# systemctl cat libvirt-guests|grep virt-guest-shutdown
Requires=virt-guest-shutdown.target
After=virt-guest-shutdown.target

1.virt-guest-shutdown.target is inactive at first.
# systemctl status virt-guest-shutdown.target
● virt-guest-shutdown.target - Libvirt guests shutdown
   Loaded: loaded (/usr/lib/systemd/system/virt-guest-shutdown.target; static; vendor preset: disabled)
   Active: inactive (dead) since Wed 2017-03-15 05:15:14 EDT; 2s ago

2.# systemctl start libvirt-guests

3.virt-guest-shutdown.target is started along with libvirt-guests.
# systemctl status virt-guest-shutdown.target
● virt-guest-shutdown.target - Libvirt guests shutdown
   Loaded: loaded (/usr/lib/systemd/system/virt-guest-shutdown.target; static; vendor preset: disabled)
   Active: active since Wed 2017-03-15 05:16:02 EDT; 22s ago

4.# virsh start guest

5.# reboot

6. After host boots up:
# ll /var/lib/libvirt/qemu/save/
total 3616
-rw-------. 1 root root 3702094 Mar 15 05:11 guest.save
 
Guest is saved on host shutdown.

Comment 4 errata-xmlrpc 2017-08-01 17:19:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 5 errata-xmlrpc 2017-08-01 23:59:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846