RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1401054 - [RFE] Gracefully shutdown Virtual Machines on Host reboot/shutdown.
Summary: [RFE] Gracefully shutdown Virtual Machines on Host reboot/shutdown.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.2
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Martin Kletzander
QA Contact: Fangge Jin
URL:
Whiteboard:
Depends On:
Blocks: 1334982
TreeView+ depends on / blocked
 
Reported: 2016-12-02 16:13 UTC by Michal Skrivanek
Modified: 2021-09-09 12:01 UTC (History)
18 users (show)

Fixed In Version: libvirt-2.5.0-1.el7
Doc Type: Enhancement
Doc Text:
Clone Of: 1334982
Environment:
Last Closed: 2017-08-01 17:19:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description Michal Skrivanek 2016-12-02 16:13:00 UTC
We need to be able to override the service registered to be run before the guest is shut down by systemd
Just making it a configurable system-wide parameter would be enough

Thanks

+++ This bug was initially created as a clone of Bug #1334982 +++

[...]

In RHEV-M GUI, have options to manage shutdown (and maybe start) options for hosts.

-> Gracefully shutdown all VMs that are running, updating RHEV-M DB
-> Switch to maintenance mode, updating RHEV-M DB

[...]

Other information:

Well, I tried to achieve this by using API calls from the host to RHEV-M. The idea was:

A) systemd service that on shutdown would call the RHEV API to get a list of VMs it has running on it
B) Issue API calls to shutdown all these VMs
C) Put the Host into Maintenance mode
D) Continue shutdown process

Problems:

1) Scopes

My systemd service was always called AFTER the VMs were destroyed, even if it was 'After' vdsmd+libvirt and tons of targets.

This is due to libvirt. When it creates a VM, it tells systemd to assign a scope to it. To make the destruction of the scope dependent on another service, libvirt needs to specifically tell it to systemd. Current implementation is hardcoded in libvirt as 'libvirt-guests.sevice".

So in RHEV, these scopes do not depend on anything, so systemd shuts them down quite early in the shutdown process, killing all the VMs.

[  175.376302] systemd-logind[489]: Power key pressed.
[  175.376312] systemd-logind[489]: Powering Off...
...
[  175.388845] systemd[1]: systemd-machined.service changed running -> stop-sigterm
[  175.388854] systemd[1]: Stopping Virtual Machine and Container Registration Service...
[  175.389604] systemd[1]: machine-qemu\x2d2\x2dCentOS7.scope changed running -> stop-sigterm
[  175.389613] systemd[1]: Stopping Virtual Machine qemu-2-CentOS7.

Looking at libvirt code, it is hard-coded to make this scopes depend only on libvirt-guests.service.

int virSystemdCreateMachine(const char *name, ...
{
....
        if (virDBusCallMethod(conn,
                              NULL,
                              &error,
                              "org.freedesktop.machine1",
                              "/org/freedesktop/machine1",
                              "org.freedesktop.machine1.Manager",
                              "CreateMachineWithNetwork",
                              "sayssusa&ia(sv)",
                              name,
                              16,
                              uuid[0], uuid[1], uuid[2], uuid[3],
                              uuid[4], uuid[5], uuid[6], uuid[7],
                              uuid[8], uuid[9], uuid[10], uuid[11],
                              uuid[12], uuid[13], uuid[14], uuid[15],
                              creatorname,
                              iscontainer ? "container" : "vm",
                              (unsigned int)pidleader,
                              rootdir ? rootdir : "",
                              nnicindexes, nicindexes,
                              3,
                              "Slice", "s", slicename,
                              "After", "as", 1, "libvirtd.service",                  
                              "Before", "as", 1, "libvirt-guests.service") < 0)
...
}

So systemd won't care about my service depending on vdsm because the VMs are assigned to these scopes and not the vdsm service. Result: only the libvirt-guests.service is able to do something here.

Comment 1 Jiri Denemark 2016-12-02 17:26:55 UTC
I believe this is already implemented upstream by

commit 01079727fe29dfeafac751a07e564e0d6bb53389
Refs: v2.4.0-186-g01079727f
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Fri Oct 14 10:13:48 2016 +0300
Commit:     Maxim Nestratov <mnestratov>
CommitDate: Wed Nov 23 11:13:53 2016 +0300

    libvirtd: systemd: add special target for system shutdown

    It is already discussed in "[RFC] daemon: remove hardcode dep on libvirt-guests" [1].

    Mgmt can use means to save/restore domains on system shutdown/boot other than
    libvirt-guests.service. Thus we need to specify appropriate ordering dependency between
    libvirtd, domains and save/restore service. This patch takes approach suggested
    in RFC and introduces a systemd target, so that ordering can be built next way:

    libvirtd -> domain -> virt-guest-shutdown.target -> save-restore.service.

    This way domains are decoupled from specific shutdown service via intermediate
    target.

    [1] https://www.redhat.com/archives/libvir-list/2016-September/msg01353.html


In other words, you can just remove libvirt-guests from virt-guest-shutdown.target and put your custom service which will do it your way.

Comment 3 Fangge Jin 2017-03-15 09:18:40 UTC
Verify PASS libvirt-3.1.0-2.el7.x86_64

0.# systemctl cat libvirt-guests|grep virt-guest-shutdown
Requires=virt-guest-shutdown.target
After=virt-guest-shutdown.target

1.virt-guest-shutdown.target is inactive at first.
# systemctl status virt-guest-shutdown.target
● virt-guest-shutdown.target - Libvirt guests shutdown
   Loaded: loaded (/usr/lib/systemd/system/virt-guest-shutdown.target; static; vendor preset: disabled)
   Active: inactive (dead) since Wed 2017-03-15 05:15:14 EDT; 2s ago

2.# systemctl start libvirt-guests

3.virt-guest-shutdown.target is started along with libvirt-guests.
# systemctl status virt-guest-shutdown.target
● virt-guest-shutdown.target - Libvirt guests shutdown
   Loaded: loaded (/usr/lib/systemd/system/virt-guest-shutdown.target; static; vendor preset: disabled)
   Active: active since Wed 2017-03-15 05:16:02 EDT; 22s ago

4.# virsh start guest

5.# reboot

6. After host boots up:
# ll /var/lib/libvirt/qemu/save/
total 3616
-rw-------. 1 root root 3702094 Mar 15 05:11 guest.save
 
Guest is saved on host shutdown.

Comment 4 errata-xmlrpc 2017-08-01 17:19:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 5 errata-xmlrpc 2017-08-01 23:59:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846


Note You need to log in before you can comment on or make changes to this bug.