Bug 1755303 - 'autostart' domain goes to running state when any client app connects under system libvirtd daemon
Summary: 'autostart' domain goes to running state when any client app connects under s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.1
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: 8.1
Assignee: Michal Privoznik
QA Contact: yafu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-25 08:38 UTC by Yanqiu Zhang
Modified: 2020-11-14 04:58 UTC (History)
9 users (show)

Fixed In Version: libvirt-5.6.0-7.el8
Doc Type: Bug Fix
Doc Text:
Cause: Libvirt offers so called 'autostart' feature which can be set on some objects that libvirt manages (e.g. domains, networks, storage pools). If an object is marked as 'autostart' then the idea is that such object is started automatically on host boot. Historically this was done by libvirtd - when it was starting up, it also started all objects that were marked as autostart. But with recent work on daemon split and socket activation this old approach was not good enough. Thing is, libvirtd can be now socket activated and after some period of inactivity it shuts down again (this is achieved by adding --timeout onto the libvirtd's command line). Therefore, libvirtd may start up and shut down multiple times which would also start all autostarted objects again. Consequence: For instance, let's assume I have a domain A, makred as autostart, the host boots up and the domain is started automatically. But then I shut the domain down (virsh shutdown A). Now, after some time of inactivity, I call a libvirt API (say virsh list) which starts the daemon again, which also starts the domain A again. Fix: On first autostart, libvirtd creates a dummy file (under a location that doesn't survive reboots), and if the file already exists autostart is suppressed. This means, that no matter how many times libvirtd starts and shuts down, the autostart is done exactly once. Result: Autostart feature works as designed again.
Clone Of:
Environment:
Last Closed: 2020-02-04 18:28:50 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:0404 0 None None None 2020-02-04 18:29:46 UTC

Description Yanqiu Zhang 2019-09-25 08:38:29 UTC
Description of problem:
'autostart' domain goes to running state when any client app connects under system libvirtd daemon

Version-Release number of selected component (if applicable):
libvirt-daemon-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64
qemu-kvm-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64

How reproducible:
100%

Steps to Reproduce:
Check /etc/sysconfig/libvirtd:
  # Default behaviour is for libvirtd.service to start on boot
  # so that VM autostart can be performed. We then want it to
  # shutdown again if nothing was started and rely on systemd
  # socket activation to start it again when some client app
  # connects.
  LIBVIRTD_ARGS="--timeout 120"

1.# virsh autostart avocado-vt-vm1 
Domain avocado-vt-vm1 marked as autostarted

# virsh list --all --autostart
 Id   Name             State
---------------------------------
 -    avocado-vt-vm1   shut off

2.# systemctl stop libvirtd (or wait for 120s until libvirtd timeout)
Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd.socket
  libvirtd-tls.socket
  libvirtd-tcp.socket

# ps aux|grep qemu-kvm
root     29846  0.0  0.0  12108   988 pts/4    S+   04:22   0:00 grep --color=auto qemu-kvm

3.# virsh list 
 Id   Name             State
--------------------------------
 1    avocado-vt-vm1   running


Actual results:
As in step3, the domain is auto started. 
Now any virsh client connecting will trigger all 'autostart' domains to be running since libvirtd is auto started in systemd socket activation mode. But those client operations may not want to use the domains.

Expected results:
There could be some fix here. The resolution for session daemon (Bug 1501450: documenting this behaviour as known issue) may not meet demand for system libvirtd mode.

Additional info:

Comment 1 Daniel Berrangé 2019-09-25 09:18:27 UTC
Yikes, that is an annoying an unintended side effect of making libvirtd automatically timeout & shutdown after 120 seconds

Comment 2 Michal Privoznik 2019-09-25 11:44:27 UTC
After thinking about this I think I might have an idea how to fix this. The autostart feature is there basically to start domains on host startup or when the daemon starts for the first time. What we could do is to have a per driver file stored under /var/run (or some other location that is cleaned between host reboots) and on the first run of the per-driver autostart callback the file is created and the managed objects are autostarted. Then, if a deamon starts for the second time, the file is already there and no autostart is attempted.

Comment 3 Daniel Berrangé 2019-09-25 12:16:27 UTC
That's a much nicer idea than what I was going to suggest which was to simply ignore the --timeout arg if any guests marked auto-started exist.  It will nicely fix the same problem we've always had with a user who manually stops & starts libvirtd, and also provide an approach that can fix the autostart problem with the session daemon too.

Comment 4 Michal Privoznik 2019-10-07 09:12:14 UTC
Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2019-October/msg00214.html

Comment 5 Michal Privoznik 2019-10-08 15:10:42 UTC
And I've just pushed patches upstream:

897d8b34c8 Revert "src: Document autostart for session demon"
19b1b14f17 news: Document autostart fix
bab464f8ea lib: autostart objects exactly once
ee16a195d9 driver: Introduce virDriverShouldAutostart()
e0b90162c9 qemu_driver: Fix comment of qemuStateCleanup()

v5.8.0-45-g897d8b34c8

Comment 8 yafu 2019-11-25 09:46:32 UTC
Verified with libvirt-5.6.0-8.el8.x86_64.

Test steps:
1.Configure a domain to be automatically started at boot:
#virsh autostart vm1

2.Reboot the host:
#reboot

3.Check the guest status after booting:
#virsh list
 Id   Name      State
--------------------------------
 1    vm1      running

4.Check the autostart dir for qemu driver:
#ll /var/run/libvirt/qemu/autostarted 
-rw-------. 1 root root 0 Nov 25 04:23 /var/run/libvirt/qemu/autostarted

5.Destroy the guest:
#virsh destroy vm1

6.Restart libvirtd service
#systemctl restart libvirtd

7.Check the guest status:
#virsh list
no running guest

8.Remove the autostart dir for qemu driver:
#rm -rf /var/run/libvirt/qemu/autostarted

9.Restart libvirtd service
#systemctl restart libvirtd

10.Check the guest status:
#virsh list
 Id   Name      State
--------------------------------
 1    vm1      running

Comment 9 yafu 2019-12-10 02:35:33 UTC
Verified with libvirt-5.6.0-10.el8.x86_64.

Comment 11 errata-xmlrpc 2020-02-04 18:28:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0404


Note You need to log in before you can comment on or make changes to this bug.