Bug 1868537

Summary:	Support libvirt-guests.service working with libvirt modular daemon
Product:	Red Hat Enterprise Linux 9	Reporter:	Yanqiu Zhang <yanqzhan>
Component:	libvirt	Assignee:	Martin Kletzander <mkletzan>
libvirt sub component:	General	QA Contact:	Yanqiu Zhang <yanqzhan>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	berrange, dyuan, fjin, jdenemar, jsuchane, lizhu, lmen, meili, mkletzan, pvlasin, virt-maint, xuzhang, yafu, yalzhang, yanqzhan
Version:	9.0	Keywords:	Automation, Regression, Reopened, Triaged
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	libvirt-8.0.0-5.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-05-17 12:45:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:	8.1.0
Embargoed:
Bug Depends On:
Bug Blocks:	2027125

Description Yanqiu Zhang 2020-08-13 03:46:07 UTC

Description of problem:
libvirt-guests.service can not work with split daemon mode

Version-Release number of selected component (if applicable):
libvirt-daemon-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64
qemu-kvm-5.0.0-2.module+el8.3.0+7379+0505d6ca.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Enable split daemon mode:
#systemctl stop libvirtd.service
#systemctl stop libvirtd{,-ro,-admin,-tcp,-tls}.socket
#systemctl disable libvirtd.service
#systemctl disable libvirtd{,-ro,-admin,-tcp,-tls}.socket
#systemctl enable virtlogd; systemctl enable virtlogd.socket; systemctl start virtlogd.socket
#for drv in qemu interface network nodedev nwfilter secret storage proxy; do systemctl unmask virt${drv}d.service; systemctl unmask virt${drv}d{,-ro,-admin}.socket; systemctl enable virt${drv}d.service; systemctl enable virt${drv}d{,-ro,-admin}.socket; systemctl start virt${drv}d{,-ro,-admin}.socket ; done 

Libvirtd and libvirtd.socket are dead and disabled, split daemon sockets such as virtqemud.socket is active.
# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: enabled)
   Active: inactive (dead) (thawing) since Wed 2020-08-12 23:30:23 EDT; 13s ago
...
# systemctl status libvirtd.socket
● libvirtd.socket - Libvirt local socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
   Active: inactive (dead) (thawing) since Wed 2020-08-12 23:30:23 EDT; 17s ago
...
# systemctl status virtqemud
● virtqemud.service - Virtualization qemu daemon
   Loaded: loaded (/usr/lib/systemd/system/virtqemud.service; enabled; vendor preset: disabled)
   Active: inactive (dead) (thawing) since Wed 2020-08-12 06:48:49 EDT; 16h ago
...
# systemctl status virtqemud.socket
● virtqemud.socket - Libvirt qemu local socket
   Loaded: loaded (/usr/lib/systemd/system/virtqemud.socket; enabled; vendor preset: disabled)
   Active: active (listening) (thawing) since Wed 2020-08-12 23:30:24 EDT; 28s ago
...

# virsh list --all
 Id   Name             State
---------------------------------
 -    avocado-vt-vm1   shut off

2. Start libvirt-guests.service, libvirtd and libvirtd.socket are activated, virtqemud.socket is dead.
# systemctl start libvirt-guests
# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: enabled)
   Active: active (running) (thawing) since Wed 2020-08-12 23:31:58 EDT; 19s ago
...
# systemctl status libvirtd.socket
● libvirtd.socket - Libvirt local socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
   Active: active (running) (thawing) since Wed 2020-08-12 23:31:58 EDT; 22s ago
...
# systemctl status virtqemud
● virtqemud.service - Virtualization qemu daemon
   Loaded: loaded (/usr/lib/systemd/system/virtqemud.service; enabled; vendor preset: disabled)
   Active: inactive (dead) (thawing) since Wed 2020-08-12 23:31:58 EDT; 29s ago
...
# systemctl status virtqemud.socket
● virtqemud.socket - Libvirt qemu local socket
   Loaded: loaded (/usr/lib/systemd/system/virtqemud.socket; enabled; vendor preset: disabled)
   Active: inactive (dead) (thawing) since Wed 2020-08-12 23:31:58 EDT; 31s ago

# virsh list --all
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/virtqemud-sock': Connection refused

3. Check libvirt-guests.service content
# vim /usr/lib/systemd/system/libvirt-guests.service 
[Unit]
Description=Suspend/Resume Running libvirt Guests
***Wants=libvirtd.service***
Requires=virt-guest-shutdown.target
After=network.target
After=time-sync.target
***After=libvirtd.service***
After=virt-guest-shutdown.target
Documentation=man:libvirtd(8)
Documentation=https://libvirt.org

[Service]
EnvironmentFile=-/etc/sysconfig/libvirt-guests
# Hack just call traditional service until we factor
# out the code
ExecStart=/usr/libexec/libvirt-guests.sh start
ExecStop=/usr/libexec/libvirt-guests.sh stop
Type=oneshot
RemainAfterExit=yes
StandardOutput=journal+console
TimeoutStopSec=0

[Install]
WantedBy=multi-user.target


Actual results:
libvirt-guests.service only works with libvirtd.service, killing split daemon sockets.

Expected results:
libvirt-guests.service should also work well with split daemons(virtqemud, etc)

Additional info:

Comment 2 John Ferlan 2021-09-08 13:30:45 UTC

Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.

Comment 4 Yanqiu Zhang 2022-01-26 04:01:38 UTC

Hi,

1. Issue still reproduces on libvirt-8.0.0-2.el9.x86_64.

# systemctl status libvirtd
○ libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)
...
# systemctl status libvirtd.socket
○ libvirtd.socket - Libvirt local socket
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
     Active: inactive (dead)
   Triggers: ● libvirtd.service

# systemctl start libvirt-guests

# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: disabled)
     Active: active (running) since Tue 2022-01-25 22:47:26 EST; 7s ago
TriggeredBy: ○ libvirtd-tls.socket
             ● libvirtd-ro.socket
             ○ libvirtd-tcp.socket
             ● libvirtd-admin.socket
             ● libvirtd.socket
...
# systemctl status libvirtd.socket
● libvirtd.socket - Libvirt local socket
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
     Active: active (running) since Tue 2022-01-25 22:47:26 EST; 22s ago
   Triggers: ● libvirtd.service

# systemctl stop libvirt-guests <--Hangs


2. Mask libvirtd will make it work well:
# systemctl mask libvirtd
Created symlink /etc/systemd/system/libvirtd.service → /dev/null.
# systemctl mask libvirtd.socket
Created symlink /etc/systemd/system/libvirtd.socket → /dev/null.

# virsh start avocado-vt-vm1
Domain 'avocado-vt-vm1' started

# systemctl start libvirt-guests
# systemctl stop libvirt-guests

# virsh list --all --managed-save
 Id   Name             State
---------------------------------
 -    avocado-vt-vm1   saved

# systemctl start libvirt-guests

# virsh list --all --managed-save
 Id   Name             State
---------------------------------
 2    avocado-vt-vm1   running

# systemctl status libvirtd
○ libvirtd.service
     Loaded: masked (Reason: Unit libvirtd.service is masked.)
     Active: inactive (dead)
TriggeredBy: ○ libvirtd-admin.socket
             ○ libvirtd-tls.socket
             ○ libvirtd-tcp.socket
             ○ libvirtd-ro.socket
# systemctl status libvirtd.socket
○ libvirtd.socket
     Loaded: masked (Reason: Unit libvirtd.socket is masked.)
     Active: inactive (dead)

But a fresh installed libvirt does not auto-mask libvirtd(/.socket). And upgraded libvirt does not informed users must mask libvirtd, neither. So this is still an issue.

Mark as high since rhel9 is using split daemon mode by default now.

Comment 5 Martin Kletzander 2022-02-05 14:28:35 UTC

Dan, do we have some documentation what is the expected way of using the split daemon?  Are users expected to be able to choose to switch back to the monolithic one?  Should the upgrade path make the switch seamless as well?  I would like to solve this according some official docs so that I do not spend time coming up with something that gets disregarded afterwards.  Thanks.

Comment 6 Xuesong Zhang 2022-02-09 08:54:05 UTC

(In reply to Martin Kletzander from comment #5)
> Dan, do we have some documentation what is the expected way of using the
> split daemon?  Are users expected to be able to choose to switch back to the
> monolithic one?  Should the upgrade path make the switch seamless as well? 
> I would like to solve this according some official docs so that I do not
> spend time coming up with something that gets disregarded afterwards. 
> Thanks.

Hi, Martin,

Here is one upstream doc: https://libvirt.org/daemons.html
Not sure if it is enough to answer your questions here.

Comment 7 RHEL Program Management 2022-02-13 07:27:11 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 8 Xuesong Zhang 2022-02-14 08:30:08 UTC

Change the status back to Assign, since still need to confirm with devel and PM whether need to fix this issue.

Comment 10 Martin Kletzander 2022-02-15 11:12:34 UTC

Thanks for reopening this, it clearly needs to be dealt with.  I was looking for a different type of documentation, but I expressed myself incorrectly, sorry for that.

Anyway, the problem is a bit tougher unless we assume that the system is set up in a sensible manner, so I leveraged that in order to propose the following fix:

https://listman.redhat.com/archives/libvir-list/2022-February/msg00566.html

Comment 11 Martin Kletzander 2022-02-22 13:27:58 UTC

Fixed upstream with v8.0.0-461-g4e42686adef8:

commit 4e42686adef8b9e9266f0099ddcd25bc95c4ed43
Author: Martin Kletzander <mkletzan>
Date:   Mon Feb 14 12:37:37 2022 +0100

    Make systemd unit ordering more robust

Comment 14 Yanqiu Zhang 2022-02-23 08:37:09 UTC

Hi Martin,

I encounter an issue when pre-verify on v8.0.0-468-ga6929d62cf, could you help check? Thank you!

# virsh start avocado-vt-vm1 
error: Failed to start domain 'avocado-vt-vm1'
error: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Invalid unit name virt%sd.service

Comment 15 Martin Kletzander 2022-02-23 12:09:50 UTC

(In reply to yanqzhan from comment #14)
Thanks, I am on it right now.

Comment 17 Yanqiu Zhang 2022-02-24 03:40:29 UTC

Hi,

I find another issue caused by removing 'wants':
Start libvirt-guests while virtqemud is stopped, then stop libvirt-guests.

○ virtqemud.service - Virtualization qemu daemon
     Loaded: loaded (/usr/lib/systemd/system/virtqemud.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2022-02-23 22:15:09 EST; 9s ago

● virtqemud.socket - Libvirt qemu local socket
     Loaded: loaded (/usr/lib/systemd/system/virtqemud.socket; enabled; vendor preset: disabled)
     Active: active (listening) since Wed 2022-02-23 21:38:32 EST; 36min ago

# systemctl start libvirt-guests

#  systemctl status virtqemud
○ virtqemud.service - Virtualization qemu daemon
     Loaded: loaded (/usr/lib/systemd/system/virtqemud.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2022-02-23 22:15:09 EST; 41s ago

● libvirt-guests.service - Suspend/Resume Running libvirt Guests
     Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; disabled; vendor preset: disabled)
     Active: active (exited) since Wed 2022-02-23 22:15:40 EST; 2min 50s ago

# systemctl stop libvirt-guests  <--Hangs here.


Can this be fixed here?  Thanks.

(It's like similar results as bz1964855 by a different trigger way, due to virtqemud not started by libvirt-guests)

Comment 18 Martin Kletzander 2022-02-24 08:43:06 UTC

(In reply to yanqzhan from comment #17)
Yes, it does look like the BZ you mentioned.  If systemctl status virtqemud shows that the socket triggers are active, then it should start the service upon connection to the socket.  Note that changing to After= without Wants= was done because we have to assume that the system is configured to start the necessary daemons that libvirt-guests is supposed to handle.

I can easily reproduce the issue you are describing even without libvirt-guests, although it might be some local configuration issue as I did not check that on a clean install, but for me in this state even "systemctl start virtqemud.service" hangs.

Comment 19 Martin Kletzander 2022-02-24 08:51:25 UTC

I figured that one out and posted a possible root cause in the other bug.  Just kill any "virsh connect" that you see and it should be fine.  It got stuck due to reproducing this BZ IMHO.

Comment 20 Yanqiu Zhang 2022-02-24 09:48:01 UTC

(In reply to Martin Kletzander from comment #18)
> (In reply to yanqzhan from comment #17)
> Yes, it does look like the BZ you mentioned.  If systemctl status virtqemud
> shows that the socket triggers are active, then it should start the service
> upon connection to the socket.  Note that changing to After= without Wants=
> was done because we have to assume that the system is configured to start
> the necessary daemons that libvirt-guests is supposed to handle.

Ok. Let's track this issue in bz1964855. 
I just have the concern that libvirt-guests's function doesn't work when guest forget to manually start virtqemud/libvirtd when 120s timeout. 

> 
> I can easily reproduce the issue you are describing even without
> libvirt-guests, although it might be some local configuration issue as I did
> not check that on a clean install, but for me in this state even "systemctl
> start virtqemud.service" hangs.

Of course 'start virtqemud' hangs, even 'virsh list' will hang when that 'virsh connect' process exists.
But if you never executed 'stop libvirt-guests' before, it doesn't hang. So it's better for libvirt-guests to work well by itself I think.
(1)Start virtqemud when libvirt-guests isn't started:
# systemctl status libvirt-guests
○ libvirt-guests.service - Suspend/Resume Running libvirt Guests
...  Active: inactive (dead) since Thu 2022-02-24 04:31:31 EST; 31s ago

● virtqemud.socket - Libvirt qemu local socket
...  Active: active (listening) since Thu 2022-02-24 04:20:19 EST; 11min ago

○ virtqemud.service - Virtualization qemu daemon
...  Active: inactive (dead) since Thu 2022-02-24 04:31:40 EST; 38s ago

# systemctl start virtqemud

● virtqemud.service - Virtualization qemu daemon
     Loaded: loaded (/usr/lib/systemd/system/virtqemud.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-02-24 04:32:31 EST; 2s ago

(2)Start virtqemud when libvirt-guests is active:
# systemctl status  virtqemud
...     Active: inactive (dead) since Thu 2022-02-24 04:22:19 EST; 2min 13s ago

# systemctl start libvirt-guests
# systemctl status libvirt-guests
● libvirt-guests.service - Suspend/Resume Running libvirt Guests
     Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; disabled; vendor preset: disabled)
     Active: active (exited) since Thu 2022-02-24 04:24:18 EST; 7s ago

# systemctl start virtqemud
# systemctl status virtqemud
● virtqemud.service - Virtualization qemu daemon
     Loaded: loaded (/usr/lib/systemd/system/virtqemud.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-02-24 04:25:03 EST; 7s ago

Comment 21 Yanqiu Zhang 2022-02-24 09:53:59 UTC

Pre-verify on v8.1.0-rc1-3-g32b9d8b0ae:

1.Guest start successfully
# virsh start avocado-vt-vm1 
Domain 'avocado-vt-vm1' started

2.Service start doesn’t break split daemon mode
○ libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

○ libvirtd.socket - Libvirt local socket
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
     Active: inactive (dead)

# systemctl start libvirt-guests
# echo $?
0
# systemctl status libvirt-guests
● libvirt-guests.service - Suspend/Resume Running libvirt Guests
     Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; disabled; vendor preset: disabled)
     Active: active (exited) since Wed 2022-02-23 21:39:25 EST; 18s ago
       Docs: man:libvirt-guests(8)
             https://libvirt.org
    Process: 148286 ExecStart=/usr/libexec/libvirt-guests.sh start (code=exited, status=0/SUCCESS)
   Main PID: 148286 (code=exited, status=0/SUCCESS)
        CPU: 13ms

Feb 23 21:39:25 hosta systemd[1]: Starting Suspend/Resume Running libvirt Guests...
Feb 23 21:39:25 hosta systemd[1]: Finished Suspend/Resume Running libvirt Guests.

○ libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

○ libvirtd.socket - Libvirt local socket
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
     Active: inactive (dead)

3.Save and restore guest successfully:
Login guest: 
# virsh console avocado-vt-vm1 
…
localhost login: root
Password: 
[root@localhost ~]# 

# systemctl stop libvirt-guests
# echo $?
0
# systemctl status libvirt-guests
○ libvirt-guests.service - Suspend/Resume Running libvirt Guests
     Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; disabled; vendor preset: disabled)
     Active: inactive (dead) since Wed 2022-02-23 21:47:34 EST; 16s ago
       Docs: man:libvirt-guests(8)
             https://libvirt.org
    Process: 148286 ExecStart=/usr/libexec/libvirt-guests.sh start (code=exited, status=0/SUCCESS)
    Process: 148558 ExecStop=/usr/libexec/libvirt-guests.sh stop (code=exited, status=0/SUCCESS)
   Main PID: 148286 (code=exited, status=0/SUCCESS)
        CPU: 274ms

Feb 23 21:39:25 hosta systemd[1]: Starting Suspend/Resume Running libvirt Guests...
Feb 23 21:39:25 hosta systemd[1]: Finished Suspend/Resume Running libvirt Guests.
Feb 23 21:47:30 hosta systemd[1]: Stopping Suspend/Resume Running libvirt Guests...
Feb 23 21:47:30 hosta libvirt-guests.sh[148566]: Running guests on default URI:
Feb 23 21:47:30 hosta libvirt-guests.sh[148558]: avocado-vt-vm1
Feb 23 21:47:30 hosta libvirt-guests.sh[148594]: Suspending guests on default URI...
Feb 23 21:47:30 hosta libvirt-guests.sh[148558]: Suspending avocado-vt-vm1: ...
Feb 23 21:47:34 hosta libvirt-guests.sh[148558]: Suspending avocado-vt-vm1: done
Feb 23 21:47:34 hosta systemd[1]: libvirt-guests.service: Deactivated successfully.
Feb 23 21:47:34 hosta systemd[1]: Stopped Suspend/Resume Running libvirt Guests.

# virsh list --all --managed-save
 Id   Name             State
------------------------------
 -    avocado-vt-vm1   saved

#  systemctl start libvirt-guests

#  virsh list --all --managed-save
 Id   Name             State
--------------------------------
 2    avocado-vt-vm1   running

○ libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

○ libvirtd.socket - Libvirt local socket
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
     Active: inactive (dead)

Visit guest os, it restored to last state which has been login:
# virsh console avocado-vt-vm1 
Connected to domain 'avocado-vt-vm1'
Escape character is ^] (Ctrl + ])

[root@localhost ~]# date

4.Restart virtqemud doesn’t change libvirt-guests pid or domain id:
# systemctl status libvirt-guests|grep PID
   Main PID: 148754 (code=exited, status=0/SUCCESS)

# systemctl restart virtqemud

# systemctl status libvirt-guests|grep PID
   Main PID: 148754 (code=exited, status=0/SUCCESS)

# virsh list --all
 Id   Name             State
--------------------------------
 2    avocado-vt-vm1   running

Comment 29 Yanqiu Zhang 2022-02-28 10:22:38 UTC

Verify on:
libvirt-8.0.0-5.el9.x86_64
qemu-kvm-6.2.0-10.el9.x86_64

1.Steps and results are same with comment 21.
2.Auto regression test passed: 
https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-9.0%20x86_64/job/libvirt-RHEL-9.0-runtest-x86_64-function-managed_save/74/testReport/
The failed one local run passed: 
 (1/1) type_specific.io-github-autotest-libvirt.virsh.managedsave.functional_test.libvirt_guests.multi_guests: PASS (113.69 s)

Comment 31 errata-xmlrpc 2022-05-17 12:45:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libvirt), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2390