Bug 1741403

Created attachment 1603976 [details]
libvirtd_log_configfile

Only one line that may be useful in libvirtd.log:
2019-08-15 04:17:07.346+0000: 3023: debug : virSystemdActivationClaimFDs:991 : No FD with name libvirtd-tcp.socket

Not reproduces on rhel-av8.0.1:
libvirt-daemon-5.0.0-11.module+el8.0.1+3459+e357ef2f.x86_64
qemu-kvm-3.1.0-27.module+el8.0.1+3253+c5371cb3.x86_64

libvirtd is now using systemd socket activation by default. When systemd activation is used, the --listen parameter no longer has any effect. The demo  /etc/sysconfig/libvirtd file provided by the libvirt RPM contains a warning to this effect.

Instead you need to activate the appropriate socket unit file eg

$  systemctl start libvirtd-tls.socket

or

$ systemctl start libvirtd-tcp.socket

NB, you must start these sockets, *before* libvirtd itself is started.

For the same reason the listen_tcp and listen_tls params in /etc/libvirt/libvirtd.conf also have no effect when systemd activation is used. Again the default config file contains warnings about this.

(In reply to Daniel Berrangé from comment #4)
> libvirtd is now using systemd socket activation by default. When systemd
> activation is used, the --listen parameter no longer has any effect. The
> demo  /etc/sysconfig/libvirtd file provided by the libvirt RPM contains a
> warning to this effect.
> 
> Instead you need to activate the appropriate socket unit file eg
> 
> $  systemctl start libvirtd-tls.socket
> 
> or
> 
> $ systemctl start libvirtd-tcp.socket
> 
> NB, you must start these sockets, *before* libvirtd itself is started.
> 
> For the same reason the listen_tcp and listen_tls params in
> /etc/libvirt/libvirtd.conf also have no effect when systemd activation is
> used. Again the default config file contains warnings about this.

This change will break the upper layer application connection unless they make
change accordingly.

Yes, any applications previously using --listen will have to adapt. Either they need to use the unit files as described, or they need to disable systemd socket activation to continue the old way.

SOme improvements upstream

 - Make it possible to disable socket activation: https://www.redhat.com/archives/libvir-list/2019-August/msg01068.html
 - Turn the use of --listen in to a fatal error so users see the back compat problem immediately instead of it being silently ignored: https://www.redhat.com/archives/libvir-list/2019-August/msg01070.html
 - For in place upgrades, if the user has used --listen, we disable socket activation to avoid breaking their existing host config: https://www.redhat.com/archives/libvir-list/2019-August/msg01071.html

Verify this bug with:
libvirt-daemon-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64
qemu-kvm-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64
systemd-239-18.el8.x86_64

Scenario1: Under systemd socket activation
1. libvirtd-tls/tcp.socket work for connection
Setup CA and issue server can client certificates, and config in files:
]# grep -v '#' /etc/libvirt/libvirtd.conf
auth_tls = "none"
auth_tcp = "none"

]# systemctl stop libvirtd
Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd.socket
  libvirtd-tls.socket
  libvirtd-tcp.socket

]# systemctl status libvirtd-tls.socket
● libvirtd-tls.socket - Libvirt TLS IP socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-tls.socket; disabled; vendor preset: disabled)
   Active: active (listening) since Fri 2019-09-20 22:42:01 EDT; 6s ago
   Listen: [::]:16514 (Stream)
   CGroup: /system.slice/libvirtd-tls.socket

Sep 20 22:42:01 lenovo-*** systemd[1]: Listening on Libvirt TLS IP socket.
]# systemctl status libvirtd-tcp.socket
● libvirtd-tcp.socket - Libvirt non-TLS IP socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-tcp.socket; disabled; vendor preset: disabled)
   Active: active (listening) since Fri 2019-09-20 22:41:57 EDT; 16s ago
   Listen: [::]:16509 (Stream)
   CGroup: /system.slice/libvirtd-tcp.socket

Sep 20 22:41:57 lenovo-*** systemd[1]: Listening on Libvirt non-TLS IP socket.

]# virsh list

]# netstat -nltp
tcp6       0      0 :::16509                :::*                    LISTEN      1/systemd           
tcp6       0      0 :::16514                :::*                    LISTEN      1/systemd           

]# virsh -c qemu+tcp://lenovo-***/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # start avocado-vt-vm1
Domain avocado-vt-vm1 started

]# virsh -c qemu+tls://lenovo-***/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # destroy avocado-vt-vm1
Domain avocado-vt-vm1 destroyed

2. Check default configurations after a fresh install of virt:8.1 module
]# grep -v '#' /etc/libvirt/libvirtd.conf
]# grep -v '#' /etc/libvirt/qemu.conf
(nothing output)
]# grep -v '#' /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--timeout 120"

]# grep -iE 'ExecStart|timeout' /usr/lib/systemd/system/libvirtd.service
ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS

# Use Wants instead of Requires so that users
# can disable these three .socket units to revert
# to a traditional non-activation deployment setup
Wants=libvirtd.socket
Wants=libvirtd-ro.socket
Wants=libvirtd-admin.socket
...

Check sockets are active and unmasked:
]# systemctl status libvirtd*socket
● libvirtd.socket - Libvirt local socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
   Active: active (listening) since Fri 2019-09-20 23:37:26 EDT; 1h 25min ago
   Listen: /var/run/libvirt/libvirt-sock (Stream)
   CGroup: /system.slice/libvirtd.socket

Sep 20 23:37:26 lenovo-*** systemd[1]: Listening on Libvirt local socket.

● libvirtd-admin.socket - Libvirt admin socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-admin.socket; disabled; vendor preset: disabled)
   Active: active (listening) since Fri 2019-09-20 23:37:26 EDT; 1h 25min ago
   Listen: /var/run/libvirt/libvirt-admin-sock (Stream)
   CGroup: /system.slice/libvirtd-admin.socket

Sep 20 23:37:26 lenovo-*** systemd[1]: Listening on Libvirt admin socket.

● libvirtd-ro.socket - Libvirt local read-only socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-ro.socket; disabled; vendor preset: disabled)
   Active: active (listening) since Fri 2019-09-20 23:37:26 EDT; 1h 25min ago
   Listen: /var/run/libvirt/libvirt-sock-ro (Stream)
   CGroup: /system.slice/libvirtd-ro.socket

Sep 20 23:37:26 lenovo-*** systemd[1]: Listening on Libvirt local read-only socket.

3. --timeout test:
(1)test default timeout
]#cat /etc/sysconfig/libvirtd
# Default behaviour is for libvirtd.service to start on boot
# so that VM autostart can be performed. We then want it to
# shutdown again if nothing was started and rely on systemd
# socket activation to start it again when some client app
# connects.
LIBVIRTD_ARGS="--timeout 120"
…
]# virsh list

root      5369  7.6  0.2 1828264 45248 ?       Ssl  01:11   0:00 /usr/sbin/libvirtd --timeout 120

Wait for 120s, libvirtd gets inactive
]# ps aux|grep libvirtd ;date
root      5481  0.0  0.0  12108  1076 pts/0    S+   01:13   0:00 grep --color=auto libvirtd
Sat Sep 21 01:13:49 EDT 2019

(2)Disable timeout, libvirtd is still active after 120s
    #LIBVIRTD_ARGS="--timeout 120"

]# ps aux|grep libvirtd ;date
root      5559  0.1  0.3 1828264 47124 ?       Ssl  01:18   0:00 /usr/sbin/libvirtd
root      5677  0.0  0.0  12108   980 pts/0    S+   01:22   0:00 grep --color=auto libvirtd
Sat Sep 21 01:22:47 EDT 2019

(3)Set timemout to other values:
    LIBVIRTD_ARGS="--timeout 10"

# systemctl stop libvirtd
]# virsh list ; date
Sat Sep 21 **01:25:22** EDT 2019

root      5801 15.0  0.2 1828268 45192 ?       Ssl  01:25   0:00 /usr/sbin/libvirtd --timeout 10

]# ps aux|grep libvirtd ;date
root      5902  0.0  0.0  12108   968 pts/0    S+   01:25   0:00 grep --color=auto libvirtd
Sat Sep 21 **01:25:32** EDT 2019

   LIBVIRTD_ARGS="--timeout 0"
Sep 21 01:34:17 lenovo-*** libvirtd[6026]: 2019-09-21 05:34:17.597+0000: 6026: error : main:1016 : Invalid value for timeout

4. --listen test
]# cat /etc/sysconfig/libvirtd
   # If systemd socket activation is disabled, then the following
   # can be used to listen on TCP/TLS sockets
   #LIBVIRTD_ARGS="--listen"
   ...
]# man libvirtd
…
SYSTEM SOCKET ACTIVATION
       The libvirtd daemon is capable of starting in two modes.

       In the traditional mode, it will create and listen on UNIX sockets itself.  If the --listen parameter is given, it will also listen on TCP/IP socket(s), according to the listen_tcp and listen_tls
       options in /etc/libvirt/libvirtd.conf

       In socket activation mode, it will rely on systemd to create and listen on the UNIX, and optionally TCP/IP, sockets and pass them as pre-opened file descriptors. In this mode, it is not permitted to
       pass the --listen parameter, and most of the socket related config options in /etc/libvirt/libvirtd.conf will no longer have any effect. To enable TCP or TLS sockets use either

       $ systemctl start libvirtd-tls.socket

       Or

       $ systemctl start libvirtd-tcp.socket

       Socket activation mode is generally the default when running on a host OS that uses systemd. To revert to the traditional mode, all the socket unit files must be masked:

       $ systemctl mask libvirtd.socket libvirtd-ro.socket \
             libvirtd-admin.socket libvirtd-tls.socket libvirtd-tcp.socket

OPTIONS
…
       -l, --listen
           Listen for TCP/IP connections. This should not be set if using systemd socket activation. Instead activate the libvirtd-tls.socket or libvirtd-tcp.socket unit files.

(1) The default use libvirtd-tcp/tls.socket was in step1, libvird process is like:
root     21936  4.0  0.2 1828268 47300 ?       Ssl  02:08   0:00 /usr/sbin/libvirtd --timeout 120

(2) Stop libvirtd-tls.socket, and try to start by "libvirtd --listen", it can still be set successfully. ??
]# libvirtd --listen

]# virsh -c qemu+tls://lenovo-sr630-05.lab.eng.pek2.redhat.com/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # quit

(3)set --listen in sysconfig, try to start libvirtd
]# cat /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--timeout 120"
LIBVIRTD_ARGS="--listen"

]# virsh list

root      8243  3.3  0.3 1828272 47952 ?       Ssl  04:12   0:01 /usr/sbin/libvirtd --listen

2019-09-21 08:12:59.889+0000: 8243: debug : virSystemdActivationClaimFDs:991 : No FD with name libvirtd-tls.socket

# virsh -c qemu+tls://lenovo-sr630-06.lab.eng.pek2.redhat.com/system
error: failed to connect to the hypervisor
error: unable to connect to server at 'lenovo-sr630-06.lab.eng.pek2.redhat.com:16514': Connection refused

Connect fails, but  without error like below??
"--listen parameter not permitted with systemd activation sockets, see 'man libvirtd' for further guidance"

Scenario2: transform to traditional mode

# systemctl stop libvirtd
# systemctl stop libvirtd.socket libvirtd-admin.socket  libvirtd-ro.socket libvirtd-tcp.socket libvirtd-tls.socket
# systemctl mask libvirtd.socket libvirtd-admin.socket  libvirtd-ro.socket libvirtd-tcp.socket libvirtd-tls.socket
Created symlink /etc/systemd/system/libvirtd.socket → /dev/null.
Created symlink /etc/systemd/system/libvirtd-admin.socket → /dev/null.
Created symlink /etc/systemd/system/libvirtd-ro.socket → /dev/null.
Created symlink /etc/systemd/system/libvirtd-tcp.socket → /dev/null.
Created symlink /etc/systemd/system/libvirtd-tls.socket → /dev/null.

1. try to start by virsh client:
# virsh list
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused

2. start by systemctl
# systemctl start libvirtd
# ps aux|grep libvirtd
root      9362  3.8  0.2 1828268 44744 ?       Ssl  04:43   0:00 /usr/sbin/libvirtd --timeout 120

3. Setup tls remote access by traditional way, succceed.
# vim /etc/libvirt/libvirtd.conf
listen_tls = 1
auth_tls = "none"
# vim /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--timeout 120"
LIBVIRTD_ARGS="--listen"

# systemctl restart libvirtd
root      9593  5.5  0.3 1828264 52496 ?       Ssl  04:47   0:01 /usr/sbin/libvirtd --listen

# virsh -c qemu+tls://lenovo-sr630-06.lab.eng.pek2.redhat.com/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh #

And the newly added --timeout does not break --listen consistent daemon:
]# ps aux|grep libvirtd; date
root      9593  0.3  0.3 1828264 52728 ?       Ssl  **04:47**   0:01 /usr/sbin/libvirtd --listen
root      9751  0.0  0.0  12108   964 pts/1    S+   04:55   0:00 grep --color=auto libvirtd
Sat Sep 21 **04:55:19** EDT 2019

4. --timeout test in traditional mode:
Only open --timeout, without --listen set in /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--timeout 120"

# systemctl restart libvirtd

root      9996  7.0  0.2 1828264 43196 ?       Ssl  05:02   0:00 /usr/sbin/libvirtd --timeout 120

# ps aux|grep libvirtd; date
root     10106  0.0  0.0  12108  1096 pts/1    S+   05:05   0:00 grep --color=auto libvirtd
Sat Sep 21 05:05:59 EDT 2019

# virsh list
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

Then libvirtd can not be used unless restart by systemctl, seems --timeout is not useful in traditional mode. Can it be disabled under traditional mode??

Scenario3: upgrade test
Upgrade from:
libvirt-daemon-5.5.0-2.module+el8.1.0+3773+7dd501bf.x86_64
qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64
systemd-239-16.el8.x86_64

]# systemctl status libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket libvirtd-tcp.socket libvirtd-tls.socket
Unit libvirtd.socket could not be found.
Unit libvirtd-ro.socket could not be found.
Unit libvirtd-admin.socket could not be found.
Unit libvirtd-tcp.socket could not be found.
Unit libvirtd-tls.socket could not be found.

]# grep -v '#' /etc/libvirt/libvirtd.conf
]# grep -v '#' /etc/sysconfig/libvirtd
]# grep -v '#' /etc/libvirt/qemu.conf
(nothing output)

root     23450  5.7  0.2 1829240 46688 ?       Ssl  06:24   0:00 /usr/sbin/libvirtd

Replace repos, then:
]# yum module reset virt
]# yum module enable virt:8.1
]# yum module update virt:8.1

1. upgrade without --listen, after upgrade:
(1) '--timeout' is newly added:
]# grep -v '#' /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--timeout 120"

root     17003  0.1  0.3 1833604 48856 ?       Ssl  06:04   0:00 /usr/sbin/libvirtd --timeout 120

(2)libvirtd sockets are newly added and in active status, means under systemd socket activation mode:
]# systemctl status libvirtd*socket
● libvirtd-ro.socket - Libvirt local read-only socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-ro.socket; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-09-21 06:04:08 EDT; 4min 10s ago..
...
● libvirtd.socket - Libvirt local socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-09-21 06:04:08 EDT; 4min 10s ago
...
● libvirtd-admin.socket - Libvirt admin socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-admin.socket; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-09-21 06:04:08 EDT; 4min 10s ago
...
(3) a previous running guest is still alive:
# virsh list --all
 Id   Name             State
--------------------------------
 1    avocado-vt-vm1   running

After shutdown the guest and wait for ~120s timeout, libvirtd gets inactive.


2. Upgrade with --listen
Before upgrade, setup tls connection:
]# vim /etc/libvirt/libvirtd.conf
listen_tls = 1
auth_tls = "none"

]# vim /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--listen"

]# systemctl restart libvirtd
]# virsh -c qemu+tls://lenovo-***/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh #

After upgrade:
root     23695  0.3  0.3 1832484 47348 ?       Ssl  06:32   0:01 /usr/sbin/libvirtd --listen

]# systemctl status libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket libvirtd-tcp.socket libvirtd-tls.socket
● libvirtd.socket
   Loaded: masked (Reason: Unit libvirtd.socket is masked.)
   Active: inactive (dead)
...
● libvirtd-ro.socket
   Loaded: masked (Reason: Unit libvirtd-ro.socket is masked.)
   Active: inactive (dead)
...
● libvirtd-admin.socket
   Loaded: masked (Reason: Unit libvirtd-admin.socket is masked.)
   Active: inactive (dead)
...
]# virsh -c qemu+tls://lenovo-***/system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # 

New sockets are inactive and masked, means maintains in traditional mode. And tls remote access env is maintained, can connect successfully.

Hi  Daniel,

Here are some questions about issues met during testing, could you help give some answers or suggestions pls?

1.In systemd socket activation mode
Step4(2) in comment15: why -l, --listen option still can be used, and works well?

Step4(3) in comment15: set '--listen' in sysconfig file, why 'virsh list' can still active libvirtd successfully? Shouldn't it be failed with error below?
   "--listen parameter not permitted with systemd activation sockets, see 'man libvirtd' for further guidance" 

Is there sth wrong with my operation or understanding?

2.Setp4 in coment16: --timeout is not useful in traditional mode. Can it be auto disabled when detecting it's in tradition mode(sockects inactive)?

3. In comment17, When upgrade by 'module update', can not find daemon/service stop/start order msg and details in libvirtd.log(even log_level=1), where can I check these details pls?

Thank you!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723

(In reply to yanqzhan from comment #18)
> Hi  Daniel,
> 
> Here are some questions about issues met during testing, could you help give
> some answers or suggestions pls?
> 
> 1.In systemd socket activation mode
> Step4(2) in comment15: why -l, --listen option still can be used, and works
> well?

You stopped the 'libvirtd-tls.socket' unit file which made systemd stop listening on the IP socket.

systemd was still listening on the UNIX sockets but libvirtd would not actually be running due to the autostart + timeout.

When you then start 'libvirtd --listen' outside of systemd, it will have deleted the UNIX sockets that systemd had open & created its own & successfully started.

This is just something we have to live with, there's not much we can viably do to protect against this scenario without negatively impacting other valid scenarios.

> Step4(3) in comment15: set '--listen' in sysconfig file, why 'virsh list'
> can still active libvirtd successfully? Shouldn't it be failed with error
> below?
>    "--listen parameter not permitted with systemd activation sockets, see
> 'man libvirtd' for further guidance" 
> 
> Is there sth wrong with my operation or understanding?

It is starting, but then immediately exiting with a different error

  "No FD with name libvirtd-tls.socket"

this was addressed in bug 1752837

> 2.Setp4 in coment16: --timeout is not useful in traditional mode. Can it be
> auto disabled when detecting it's in tradition mode(sockects inactive)?

We need --timeout working in non-systemd modes for when libvirtd is running unprivileged. It isn't worth trying to get too clever in blocking features as it risks breaking some unexpected usecases.

> 3. In comment17, When upgrade by 'module update', can not find
> daemon/service stop/start order msg and details in libvirtd.log(even
> log_level=1), where can I check these details pls?

The systemd journal should record whenever the service restarts.