Description of problem: restart libvirtd with LIBVIRTD_ARGS="--listen" cannot create TCP server Version-Release number of selected component (if applicable): libvirt-daemon-5.6.0-1.module+el8.1.0+3890+4d3d259c.x86_64 qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64 How reproducible: 100% Steps to Reproduce: 1. setup tcp env: #cat /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--listen" #cat /etc/libvirt/libvirtd.conf auth_tcp = "none" listen_tls = 0 listen_tcp = 1 # firewall-cmd --add-port=16509/tcp 2.# systemctl restart libvirtd # ps aux|grep libvirtd root 17955 7.7 0.2 1828252 44852 ? Ssl 23:13 0:00 /usr/sbin/libvirtd --timeout 120 --listen # netstat -nltp|grep 16509 (nothing output) 3.try to client on a client side: # virsh -c qemu+tcp://lenovo-***/system error: failed to connect to the hypervisor error: unable to connect to server at 'lenovo-***:16509': Connection refused # cat configfile.log |grep -E 'virNetServerServiceNewTCP|16509' (nothing output) Actual results: As in step3 Expected results: Should create tcp server, refer to 'additional info' part. Additional info: 1. if use '--listen' to start libvirtd in cmd line, it will work: # systemctl stop libvirtd # /usr/sbin/libvirtd --timeout 120 --listen # ps aux|grep libvirtd root 18347 2.2 0.2 1828256 45080 pts/2 Sl+ 23:37 0:00 /usr/sbin/libvirtd --timeout 120 --listen # netstat -nltp|grep 16509 tcp 0 0 0.0.0.0:16509 0.0.0.0:* LISTEN 18347/libvirtd tcp6 0 0 :::16509 :::* LISTEN 18347/libvirtd # virsh -c qemu+tcp://lenovo-s***/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # # cat cmdline.log |grep -E 'virNetServerServiceNewTCP|16509' 2019-08-15 03:11:59.463+0000: 17842: debug : virNetServerServiceNewTCP:162 : Creating new TCP server nodename='<null>' service='16509' 2019-08-15 03:11:59.464+0000: 17842: info : virNetSocketNew:293 : RPC_SOCKET_NEW: sock=0x5623fc37f1e0 fd=13 errfd=-1 pid=0 localAddr=0.0.0.0;16509, remoteAddr=<null> 2019-08-15 03:11:59.464+0000: 17842: info : virNetSocketNew:293 : RPC_SOCKET_NEW: sock=0x5623fc3763b0 fd=14 errfd=-1 pid=0 localAddr=::;16509, remoteAddr=<null>
Created attachment 1603976 [details] libvirtd_log_configfile
Only one line that may be useful in libvirtd.log: 2019-08-15 04:17:07.346+0000: 3023: debug : virSystemdActivationClaimFDs:991 : No FD with name libvirtd-tcp.socket
Not reproduces on rhel-av8.0.1: libvirt-daemon-5.0.0-11.module+el8.0.1+3459+e357ef2f.x86_64 qemu-kvm-3.1.0-27.module+el8.0.1+3253+c5371cb3.x86_64
libvirtd is now using systemd socket activation by default. When systemd activation is used, the --listen parameter no longer has any effect. The demo /etc/sysconfig/libvirtd file provided by the libvirt RPM contains a warning to this effect. Instead you need to activate the appropriate socket unit file eg $ systemctl start libvirtd-tls.socket or $ systemctl start libvirtd-tcp.socket NB, you must start these sockets, *before* libvirtd itself is started. For the same reason the listen_tcp and listen_tls params in /etc/libvirt/libvirtd.conf also have no effect when systemd activation is used. Again the default config file contains warnings about this.
(In reply to Daniel Berrangé from comment #4) > libvirtd is now using systemd socket activation by default. When systemd > activation is used, the --listen parameter no longer has any effect. The > demo /etc/sysconfig/libvirtd file provided by the libvirt RPM contains a > warning to this effect. > > Instead you need to activate the appropriate socket unit file eg > > $ systemctl start libvirtd-tls.socket > > or > > $ systemctl start libvirtd-tcp.socket > > NB, you must start these sockets, *before* libvirtd itself is started. > > For the same reason the listen_tcp and listen_tls params in > /etc/libvirt/libvirtd.conf also have no effect when systemd activation is > used. Again the default config file contains warnings about this. This change will break the upper layer application connection unless they make change accordingly.
Yes, any applications previously using --listen will have to adapt. Either they need to use the unit files as described, or they need to disable systemd socket activation to continue the old way.
SOme improvements upstream - Make it possible to disable socket activation: https://www.redhat.com/archives/libvir-list/2019-August/msg01068.html - Turn the use of --listen in to a fatal error so users see the back compat problem immediately instead of it being silently ignored: https://www.redhat.com/archives/libvir-list/2019-August/msg01070.html - For in place upgrades, if the user has used --listen, we disable socket activation to avoid breaking their existing host config: https://www.redhat.com/archives/libvir-list/2019-August/msg01071.html
Verify this bug with: libvirt-daemon-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64 qemu-kvm-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64 systemd-239-18.el8.x86_64 Scenario1: Under systemd socket activation 1. libvirtd-tls/tcp.socket work for connection Setup CA and issue server can client certificates, and config in files: ]# grep -v '#' /etc/libvirt/libvirtd.conf auth_tls = "none" auth_tcp = "none" ]# systemctl stop libvirtd Warning: Stopping libvirtd.service, but it can still be activated by: libvirtd.socket libvirtd-tls.socket libvirtd-tcp.socket ]# systemctl status libvirtd-tls.socket ● libvirtd-tls.socket - Libvirt TLS IP socket Loaded: loaded (/usr/lib/systemd/system/libvirtd-tls.socket; disabled; vendor preset: disabled) Active: active (listening) since Fri 2019-09-20 22:42:01 EDT; 6s ago Listen: [::]:16514 (Stream) CGroup: /system.slice/libvirtd-tls.socket Sep 20 22:42:01 lenovo-*** systemd[1]: Listening on Libvirt TLS IP socket. ]# systemctl status libvirtd-tcp.socket ● libvirtd-tcp.socket - Libvirt non-TLS IP socket Loaded: loaded (/usr/lib/systemd/system/libvirtd-tcp.socket; disabled; vendor preset: disabled) Active: active (listening) since Fri 2019-09-20 22:41:57 EDT; 16s ago Listen: [::]:16509 (Stream) CGroup: /system.slice/libvirtd-tcp.socket Sep 20 22:41:57 lenovo-*** systemd[1]: Listening on Libvirt non-TLS IP socket. ]# virsh list ]# netstat -nltp tcp6 0 0 :::16509 :::* LISTEN 1/systemd tcp6 0 0 :::16514 :::* LISTEN 1/systemd ]# virsh -c qemu+tcp://lenovo-***/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # start avocado-vt-vm1 Domain avocado-vt-vm1 started ]# virsh -c qemu+tls://lenovo-***/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # destroy avocado-vt-vm1 Domain avocado-vt-vm1 destroyed 2. Check default configurations after a fresh install of virt:8.1 module ]# grep -v '#' /etc/libvirt/libvirtd.conf ]# grep -v '#' /etc/libvirt/qemu.conf (nothing output) ]# grep -v '#' /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--timeout 120" ]# grep -iE 'ExecStart|timeout' /usr/lib/systemd/system/libvirtd.service ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS # Use Wants instead of Requires so that users # can disable these three .socket units to revert # to a traditional non-activation deployment setup Wants=libvirtd.socket Wants=libvirtd-ro.socket Wants=libvirtd-admin.socket ... Check sockets are active and unmasked: ]# systemctl status libvirtd*socket ● libvirtd.socket - Libvirt local socket Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled) Active: active (listening) since Fri 2019-09-20 23:37:26 EDT; 1h 25min ago Listen: /var/run/libvirt/libvirt-sock (Stream) CGroup: /system.slice/libvirtd.socket Sep 20 23:37:26 lenovo-*** systemd[1]: Listening on Libvirt local socket. ● libvirtd-admin.socket - Libvirt admin socket Loaded: loaded (/usr/lib/systemd/system/libvirtd-admin.socket; disabled; vendor preset: disabled) Active: active (listening) since Fri 2019-09-20 23:37:26 EDT; 1h 25min ago Listen: /var/run/libvirt/libvirt-admin-sock (Stream) CGroup: /system.slice/libvirtd-admin.socket Sep 20 23:37:26 lenovo-*** systemd[1]: Listening on Libvirt admin socket. ● libvirtd-ro.socket - Libvirt local read-only socket Loaded: loaded (/usr/lib/systemd/system/libvirtd-ro.socket; disabled; vendor preset: disabled) Active: active (listening) since Fri 2019-09-20 23:37:26 EDT; 1h 25min ago Listen: /var/run/libvirt/libvirt-sock-ro (Stream) CGroup: /system.slice/libvirtd-ro.socket Sep 20 23:37:26 lenovo-*** systemd[1]: Listening on Libvirt local read-only socket. 3. --timeout test: (1)test default timeout ]#cat /etc/sysconfig/libvirtd # Default behaviour is for libvirtd.service to start on boot # so that VM autostart can be performed. We then want it to # shutdown again if nothing was started and rely on systemd # socket activation to start it again when some client app # connects. LIBVIRTD_ARGS="--timeout 120" … ]# virsh list root 5369 7.6 0.2 1828264 45248 ? Ssl 01:11 0:00 /usr/sbin/libvirtd --timeout 120 Wait for 120s, libvirtd gets inactive ]# ps aux|grep libvirtd ;date root 5481 0.0 0.0 12108 1076 pts/0 S+ 01:13 0:00 grep --color=auto libvirtd Sat Sep 21 01:13:49 EDT 2019 (2)Disable timeout, libvirtd is still active after 120s #LIBVIRTD_ARGS="--timeout 120" ]# ps aux|grep libvirtd ;date root 5559 0.1 0.3 1828264 47124 ? Ssl 01:18 0:00 /usr/sbin/libvirtd root 5677 0.0 0.0 12108 980 pts/0 S+ 01:22 0:00 grep --color=auto libvirtd Sat Sep 21 01:22:47 EDT 2019 (3)Set timemout to other values: LIBVIRTD_ARGS="--timeout 10" # systemctl stop libvirtd ]# virsh list ; date Sat Sep 21 **01:25:22** EDT 2019 root 5801 15.0 0.2 1828268 45192 ? Ssl 01:25 0:00 /usr/sbin/libvirtd --timeout 10 ]# ps aux|grep libvirtd ;date root 5902 0.0 0.0 12108 968 pts/0 S+ 01:25 0:00 grep --color=auto libvirtd Sat Sep 21 **01:25:32** EDT 2019 LIBVIRTD_ARGS="--timeout 0" Sep 21 01:34:17 lenovo-*** libvirtd[6026]: 2019-09-21 05:34:17.597+0000: 6026: error : main:1016 : Invalid value for timeout
4. --listen test ]# cat /etc/sysconfig/libvirtd # If systemd socket activation is disabled, then the following # can be used to listen on TCP/TLS sockets #LIBVIRTD_ARGS="--listen" ... ]# man libvirtd … SYSTEM SOCKET ACTIVATION The libvirtd daemon is capable of starting in two modes. In the traditional mode, it will create and listen on UNIX sockets itself. If the --listen parameter is given, it will also listen on TCP/IP socket(s), according to the listen_tcp and listen_tls options in /etc/libvirt/libvirtd.conf In socket activation mode, it will rely on systemd to create and listen on the UNIX, and optionally TCP/IP, sockets and pass them as pre-opened file descriptors. In this mode, it is not permitted to pass the --listen parameter, and most of the socket related config options in /etc/libvirt/libvirtd.conf will no longer have any effect. To enable TCP or TLS sockets use either $ systemctl start libvirtd-tls.socket Or $ systemctl start libvirtd-tcp.socket Socket activation mode is generally the default when running on a host OS that uses systemd. To revert to the traditional mode, all the socket unit files must be masked: $ systemctl mask libvirtd.socket libvirtd-ro.socket \ libvirtd-admin.socket libvirtd-tls.socket libvirtd-tcp.socket OPTIONS … -l, --listen Listen for TCP/IP connections. This should not be set if using systemd socket activation. Instead activate the libvirtd-tls.socket or libvirtd-tcp.socket unit files. (1) The default use libvirtd-tcp/tls.socket was in step1, libvird process is like: root 21936 4.0 0.2 1828268 47300 ? Ssl 02:08 0:00 /usr/sbin/libvirtd --timeout 120 (2) Stop libvirtd-tls.socket, and try to start by "libvirtd --listen", it can still be set successfully. ?? ]# libvirtd --listen ]# virsh -c qemu+tls://lenovo-sr630-05.lab.eng.pek2.redhat.com/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # quit (3)set --listen in sysconfig, try to start libvirtd ]# cat /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--timeout 120" LIBVIRTD_ARGS="--listen" ]# virsh list root 8243 3.3 0.3 1828272 47952 ? Ssl 04:12 0:01 /usr/sbin/libvirtd --listen 2019-09-21 08:12:59.889+0000: 8243: debug : virSystemdActivationClaimFDs:991 : No FD with name libvirtd-tls.socket # virsh -c qemu+tls://lenovo-sr630-06.lab.eng.pek2.redhat.com/system error: failed to connect to the hypervisor error: unable to connect to server at 'lenovo-sr630-06.lab.eng.pek2.redhat.com:16514': Connection refused Connect fails, but without error like below?? "--listen parameter not permitted with systemd activation sockets, see 'man libvirtd' for further guidance"
Scenario2: transform to traditional mode # systemctl stop libvirtd # systemctl stop libvirtd.socket libvirtd-admin.socket libvirtd-ro.socket libvirtd-tcp.socket libvirtd-tls.socket # systemctl mask libvirtd.socket libvirtd-admin.socket libvirtd-ro.socket libvirtd-tcp.socket libvirtd-tls.socket Created symlink /etc/systemd/system/libvirtd.socket → /dev/null. Created symlink /etc/systemd/system/libvirtd-admin.socket → /dev/null. Created symlink /etc/systemd/system/libvirtd-ro.socket → /dev/null. Created symlink /etc/systemd/system/libvirtd-tcp.socket → /dev/null. Created symlink /etc/systemd/system/libvirtd-tls.socket → /dev/null. 1. try to start by virsh client: # virsh list error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused 2. start by systemctl # systemctl start libvirtd # ps aux|grep libvirtd root 9362 3.8 0.2 1828268 44744 ? Ssl 04:43 0:00 /usr/sbin/libvirtd --timeout 120 3. Setup tls remote access by traditional way, succceed. # vim /etc/libvirt/libvirtd.conf listen_tls = 1 auth_tls = "none" # vim /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--timeout 120" LIBVIRTD_ARGS="--listen" # systemctl restart libvirtd root 9593 5.5 0.3 1828264 52496 ? Ssl 04:47 0:01 /usr/sbin/libvirtd --listen # virsh -c qemu+tls://lenovo-sr630-06.lab.eng.pek2.redhat.com/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # And the newly added --timeout does not break --listen consistent daemon: ]# ps aux|grep libvirtd; date root 9593 0.3 0.3 1828264 52728 ? Ssl **04:47** 0:01 /usr/sbin/libvirtd --listen root 9751 0.0 0.0 12108 964 pts/1 S+ 04:55 0:00 grep --color=auto libvirtd Sat Sep 21 **04:55:19** EDT 2019 4. --timeout test in traditional mode: Only open --timeout, without --listen set in /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--timeout 120" # systemctl restart libvirtd root 9996 7.0 0.2 1828264 43196 ? Ssl 05:02 0:00 /usr/sbin/libvirtd --timeout 120 # ps aux|grep libvirtd; date root 10106 0.0 0.0 12108 1096 pts/1 S+ 05:05 0:00 grep --color=auto libvirtd Sat Sep 21 05:05:59 EDT 2019 # virsh list error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory Then libvirtd can not be used unless restart by systemctl, seems --timeout is not useful in traditional mode. Can it be disabled under traditional mode??
Scenario3: upgrade test Upgrade from: libvirt-daemon-5.5.0-2.module+el8.1.0+3773+7dd501bf.x86_64 qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64 systemd-239-16.el8.x86_64 ]# systemctl status libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket libvirtd-tcp.socket libvirtd-tls.socket Unit libvirtd.socket could not be found. Unit libvirtd-ro.socket could not be found. Unit libvirtd-admin.socket could not be found. Unit libvirtd-tcp.socket could not be found. Unit libvirtd-tls.socket could not be found. ]# grep -v '#' /etc/libvirt/libvirtd.conf ]# grep -v '#' /etc/sysconfig/libvirtd ]# grep -v '#' /etc/libvirt/qemu.conf (nothing output) root 23450 5.7 0.2 1829240 46688 ? Ssl 06:24 0:00 /usr/sbin/libvirtd Replace repos, then: ]# yum module reset virt ]# yum module enable virt:8.1 ]# yum module update virt:8.1 1. upgrade without --listen, after upgrade: (1) '--timeout' is newly added: ]# grep -v '#' /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--timeout 120" root 17003 0.1 0.3 1833604 48856 ? Ssl 06:04 0:00 /usr/sbin/libvirtd --timeout 120 (2)libvirtd sockets are newly added and in active status, means under systemd socket activation mode: ]# systemctl status libvirtd*socket ● libvirtd-ro.socket - Libvirt local read-only socket Loaded: loaded (/usr/lib/systemd/system/libvirtd-ro.socket; disabled; vendor preset: disabled) Active: active (running) since Sat 2019-09-21 06:04:08 EDT; 4min 10s ago.. ... ● libvirtd.socket - Libvirt local socket Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; disabled; vendor preset: disabled) Active: active (running) since Sat 2019-09-21 06:04:08 EDT; 4min 10s ago ... ● libvirtd-admin.socket - Libvirt admin socket Loaded: loaded (/usr/lib/systemd/system/libvirtd-admin.socket; disabled; vendor preset: disabled) Active: active (running) since Sat 2019-09-21 06:04:08 EDT; 4min 10s ago ... (3) a previous running guest is still alive: # virsh list --all Id Name State -------------------------------- 1 avocado-vt-vm1 running After shutdown the guest and wait for ~120s timeout, libvirtd gets inactive. 2. Upgrade with --listen Before upgrade, setup tls connection: ]# vim /etc/libvirt/libvirtd.conf listen_tls = 1 auth_tls = "none" ]# vim /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--listen" ]# systemctl restart libvirtd ]# virsh -c qemu+tls://lenovo-***/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # After upgrade: root 23695 0.3 0.3 1832484 47348 ? Ssl 06:32 0:01 /usr/sbin/libvirtd --listen ]# systemctl status libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket libvirtd-tcp.socket libvirtd-tls.socket ● libvirtd.socket Loaded: masked (Reason: Unit libvirtd.socket is masked.) Active: inactive (dead) ... ● libvirtd-ro.socket Loaded: masked (Reason: Unit libvirtd-ro.socket is masked.) Active: inactive (dead) ... ● libvirtd-admin.socket Loaded: masked (Reason: Unit libvirtd-admin.socket is masked.) Active: inactive (dead) ... ]# virsh -c qemu+tls://lenovo-***/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # New sockets are inactive and masked, means maintains in traditional mode. And tls remote access env is maintained, can connect successfully.
Hi Daniel, Here are some questions about issues met during testing, could you help give some answers or suggestions pls? 1.In systemd socket activation mode Step4(2) in comment15: why -l, --listen option still can be used, and works well? Step4(3) in comment15: set '--listen' in sysconfig file, why 'virsh list' can still active libvirtd successfully? Shouldn't it be failed with error below? "--listen parameter not permitted with systemd activation sockets, see 'man libvirtd' for further guidance" Is there sth wrong with my operation or understanding? 2.Setp4 in coment16: --timeout is not useful in traditional mode. Can it be auto disabled when detecting it's in tradition mode(sockects inactive)? 3. In comment17, When upgrade by 'module update', can not find daemon/service stop/start order msg and details in libvirtd.log(even log_level=1), where can I check these details pls? Thank you!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3723
(In reply to yanqzhan from comment #18) > Hi Daniel, > > Here are some questions about issues met during testing, could you help give > some answers or suggestions pls? > > 1.In systemd socket activation mode > Step4(2) in comment15: why -l, --listen option still can be used, and works > well? You stopped the 'libvirtd-tls.socket' unit file which made systemd stop listening on the IP socket. systemd was still listening on the UNIX sockets but libvirtd would not actually be running due to the autostart + timeout. When you then start 'libvirtd --listen' outside of systemd, it will have deleted the UNIX sockets that systemd had open & created its own & successfully started. This is just something we have to live with, there's not much we can viably do to protect against this scenario without negatively impacting other valid scenarios. > Step4(3) in comment15: set '--listen' in sysconfig file, why 'virsh list' > can still active libvirtd successfully? Shouldn't it be failed with error > below? > "--listen parameter not permitted with systemd activation sockets, see > 'man libvirtd' for further guidance" > > Is there sth wrong with my operation or understanding? It is starting, but then immediately exiting with a different error "No FD with name libvirtd-tls.socket" this was addressed in bug 1752837 > 2.Setp4 in coment16: --timeout is not useful in traditional mode. Can it be > auto disabled when detecting it's in tradition mode(sockects inactive)? We need --timeout working in non-systemd modes for when libvirtd is running unprivileged. It isn't worth trying to get too clever in blocking features as it risks breaking some unexpected usecases. > 3. In comment17, When upgrade by 'module update', can not find > daemon/service stop/start order msg and details in libvirtd.log(even > log_level=1), where can I check these details pls? The systemd journal should record whenever the service restarts.