Bug 1752837 - libvirtd fails to start with TLS socket activation
Summary: libvirtd fails to start with TLS socket activation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.1
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: 8.1
Assignee: Daniel Berrangé
QA Contact: Yanqiu Zhang
URL:
Whiteboard:
: 1776323 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-17 11:12 UTC by Daniel Berrangé
Modified: 2020-11-14 05:03 UTC (History)
10 users (show)

Fixed In Version: libvirt-5.6.0-6.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-06 07:19:21 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 0 None None None 2019-11-06 07:19:50 UTC

Description Daniel Berrangé 2019-09-17 11:12:20 UTC
Description of problem:
With new socket activation support, enable TLS and or TCP

$ systemctl stop libvirtd.service
..setup certs...
$ systemctl start libvirtd-tls.socket

Then try to run virsh

$ virsh list
error: failed to connect to the hypervisor
error: Cannot recv data: Connection reset by peer

# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-09-17 10:31:22 BST; 6s ago
     Docs: man:libvirtd(8)
           https://libvirt.org
  Process: 6034 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=6)
 Main PID: 6034 (code=exited, status=6)

Sep 17 10:31:21 localhost.localdomain systemd[1]: Failed to start Virtualization daemon.
Sep 17 10:31:22 localhost.localdomain systemd[1]: libvirtd.service: Service RestartSec=100ms expired, scheduling restart.
Sep 17 10:31:22 localhost.localdomain systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 5.

Need two fixes from upstream

commit 522b3d2b24d0f7ac78dad442c990d4e34db0eaf2
Author: Michael Chapman <mike.org>
Date:   Tue Sep 17 17:03:57 2019 +1000

    remote: fix registration of TLS socket
    
    Reviewed-by: Daniel P. Berrangé <berrange>
    Signed-off-by: Michael Chapman <mike.org>



Version-Release number of selected component (if applicable):
libvirt-daemon-5.6.0-3.module+el8.1.0+4110+a6d45c3d.x86_64

How reproducible:
Always

Comment 1 Yanqiu Zhang 2019-09-18 12:14:44 UTC
1. Not reproduces for tcp:
# systemctl stop libvirtd
Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd.socket
# systemctl start libvirtd-tcp.socket
# systemctl status libvirtd-tcp.socket
● libvirtd-tcp.socket - Libvirt non-TLS IP socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-tcp.socket; disabled; vendor preset: disabled)
   Active: active (listening) since Wed 2019-09-18 07:30:49 EDT; 12s ago
   Listen: [::]:16509 (Stream)
   CGroup: /system.slice/libvirtd-tcp.socket

Sep 18 07:30:49 lenovo-*** systemd[1]: Listening on Libvirt non-TLS IP socket.
# virsh list
 Id   Name   State
--------------------

# netstat -nltp|grep 16509
tcp6       0      0 :::16509                :::*                    LISTEN      1/systemd           
# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-09-18 07:31:31 EDT; 1min 27s ago


2.Reproduces for tls, the libvirtd-tls.socket returns to inactive status when try to start the libvirtd.service:
# systemctl stop libvirtd
Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd.socket
# systemctl start libvirtd-tls.socket
# systemctl status libvirtd-tls.socket
● libvirtd-tls.socket - Libvirt TLS IP socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-tls.socket; disabled; vendor preset: disabled)
   Active: active (listening) since Wed 2019-09-18 08:01:28 EDT; 4s ago
   Listen: [::]:16514 (Stream)
   CGroup: /system.slice/libvirtd-tls.socket

Sep 18 08:01:28 lenovo-*** systemd[1]: Listening on Libvirt TLS IP socket.

# netstat -nltp|grep 16514
tcp6       0      0 :::16514                :::*                    LISTEN      1/systemd           

# virsh list
error: failed to connect to the hypervisor
error: Cannot recv data: Connection reset by peer

# netstat -nltp|grep 16514

# systemctl status libvirtd-tls.socket
● libvirtd-tls.socket - Libvirt TLS IP socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-tls.socket; disabled; vendor preset: disabled)
   Active: failed (Result: service-start-limit-hit) since Wed 2019-09-18 08:02:53 EDT; 4s ago
   Listen: [::]:16514 (Stream)

Sep 18 08:01:28 lenovo-*** systemd[1]: Listening on Libvirt TLS IP socket.
Sep 18 08:02:53 lenovo-*** systemd[1]: libvirtd-tls.socket: Failed with result 'service-start-limit-hit'.

# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-09-18 08:02:53 EDT; 46s ago
     Docs: man:libvirtd(8)
           https://libvirt.org
  Process: 5383 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=6)
 Main PID: 5383 (code=exited, status=6)
    Tasks: 2 (limit: 32768)
   Memory: 85.2M
   CGroup: /system.slice/libvirtd.service
           ├─24364 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshe>
           └─24365 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshe>

Sep 18 08:02:53 lenovo-*** systemd[1]: libvirtd.service: Service RestartSec=100ms expired, scheduling restart.
Sep 18 08:02:53 lenovo-*** systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 5.
Sep 18 08:02:53 lenovo-*** systemd[1]: Stopped Virtualization daemon.
Sep 18 08:02:53 lenovo-*** systemd[1]: libvirtd.service: Start request repeated too quickly.
Sep 18 08:02:53 lenovo-*** systemd[1]: libvirtd.service: Failed with result 'exit-code'.
Sep 18 08:02:53 lenovo-*** systemd[1]: Failed to start Virtualization daemon.

Here, if manually do "# systemctl start libvirtd" will make libvirtd.service back to work:

# virsh list
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
...
# virsh list
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused

# systemctl start libvirtd
# virsh list
 Id   Name   State
--------------------

Comment 2 Yanqiu Zhang 2019-09-19 02:22:40 UTC
Pkgs version for comment1:
libvirt-daemon-5.6.0-5.module+el8.1.0+4229+2e4e348c.x86_64
qemu-kvm-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64

Comment 7 Yanqiu Zhang 2019-09-20 07:07:40 UTC
Verify with:
libvirt-daemon-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64
qemu-kvm-4.1.0-10.module+el8.1.0+4234+33aa4f57.x86_64

#cat /etc/libvirt/libvirtd.conf
auth_tls = "none"

And setup certificates.

# systemctl stop libvirtd
Warning: Stopping libvirtd.service, but it can still be activated by:
  libvirtd-tls.socket
  libvirtd.socket
  libvirtd-tcp.socket

# systemctl start libvirtd-tls.socket 

# virsh list
 Id   Name   State
--------------------

# netstat -nltp|grep 16514
tcp6       0      0 :::16514                :::*                    LISTEN      1/systemd           

# systemctl status libvirtd-tls.socket
● libvirtd-tls.socket - Libvirt TLS IP socket
   Loaded: loaded (/usr/lib/systemd/system/libvirtd-tls.socket; disabled; vendor preset: disabled)
   Active: active (listening) since Fri 2019-09-20 02:52:02 EDT; 14min ago
   Listen: [::]:16514 (Stream)
   CGroup: /system.slice/libvirtd-tls.socket

Sep 20 02:52:02 lenovo-*** systemd[1]: Listening on Libvirt TLS IP socket.


# virsh -c qemu+tls://lenovo-***/system 
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # quit

Comment 9 errata-xmlrpc 2019-11-06 07:19:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723

Comment 10 Daniel Berrangé 2019-11-26 09:57:45 UTC
*** Bug 1776323 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.