Description of problem: The libvirtd and virtproxyd daemons both support listening on a TCP socket, with TLS & x509 certificates for encryption/authentication. Currently those certificates are loaded at daemon startup and used forever. It is increasingly common to use short lived certificates and thus users want to be able to refresh certificates for services, ideally without taking downtime for a restart. Libvirt has an enhancement made upstream to support reloading of x509 certificates without restarting libvirtd/virtproxyd The way to test this is as follows (see https://libvirt.org/tlscerts.html for precise commands) 1. Create a new CA (ca-cert.pem) 2. Create a server certificate (server-cert.pem) using the first CA 3. Create a second CA (ca-cert.pem) 4. Create a client certificate (client-cert.pem) using the second CA 2. Create a second server certificate (server-cert.pem) using the second CA Configure libvirtd or virtproxyd on the virtualization host using the first CA and server certificate Configure virsh on a second machine to use the second CA and server certificate Attempt to connect with virsh. It should fail due to the mismatched CA Configure virtualization host to use the second CA and second server certificate, but do *NOT* restart libvirtd/virtproxdy. Attempt to connect with virsh. It should fail as libvirtd/virtproxyd are still using the original first CA/server cert Now run "$ virt-admin server-update-tls libvirtd" or "$ virt-admin server-update-tls virtproxyd" Attempt to connect with virsh. It should now succeed, providing the certs were reloaded. This feature is introduced in libvirt upstream in the forthcoming 6.2.0 release and will thus become available in 8.3 stream. Version-Release number of selected component (if applicable): 6.2.0
Patches contributed by Huawei for this feature were merged upstream as: c43969e1646b39bf4dc07b585de7e2720b83519f docs: update virt-admin.rst for server-update-tls a0c8d49e60a552507fe3f88ee08df32af96a963b virt-admin: Introduce command srv-update-tls 29e68c932fe4509aaa8dd1c84d54d0137de32f45 admin: Introduce virAdmServerUpdateTlsFiles b461178639ba6c37ab851717f1e5a05532db2da6 tls: Add a mutex lock on 'tlsCtxt' 15d280fa97b041cc9b847e5540ed51333c97fcc4 virnetserver: Introduce virNetServerUpdateTlsFiles
Test with libvirt-6.3.0-1.virtcov.el8.x86_64. Test steps: Scenario 1: Test with libvirtd 1.Setup tls env: #sh server.sh #sh client.sh 2.# systemctl stop libvirtd Warning: Stopping libvirtd.service, but it can still be activated by: libvirtd-ro.socket libvirtd-admin.socket libvirtd.socket # systemctl start libvirtd-tls.socket 3.Do tls connection: # virsh -c qemu+tls://dell-*.com/system list Id Name State -------------------- 4.Regenerate servercert.pem and clientcert.pem #sh server.sh #sh client.sh 5.Do tls connection: # virsh -c qemu+tls://dell-*.com/system list 2020-05-09 09:13:26.584+0000: 5323: info : libvirt version: 6.3.0, package: 1.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-05-05-16:30:13, ) 2020-05-09 09:13:26.584+0000: 5323: info : hostname: dell-*.com 2020-05-09 09:13:26.584+0000: 5323: warning : virNetTLSContextCheckCertificate:1128 : Certificate check failed Certificate failed validation: The certificate hasn't got a known issuer. error: failed to connect to the hypervisor error: authentication failed: Failed to verify peer's certificate 5.Update tls certificates: ## virt-admin server-update-tls libvirtd update tls related files succeed 6.Reconnect tls socket: # virsh -c qemu+tls://dell-*.com/system list Id Name State -------------------- Scenario 1: Test with virtproxyd 1.Setup tls env #sh server.sh #sh client.sh 2.Start virtproxyd-tls.socket and virtqemud: # systemctl stop libvirtd # systemctl start virtproxyd-tls.socket # systemctl start virtqemud 3.Do tls connection: # virsh -c qemu+tls://dell-*.com/system list Id Name State -------------------- 4.Regenerate servercert.pem and clientcert.pem #sh server.sh #sh client.sh 5.Do tls connection: # virsh -c qemu+tls://dell-*.com/system list 2020-05-09 09:23:53.276+0000: 5613: info : libvirt version: 6.3.0, package: 1.virtcov.el8 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-05-05-16:30:13, ) 2020-05-09 09:23:53.276+0000: 5613: info : hostname: dell-*.com 2020-05-09 09:23:53.276+0000: 5613: warning : virNetTLSContextCheckCertificate:1128 : Certificate check failed Certificate failed validation: The certificate hasn't got a known issuer. error: failed to connect to the hypervisor error: authentication failed: Failed to verify peer's certificate 6.Update tls certificats: ## virt-admin server-update-tls virtproxyd update tls related files succeed 7.Reconnect tls socket: # virsh -c qemu+tls://dell-*.com/system list Id Name State --------------------
Negative test: 1.# systemctl stop virtproxyd-tls.socket # netstat -nltp | grep 16514 no output 2.# virt-admin server-update-tls virtproxyd error: Unable to update server's tls related files. error: internal error: no tls service found, unable to update tls files 3.Setup tls env and start virtproxyd-tls.socket/virtqemud; 4.Remove servercert.pem: #rm -rf /etc/pki/libvirt/servercert.pem 5.Update tls certificates: # virt-admin server-update-tls virtproxyd error: Unable to update server's tls related files. error: Cannot read certificate '/etc/pki/libvirt/servercert.pem': No such file or directory
Hi Daniel, the man manual for 'virt-admin server-update-tls' only mentions 'libivrtd' daemon. It's better to also add 'virtproxyd' daemon. # virt-admin help server-update-tls NAME server-update-tls - notify server to update TLS related files online. SYNOPSIS server-update-tls <server> DESCRIPTION notify server to update the CA cert, CA CRL, server cert / key without restarts. See OPTIONS for currently supported attributes. OPTIONS [--server] <string> Available servers on a daemon. Currently ***only supports**** 'libvirtd'. #man virt-admin server-update-tls Syntax: server-update-tls server Update tls context on server. · server Available servers on a daemon. Currently ***only supports*** 'libvirtd'.
Created attachment 1687123 [details] server.sh
Created attachment 1687124 [details] client.sh
(In reply to yafu from comment #7) > Hi Daniel, the man manual for 'virt-admin server-update-tls' only mentions > 'libivrtd' daemon. It's better to also add 'virtproxyd' daemon. Yes, we should mention both.
https://www.redhat.com/archives/libvir-list/2020-May/msg00447.html
Fixed upstream by commit d4380be6a763bdd51656641cefe02e9ddfade433 Refs: v6.3.0-86-gd4380be6a7 Author: Daniel P. Berrangé <berrange> AuthorDate: Tue May 12 10:49:10 2020 +0100 Commit: Daniel P. Berrangé <berrange> CommitDate: Wed May 13 16:28:14 2020 +0100 tools: mention server-update-tls supports virtproxyd Reviewed-by: Erik Skultety <eskultet> Signed-off-by: Daniel P. Berrangé <berrange>
Created attachment 1710726 [details] server.sh
Created attachment 1710727 [details] client.sh
Verified with libvirt-6.6.0-2.el8.x86_64. Test steps are the same with comment 5 and comment 6. Also checked the man manual of virt-admin: #man virt-admin ... server-update-tls Syntax: server-update-tls server Update tls context on server. · server Available servers on a daemon. Currently only supports 'libvirtd' or 'virtproxyd'. ...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137