Bug 1651882
Summary: | Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | yafu <yafu> | ||||
Component: | spice | Assignee: | Default Assignee for SPICE Bugs <rh-spice-bugs> | ||||
Status: | CLOSED NOTABUG | QA Contact: | SPICE QE bug list <spice-qe-bugs> | ||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 8.0 | CC: | blc, cfergeau, fdelorey, fjin, knoel, kraxel, lizhu, rbalakri, ribarry, virt-maint, xiaodwan, xuzhang, yanqzhan, zhguo | ||||
Target Milestone: | rc | Keywords: | Automation, Regression | ||||
Target Release: | 8.0 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2019-01-14 09:09:46 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
My first guess would be a permission issue. Definitively not in qemu. Maybe spice-server, but unlikely. Could be libvirt not setting up cgroups/svirt correctly. Could also be selinux policy bug. Assigning to libvirt for investigation. mid-air collision detected, so trying to re-assign again (can someone fix bugzilla to allow commit non-conflicting changes, *please*?) This looks like a regression caused somewhere in spice-server or below. It's definitely not a permission issue. I managed to reproduce this on RHEL-8 and I can confirm everything works on RHEL-7.6. The libvirt-spice directory and files inside have the same labels and permissions in both cases. Thus I used strace to see what's going on here: 26712 openat(AT_FDCWD, "/etc/pki/libvirt-spice/server-cert.pem", O_RDONLY) = 17 26712 fstat(17, {st_mode=S_IFREG|0655, st_size=745, ...}) = 0 26712 read(17, "-----BEGIN CERTIFICATE-----\n"..."\n-----END CERTIFICATE-----\n", 4096) = 745 26712 close(17) = 0 26712 getpeername(2, 0x7ffe7b241b80, [128]) = -1 ENOTSOCK (Socket operation on non-socket) 26712 futex(0x7f9ea50daf78, FUTEX_WAKE_PRIVATE, 2147483647) = 0 26712 ioctl(2, TCGETS, 0x7ffe7b241b60) = -1 ENOTTY (Inappropriate ioctl for device) 26712 getpid() = 26712 26712 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2338, ...}) = 0 26712 openat(AT_FDCWD, "/usr/lib64/charset.alias", O_RDONLY) = -1 ENOENT (No such file or directory) 26712 openat(AT_FDCWD, "/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 17 26712 fstat(17, {st_mode=S_IFREG|0644, st_size=26398, ...}) = 0 26712 mmap(NULL, 26398, PROT_READ, MAP_SHARED, 17, 0) = 0x7f9ea5753000 26712 close(17) = 0 26712 futex(0x7f9ea06b7a28, FUTEX_WAKE_PRIVATE, 2147483647) = 0 26712 brk(NULL) = 0x555ea90ff000 26712 brk(0x555ea9121000) = 0x555ea9121000 26712 brk(NULL) = 0x555ea9121000 26712 brk(NULL) = 0x555ea9121000 26712 brk(0x555ea9119000) = 0x555ea9119000 26712 brk(NULL) = 0x555ea9119000 26712 write(2, "\n(process:26712): Spice-WARNING **: 15:39:40.667:reds.c:2860:reds_init_ssl: " "Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem\n", 149) = 149 Looking at reds_init_ssl() this error is reported when SSL_CTX_use_certificate_chain_file() fails. But according to the strace output we can see the file was loaded successfully. Let's see if RHEL-7.6 did something else: $ diff -up spice-{7.6,8.0}/server/reds.c --- spice-7.6/server/reds.c 2019-01-11 16:20:57.776396367 +0100 +++ spice-8.0/server/reds.c 2017-10-11 10:32:40.000000000 +0200 ... @@ -2933,10 +2830,9 @@ static int reds_init_ssl(RedsState *reds SSL_METHOD *ssl_method; #endif int return_code; - /* Limit connection to TLSv1.1 or newer. - * When some other SSL/TLS version becomes obsolete, add it to this + /* When some other SSL/TLS version becomes obsolete, add it to this * variable. */ - long ssl_options = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_TLSv1; + long ssl_options = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3; /* Global system initialization*/ g_once(&openssl_once, openssl_global_init, NULL); @@ -2955,7 +2851,6 @@ static int reds_init_ssl(RedsState *reds ssl_options |= SSL_OP_NO_COMPRESSION; #endif SSL_CTX_set_options(reds->ctx, ssl_options); - SSL_CTX_set_ecdh_auto(reds->ctx, 1); /* Load our keys and certificates*/ return_code = SSL_CTX_use_certificate_chain_file(reds->ctx, reds->config->ssl_parameters.certs_file); @@ -3009,11 +2904,6 @@ static int reds_init_ssl(RedsState *reds sk_zero(cmp_stack); #endif - /* must be last to override whatever was configured previously */ - if (reds_ssl_config_file_try_load(reds) != 0) { - return -1; - } - return 0; } Looks like both versions call the same function. That's interesting as the documentation for SSL_CTX_use_certificate_chain_file says: SSL_CTX_use_certificate_chain_file() loads a certificate chain from file into ctx. The certificates must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable, and ending at the highest level (root) CA. SSL_use_certificate_chain_file() is similar except it loads the certificate chain into ssl. But the /etc/pki/libvirt-spice/server-cert.pem file contains only the server certificate without CA certificate (which is in ca-cert.pem). But the setup is the same both in 7.6 and 8.0 so the main question is why it works in 7.6 and fails in 8.0. Perhaps openssl was fixed to actually work according to its documentation. But in that case Spice should be modified to be able to load the files which used to work before. The certificates which are generated by the script are too weak for openssl default's configuration, and thus they get rejected. With a "sed -i s/1024/2048/g spice-tls.sh", I don't get the error described in this bug. An alternative is to change the system-wide crypto policy (see 'update-crypto-policies'), but I don't know the correct way of doing it. Changing @SECLEVEL=2 to @SECLEVEL=1 in /etc/crypto-policies/back-ends/openssl* fixes the startup issue even with a 1024 bit certificate, but I'm quite sure poking directly at these files is not how one should change crypto policies. I sent https://lists.freedesktop.org/archives/spice-devel/2019-January/047173.html upstream to try to have slightly better error reporting in these situations. |
Created attachment 1507556 [details] tls env setup script Description of problem: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled. Version-Release number of selected component: libvirt-4.9.0-1.module+el8+2241+fdba4e12.x86_64 qemu-kvm-3.0.0-2.module+el8+2208+e41b12e0.x86_64 spice-server-0.14.0-6.el8.x86_64 How reproducible: 100% Steps to Reproduce: 1.Setup tls env using the scripts in attachment: #sh spice-tls.sh 2.Enable spice_tls in qemu.conf: spice_tls = 1 spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice" 3.Restart libvirtd: #systemctl restart libvirtd 4.Define a guest with spice graphics device: #virsh edit vm1 <graphics type='spice' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> 5.Start the guest: #virsh start vm1 error: Failed to start domain vm1 error: internal error: qemu unexpectedly closed the monitor: (process:20653): Spice-WARNING **: 23:17:28.643: reds.c:2860:reds_init_ssl: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem 2018-11-21T04:17:28.643520Z qemu-kvm: failed to initialize spice server Actual results: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled. Expected results: Should start guest successfully when spice_tls enabled. Additional info: It works well with: qemu-kvm-rhev-2.12.0-18.el7.x86_64 spice-server-0.14.0-6.el7.x86_64