Bug 1651882
| Summary: | Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | yafu <yafu> | ||||
| Component: | spice | Assignee: | Default Assignee for SPICE Bugs <rh-spice-bugs> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | SPICE QE bug list <spice-qe-bugs> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 8.0 | CC: | blc, cfergeau, fdelorey, fjin, knoel, kraxel, lizhu, rbalakri, ribarry, virt-maint, xiaodwan, xuzhang, yanqzhan, zhguo | ||||
| Target Milestone: | rc | Keywords: | Automation, Regression | ||||
| Target Release: | 8.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-01-14 09:09:46 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
My first guess would be a permission issue. Definitively not in qemu. Maybe spice-server, but unlikely. Could be libvirt not setting up cgroups/svirt correctly. Could also be selinux policy bug. Assigning to libvirt for investigation. mid-air collision detected, so trying to re-assign again (can someone fix bugzilla to allow commit non-conflicting changes, *please*?) This looks like a regression caused somewhere in spice-server or below. It's
definitely not a permission issue. I managed to reproduce this on RHEL-8 and I
can confirm everything works on RHEL-7.6. The libvirt-spice directory and
files inside have the same labels and permissions in both cases. Thus I used
strace to see what's going on here:
26712 openat(AT_FDCWD, "/etc/pki/libvirt-spice/server-cert.pem", O_RDONLY) = 17
26712 fstat(17, {st_mode=S_IFREG|0655, st_size=745, ...}) = 0
26712 read(17, "-----BEGIN CERTIFICATE-----\n"..."\n-----END CERTIFICATE-----\n", 4096) = 745
26712 close(17) = 0
26712 getpeername(2, 0x7ffe7b241b80, [128]) = -1 ENOTSOCK (Socket operation on non-socket)
26712 futex(0x7f9ea50daf78, FUTEX_WAKE_PRIVATE, 2147483647) = 0
26712 ioctl(2, TCGETS, 0x7ffe7b241b60) = -1 ENOTTY (Inappropriate ioctl for device)
26712 getpid() = 26712
26712 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2338, ...}) = 0
26712 openat(AT_FDCWD, "/usr/lib64/charset.alias", O_RDONLY) = -1 ENOENT (No such file or directory)
26712 openat(AT_FDCWD, "/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 17
26712 fstat(17, {st_mode=S_IFREG|0644, st_size=26398, ...}) = 0
26712 mmap(NULL, 26398, PROT_READ, MAP_SHARED, 17, 0) = 0x7f9ea5753000
26712 close(17) = 0
26712 futex(0x7f9ea06b7a28, FUTEX_WAKE_PRIVATE, 2147483647) = 0
26712 brk(NULL) = 0x555ea90ff000
26712 brk(0x555ea9121000) = 0x555ea9121000
26712 brk(NULL) = 0x555ea9121000
26712 brk(NULL) = 0x555ea9121000
26712 brk(0x555ea9119000) = 0x555ea9119000
26712 brk(NULL) = 0x555ea9119000
26712 write(2, "\n(process:26712): Spice-WARNING **: 15:39:40.667:reds.c:2860:reds_init_ssl: "
"Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem\n", 149) = 149
Looking at reds_init_ssl() this error is reported when
SSL_CTX_use_certificate_chain_file() fails. But according to the strace output
we can see the file was loaded successfully. Let's see if RHEL-7.6 did
something else:
$ diff -up spice-{7.6,8.0}/server/reds.c
--- spice-7.6/server/reds.c 2019-01-11 16:20:57.776396367 +0100
+++ spice-8.0/server/reds.c 2017-10-11 10:32:40.000000000 +0200
...
@@ -2933,10 +2830,9 @@ static int reds_init_ssl(RedsState *reds
SSL_METHOD *ssl_method;
#endif
int return_code;
- /* Limit connection to TLSv1.1 or newer.
- * When some other SSL/TLS version becomes obsolete, add it to this
+ /* When some other SSL/TLS version becomes obsolete, add it to this
* variable. */
- long ssl_options = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_TLSv1;
+ long ssl_options = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3;
/* Global system initialization*/
g_once(&openssl_once, openssl_global_init, NULL);
@@ -2955,7 +2851,6 @@ static int reds_init_ssl(RedsState *reds
ssl_options |= SSL_OP_NO_COMPRESSION;
#endif
SSL_CTX_set_options(reds->ctx, ssl_options);
- SSL_CTX_set_ecdh_auto(reds->ctx, 1);
/* Load our keys and certificates*/
return_code = SSL_CTX_use_certificate_chain_file(reds->ctx, reds->config->ssl_parameters.certs_file);
@@ -3009,11 +2904,6 @@ static int reds_init_ssl(RedsState *reds
sk_zero(cmp_stack);
#endif
- /* must be last to override whatever was configured previously */
- if (reds_ssl_config_file_try_load(reds) != 0) {
- return -1;
- }
-
return 0;
}
Looks like both versions call the same function. That's interesting as the
documentation for SSL_CTX_use_certificate_chain_file says:
SSL_CTX_use_certificate_chain_file() loads a certificate chain from file
into ctx. The certificates must be in PEM format and must be sorted
starting with the subject's certificate (actual client or server
certificate), followed by intermediate CA certificates if applicable, and
ending at the highest level (root) CA. SSL_use_certificate_chain_file() is
similar except it loads the certificate chain into ssl.
But the /etc/pki/libvirt-spice/server-cert.pem file contains only the server
certificate without CA certificate (which is in ca-cert.pem). But the setup is
the same both in 7.6 and 8.0 so the main question is why it works in 7.6 and
fails in 8.0. Perhaps openssl was fixed to actually work according to its
documentation. But in that case Spice should be modified to be able to load
the files which used to work before.
The certificates which are generated by the script are too weak for openssl default's configuration, and thus they get rejected. With a "sed -i s/1024/2048/g spice-tls.sh", I don't get the error described in this bug. An alternative is to change the system-wide crypto policy (see 'update-crypto-policies'), but I don't know the correct way of doing it. Changing @SECLEVEL=2 to @SECLEVEL=1 in /etc/crypto-policies/back-ends/openssl* fixes the startup issue even with a 1024 bit certificate, but I'm quite sure poking directly at these files is not how one should change crypto policies. I sent https://lists.freedesktop.org/archives/spice-devel/2019-January/047173.html upstream to try to have slightly better error reporting in these situations. |
Created attachment 1507556 [details] tls env setup script Description of problem: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled. Version-Release number of selected component: libvirt-4.9.0-1.module+el8+2241+fdba4e12.x86_64 qemu-kvm-3.0.0-2.module+el8+2208+e41b12e0.x86_64 spice-server-0.14.0-6.el8.x86_64 How reproducible: 100% Steps to Reproduce: 1.Setup tls env using the scripts in attachment: #sh spice-tls.sh 2.Enable spice_tls in qemu.conf: spice_tls = 1 spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice" 3.Restart libvirtd: #systemctl restart libvirtd 4.Define a guest with spice graphics device: #virsh edit vm1 <graphics type='spice' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> 5.Start the guest: #virsh start vm1 error: Failed to start domain vm1 error: internal error: qemu unexpectedly closed the monitor: (process:20653): Spice-WARNING **: 23:17:28.643: reds.c:2860:reds_init_ssl: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem 2018-11-21T04:17:28.643520Z qemu-kvm: failed to initialize spice server Actual results: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled. Expected results: Should start guest successfully when spice_tls enabled. Additional info: It works well with: qemu-kvm-rhev-2.12.0-18.el7.x86_64 spice-server-0.14.0-6.el7.x86_64