Bug 1651882

Summary: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled
Product: Red Hat Enterprise Linux 8 Reporter: yafu <yafu>
Component: spiceAssignee: Default Assignee for SPICE Bugs <rh-spice-bugs>
Status: CLOSED NOTABUG QA Contact: SPICE QE bug list <spice-qe-bugs>
Severity: unspecified Docs Contact:
Priority: high    
Version: 8.0CC: blc, cfergeau, fdelorey, fjin, knoel, kraxel, lizhu, rbalakri, ribarry, virt-maint, xiaodwan, xuzhang, yanqzhan, zhguo
Target Milestone: rcKeywords: Automation, Regression
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-14 09:09:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
tls env setup script none

Description yafu 2018-11-21 06:06:34 UTC
Created attachment 1507556 [details]
tls env setup script

Description of problem:
Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled.

Version-Release number of selected component:
libvirt-4.9.0-1.module+el8+2241+fdba4e12.x86_64
qemu-kvm-3.0.0-2.module+el8+2208+e41b12e0.x86_64
spice-server-0.14.0-6.el8.x86_64


How reproducible:
100%

Steps to Reproduce:
1.Setup tls env using the scripts in attachment:
#sh spice-tls.sh

2.Enable spice_tls in qemu.conf:
spice_tls = 1
spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"

3.Restart libvirtd:
#systemctl restart libvirtd

4.Define a guest with spice graphics device:
#virsh edit vm1
 <graphics type='spice' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
 </graphics>

5.Start the guest:
#virsh start vm1
error: Failed to start domain vm1
error: internal error: qemu unexpectedly closed the monitor: 
(process:20653): Spice-WARNING **: 23:17:28.643: reds.c:2860:reds_init_ssl: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem
2018-11-21T04:17:28.643520Z qemu-kvm: failed to initialize spice server

Actual results:
Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem when starting vm with spice_tls enabled.

Expected results:
Should start guest successfully when spice_tls enabled.

Additional info:
It works well with:
qemu-kvm-rhev-2.12.0-18.el7.x86_64
spice-server-0.14.0-6.el7.x86_64

Comment 2 Gerd Hoffmann 2018-12-05 07:03:35 UTC
My first guess would be a permission issue.
Definitively not in qemu.  Maybe spice-server, but unlikely.
Could be libvirt not setting up cgroups/svirt correctly.
Could also be selinux policy bug.
Assigning to libvirt for investigation.

Comment 3 Gerd Hoffmann 2018-12-05 07:05:23 UTC
mid-air collision detected, so trying to re-assign again
(can someone fix bugzilla to allow commit non-conflicting changes, *please*?)

Comment 5 Jiri Denemark 2019-01-11 15:54:26 UTC
This looks like a regression caused somewhere in spice-server or below. It's
definitely not a permission issue. I managed to reproduce this on RHEL-8 and I
can confirm everything works on RHEL-7.6. The libvirt-spice directory and
files inside have the same labels and permissions in both cases. Thus I used
strace to see what's going on here:

    26712 openat(AT_FDCWD, "/etc/pki/libvirt-spice/server-cert.pem", O_RDONLY) = 17
    26712 fstat(17, {st_mode=S_IFREG|0655, st_size=745, ...}) = 0
    26712 read(17, "-----BEGIN CERTIFICATE-----\n"..."\n-----END CERTIFICATE-----\n", 4096) = 745
    26712 close(17)                         = 0
    26712 getpeername(2, 0x7ffe7b241b80, [128]) = -1 ENOTSOCK (Socket operation on non-socket)
    26712 futex(0x7f9ea50daf78, FUTEX_WAKE_PRIVATE, 2147483647) = 0
    26712 ioctl(2, TCGETS, 0x7ffe7b241b60)  = -1 ENOTTY (Inappropriate ioctl for device)
    26712 getpid()                          = 26712
    26712 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2338, ...}) = 0
    26712 openat(AT_FDCWD, "/usr/lib64/charset.alias", O_RDONLY) = -1 ENOENT (No such file or directory)
    26712 openat(AT_FDCWD, "/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 17
    26712 fstat(17, {st_mode=S_IFREG|0644, st_size=26398, ...}) = 0
    26712 mmap(NULL, 26398, PROT_READ, MAP_SHARED, 17, 0) = 0x7f9ea5753000
    26712 close(17)                         = 0
    26712 futex(0x7f9ea06b7a28, FUTEX_WAKE_PRIVATE, 2147483647) = 0
    26712 brk(NULL)                         = 0x555ea90ff000
    26712 brk(0x555ea9121000)               = 0x555ea9121000
    26712 brk(NULL)                         = 0x555ea9121000
    26712 brk(NULL)                         = 0x555ea9121000
    26712 brk(0x555ea9119000)               = 0x555ea9119000
    26712 brk(NULL)                         = 0x555ea9119000
    26712 write(2, "\n(process:26712): Spice-WARNING **: 15:39:40.667:reds.c:2860:reds_init_ssl: "
                   "Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem\n", 149) = 149

Looking at reds_init_ssl() this error is reported when
SSL_CTX_use_certificate_chain_file() fails. But according to the strace output
we can see the file was loaded successfully. Let's see if RHEL-7.6 did
something else:

    $ diff -up spice-{7.6,8.0}/server/reds.c
    --- spice-7.6/server/reds.c	2019-01-11 16:20:57.776396367 +0100
    +++ spice-8.0/server/reds.c	2017-10-11 10:32:40.000000000 +0200
    ...
    @@ -2933,10 +2830,9 @@ static int reds_init_ssl(RedsState *reds
         SSL_METHOD *ssl_method;
     #endif
         int return_code;
    -    /* Limit connection to TLSv1.1 or newer.
    -     * When some other SSL/TLS version becomes obsolete, add it to this
    +    /* When some other SSL/TLS version becomes obsolete, add it to this
          * variable. */
    -    long ssl_options = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_TLSv1;
    +    long ssl_options = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3;

         /* Global system initialization*/
         g_once(&openssl_once, openssl_global_init, NULL);
    @@ -2955,7 +2851,6 @@ static int reds_init_ssl(RedsState *reds
         ssl_options |= SSL_OP_NO_COMPRESSION;
     #endif
         SSL_CTX_set_options(reds->ctx, ssl_options);
    -    SSL_CTX_set_ecdh_auto(reds->ctx, 1);

         /* Load our keys and certificates*/
         return_code = SSL_CTX_use_certificate_chain_file(reds->ctx, reds->config->ssl_parameters.certs_file);
    @@ -3009,11 +2904,6 @@ static int reds_init_ssl(RedsState *reds
         sk_zero(cmp_stack);
     #endif

    -    /* must be last to override whatever was configured previously */
    -    if (reds_ssl_config_file_try_load(reds) != 0) {
    -        return -1;
    -    }
    -
         return 0;
     }

Looks like both versions call the same function. That's interesting as the
documentation for SSL_CTX_use_certificate_chain_file says:

    SSL_CTX_use_certificate_chain_file() loads a certificate chain from file
    into ctx. The certificates must be in PEM format and must be sorted
    starting with the subject's certificate (actual client or server
    certificate), followed by intermediate CA certificates if applicable, and
    ending at the highest level (root) CA. SSL_use_certificate_chain_file() is
    similar except it loads the certificate chain into ssl.

But the /etc/pki/libvirt-spice/server-cert.pem file contains only the server
certificate without CA certificate (which is in ca-cert.pem). But the setup is
the same both in 7.6 and 8.0 so the main question is why it works in 7.6 and
fails in 8.0. Perhaps openssl was fixed to actually work according to its
documentation. But in that case Spice should be modified to be able to load
the files which used to work before.

Comment 6 Christophe Fergeau 2019-01-14 09:09:46 UTC
The certificates which are generated by the script are too weak for openssl default's configuration, and thus they get rejected. With a "sed -i s/1024/2048/g spice-tls.sh", I don't get the error described in this bug.
An alternative is to change the system-wide crypto policy (see 'update-crypto-policies'), but I don't know the correct way of doing it. Changing @SECLEVEL=2 to @SECLEVEL=1 in /etc/crypto-policies/back-ends/openssl* fixes the startup issue even with a 1024 bit certificate, but I'm quite sure poking directly at these files is not how one should change crypto policies.

Comment 7 Christophe Fergeau 2019-01-14 11:00:34 UTC
I sent https://lists.freedesktop.org/archives/spice-devel/2019-January/047173.html upstream to try to have slightly better error reporting in these situations.