Bug 951630

Description of problem:

We are facing a libvirtd crash when we are trying to connect to qemu by default TLS transport.
 
# virsh -c qemu+tls://localhost/system version
error: authentication failed: TLS handshake failed A TLS packet with unexpected length was received.
error: failed to connect to the hypervisor
 
I used my own CA and certificates (generated using http://libvirt.org/remote.html#Remote_libvirtd_configuration on Redhat PC) on both Kontron PC and our Board (with patched libvirt). Libvirtd.conf was modified so that libvirt is listening all IPs using default IP (so that it was possible to use same certificates on all machines) 
 
These directories and files created and used.
/etc/pki/CA/cacert.pem
/etc/pki/libvirt/private/serverkey.pem
/etc/pki/libvirt/servercert.pem
/etc/pki/libvirt/private/clientkey.pem 
/etc/pki/libvirt/clientcert.pem 
 
TLS connection worked fine with Kontron PC
 
# virsh -c qemu+tls://localhost/system version
Compiled against library: libvir 0.9.5
Using library: libvir 0.9.5
Using API: QEMU 0.9.5
Running hypervisor: QEMU 0.12.1
 
But libvirt crashed on our Board (using libvirt 0.10.2, gnutls-2.10.5-1_WR4.3.x86_64 and libudev-161-4 rpms )
 
# virsh -c qemu+tls://localhost/system version
error: authentication failed: TLS handshake failed A TLS packet with unexpected length was received.
error: failed to connect to the hypervisor



Version-Release number of selected component (if applicable):

libvirt 0.10.2


How reproducible:

The issue is easily reproducable when we are trying to securely connect to qemu using default TLS transport.

virsh -c qemu+tls://localhost/system version
  
Actual results:

Libvirt crash, dumping some internal log buffer

Expected results:

The command should not dump any core.
Additional info:

GDB :

Program received signal SIGABRT, Aborted.
0x00007f8b786e1005 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f8b786e1005 in raise () from /lib64/libc.so.6
#1  0x00007f8b786e3e40 in abort () from /lib64/libc.so.6
#2  0x00007f8b79ecadc5 in _gcry_logv (level=50, fmt=0x7f8b79f11170 "operation is not possible without initialized secure memory\n",
    arg_ptr=0x7fffab7a40a0) at misc.c:136
#3  0x00007f8b79ecb3d5 in _gcry_log_bug (fmt=0x870 <Address 0x870 out of bounds>) at misc.c:220
#4  0x00007f8b79ed0697 in _gcry_secmem_malloc_internal (size=<value optimized out>) at secmem.c:497
#5  0x00007f8b79ed079c in _gcry_secmem_malloc (size=136) at secmem.c:522
#6  0x00007f8b79ecba65 in do_malloc (n=2160, flags=<value optimized out>, mem=0x7fffab7a4200) at global.c:553
#7  0x00007f8b79ecbaa9 in _gcry_malloc_secure (n=2160) at global.c:592
#8  0x00007f8b79ecbb19 in _gcry_xmalloc_secure (n=136) at global.c:746
#9  0x00007f8b79f0e5df in _gcry_mpi_alloc_limb_space (nlimbs=17, secure=2160) at mpiutil.c:92
#10 0x00007f8b79f0e65f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75
#11 0x00007f8b79efb25a in secret (output=0xcc71e0, input=0xcc6f80, skey=0x6) at rsa.c:365
#12 0x00007f8b79efb45a in _gcry_rsa_sign (algo=<value optimized out>, resarr=0xcc7130, data=0xcc6f80, skey=<value optimized out>) at rsa.c:608
#13 0x00007f8b79ed71ef in pubkey_sign (r_sig=0x7fffab7a43f8, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:692
#14 _gcry_pk_sign (r_sig=0x7fffab7a43f8, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:1807
---Type <return> to continue, or q <return> to quit---
#15 0x00007f8b7a174f9c in _wrap_gcry_pk_sign (algo=GNUTLS_PK_RSA, signature=0x7fffab7a4480, vdata=<value optimized out>, pk_params=0x7fffab7a44a0)
    at pk-libgcrypt.c:308
#16 0x00007f8b7a16108a in _gnutls_pkcs1_rsa_encrypt (ciphertext=<value optimized out>, plaintext=<value optimized out>, params=<value optimized out>,
    params_len=6, btype=<value optimized out>) at gnutls_pk.c:150
#17 0x00007f8b7a168fe6 in _gnutls_sign (algo=<value optimized out>, params=<value optimized out>, params_size=<value optimized out>,
    data=0x7fffab7a45e0, signature=0x0) at gnutls_sig.c:251
#18 0x00007f8b7a16988f in _gnutls_handshake_sign_data (session=0xced3a0, cert=0xce9270, pkey=<value optimized out>, params=<value optimized out>,
    signature=0x7fffab7a46c0, sign_algo=<value optimized out>) at gnutls_sig.c:226
#19 0x00007f8b7a169fbf in gen_dhe_server_kx (session=0xced3a0, data=0x7fffab7a4730) at auth_dhe.c:152
#20 0x00007f8b7a156195 in _gnutls_send_server_kx_message (session=0x870, again=<value optimized out>) at gnutls_kx.c:207
#21 0x00007f8b7a151c55 in _gnutls_handshake_server (session=0xced3a0) at gnutls_handshake.c:3047
#22 0x00007f8b7a152481 in gnutls_handshake (session=0xced3a0) at gnutls_handshake.c:2709



#23 0x00007f8b7aef4744 in virNetTLSSessionHandshake (sess=0xce9d70) at rpc/virnettlscontext.c:1377
#24 0x00007f8b7aee9a2b in virNetServerClientInit (client=0xce9800) at rpc/virnetserverclient.c:730
#25 0x00007f8b7aee7821 in virNetServerAddClient (svc=0xcef150, clientsock=<value optimized out>, opaque=0xce3cc0) at rpc/virnetserver.c:277
#26 virNetServerDispatchNewClient (svc=0xcef150, clientsock=<value optimized out>, opaque=0xce3cc0) at rpc/virnetserver.c:319

--------------------------

# ulimit -a
core file size          (blocks, -c) 1000000
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63706
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

-------------------------

Daniel P Berrange is aware of this issue. Please find his comments below.

Ordinarily gnutls would initialize libgcrypt disabling secmem.
Libvirt, however, needs to register thread callbacks with gcrypt. Doing
this in turn disables gnutls' setup code. So secmem is left enabled.
This is not an issue on most distros, since they allow users to mlock
sufficient memory.

Anyway we need to fix libvirt to disable secmem, since we've blocked
gnutls' own setup from running