Description of problem: We are facing a libvirtd crash when we are trying to connect to qemu by default TLS transport. # virsh -c qemu+tls://localhost/system version error: authentication failed: TLS handshake failed A TLS packet with unexpected length was received. error: failed to connect to the hypervisor I used my own CA and certificates (generated using http://libvirt.org/remote.html#Remote_libvirtd_configuration on Redhat PC) on both Kontron PC and our Board (with patched libvirt). Libvirtd.conf was modified so that libvirt is listening all IPs using default IP (so that it was possible to use same certificates on all machines) These directories and files created and used. /etc/pki/CA/cacert.pem /etc/pki/libvirt/private/serverkey.pem /etc/pki/libvirt/servercert.pem /etc/pki/libvirt/private/clientkey.pem /etc/pki/libvirt/clientcert.pem TLS connection worked fine with Kontron PC # virsh -c qemu+tls://localhost/system version Compiled against library: libvir 0.9.5 Using library: libvir 0.9.5 Using API: QEMU 0.9.5 Running hypervisor: QEMU 0.12.1 But libvirt crashed on our Board (using libvirt 0.10.2, gnutls-2.10.5-1_WR4.3.x86_64 and libudev-161-4 rpms ) # virsh -c qemu+tls://localhost/system version error: authentication failed: TLS handshake failed A TLS packet with unexpected length was received. error: failed to connect to the hypervisor Version-Release number of selected component (if applicable): libvirt 0.10.2 How reproducible: The issue is easily reproducable when we are trying to securely connect to qemu using default TLS transport. virsh -c qemu+tls://localhost/system version Actual results: Libvirt crash, dumping some internal log buffer Expected results: The command should not dump any core. Additional info: GDB : Program received signal SIGABRT, Aborted. 0x00007f8b786e1005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f8b786e1005 in raise () from /lib64/libc.so.6 #1 0x00007f8b786e3e40 in abort () from /lib64/libc.so.6 #2 0x00007f8b79ecadc5 in _gcry_logv (level=50, fmt=0x7f8b79f11170 "operation is not possible without initialized secure memory\n", arg_ptr=0x7fffab7a40a0) at misc.c:136 #3 0x00007f8b79ecb3d5 in _gcry_log_bug (fmt=0x870 <Address 0x870 out of bounds>) at misc.c:220 #4 0x00007f8b79ed0697 in _gcry_secmem_malloc_internal (size=<value optimized out>) at secmem.c:497 #5 0x00007f8b79ed079c in _gcry_secmem_malloc (size=136) at secmem.c:522 #6 0x00007f8b79ecba65 in do_malloc (n=2160, flags=<value optimized out>, mem=0x7fffab7a4200) at global.c:553 #7 0x00007f8b79ecbaa9 in _gcry_malloc_secure (n=2160) at global.c:592 #8 0x00007f8b79ecbb19 in _gcry_xmalloc_secure (n=136) at global.c:746 #9 0x00007f8b79f0e5df in _gcry_mpi_alloc_limb_space (nlimbs=17, secure=2160) at mpiutil.c:92 #10 0x00007f8b79f0e65f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75 #11 0x00007f8b79efb25a in secret (output=0xcc71e0, input=0xcc6f80, skey=0x6) at rsa.c:365 #12 0x00007f8b79efb45a in _gcry_rsa_sign (algo=<value optimized out>, resarr=0xcc7130, data=0xcc6f80, skey=<value optimized out>) at rsa.c:608 #13 0x00007f8b79ed71ef in pubkey_sign (r_sig=0x7fffab7a43f8, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:692 #14 _gcry_pk_sign (r_sig=0x7fffab7a43f8, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:1807 ---Type <return> to continue, or q <return> to quit--- #15 0x00007f8b7a174f9c in _wrap_gcry_pk_sign (algo=GNUTLS_PK_RSA, signature=0x7fffab7a4480, vdata=<value optimized out>, pk_params=0x7fffab7a44a0) at pk-libgcrypt.c:308 #16 0x00007f8b7a16108a in _gnutls_pkcs1_rsa_encrypt (ciphertext=<value optimized out>, plaintext=<value optimized out>, params=<value optimized out>, params_len=6, btype=<value optimized out>) at gnutls_pk.c:150 #17 0x00007f8b7a168fe6 in _gnutls_sign (algo=<value optimized out>, params=<value optimized out>, params_size=<value optimized out>, data=0x7fffab7a45e0, signature=0x0) at gnutls_sig.c:251 #18 0x00007f8b7a16988f in _gnutls_handshake_sign_data (session=0xced3a0, cert=0xce9270, pkey=<value optimized out>, params=<value optimized out>, signature=0x7fffab7a46c0, sign_algo=<value optimized out>) at gnutls_sig.c:226 #19 0x00007f8b7a169fbf in gen_dhe_server_kx (session=0xced3a0, data=0x7fffab7a4730) at auth_dhe.c:152 #20 0x00007f8b7a156195 in _gnutls_send_server_kx_message (session=0x870, again=<value optimized out>) at gnutls_kx.c:207 #21 0x00007f8b7a151c55 in _gnutls_handshake_server (session=0xced3a0) at gnutls_handshake.c:3047 #22 0x00007f8b7a152481 in gnutls_handshake (session=0xced3a0) at gnutls_handshake.c:2709 #23 0x00007f8b7aef4744 in virNetTLSSessionHandshake (sess=0xce9d70) at rpc/virnettlscontext.c:1377 #24 0x00007f8b7aee9a2b in virNetServerClientInit (client=0xce9800) at rpc/virnetserverclient.c:730 #25 0x00007f8b7aee7821 in virNetServerAddClient (svc=0xcef150, clientsock=<value optimized out>, opaque=0xce3cc0) at rpc/virnetserver.c:277 #26 virNetServerDispatchNewClient (svc=0xcef150, clientsock=<value optimized out>, opaque=0xce3cc0) at rpc/virnetserver.c:319 -------------------------- # ulimit -a core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63706 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ------------------------- Daniel P Berrange is aware of this issue. Please find his comments below. Ordinarily gnutls would initialize libgcrypt disabling secmem. Libvirt, however, needs to register thread callbacks with gcrypt. Doing this in turn disables gnutls' setup code. So secmem is left enabled. This is not an issue on most distros, since they allow users to mlock sufficient memory. Anyway we need to fix libvirt to disable secmem, since we've blocked gnutls' own setup from running
We need this fix on high priority. Please give us some more leads to us so that we could try to fix this. Thanks a lot !
Shree, could you put your board in the internet so I can reproduce. I am not able to reproduce I cannot help. This seems more related to your gnutls implementation.
Please try reproducing this on latest upstream release. Also please describe what do you mean by "patched libvirt". If that's your patch applied on top of libvirt's source than we cannot find where the problem is. Thank you.