Hide Forgot
Created attachment 517741 [details] script to generate spice cert files Description of problem: Tried migration testing with spice client connected on source host . But after migration to target host , the spice session get disconnected . And there is the error log message in the target /var/log/libvirt/libvirt.log Version-Release number of selected component (if applicable): libvirt-0.9.4-2.el6.x86_64 qemu-kvm-0.12.1.2-2.177.el6.x86_64 kernel-2.6.32-178.el6.x86_64 spice-server-0.8.2-3.el6.x86_64 spice-client-0.8.2-3.el6.x86_64 How reproducible Always Steps: 1. edit spice client with following xml section <graphics type='spice' autoport='yes' listen='0.0.0.0' passwd='redhat'> <listen type='address' address='0.0.0.0'/> <channel name='main' mode='secure'/> <channel name='record' mode='insecure'/> </graphics> 2. edit /etc/libvirt/qemu.conf enable following options : a. spice_tls = 1 b. spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice" 3. perform spice.sh to generate spice ssl cert files , and scp /etc/pki/libvirt-spice/* to the target machine /etc/pki/libvirt-spice 4. restart libvirtd on both source machine and target machine 5. start spice guest 6. perform this command on source machine to connect spice session on source machine # spicec -h 10.66.4.220 -p 5900 -s 5901 --host-subject "C=IL,L=Raanana,O=Red Hat,CN=my server" --ca-file /etc/pki/libvirt-spice/ca-cert.pem --secure-channels main --enable-channels all -w redhat 7. migrate guest # virsh migrate --live spice qemu+ssh://10.66.4.241/system Actual result : 1. spice session disconnected 2. error message in target machine 19:55:34.660: 24254: info : libvirt version: 0.9.4, package: 2.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2011-08-09-04:48:09, hs20-bc2-3.build.redhat.com) 19:55:34.660: 24254: error : qemuDomainExtractTLSSubject:164 : internal error cannot initialize cert object: ASN1 parser: Element was not found. 19:55:34.660: 24254: warning : qemuMigrationPrepareAny:1078 : Unable to encode migration cookie 3. No extra error message generated on source machine , and no error message record in messages log Additional info 1. both on the source machine and target machine , the spice session could be accessed successfully . After migration , the spice session could be accessed by target host and source host . 2. tested with following package, spice session WON'T be cut during migration libvirt-0.8.7-18.el6.x86_64.rpm qemu-kvm-0.12.1.2-2.177.el6.x86_64 kernel-2.6.32-178.el6.x86_64 spice-server-0.8.2-3.el6.x86_64 spice-client-0.8.2-3.el6.x86_64
Btw , we keep the same spice cert files when doing the testing on both the old libvirt version and the new libvirt version . And with the old libvirt version , the spice session could be kept . With the new libvirt version , spice session would be interrupted .
19:55:34.660: 24254: error : qemuDomainExtractTLSSubject:164 : internal error cannot initialize cert object: ASN1 parser: Element was not found. This error comes from ret = gnutls_x509_crt_init(&cert); if (ret < 0) { qemuReportError(VIR_ERR_INTERNAL_ERROR, _("cannot initialize cert object: %s"), gnutls_strerror(ret)); goto error; } so it happens before checking the spice certificate. gnutls_x509_crt_init is int gnutls_x509_crt_init (gnutls_x509_crt_t * cert) { gnutls_x509_crt_t tmp = gnutls_calloc (1, sizeof (gnutls_x509_crt_int)); int result; if (!tmp) return GNUTLS_E_MEMORY_ERROR; result = asn1_create_element (_gnutls_get_pkix (), "PKIX1.Certificate", &tmp->cert); if (result != ASN1_SUCCESS) { gnutls_assert (); gnutls_free (tmp); return _gnutls_asn2err (result); } /* If you add anything here, be sure to check if it has to be added to gnutls_x509_crt_import as well. */ *cert = tmp; return 0; /* success */ } it it seems it's failing in the asn1_create_element call. Given the error message, I'd guess that _gnutls_get_pkix() doesn't contain the "PKIX1.Certificate" element. _gnutls_get_pkix() is initialized in gnutls_global_init(), and doing a git grep on libvirt source, I can only find 2 occurrences: src/rpc/virnettlscontext.c: gnutls_global_init(); tests/virnettlscontexttest.c: gnutls_global_init(); Could it be that a call to gnutls_global_init() is missing in libvirt? Or did I get misled somewhere on the way?
> Could it be that a call to gnutls_global_init() is missing in libvirt? Or did I > get misled somewhere on the way? This sounds like most likely cause. If libvirtd itself is not configured to use TLS, they we could be missing the call to gnutls_global_init().
Patch sent upstream: https://www.redhat.com/archives/libvir-list/2011-August/msg00785.html
Moving to POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2011-August/msg00540.html commit 4e9ad9a46dc8bb50e65c7b544072f66377fd54a7 Author: Michal Privoznik <mprivozn> Date: Thu Aug 18 10:44:08 2011 +0200 daemon: initialize GnuTLS When spice_tls is set but listen_tls is not, we don't initialize GnuTLS library. So any later gnutls call (e.g. during migration, where we initialize a certificate) will access uninitialized GnuTLS internal structs and throws an error. Although, we might now initialize GnuTLS twice, it is safe according to the documentation: This function can be called many times, but will only do something the first time. This patch creates 2 functions: virNetTLSInit and virNetTLSDeinit with respect to written above. (cherry picked from commit 74c75671331d284e1f777f9692b72e9737520bf0)
verify pass on kernel-2.6.32-191.el6.x86_64 qemu-kvm-0.12.1.2-2.184.el6.x86_64 libvirt-0.9.4-5.el6.x86_64 After live migration, spice session is still alive and no error in libvirtd.log.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2011-1513.html