Bug 729874 - SSL spice session can't be kept during migration
Summary: SSL spice session can't be kept during migration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.2
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 728234
TreeView+ depends on / blocked
 
Reported: 2011-08-11 06:38 UTC by Vivian Bian
Modified: 2011-12-06 11:26 UTC (History)
15 users (show)

Fixed In Version: libvirt-0.9.4-5.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-12-06 11:26:08 UTC


Attachments (Terms of Use)
script to generate spice cert files (1.54 KB, application/x-shellscript)
2011-08-11 06:38 UTC, Vivian Bian
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:1513 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2011-12-06 01:23:30 UTC

Description Vivian Bian 2011-08-11 06:38:05 UTC
Created attachment 517741 [details]
script to generate spice cert files

Description of problem:
Tried migration testing with spice client connected on source host . But after migration to target host , the spice session get disconnected . And there is the error log message in the target /var/log/libvirt/libvirt.log


Version-Release number of selected component (if applicable):
libvirt-0.9.4-2.el6.x86_64
qemu-kvm-0.12.1.2-2.177.el6.x86_64
kernel-2.6.32-178.el6.x86_64
spice-server-0.8.2-3.el6.x86_64
spice-client-0.8.2-3.el6.x86_64

How reproducible 
Always

Steps:
1. edit spice client with following xml section
    <graphics type='spice' autoport='yes' listen='0.0.0.0' passwd='redhat'>
      <listen type='address' address='0.0.0.0'/>
      <channel name='main' mode='secure'/>
      <channel name='record' mode='insecure'/>
    </graphics>
2. edit /etc/libvirt/qemu.conf
    enable  following options :
    a.  spice_tls = 1
    b.  spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"
  
3. perform spice.sh to generate spice ssl cert files , and scp /etc/pki/libvirt-spice/* to the target machine /etc/pki/libvirt-spice 
 
4. restart libvirtd on both source machine and target machine
5. start spice guest
6. perform this command on source machine to connect spice session on source machine
   # spicec -h 10.66.4.220 -p 5900 -s 5901 --host-subject "C=IL,L=Raanana,O=Red Hat,CN=my server" --ca-file /etc/pki/libvirt-spice/ca-cert.pem --secure-channels main --enable-channels all -w redhat

7. migrate guest
   #  virsh migrate --live spice qemu+ssh://10.66.4.241/system

Actual result :
1. spice session disconnected
2. error message in target machine
    19:55:34.660: 24254: info : libvirt version: 0.9.4, package: 2.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2011-08-09-04:48:09, hs20-bc2-3.build.redhat.com)
    19:55:34.660: 24254: error : qemuDomainExtractTLSSubject:164 : internal error cannot initialize cert object: ASN1 parser: Element was not found.
    19:55:34.660: 24254: warning : qemuMigrationPrepareAny:1078 : Unable to encode migration cookie
3. No extra error message generated on source machine , and no error message record in messages log

Additional info
1. both on the source machine and target machine , the spice session could be accessed successfully . After migration , the spice session could be accessed by target host and source host .

2. tested with following package, spice session WON'T be cut during migration 
libvirt-0.8.7-18.el6.x86_64.rpm
qemu-kvm-0.12.1.2-2.177.el6.x86_64
kernel-2.6.32-178.el6.x86_64
spice-server-0.8.2-3.el6.x86_64
spice-client-0.8.2-3.el6.x86_64

Comment 2 Vivian Bian 2011-08-11 07:49:22 UTC
Btw , we keep the same spice cert files when doing the testing on both the old libvirt version and the new libvirt version . And with the old libvirt version , the spice session could be kept . With the new libvirt version , spice session would be interrupted .

Comment 3 Christophe Fergeau 2011-08-11 08:30:06 UTC
19:55:34.660: 24254: error : qemuDomainExtractTLSSubject:164 : internal
error cannot initialize cert object: ASN1 parser: Element was not found.

This error comes from 

    ret = gnutls_x509_crt_init(&cert);
    if (ret < 0) {
        qemuReportError(VIR_ERR_INTERNAL_ERROR,
                        _("cannot initialize cert object: %s"),
                        gnutls_strerror(ret));
        goto error;
    }

so it happens before checking the spice certificate.
gnutls_x509_crt_init is 

int
gnutls_x509_crt_init (gnutls_x509_crt_t * cert)
{
  gnutls_x509_crt_t tmp = gnutls_calloc (1, sizeof (gnutls_x509_crt_int));
  int result;

  if (!tmp)
    return GNUTLS_E_MEMORY_ERROR;

  result = asn1_create_element (_gnutls_get_pkix (),
                                "PKIX1.Certificate", &tmp->cert);
  if (result != ASN1_SUCCESS)
    {
      gnutls_assert ();
      gnutls_free (tmp);
      return _gnutls_asn2err (result);
    }

  /* If you add anything here, be sure to check if it has to be added
     to gnutls_x509_crt_import as well. */

  *cert = tmp;

  return 0;                     /* success */
}

it it seems it's failing in the asn1_create_element call. Given the error message, I'd guess that _gnutls_get_pkix() doesn't contain the "PKIX1.Certificate" element. _gnutls_get_pkix() is initialized in gnutls_global_init(), and doing a git grep on libvirt source, I can only find 2 occurrences:
src/rpc/virnettlscontext.c:    gnutls_global_init();
tests/virnettlscontexttest.c:    gnutls_global_init();

Could it be that a call to gnutls_global_init() is missing in libvirt? Or did I get misled somewhere on the way?

Comment 4 Daniel Berrange 2011-08-11 09:27:57 UTC
> Could it be that a call to gnutls_global_init() is missing in libvirt? Or did I
> get misled somewhere on the way?

This sounds like most likely cause. If  libvirtd itself is not configured to use TLS, they we could be missing the call to gnutls_global_init().

Comment 5 Michal Privoznik 2011-08-17 16:58:14 UTC
Patch sent upstream:

https://www.redhat.com/archives/libvir-list/2011-August/msg00785.html

Comment 6 Michal Privoznik 2011-08-19 09:37:40 UTC
Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2011-August/msg00540.html


commit 4e9ad9a46dc8bb50e65c7b544072f66377fd54a7
Author: Michal Privoznik <mprivozn@redhat.com>
Date:   Thu Aug 18 10:44:08 2011 +0200

    daemon: initialize GnuTLS
    
    When spice_tls is set but listen_tls is not, we don't initialize
    GnuTLS library. So any later gnutls call (e.g. during migration,
    where we initialize a certificate) will access uninitialized GnuTLS
    internal structs and throws an error.
    
    Although, we might now initialize GnuTLS twice, it is safe according
    to the documentation:
    
        This function can be called many times,
        but will only do something the first time.
    
    This patch creates 2 functions: virNetTLSInit and virNetTLSDeinit
    with respect to written above.
    (cherry picked from commit 74c75671331d284e1f777f9692b72e9737520bf0)

Comment 8 weizhang 2011-08-23 06:20:38 UTC
verify pass on 
kernel-2.6.32-191.el6.x86_64
qemu-kvm-0.12.1.2-2.184.el6.x86_64
libvirt-0.9.4-5.el6.x86_64

After live migration, spice session is still alive and no error in libvirtd.log.

Comment 9 errata-xmlrpc 2011-12-06 11:26:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html


Note You need to log in before you can comment on or make changes to this bug.