Bug 698133

Please verify that spice migration works on the libvirt level. vdsm never communicate with spice client, and as far as i recall, neither do rhev-m (after invoking spicec).

If that's a regression indeed, could you tell when it was introduced?

*** Bug 700530 has been marked as a duplicate of this bug. ***

(In reply to comment #3)

> If that's a regression indeed, could you tell when it was introduced?

Between RHEV 2.2 and IC114 - it worked in 2.2.
The question is, does it work when the VMs are migrating between RHEL 5.x hosts. I think that'll narrow down the issue a bit.

(In reply to comment #5)
> (In reply to comment #3)
> 
> > If that's a regression indeed, could you tell when it was introduced?
> 
> Between RHEV 2.2 and IC114 - it worked in 2.2.

I'm pretty sure that spice migration was tested with RHEL6 hosts as well. Hasn't it, Ofer?

I tested, manually, migration of a VM with a spice client connected (on a single
host).
Works for me.

("manually" means running qemu-kvm command line from shell, and providing
"__com.redhat_spice_migrate_info" and "migrate" monitor commands directly to qemu-kvm)

spice-server-0.8.0-1.el6.x86_64
spice-client-0.8.0-2.el6.x86_64
qemu-kvm-0.12.1.2-2.160.el6.x86_64

Please provide the /var/log/libvirt/libvirtd.log file for the source and destination hosts covering the time migration was active.

Also please provide the output of

  # certtool -i --infile /etc/pki/libvirt/server-cert.pem

Created attachment 497115 [details]
vdsm.log

Created attachment 497117 [details]
libvirt log from the target

Created attachment 497118 [details]
libvirt log from the source

Those aren't the right logfiles. I want the *libvirtd* logfile from both hosts, not the QEMU logfiles. A RHEVH node normally has these as /var/log/libvirt/libvirtd.log or /var/log/libvirtd.log

*** Bug 703048 has been marked as a duplicate of this bug. ***

Created attachment 499321 [details]
libvirtd_source.log

requested /var/log/libvirt/libvirtd.log from the source host during migration migrated guest = WIN7x32-lk

Created attachment 499323 [details]
libvirtd_target.log

requested /var/log/libvirt/libvirtd.log from the target host during migration migrated guest = WIN7x32-lk

While testing the same code in upstream libvirt I found a bug in the XPath expression used to extract the TLS subject from XML


+    grap->tlsSubject = virXPathString("string(./graphics/cert[ info='subject']/@value)", ctxt);

should be

+    grap->tlsSubject = virXPathString("string(./graphics/cert[@info='subject']/@value)", ctxt);

Note '@info' instead of 'info'.  This is almost certainly what's causing this bug, because if we don't match the subject when parsing XML, then the SPICE client won't receive any data via the monitor.

Pushed upstream:

commit 45b28f7c4f1876c341c97d24c368427c0f26f344
Author: Michal Privoznik <mprivozn>
Date:   Wed May 18 11:57:07 2011 +0200

    qemu: fix typo in spice migration code
    
    This typo caused XPath returning improper value and thus not
    working spice after migration.

v0.9.1-195-g45b28f7

This should be fixed by the libvirt-0.9.2-1.el6 rebase

For code inspection verification it pass on libvirt-0.9.2-1.el6

Checked this bug 

==== Steps ====

1. install a new guest and configure it with spice graphic framebuffer
2. open the spice client console to access the graphic interface on the source machine 
3. try to migrate the guest to the remote destination machine

==== tested with ====
libvirt-0.9.3-1.el6.x86_64
kernel-2.6.32-164.el6.x86_64
qemu-kvm-0.12.1.2-2.169.el6.x86_64
spice-server-0.8.0-1.el6.x86_64
spice-client-0.8.0-2.el6.x86_64

==== result ====
at step 3, we get the guest migrated to the remote machine , BUT the spice client window get closed . And also we get the following error record in libvirtd.log 

19:22:24.992: 15397: info : libvirt version: 0.9.3, package: 1.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2011-07-04-23:20:52, x86-002.build.bos.redhat.com)
19:22:24.992: 15397: error : qemuDomainExtractTLSSubject:151 : internal error cannot initialize cert object: ASN1 parser: Element was not found.
19:22:24.992: 15397: warning : qemuMigrationPrepareDirect:1386 : Unable to encode migration cookie


==== tested with ====
libvirt-0.8.7-18.el6.x86_64.rpm
kernel-2.6.32-164.el6.x86_64
qemu-kvm-0.12.1.2-2.169.el6.x86_64
spice-server-0.8.0-1.el6.x86_64
spice-client-0.8.0-2.el6.x86_64

==== result ====
at step 3, we can get the spice seemless migration successfully . Spice client window kept open all the time .And there is no error record in libvirtd.log 

So there might me a regression when trying to fix the tls problem for this bug

Vivian, why are you asking Dan Berrange for information?  This BZ is assigned to Michal, he should be your source of information.

Yes. It was a regression but it should be fixed in 0.9.3-2

https://www.redhat.com/archives/libvir-list/2011-July/msg00413.html

Please try again with the recent build.

Hi Michal ,
Today , we tried bug https://bugzilla.redhat.com/show_bug.cgi?id=698133
And as we mentioned before , on our side , we haven't got this bug reproduced even with the old buggy version of libvirt .


But now , with this bug moved to ON_QA , we get following result , please confirm , if it is the regression caused by incomplete patch , or it is another new regression bug . Btw, please confirm , if there are something missed in the steps we tried to reproduce this bug with libvirt only .

[steps]
1. install a new guest and configure it with spice graphic framebuffer
2. open the spice client console to access the graphic interface on the source
machine
3. try to migrate the guest to the remote destination machine

[results]
libvirt-0.9.3-3.el6.x86_64

at step 3, we get the guest migrated to the remote machine , BUT the spice
client window get closed . And also we get the following error record in
libvirtd.log

16:22:24.992: 15397: info : libvirt version: 0.9.3, package: 3.el6 (Red Hat,
Inc. <http://bugzilla.redhat.com/bugzilla>, 2011-07-13-23:20:52,
x86-002.build.bos.redhat.com)
16:22:24.992: 15397: error : qemuDomainExtractTLSSubject:151 : internal error
cannot initialize cert object: ASN1 parser: Element was not found.
16:22:24.992: 15397: warning : qemuMigrationPrepareDirect:1386 : Unable to
encode migration cookie

# spicec -h 10.66.4.220 -p 5900 -s 5901 --host-subject "C=IL,L=Raanana,O=Red Hat,CN=my server" --ca-file /etc/pki/libvirt-spice/ca-cert.pem --secure-channels main --enable-channels all -w redhat
Warning: no factory for 8
ALSA lib pulse.c:229:(pulse_connect) PulseAudio: Unable to connect: Connection refused


libvirt-devel-0.9.3-1.el6.x86_64.rpm

The same with libvirt-0.9.3-3.el6.x86_64.rpm


libvirt-0.8.7-18.el6.x86_64.rpm
15:14:32.757: 7695: warning : qemudStartVMDaemon:3336 : Executing /usr/libexec/qemu-kvm
15:14:32.764: 7695: warning : qemudStartVMDaemon:3346 : Executing done /usr/libexec/qemu-kvm

And the spicec monitor didn't get disconnected .
# spicec -h 10.66.4.220 -p 5900 -s 5901 --host-subject "C=IL,L=Raanana,O=Red Hat,CN=my server" --ca-file /etc/pki/libvirt-spice/ca-cert.pem --secure-channels main --enable-channels all -w redhat
Warning: no factory for 8
Warning: no factory for 8


Thanks
Vivian

Created attachment 513916 [details]
attach detail setup and migrate command here

tested with libvirt-0.9.3-6.el6.x86_64

With the same steps as comment 36 , with ssl spice client connection didn't get cut . And there is no error msg in libvirtd.log and messages log .

Can't help you with verification:

got following problem:

1311166702 INFO [26756:26756] Application::switch_host: host=hyper03.spice.lab.eng.brq.redhat.com port=5906 sport=5907
1311166703 INFO [26756:26757] RedPeer::connect_unsecure: Trying 10.34.58.4 5907
1311166703 INFO [26756:26757] RedPeer::connect_unsecure: Connect failed: Connection refused (111)
1311166703 WARN [26756:26757] RedChannel::run: failed to connect: Connection refused (111)
1311166703 INFO [26756:26756] main: Spice client terminated (exitcode = 3)

seems like spice gets an order to migrate to a new host, but vm was not migrated at all.

Haven't you changed anything that could change behaviour to this?

libvirt-0.9.3-3.el6.x86_64
qemu-kvm-0.12.1.2-2.169.el6.x86_64

Lubos, please try again butwith -6 build. It fixes some serious TLS issues.

tested with libvirt-0.9.3-7.el6.x86_64

guest with spice graphical framebuffer could be migrated successfully . And there is no host subject error record in the libvirtd.log and messages . Spice client can keep connecting , we got the seamless migration . 

So set bug status to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html