Hide Forgot
Description of problem: If you'll migrate guest in ic114 then spice-client exits during switching hosts with following error message: 1303126299 ERROR [3406:3407] RedPeer::verify_subject: host-subject mismatch This indicates that invalid or none host-subject is passed to the controller. Manual connection to the migrated guest works fine (destination node) => No host-subject issue has been raised. It is caused by not-passing valid host-subject. This can be caused either by vdsm or rhevm itself. Version-Release number of selected component (if applicable): spice-client-0.8.0-2.el6.x86_64 vdsm-4.9-61.el6.x86_64 spice-server-0.8.0-1.el6.x86_64 qemu-kvm-0.12.1.2-2.158.el6.x86_64 libvirt-0.8.7-17.el6.x86_64 ic114 How reproducible: reproducible always Steps to Reproduce: 1. connect with spice to the 2. 3. Actual results: client disconnects with 1303126299 ERROR [3406:3407] RedPeer::verify_subject: host-subject mismatch Expected results: valid host subject will be passed so the session will not be interrupted Additional info: Folowing output seems to be producet by connecting to the guest from rhevm after migration. Manual connection still works. 1303291829 INFO [5194:5194] ForeignMenu::ForeignMenu: Creating a foreign menu connection /tmp/SpiceForeignMenu-5194.uds 1303291829 INFO [5194:5194] Controller::Controller: Creating a controller connection /home/lkocman.el6/.spicec/spice-xpi-qQqqHT 1303291829 INFO [5194:5195] RedPeer::connect_unsecure: Trying 10.34.58.5 5901 1303291829 INFO [5194:5195] RedPeer::connect_unsecure: Connect failed: Connection refused (111) 1303291829 WARN [5194:5195] RedChannel::run: failed to connect: Connection refused (111) 1303291829 INFO [5194:5194] main: Spice client terminated (exitcode = 3)
Please verify that spice migration works on the libvirt level. vdsm never communicate with spice client, and as far as i recall, neither do rhev-m (after invoking spicec). If that's a regression indeed, could you tell when it was introduced?
*** Bug 700530 has been marked as a duplicate of this bug. ***
(In reply to comment #3) > If that's a regression indeed, could you tell when it was introduced? Between RHEV 2.2 and IC114 - it worked in 2.2. The question is, does it work when the VMs are migrating between RHEL 5.x hosts. I think that'll narrow down the issue a bit.
(In reply to comment #5) > (In reply to comment #3) > > > If that's a regression indeed, could you tell when it was introduced? > > Between RHEV 2.2 and IC114 - it worked in 2.2. I'm pretty sure that spice migration was tested with RHEL6 hosts as well. Hasn't it, Ofer?
I tested, manually, migration of a VM with a spice client connected (on a single host). Works for me. ("manually" means running qemu-kvm command line from shell, and providing "__com.redhat_spice_migrate_info" and "migrate" monitor commands directly to qemu-kvm) spice-server-0.8.0-1.el6.x86_64 spice-client-0.8.0-2.el6.x86_64 qemu-kvm-0.12.1.2-2.160.el6.x86_64
Please provide the /var/log/libvirt/libvirtd.log file for the source and destination hosts covering the time migration was active. Also please provide the output of # certtool -i --infile /etc/pki/libvirt/server-cert.pem
Created attachment 497115 [details] vdsm.log
Created attachment 497117 [details] libvirt log from the target
Created attachment 497118 [details] libvirt log from the source
Those aren't the right logfiles. I want the *libvirtd* logfile from both hosts, not the QEMU logfiles. A RHEVH node normally has these as /var/log/libvirt/libvirtd.log or /var/log/libvirtd.log
*** Bug 703048 has been marked as a duplicate of this bug. ***
Created attachment 499321 [details] libvirtd_source.log requested /var/log/libvirt/libvirtd.log from the source host during migration migrated guest = WIN7x32-lk
Created attachment 499323 [details] libvirtd_target.log requested /var/log/libvirt/libvirtd.log from the target host during migration migrated guest = WIN7x32-lk
While testing the same code in upstream libvirt I found a bug in the XPath expression used to extract the TLS subject from XML + grap->tlsSubject = virXPathString("string(./graphics/cert[ info='subject']/@value)", ctxt); should be + grap->tlsSubject = virXPathString("string(./graphics/cert[@info='subject']/@value)", ctxt); Note '@info' instead of 'info'. This is almost certainly what's causing this bug, because if we don't match the subject when parsing XML, then the SPICE client won't receive any data via the monitor.
Pushed upstream: commit 45b28f7c4f1876c341c97d24c368427c0f26f344 Author: Michal Privoznik <mprivozn> Date: Wed May 18 11:57:07 2011 +0200 qemu: fix typo in spice migration code This typo caused XPath returning improper value and thus not working spice after migration. v0.9.1-195-g45b28f7
This should be fixed by the libvirt-0.9.2-1.el6 rebase
For code inspection verification it pass on libvirt-0.9.2-1.el6
Checked this bug ==== Steps ==== 1. install a new guest and configure it with spice graphic framebuffer 2. open the spice client console to access the graphic interface on the source machine 3. try to migrate the guest to the remote destination machine ==== tested with ==== libvirt-0.9.3-1.el6.x86_64 kernel-2.6.32-164.el6.x86_64 qemu-kvm-0.12.1.2-2.169.el6.x86_64 spice-server-0.8.0-1.el6.x86_64 spice-client-0.8.0-2.el6.x86_64 ==== result ==== at step 3, we get the guest migrated to the remote machine , BUT the spice client window get closed . And also we get the following error record in libvirtd.log 19:22:24.992: 15397: info : libvirt version: 0.9.3, package: 1.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2011-07-04-23:20:52, x86-002.build.bos.redhat.com) 19:22:24.992: 15397: error : qemuDomainExtractTLSSubject:151 : internal error cannot initialize cert object: ASN1 parser: Element was not found. 19:22:24.992: 15397: warning : qemuMigrationPrepareDirect:1386 : Unable to encode migration cookie ==== tested with ==== libvirt-0.8.7-18.el6.x86_64.rpm kernel-2.6.32-164.el6.x86_64 qemu-kvm-0.12.1.2-2.169.el6.x86_64 spice-server-0.8.0-1.el6.x86_64 spice-client-0.8.0-2.el6.x86_64 ==== result ==== at step 3, we can get the spice seemless migration successfully . Spice client window kept open all the time .And there is no error record in libvirtd.log So there might me a regression when trying to fix the tls problem for this bug
Vivian, why are you asking Dan Berrange for information? This BZ is assigned to Michal, he should be your source of information.
Yes. It was a regression but it should be fixed in 0.9.3-2 https://www.redhat.com/archives/libvir-list/2011-July/msg00413.html Please try again with the recent build.
Hi Michal , Today , we tried bug https://bugzilla.redhat.com/show_bug.cgi?id=698133 And as we mentioned before , on our side , we haven't got this bug reproduced even with the old buggy version of libvirt . But now , with this bug moved to ON_QA , we get following result , please confirm , if it is the regression caused by incomplete patch , or it is another new regression bug . Btw, please confirm , if there are something missed in the steps we tried to reproduce this bug with libvirt only . [steps] 1. install a new guest and configure it with spice graphic framebuffer 2. open the spice client console to access the graphic interface on the source machine 3. try to migrate the guest to the remote destination machine [results] libvirt-0.9.3-3.el6.x86_64 at step 3, we get the guest migrated to the remote machine , BUT the spice client window get closed . And also we get the following error record in libvirtd.log 16:22:24.992: 15397: info : libvirt version: 0.9.3, package: 3.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2011-07-13-23:20:52, x86-002.build.bos.redhat.com) 16:22:24.992: 15397: error : qemuDomainExtractTLSSubject:151 : internal error cannot initialize cert object: ASN1 parser: Element was not found. 16:22:24.992: 15397: warning : qemuMigrationPrepareDirect:1386 : Unable to encode migration cookie # spicec -h 10.66.4.220 -p 5900 -s 5901 --host-subject "C=IL,L=Raanana,O=Red Hat,CN=my server" --ca-file /etc/pki/libvirt-spice/ca-cert.pem --secure-channels main --enable-channels all -w redhat Warning: no factory for 8 ALSA lib pulse.c:229:(pulse_connect) PulseAudio: Unable to connect: Connection refused libvirt-devel-0.9.3-1.el6.x86_64.rpm The same with libvirt-0.9.3-3.el6.x86_64.rpm libvirt-0.8.7-18.el6.x86_64.rpm 15:14:32.757: 7695: warning : qemudStartVMDaemon:3336 : Executing /usr/libexec/qemu-kvm 15:14:32.764: 7695: warning : qemudStartVMDaemon:3346 : Executing done /usr/libexec/qemu-kvm And the spicec monitor didn't get disconnected . # spicec -h 10.66.4.220 -p 5900 -s 5901 --host-subject "C=IL,L=Raanana,O=Red Hat,CN=my server" --ca-file /etc/pki/libvirt-spice/ca-cert.pem --secure-channels main --enable-channels all -w redhat Warning: no factory for 8 Warning: no factory for 8 Thanks Vivian
Created attachment 513916 [details] attach detail setup and migrate command here
tested with libvirt-0.9.3-6.el6.x86_64 With the same steps as comment 36 , with ssl spice client connection didn't get cut . And there is no error msg in libvirtd.log and messages log .
Can't help you with verification: got following problem: 1311166702 INFO [26756:26756] Application::switch_host: host=hyper03.spice.lab.eng.brq.redhat.com port=5906 sport=5907 1311166703 INFO [26756:26757] RedPeer::connect_unsecure: Trying 10.34.58.4 5907 1311166703 INFO [26756:26757] RedPeer::connect_unsecure: Connect failed: Connection refused (111) 1311166703 WARN [26756:26757] RedChannel::run: failed to connect: Connection refused (111) 1311166703 INFO [26756:26756] main: Spice client terminated (exitcode = 3) seems like spice gets an order to migrate to a new host, but vm was not migrated at all. Haven't you changed anything that could change behaviour to this? libvirt-0.9.3-3.el6.x86_64 qemu-kvm-0.12.1.2-2.169.el6.x86_64
Lubos, please try again butwith -6 build. It fixes some serious TLS issues.
tested with libvirt-0.9.3-7.el6.x86_64 guest with spice graphical framebuffer could be migrated successfully . And there is no host subject error record in the libvirtd.log and messages . Spice client can keep connecting , we got the seamless migration . So set bug status to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2011-1513.html