Red Hat Bugzilla – Bug 1147331
[migration] Tunnelled migration failed
Last modified: 2015-03-05 02:45:51 EST
Description of problem: Tunnelled spice migration failed Version-Release number of selected component (if applicable): libvirt-1.2.8-4.el7.x86_64 qemu-kvm-rhev-2.1.2-1.el7.x86_64 How reproducible: 100% Steps to Reproduce: setup: Dispatch ssh public key of source host to target host. Creating your local public key pair # ssh-keygen -t rsa Copying the public key to remote host # ssh-copy-id -i ~/.ssh/id_rsa.pub root@{target ip} 1. Define a domain with spice graphic. ... <graphics type='spice' port='5900' autoport='no' keymap='en-us'/> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> 2. Edit the following line in /etc/libvirt/qemu.conf both in source host and target host: from: # spice_listen = "0.0.0.0" to spice_listen = "0.0.0.0" Restart libvirtd service 3. start guest # virsh start qe-con Domain qe-con started 4. Migrate the domain to destination host # virsh migrate --live --p2p --tunnelled guest qemu+ssh://$ip/system error: End of file while reading data: 2014-09-28 11:46:17.020+0000: 2580: debug : virFileClose:99 : Closed fd 23 2014-09-28 11:46:17.020+0000: 2580: debug : virFileClose:99 : Closed fd 26 2014-09-28 11:46:17.020+0000: 2580: debug : virExec:728 : Setting child uid:gid to -1:-1 with caps 0 Ncat: Broken pipe.: Input/output error Actual result: In step 4: migration failed Expected result: fix it Additional info: migration can complete without tunnelled option passing # virsh migrate --live --p2p qe-con qemu+ssh://10.66.4.164/system --verbose Migration: [100 %] The issue can NOT reproduced on libvirt-1.2.8-3.el7.x86_64
Can you attach libvirtd debug logs and /var/log/libvirt/qemu/$VM.log files from both source and destination machines?
Created attachment 942225 [details] qemu log
Created attachment 942229 [details] libvirtd debug log
Created attachment 942230 [details] libvirtd debug log on destination machine
Still missing qemu log from the destination.
(In reply to Jiri Denemark from comment #6) > Still missing qemu log from the destination. Hi Jiri, It failed at the beginning of migration, so there is no qemu log on the destination. On source: # virsh list --all Id Name State ---------------------------------------------------- 26 qe-mig running # virsh migrate --live --p2p --tunnelled qe-mig qemu+ssh://$ip/system --verbose error: End of file while reading data: 2014-09-29 09:51:26.097+0000: 10928: debug : virFileClose:99 : Closed fd 28 2014-09-29 09:51:26.097+0000: 10928: debug : virFileClose:99 : Closed fd 32 2014-09-29 09:51:26.097+0000: 10928: debug : virExec:728 : Setting child uid:gid to -1:-1 with caps 0 Ncat: Broken pipe.: Input/output error # virsh list --all Id Name State ---------------------------------------------------- 26 qe-mig running On destination: # virsh list --all Id Name State ----------------------------------------------------
Hi, Jiri I test tunnelled migration with vnc and spice using ssh tcp and tls, they are all failed. This bug has blocked all tunnelled migration test case, so I set serverity to high.
Fixed upstream by v1.2.10-144-g52691f9: commit 52691f99fa016ac46c9546c37706e57a5180d4c6 Author: Jiri Denemark <jdenemar@redhat.com> Date: Thu Nov 20 13:09:16 2014 +0100 qemu: Fix crash in tunnelled migration Any attempt to start a tunnelled migration with libvirtd that supports RDMA migration (specifically commit v1.2.8-226-ged22a47) crashes libvirtd on the destination host. The crash is inevitable because qemuMigrationPrepareAny is always called with NULL protocol in case of tunnelled migration. https://bugzilla.redhat.com/show_bug.cgi?id=1147331 Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Created attachment 959308 [details] backtrace
And commit v1.2.10-146-g800454e is also needed: commit 800454e45ec9ad1ee495bfa7ffe617453f349d3e Author: Jiri Denemark <jdenemar@redhat.com> Date: Thu Nov 20 13:44:18 2014 +0100 qemu: Really fix crash in tunnelled migration Oops, I forgot to squash one more instance of the same check in the previous commit (v1.2.10-144-g52691f9). https://bugzilla.redhat.com/show_bug.cgi?id=1147331 Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
I can produce this bug on build libvirt-1.2.8-4.el7.x86_64 verify it on build libvirt-1.2.8-9.el7.x86_64 verify steps 1. prepare a p2p migration environment Dispatch ssh public key of source host to target host. Creating your local public key pair # ssh-keygen -t rsa Copying the public key to remote host # ssh-copy-id -i ~/.ssh/id_rsa.pub root@{target ip} 2. prepare a guest with spice graphics <graphics type='spice' port='5900' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> 3. start tunnelled migration with ssh tcp and tls success # virsh migrate rhel7 --live --p2p --tunnelled qemu+ssh://10.66.7.206/system --verbose Migration: [100 %] # virsh migrate rhel7 --live --p2p --tunnelled qemu+tcp://10.66.7.206/system --verbose Migration: [100 %] # virsh migrate --live --p2p --tunnelled rhel7 qemu+tls://server.englab.nay.redhat.com/system --verbose Migration: [100 %] 4. check guest on target host work well after login 5. prepare a guest with vnc graphics <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> 6. start tunnelled migration with ssh tcp and tls success # virsh migrate rhel7 --live --p2p --tunnelled qemu+ssh://10.66.7.206/system --verbose Migration: [100 %] # virsh migrate rhel7 --live --p2p --tunnelled qemu+tcp://10.66.7.206/system --verbose Migration: [100 %] # virsh migrate --live --p2p --tunnelled rhel7 qemu+tls://server.englab.nay.redhat.com/system --verbose Migration: [100 %] 7. check guest work well after migration move to verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html