Bug 1147331
| Summary: | [migration] Tunnelled migration failed | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Yang Yang <yanyang> | ||||||||||
| Component: | libvirt | Assignee: | Jiri Denemark <jdenemar> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||||||
| Severity: | high | Docs Contact: | |||||||||||
| Priority: | high | ||||||||||||
| Version: | 7.1 | CC: | dyuan, jdenemar, mzhan, rbalakri, vivianzhang, xuzhang, yanyang, zhwang, zpeng | ||||||||||
| Target Milestone: | rc | ||||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | x86_64 | ||||||||||||
| OS: | Linux | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | libvirt-1.2.8-8.el7 | Doc Type: | Bug Fix | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2015-03-05 07:45:51 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
Can you attach libvirtd debug logs and /var/log/libvirt/qemu/$VM.log files from both source and destination machines? Created attachment 942225 [details]
qemu log
Created attachment 942229 [details]
libvirtd debug log
Created attachment 942230 [details]
libvirtd debug log on destination machine
Still missing qemu log from the destination. (In reply to Jiri Denemark from comment #6) > Still missing qemu log from the destination. Hi Jiri, It failed at the beginning of migration, so there is no qemu log on the destination. On source: # virsh list --all Id Name State ---------------------------------------------------- 26 qe-mig running # virsh migrate --live --p2p --tunnelled qe-mig qemu+ssh://$ip/system --verbose error: End of file while reading data: 2014-09-29 09:51:26.097+0000: 10928: debug : virFileClose:99 : Closed fd 28 2014-09-29 09:51:26.097+0000: 10928: debug : virFileClose:99 : Closed fd 32 2014-09-29 09:51:26.097+0000: 10928: debug : virExec:728 : Setting child uid:gid to -1:-1 with caps 0 Ncat: Broken pipe.: Input/output error # virsh list --all Id Name State ---------------------------------------------------- 26 qe-mig running On destination: # virsh list --all Id Name State ---------------------------------------------------- Hi, Jiri I test tunnelled migration with vnc and spice using ssh tcp and tls, they are all failed. This bug has blocked all tunnelled migration test case, so I set serverity to high. Fixed upstream by v1.2.10-144-g52691f9:
commit 52691f99fa016ac46c9546c37706e57a5180d4c6
Author: Jiri Denemark <jdenemar>
Date: Thu Nov 20 13:09:16 2014 +0100
qemu: Fix crash in tunnelled migration
Any attempt to start a tunnelled migration with libvirtd that supports
RDMA migration (specifically commit v1.2.8-226-ged22a47) crashes
libvirtd on the destination host.
The crash is inevitable because qemuMigrationPrepareAny is always called
with NULL protocol in case of tunnelled migration.
https://bugzilla.redhat.com/show_bug.cgi?id=1147331
Signed-off-by: Jiri Denemark <jdenemar>
Created attachment 959308 [details]
backtrace
And commit v1.2.10-146-g800454e is also needed:
commit 800454e45ec9ad1ee495bfa7ffe617453f349d3e
Author: Jiri Denemark <jdenemar>
Date: Thu Nov 20 13:44:18 2014 +0100
qemu: Really fix crash in tunnelled migration
Oops, I forgot to squash one more instance of the same check in the
previous commit (v1.2.10-144-g52691f9).
https://bugzilla.redhat.com/show_bug.cgi?id=1147331
Signed-off-by: Jiri Denemark <jdenemar>
I can produce this bug on build libvirt-1.2.8-4.el7.x86_64
verify it on build libvirt-1.2.8-9.el7.x86_64
verify steps
1. prepare a p2p migration environment
Dispatch ssh public key of source host to target host.
Creating your local public key pair
# ssh-keygen -t rsa
Copying the public key to remote host
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@{target ip}
2. prepare a guest with spice graphics
<graphics type='spice' port='5900' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
3. start tunnelled migration with ssh tcp and tls success
# virsh migrate rhel7 --live --p2p --tunnelled qemu+ssh://10.66.7.206/system --verbose
Migration: [100 %]
# virsh migrate rhel7 --live --p2p --tunnelled qemu+tcp://10.66.7.206/system --verbose
Migration: [100 %]
# virsh migrate --live --p2p --tunnelled rhel7 qemu+tls://server.englab.nay.redhat.com/system --verbose
Migration: [100 %]
4. check guest on target host work well after login
5. prepare a guest with vnc graphics
<graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
6. start tunnelled migration with ssh tcp and tls success
# virsh migrate rhel7 --live --p2p --tunnelled qemu+ssh://10.66.7.206/system --verbose
Migration: [100 %]
# virsh migrate rhel7 --live --p2p --tunnelled qemu+tcp://10.66.7.206/system --verbose
Migration: [100 %]
# virsh migrate --live --p2p --tunnelled rhel7 qemu+tls://server.englab.nay.redhat.com/system --verbose
Migration: [100 %]
7. check guest work well after migration
move to verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html |
Description of problem: Tunnelled spice migration failed Version-Release number of selected component (if applicable): libvirt-1.2.8-4.el7.x86_64 qemu-kvm-rhev-2.1.2-1.el7.x86_64 How reproducible: 100% Steps to Reproduce: setup: Dispatch ssh public key of source host to target host. Creating your local public key pair # ssh-keygen -t rsa Copying the public key to remote host # ssh-copy-id -i ~/.ssh/id_rsa.pub root@{target ip} 1. Define a domain with spice graphic. ... <graphics type='spice' port='5900' autoport='no' keymap='en-us'/> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> 2. Edit the following line in /etc/libvirt/qemu.conf both in source host and target host: from: # spice_listen = "0.0.0.0" to spice_listen = "0.0.0.0" Restart libvirtd service 3. start guest # virsh start qe-con Domain qe-con started 4. Migrate the domain to destination host # virsh migrate --live --p2p --tunnelled guest qemu+ssh://$ip/system error: End of file while reading data: 2014-09-28 11:46:17.020+0000: 2580: debug : virFileClose:99 : Closed fd 23 2014-09-28 11:46:17.020+0000: 2580: debug : virFileClose:99 : Closed fd 26 2014-09-28 11:46:17.020+0000: 2580: debug : virExec:728 : Setting child uid:gid to -1:-1 with caps 0 Ncat: Broken pipe.: Input/output error Actual result: In step 4: migration failed Expected result: fix it Additional info: migration can complete without tunnelled option passing # virsh migrate --live --p2p qe-con qemu+ssh://10.66.4.164/system --verbose Migration: [100 %] The issue can NOT reproduced on libvirt-1.2.8-3.el7.x86_64