RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1147331 - [migration] Tunnelled migration failed
Summary: [migration] Tunnelled migration failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-29 03:33 UTC by Yang Yang
Modified: 2015-03-05 07:45 UTC (History)
9 users (show)

Fixed In Version: libvirt-1.2.8-8.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-05 07:45:51 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
qemu log (6.24 KB, text/plain)
2014-09-29 08:15 UTC, Yang Yang 		no flags Details
libvirtd debug log (9.30 MB, text/plain)
2014-09-29 08:22 UTC, Yang Yang 		no flags Details
libvirtd debug log on destination machine (1.01 MB, text/plain)
2014-09-29 08:27 UTC, Yang Yang 		no flags Details
backtrace (4.54 KB, text/plain)
2014-11-20 12:34 UTC, Jiri Denemark 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:0323 0 normal SHIPPED_LIVE Low: libvirt security, bug fix, and enhancement update 2015-03-05 12:10:54 UTC

Description Yang Yang 2014-09-29 03:33:58 UTC 
Description of problem:
Tunnelled spice migration failed

Version-Release number of selected component (if applicable):
libvirt-1.2.8-4.el7.x86_64
qemu-kvm-rhev-2.1.2-1.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
setup:
Dispatch ssh public key of source host to target host.
    Creating your local public key pair
    # ssh-keygen -t rsa

    Copying the public key to remote host
    # ssh-copy-id -i ~/.ssh/id_rsa.pub root@{target ip}

1. Define a domain with spice graphic.
...
<graphics type='spice' port='5900' autoport='no' keymap='en-us'/>

 <channel type='spicevmc'>
 <target type='virtio' name='com.redhat.spice.0'/>
 <address type='virtio-serial' controller='0' bus='0' port='1'/>
 </channel>

2. Edit the following line in /etc/libvirt/qemu.conf both in source host and target host:
from:
# spice_listen = "0.0.0.0"
to
spice_listen = "0.0.0.0"

Restart libvirtd service

3. start guest
# virsh start qe-con
Domain qe-con started

4. Migrate the domain to destination host
# virsh migrate --live --p2p --tunnelled guest qemu+ssh://$ip/system
error: End of file while reading data: 2014-09-28 11:46:17.020+0000: 2580: debug : virFileClose:99 : Closed fd 23
2014-09-28 11:46:17.020+0000: 2580: debug : virFileClose:99 : Closed fd 26
2014-09-28 11:46:17.020+0000: 2580: debug : virExec:728 : Setting child uid:gid to -1:-1 with caps 0
Ncat: Broken pipe.: Input/output error


Actual result:
In step 4: migration failed

Expected result:
fix it

Additional info:
migration can complete without tunnelled option passing
# virsh migrate --live --p2p qe-con qemu+ssh://10.66.4.164/system --verbose
Migration: [100 %]

The issue can NOT reproduced on libvirt-1.2.8-3.el7.x86_64
Comment 2 Jiri Denemark 2014-09-29 07:38:32 UTC 
Can you attach libvirtd debug logs and /var/log/libvirt/qemu/$VM.log files from both source and destination machines?
Comment 3 Yang Yang 2014-09-29 08:15:58 UTC 
Created attachment 942225 [details]
qemu log
Comment 4 Yang Yang 2014-09-29 08:22:36 UTC 
Created attachment 942229 [details]
libvirtd debug log
Comment 5 Yang Yang 2014-09-29 08:27:15 UTC 
Created attachment 942230 [details]
libvirtd debug log on destination machine
Comment 6 Jiri Denemark 2014-09-29 09:44:07 UTC 
Still missing qemu log from the destination.
Comment 7 Yang Yang 2014-09-29 09:58:00 UTC 
(In reply to Jiri Denemark from comment #6)
> Still missing qemu log from the destination.

Hi Jiri,

It failed at the beginning of migration, so there is no qemu log on the destination.

On source:
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 26    qe-mig                         running

# virsh migrate --live --p2p --tunnelled qe-mig qemu+ssh://$ip/system --verbose

error: End of file while reading data: 2014-09-29 09:51:26.097+0000: 10928: debug : virFileClose:99 : Closed fd 28
2014-09-29 09:51:26.097+0000: 10928: debug : virFileClose:99 : Closed fd 32
2014-09-29 09:51:26.097+0000: 10928: debug : virExec:728 : Setting child uid:gid to -1:-1 with caps 0
Ncat: Broken pipe.: Input/output error

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 26    qe-mig                         running

On destination:

# virsh list --all
 Id    Name                           State
----------------------------------------------------
Comment 8 vivian zhang 2014-11-20 04:04:49 UTC 
Hi, Jiri
I test tunnelled migration with vnc and spice using ssh tcp and tls, they are all failed.
This bug has blocked all tunnelled migration test case, so I set serverity to high.
Comment 9 Jiri Denemark 2014-11-20 12:32:53 UTC 
Fixed upstream by v1.2.10-144-g52691f9:

commit 52691f99fa016ac46c9546c37706e57a5180d4c6
Author: Jiri Denemark <jdenemar>
Date:   Thu Nov 20 13:09:16 2014 +0100

    qemu: Fix crash in tunnelled migration
    
    Any attempt to start a tunnelled migration with libvirtd that supports
    RDMA migration (specifically commit v1.2.8-226-ged22a47) crashes
    libvirtd on the destination host.
    
    The crash is inevitable because qemuMigrationPrepareAny is always called
    with NULL protocol in case of tunnelled migration.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1147331
    Signed-off-by: Jiri Denemark <jdenemar>
Comment 10 Jiri Denemark 2014-11-20 12:34:32 UTC 
Created attachment 959308 [details]
backtrace
Comment 11 Jiri Denemark 2014-11-20 12:54:04 UTC 
And commit v1.2.10-146-g800454e is also needed:

commit 800454e45ec9ad1ee495bfa7ffe617453f349d3e
Author: Jiri Denemark <jdenemar>
Date:   Thu Nov 20 13:44:18 2014 +0100

    qemu: Really fix crash in tunnelled migration
    
    Oops, I forgot to squash one more instance of the same check in the
    previous commit (v1.2.10-144-g52691f9).
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1147331
    Signed-off-by: Jiri Denemark <jdenemar>
Comment 14 vivian zhang 2014-11-28 06:30:01 UTC 
I can produce this bug on build libvirt-1.2.8-4.el7.x86_64

verify it on build libvirt-1.2.8-9.el7.x86_64

verify steps

1. prepare a p2p migration environment 
    Dispatch ssh public key of source host to target host.
    Creating your local public key pair
    # ssh-keygen -t rsa

    Copying the public key to remote host
    # ssh-copy-id -i ~/.ssh/id_rsa.pub root@{target ip}

2. prepare a guest with spice graphics
<graphics type='spice' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>

3. start tunnelled migration with ssh tcp and tls success
# virsh migrate rhel7 --live --p2p --tunnelled qemu+ssh://10.66.7.206/system --verbose
Migration: [100 %]

# virsh migrate rhel7 --live --p2p --tunnelled qemu+tcp://10.66.7.206/system --verbose
Migration: [100 %]

# virsh migrate --live --p2p --tunnelled rhel7 qemu+tls://server.englab.nay.redhat.com/system --verbose
Migration: [100 %]

4. check guest on target host work well after login


5. prepare a guest with vnc graphics

<graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>

6. start tunnelled migration with ssh tcp and tls success

# virsh migrate rhel7 --live --p2p --tunnelled qemu+ssh://10.66.7.206/system --verbose
Migration: [100 %]

# virsh migrate rhel7 --live --p2p --tunnelled qemu+tcp://10.66.7.206/system --verbose
Migration: [100 %]

# virsh migrate --live --p2p --tunnelled rhel7 qemu+tls://server.englab.nay.redhat.com/system --verbose
Migration: [100 %]

7. check guest work well after migration

move to verified
Comment 16 errata-xmlrpc 2015-03-05 07:45:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html
Note You need to log in before you can comment on or make changes to this bug.