Bug 1075174 - Migration failure occurring with VIR_DOMAIN_XML_MIGRATABLE (Target device address type none does not match source pci)
Summary: Migration failure occurring with VIR_DOMAIN_XML_MIGRATABLE (Target device add...
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 20
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Jiri Denemark
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2014-03-11 16:10 UTC by Solly Ross
Modified: 2014-04-14 22:37 UTC (History)
12 users (show)

Fixed In Version: libvirt-
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2014-04-14 22:37:49 UTC

Attachments (Terms of Use)
libvirtd.log from the source host (643.47 KB, text/plain)
2014-03-12 09:40 UTC, Jiri Denemark
no flags Details
libvirtd.log from the destination host (310.29 KB, text/plain)
2014-03-12 09:40 UTC, Jiri Denemark
no flags Details
virsh dumpxml (3.41 KB, text/plain)
2014-03-12 14:49 UTC, Solly Ross
no flags Details
virsh dumpxml --migratable (3.15 KB, text/plain)
2014-03-12 14:50 UTC, Solly Ross
no flags Details
xml used by nova (with quotes changed, basically same as virsh dumpxml --migratable) (3.15 KB, text/plain)
2014-03-12 14:51 UTC, Solly Ross
no flags Details

Description Solly Ross 2014-03-11 16:10:49 UTC
Description of problem:

In OpenStack Nova, we are trying to move to using migrateToURI2, and pass in some modified XML.  The only modification we are currently making is to change the listen address on the spice and vnc elements.

If we use the VIR_DOMAIN_XML_MIGRATABLE flag when calling xmlDesc, migrateToURI2 fails to migrate the domain, complaining of a migration failure: "Target device address type none does not match source pci".

The libvirt logs from the source machine are here:https://gist.github.com/9488767

The libvirt logs from the destination machine are here: https://gist.github.com/9488763

The migration flags being used are `VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PEER2PEER`

Version-Release number of selected component (if applicable):


Comment 1 Jiri Denemark 2014-03-12 09:40:18 UTC
Created attachment 873410 [details]
libvirtd.log from the source host

Comment 2 Jiri Denemark 2014-03-12 09:40:52 UTC
Created attachment 873412 [details]
libvirtd.log from the destination host

Comment 3 Jiri Denemark 2014-03-12 12:10:07 UTC
Oh, I forgot the original XML won't be there when xmlin is used. Could you also attach domain XML produced by "virsh dumpxml $DOMAIN" and "virsh dumpxml --migratable $DOMAIN"? And please, attach them to this bugzilla rather than gist.github.com or another service.

Comment 4 Jiri Denemark 2014-03-12 12:11:45 UTC
Anyway, looking at the code it seems we are not doing the right thing when checking ABI stability of the provided migration XML. However, I wasn't able to reproduce the error with my domain.

Comment 5 Solly Ross 2014-03-12 14:49:38 UTC
Created attachment 873614 [details]
virsh dumpxml

Comment 6 Solly Ross 2014-03-12 14:50:08 UTC
Created attachment 873616 [details]
virsh dumpxml --migratable

Comment 7 Solly Ross 2014-03-12 14:51:09 UTC
Created attachment 873618 [details]
xml used by nova (with quotes changed, basically same as virsh dumpxml --migratable)

Comment 8 Jiri Denemark 2014-03-13 11:41:35 UTC
I made a change to virDomainDefCheckABIStability function to log both source and destination XMLs when the check fails so that it's easier to debug similar issues in the future:

commit 287e2b395aa5357491e80e06ea82ba63c3a1075e
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Wed Mar 12 11:50:12 2014 +0100

    Make ABI stability issue easier to debug
    When ABI stability check fails, we only log the error message describing
    the incompatibility. Let's log both XMLs in case of an error to make it
    easier to analyze where and why the stability check failed.
    Signed-off-by: Jiri Denemark <jdenemar@redhat.com>

Comment 9 Jiri Denemark 2014-03-13 18:58:56 UTC
The bug is already fixed upstream by commit v1.1.3-89-g7d70481, which was made for bug 994364.

*** This bug has been marked as a duplicate of bug 994364 ***

Comment 10 Jiri Denemark 2014-03-14 14:47:17 UTC
Actually, since the bug description indicates, this was observed on Fedora 20, let's reopen the bug and move it Fedora component.

Comment 11 Jiri Denemark 2014-03-14 14:49:56 UTC
The bug should be fixed by the following upstream commit:

commit 7d704812b9c50cd3804dd1e7f9e2ea3e75fdc847
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Oct 10 10:53:56 2013 +0200
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Fri Oct 11 10:31:35 2013 +0200

    qemu: Introduce qemuDomainDefCheckABIStability
    Whenever we check for ABI stability, we have new xml (e.g. provided by
    user, or obtained from snapshot, whatever) which we compare to old xml
    and see if ABI won't break. However, if the new xml was produced via
    virDomainGetXMLDesc(..., VIR_DOMAIN_XML_MIGRATABLE) it lacks some
    devices, e.g. 'pci-root' controller. Hence, the ABI stability check
    fails even though it is stable. Moreover, we can't simply fix
    virDomainDefCheckABIStability because removing the correct devices is
    task for the driver. For instance, qemu driver wants to remove the usb
    controller too, while LXC driver doesn't. That's why we need special
    qemu wrapper over virDomainDefCheckABIStability which removes the
    correct devices from domain XML, produces MIGRATABLE xml and calls the
    check ABI stability function.
    Signed-off-by: Michal Privoznik <mprivozn@redhat.com>

A scratch build for testing this bug (containing both the above commit and the commit from comment #8) can be downloaded from http://koji.fedoraproject.org/koji/taskinfo?taskID=6632713

Comment 12 Solly Ross 2014-03-17 14:32:38 UTC
This appears to have fixed all of the issues we were experiencing in Nova when trying to use retrieved XML to do migration (after changing the listen addresses).

Comment 13 Fedora Update System 2014-03-18 19:30:00 UTC
libvirt- has been submitted as an update for Fedora 20.

Comment 14 Fedora Update System 2014-03-19 08:40:24 UTC
Package libvirt-
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).

Comment 15 Fedora Update System 2014-04-14 22:37:49 UTC
libvirt- has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.