Bug 1075174

Summary: Migration failure occurring with VIR_DOMAIN_XML_MIGRATABLE (Target device address type none does not match source pci)
Product: [Fedora] Fedora Reporter: Solly Ross <sross>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: acathrow, berrange, clalancette, crobinso, dallan, itamar, jforbes, laine, libvirt-maint, sross, veillard, virt-maint
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.1.3.4-4.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-04-14 22:37:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd.log from the source host
none
libvirtd.log from the destination host
none
virsh dumpxml
none
virsh dumpxml --migratable
none
xml used by nova (with quotes changed, basically same as virsh dumpxml --migratable) none

Description Solly Ross 2014-03-11 16:10:49 UTC
Description of problem:

In OpenStack Nova, we are trying to move to using migrateToURI2, and pass in some modified XML.  The only modification we are currently making is to change the listen address on the spice and vnc elements.

If we use the VIR_DOMAIN_XML_MIGRATABLE flag when calling xmlDesc, migrateToURI2 fails to migrate the domain, complaining of a migration failure: "Target device address type none does not match source pci".

The libvirt logs from the source machine are here:https://gist.github.com/9488767

The libvirt logs from the destination machine are here: https://gist.github.com/9488763

The migration flags being used are `VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PEER2PEER`

Version-Release number of selected component (if applicable):

libvirt-1.1.3.3-5.fc20.x86_64

Comment 1 Jiri Denemark 2014-03-12 09:40:18 UTC
Created attachment 873410 [details]
libvirtd.log from the source host

Comment 2 Jiri Denemark 2014-03-12 09:40:52 UTC
Created attachment 873412 [details]
libvirtd.log from the destination host

Comment 3 Jiri Denemark 2014-03-12 12:10:07 UTC
Oh, I forgot the original XML won't be there when xmlin is used. Could you also attach domain XML produced by "virsh dumpxml $DOMAIN" and "virsh dumpxml --migratable $DOMAIN"? And please, attach them to this bugzilla rather than gist.github.com or another service.

Comment 4 Jiri Denemark 2014-03-12 12:11:45 UTC
Anyway, looking at the code it seems we are not doing the right thing when checking ABI stability of the provided migration XML. However, I wasn't able to reproduce the error with my domain.

Comment 5 Solly Ross 2014-03-12 14:49:38 UTC
Created attachment 873614 [details]
virsh dumpxml

Comment 6 Solly Ross 2014-03-12 14:50:08 UTC
Created attachment 873616 [details]
virsh dumpxml --migratable

Comment 7 Solly Ross 2014-03-12 14:51:09 UTC
Created attachment 873618 [details]
xml used by nova (with quotes changed, basically same as virsh dumpxml --migratable)

Comment 8 Jiri Denemark 2014-03-13 11:41:35 UTC
I made a change to virDomainDefCheckABIStability function to log both source and destination XMLs when the check fails so that it's easier to debug similar issues in the future:

commit 287e2b395aa5357491e80e06ea82ba63c3a1075e
Author: Jiri Denemark <jdenemar>
Date:   Wed Mar 12 11:50:12 2014 +0100

    Make ABI stability issue easier to debug
    
    When ABI stability check fails, we only log the error message describing
    the incompatibility. Let's log both XMLs in case of an error to make it
    easier to analyze where and why the stability check failed.
    
    Signed-off-by: Jiri Denemark <jdenemar>

Comment 9 Jiri Denemark 2014-03-13 18:58:56 UTC
The bug is already fixed upstream by commit v1.1.3-89-g7d70481, which was made for bug 994364.

*** This bug has been marked as a duplicate of bug 994364 ***

Comment 10 Jiri Denemark 2014-03-14 14:47:17 UTC
Actually, since the bug description indicates, this was observed on Fedora 20, let's reopen the bug and move it Fedora component.

Comment 11 Jiri Denemark 2014-03-14 14:49:56 UTC
The bug should be fixed by the following upstream commit:

commit 7d704812b9c50cd3804dd1e7f9e2ea3e75fdc847
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Oct 10 10:53:56 2013 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 11 10:31:35 2013 +0200

    qemu: Introduce qemuDomainDefCheckABIStability
    
    https://bugzilla.redhat.com/show_bug.cgi?id=994364
    
    Whenever we check for ABI stability, we have new xml (e.g. provided by
    user, or obtained from snapshot, whatever) which we compare to old xml
    and see if ABI won't break. However, if the new xml was produced via
    virDomainGetXMLDesc(..., VIR_DOMAIN_XML_MIGRATABLE) it lacks some
    devices, e.g. 'pci-root' controller. Hence, the ABI stability check
    fails even though it is stable. Moreover, we can't simply fix
    virDomainDefCheckABIStability because removing the correct devices is
    task for the driver. For instance, qemu driver wants to remove the usb
    controller too, while LXC driver doesn't. That's why we need special
    qemu wrapper over virDomainDefCheckABIStability which removes the
    correct devices from domain XML, produces MIGRATABLE xml and calls the
    check ABI stability function.
    
    Signed-off-by: Michal Privoznik <mprivozn>

A scratch build for testing this bug (containing both the above commit and the commit from comment #8) can be downloaded from http://koji.fedoraproject.org/koji/taskinfo?taskID=6632713

Comment 12 Solly Ross 2014-03-17 14:32:38 UTC
This appears to have fixed all of the issues we were experiencing in Nova when trying to use retrieved XML to do migration (after changing the listen addresses).

Comment 13 Fedora Update System 2014-03-18 19:30:00 UTC
libvirt-1.1.3.4-4.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/libvirt-1.1.3.4-4.fc20

Comment 14 Fedora Update System 2014-03-19 08:40:24 UTC
Package libvirt-1.1.3.4-4.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.1.3.4-4.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-4032/libvirt-1.1.3.4-4.fc20
then log in and leave karma (feedback).

Comment 15 Fedora Update System 2014-04-14 22:37:49 UTC
libvirt-1.1.3.4-4.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.