Bug 1265111 - migration from libvirt 1.2.17 fails with "graphics listen attribute $IP must match address attribute of first listen element (found none)"
migration from libvirt 1.2.17 fails with "graphics listen attribute $IP must ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.1
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Jiri Denemark
Virtualization Bugs
:
Depends On:
Blocks: 1172230 1154205 1260409
  Show dependency treegraph
 
Reported: 2015-09-22 03:24 EDT by Francesco Romani
Modified: 2015-11-19 01:54 EST (History)
11 users (show)

See Also:
Fixed In Version: libvirt-1.2.17-11.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-19 01:54:57 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
libvirt debug logs (174.04 KB, application/x-gzip)
2015-09-22 12:03 EDT, Francesco Romani
no flags Details

  None (edit)
Description Francesco Romani 2015-09-22 03:24:29 EDT
Description of problem:
Found testing oVirt 3.6.0. Migrations of a VM using spice graphics device fails with this error

libvirtError: XML error: graphics listen attribute ${IP} must match address attribute of first listen element (found none)

when migrating from libvirt 1.2.17-9 (RHEL 7.2) to 1.2.8-16 (RHEL 7.1)

The other way around, RHEL 7.1 to RHEL 7.2 works.

Version-Release number of selected component (if applicable):
-RHEL 7.2:
libvirt-1.2.17-9.el7.x86_64

-RHEV-H 7.1 for RHEV 3.5.4-1 ASYNC (rhev-hypervisor7-7.1-20150911.0):
libvirt-1.2.8-16.el7_1.3.x86_64

How reproducible:
tried 3-4 times, always happened. Seems 100% reproduceable.

Steps to Reproduce:
1. setup one RHEL 7.2 and one RHEL 7.1 box
2. boot up VM on RHEL 7.2 box, make sure to use spice graphics device
3. migrate the VM to RHEL 7.1 box

Actual results:
migration fails

Expected results:
migration succeeds

Additional info:
Migration from 7.1 to 7.1 and from 7.2 to 7.2 reportedly works as expected.
I dug in history, and this should be caused by
http://libvirt.org/git/?p=libvirt.git;a=commit;h=699299419b4d079a57fc7faa5e819403ba198381
which was merged during the 1.2.13 cycle, and fixed shortly later by commit 
http://libvirt.org/git/?p=libvirt.git;a=commit;h=8c45e8bec887fba6a5b0f8cc5c2c9d310cfa3deb

I believe that libvirt >= 1.2.13 does this attribute backfilling and thanks to the second commit can properly handle it. When migrating, it sends backfilled element to an older libvirt which doesn't know how to do that, hence the failure.
Comment 1 Michal Skrivanek 2015-09-22 07:21:21 EDT
note: we suspect the same problem in 6.7 - worth checking
Comment 2 Jiri Denemark 2015-09-22 08:41:08 EDT
Could you provide some more details, such as domain XML and debug logs from hosts?
Comment 3 Francesco Romani 2015-09-22 12:03 EDT
Created attachment 1075901 [details]
libvirt debug logs
Comment 4 Francesco Romani 2015-09-22 12:05:26 EDT
(In reply to Jiri Denemark from comment #2)
> Could you provide some more details, such as domain XML and debug logs from
> hosts?

Sure thing. Please find attached. Domain XML is in the libvirt debug logs (as usual), here's how it will look:

<domain type="kvm" xmlns:ovirt="http://ovirt.org/vm/tune/1.0">
        <name>test7</name>
        <uuid>52a749ab-21fd-4793-9f4c-5589cc287bd3</uuid>
        <memory>4194304</memory>
        <currentMemory>4194304</currentMemory>
        <maxMemory slots="16">4294967296</maxMemory>
        <vcpu current="2">16</vcpu>
        <devices>
                <channel type="unix">
                        <target name="com.redhat.rhevm.vdsm" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/52a749ab-21fd-4793-9f4c-5589cc287bd3.com.redhat.rhevm.vdsm"/>
                </channel>
                <channel type="unix">
                        <target name="org.qemu.guest_agent.0" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/52a749ab-21fd-4793-9f4c-5589cc287bd3.org.qemu.guest_agent.0"/>
                </channel>
                <input bus="ps2" type="mouse"/>
                <console type="unix">
                        <source mode="bind" path="/var/run/ovirt-vmconsole-console/52a749ab-21fd-4793-9f4c-5589cc287bd3.sock"/>
                        <target port="0" type="serial"/>
                </console>
                <memballoon model="none"/>
                <video>
                        <address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"/>
                        <model heads="1" ram="65536" type="qxl" vram="32768"/>
                </video>
                <graphics autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
                        <listen network="vdsm-ovirtmgmt" type="network"/>
                </graphics>
                <interface type="bridge">
                        <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/>
                        <mac address="00:1a:4a:16:01:52"/>
                        <model type="virtio"/>
                        <source bridge="ovirtmgmt"/>
                        <filterref filter="vdsm-no-mac-spoofing"/>
                        <link state="up"/>
                        <boot order="3"/>
                        <bandwidth/>
                </interface>
                <disk device="cdrom" snapshot="no" type="file">
                        <source file="/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_generic_iso/b0390d9e-2f1d-43c0-a56c-a20af032e934/images/11111111-1111-1111-1111-111111111111/Fedora-Live-Workstation-x86_64-21-5.iso" startupPolicy="optional"/>
                        <target bus="ide" dev="hdc"/>
                        <readonly/>
                        <serial/>
                        <boot order="2"/>
                </disk>
                <disk device="disk" snapshot="no" type="file">
                        <address bus="0x00" domain="0x0000" function="0x0" slot="0x05" type="pci"/>
                        <source file="/rhev/data-center/00000001-0001-0001-0001-00000000008a/11accb73-9afa-4ed4-a6d9-d19a20f842bc/images/ed0a75bf-500a-46d7-be9f-8f8af8272a30/0817cc80-09c3-4b66-beda-d8a8b4a4ce5d"/>
                        <target bus="virtio" dev="vda"/>
                        <serial>ed0a75bf-500a-46d7-be9f-8f8af8272a30</serial>
                        <boot order="1"/>
                        <driver cache="none" error_policy="stop" io="threads" name="qemu" type="raw"/>
                </disk>
                <channel type="spicevmc">
                        <target name="com.redhat.spice.0" type="virtio"/>
                </channel>
                <serial type="unix">
                        <source mode="bind" path="/var/run/ovirt-vmconsole-console/52a749ab-21fd-4793-9f4c-5589cc287bd3.sock"/>
                        <target port="0"/>
                </serial>
        </devices>
        <metadata>
                <ovirt:qos/>
        </metadata>
        <os>
                <type arch="x86_64" machine="pc-i440fx-rhel7.1.0">hvm</type>
                <smbios mode="sysinfo"/>
        </os>
        <sysinfo type="smbios">
                <system>
                        <entry name="manufacturer">oVirt</entry>
                        <entry name="product">oVirt Node</entry>
                        <entry name="version">7.2-7.el7</entry>
                        <entry name="serial">4C4C4544-0058-4410-8043-B5C04F4E5A31</entry>
                        <entry name="uuid">52a749ab-21fd-4793-9f4c-5589cc287bd3</entry>
                </system>
        </sysinfo>
        <clock adjustment="0" offset="variable">
                <timer name="rtc" tickpolicy="catchup"/>
                <timer name="pit" tickpolicy="delay"/>
                <timer name="hpet" present="no"/>
        </clock>
        <features>
                <acpi/>
        </features>
        <cpu match="exact">
                <model>Nehalem</model>
                <topology cores="1" sockets="16" threads="1"/>
                <numa>
                        <cell cpus="0,1" memory="4194304"/>
                </numa>
        </cpu>
        <numatune>
                <memory mode="interleave" nodeset="0"/>
        </numatune>
</domain>
Comment 5 zhe peng 2015-09-23 02:12:31 EDT
I can reproduce this with:
rhel7.2 libvirt-1.2.17-9.el7.x86_64
rhel7.1 libvirt-1.2.8-12.el7.x86_64
step:
1. prepare two host:rhel7.2 and rhel7.1 
2: prepare a guest with spice
 <graphics type='spice' port='5900' autoport='yes'>
      <listen type='network' network='default'/>
    </graphics>
3: do migration from rhel7.2->rhel7.1
get error:
error: XML error: graphics listen attribute $ip must match address attribute of first listen element (found none)
Comment 6 Jiri Denemark 2015-09-23 17:10:28 EDT
Patch sent upstream for review: https://www.redhat.com/archives/libvir-list/2015-September/msg00837.html
Comment 9 Jiri Denemark 2015-09-24 11:14:48 EDT
This is now fixed upstream by v1.2.19-132-gc0806dc:

commit c0806dc30bda562810b0d686e33c903862e3c8f1
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Wed Sep 23 14:48:06 2015 +0200

    domain: Fix migratable XML with graphics/@listen
    
    As of commit 6992994, we set graphics/@listen attribute according to the
    first listen child element even if that element is of type='network'.
    This was done for backward compatibility with applications which only
    support the original listen attribute. However, by doing so we broke
    migration to older libvirt which tried to check that the listen
    attribute matches one of the listen child elements but which did not
    take type='network' elements into account.
    
    We are not concerned about compatibility with old applications when
    formatting domain XML for migration for two reasons. The XML is consumed
    only by libvirtd and the IP address associated with type='network'
    listen address on the source host is just useless on the destination
    host. Thus, we can safely avoid propagating the type='network' IP
    address to graphics/@listen attribute when creating migratable XML.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1265111
    
    Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Comment 11 Fangge Jin 2015-09-25 01:22:16 EDT
Test pass on build 
rhel7.2 libvirt-1.2.17-11.el7_rc.6b1e830c.x86_64
rhel7.1 libvirt-1.2.8-16.el7_1.3.x86_64


1.Prepare two hosts: rhel7.2 and rhel7.1, both have network "default".

2.Prepare a running guest on rhel7.2 with spice graphic ,type='network':
...
    <graphics type='spice' autoport='yes'>
      <listen type='network' network='default'/>
    </graphics>
...

3.Migrate the guest to rhel7.1 host:
# virsh migrate test3 qemu+ssh://10.66.4.227/system --verbose --live
root@10.66.4.227's password: 
Migration: [100 %]

4.Check the guest dumpxml on rhel7.1 host:
...
    <graphics type='spice' port='5900' autoport='yes'>
      <listen type='network' address='192.168.122.1' network='default'/>
    </graphics>
...
And the guest graphic can be connected by remote-viewer:
remote-viewer spice://192.168.122.1:5900

5.Re-do step2~4 with with spice graphic ,type='address', also passed.
Comment 13 Fangge Jin 2015-09-25 06:37:30 EDT
I can reproduce on build libvirt-1.2.17-10.el7.x86_64

Verify pass on build libvirt-1.2.17-11.el7.x86_64

1.As step1~5 in comment11
2.Replace the graphic type with vnc, redo step1.
Comment 15 errata-xmlrpc 2015-11-19 01:54:57 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html

Note You need to log in before you can comment on or make changes to this bug.