Bug 1265111

Summary: migration from libvirt 1.2.17 fails with "graphics listen attribute $IP must match address attribute of first listen element (found none)"
Product: Red Hat Enterprise Linux 7 Reporter: Francesco Romani <fromani>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.1CC: dyuan, dzheng, fjin, fromani, gsun, lmiksik, michal.skrivanek, mzhan, rbalakri, zhwang, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.2.17-11.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 06:54:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1154205, 1172230, 1260409    
Attachments:
Description Flags
libvirt debug logs none

Description Francesco Romani 2015-09-22 07:24:29 UTC
Description of problem:
Found testing oVirt 3.6.0. Migrations of a VM using spice graphics device fails with this error

libvirtError: XML error: graphics listen attribute ${IP} must match address attribute of first listen element (found none)

when migrating from libvirt 1.2.17-9 (RHEL 7.2) to 1.2.8-16 (RHEL 7.1)

The other way around, RHEL 7.1 to RHEL 7.2 works.

Version-Release number of selected component (if applicable):
-RHEL 7.2:
libvirt-1.2.17-9.el7.x86_64

-RHEV-H 7.1 for RHEV 3.5.4-1 ASYNC (rhev-hypervisor7-7.1-20150911.0):
libvirt-1.2.8-16.el7_1.3.x86_64

How reproducible:
tried 3-4 times, always happened. Seems 100% reproduceable.

Steps to Reproduce:
1. setup one RHEL 7.2 and one RHEL 7.1 box
2. boot up VM on RHEL 7.2 box, make sure to use spice graphics device
3. migrate the VM to RHEL 7.1 box

Actual results:
migration fails

Expected results:
migration succeeds

Additional info:
Migration from 7.1 to 7.1 and from 7.2 to 7.2 reportedly works as expected.
I dug in history, and this should be caused by
http://libvirt.org/git/?p=libvirt.git;a=commit;h=699299419b4d079a57fc7faa5e819403ba198381
which was merged during the 1.2.13 cycle, and fixed shortly later by commit 
http://libvirt.org/git/?p=libvirt.git;a=commit;h=8c45e8bec887fba6a5b0f8cc5c2c9d310cfa3deb

I believe that libvirt >= 1.2.13 does this attribute backfilling and thanks to the second commit can properly handle it. When migrating, it sends backfilled element to an older libvirt which doesn't know how to do that, hence the failure.

Comment 1 Michal Skrivanek 2015-09-22 11:21:21 UTC
note: we suspect the same problem in 6.7 - worth checking

Comment 2 Jiri Denemark 2015-09-22 12:41:08 UTC
Could you provide some more details, such as domain XML and debug logs from hosts?

Comment 3 Francesco Romani 2015-09-22 16:03:56 UTC
Created attachment 1075901 [details]
libvirt debug logs

Comment 4 Francesco Romani 2015-09-22 16:05:26 UTC
(In reply to Jiri Denemark from comment #2)
> Could you provide some more details, such as domain XML and debug logs from
> hosts?

Sure thing. Please find attached. Domain XML is in the libvirt debug logs (as usual), here's how it will look:

<domain type="kvm" xmlns:ovirt="http://ovirt.org/vm/tune/1.0">
        <name>test7</name>
        <uuid>52a749ab-21fd-4793-9f4c-5589cc287bd3</uuid>
        <memory>4194304</memory>
        <currentMemory>4194304</currentMemory>
        <maxMemory slots="16">4294967296</maxMemory>
        <vcpu current="2">16</vcpu>
        <devices>
                <channel type="unix">
                        <target name="com.redhat.rhevm.vdsm" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/52a749ab-21fd-4793-9f4c-5589cc287bd3.com.redhat.rhevm.vdsm"/>
                </channel>
                <channel type="unix">
                        <target name="org.qemu.guest_agent.0" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/52a749ab-21fd-4793-9f4c-5589cc287bd3.org.qemu.guest_agent.0"/>
                </channel>
                <input bus="ps2" type="mouse"/>
                <console type="unix">
                        <source mode="bind" path="/var/run/ovirt-vmconsole-console/52a749ab-21fd-4793-9f4c-5589cc287bd3.sock"/>
                        <target port="0" type="serial"/>
                </console>
                <memballoon model="none"/>
                <video>
                        <address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"/>
                        <model heads="1" ram="65536" type="qxl" vram="32768"/>
                </video>
                <graphics autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
                        <listen network="vdsm-ovirtmgmt" type="network"/>
                </graphics>
                <interface type="bridge">
                        <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/>
                        <mac address="00:1a:4a:16:01:52"/>
                        <model type="virtio"/>
                        <source bridge="ovirtmgmt"/>
                        <filterref filter="vdsm-no-mac-spoofing"/>
                        <link state="up"/>
                        <boot order="3"/>
                        <bandwidth/>
                </interface>
                <disk device="cdrom" snapshot="no" type="file">
                        <source file="/rhev/data-center/mnt/192.168.1.20:_srv_virtstore_nfs_generic_iso/b0390d9e-2f1d-43c0-a56c-a20af032e934/images/11111111-1111-1111-1111-111111111111/Fedora-Live-Workstation-x86_64-21-5.iso" startupPolicy="optional"/>
                        <target bus="ide" dev="hdc"/>
                        <readonly/>
                        <serial/>
                        <boot order="2"/>
                </disk>
                <disk device="disk" snapshot="no" type="file">
                        <address bus="0x00" domain="0x0000" function="0x0" slot="0x05" type="pci"/>
                        <source file="/rhev/data-center/00000001-0001-0001-0001-00000000008a/11accb73-9afa-4ed4-a6d9-d19a20f842bc/images/ed0a75bf-500a-46d7-be9f-8f8af8272a30/0817cc80-09c3-4b66-beda-d8a8b4a4ce5d"/>
                        <target bus="virtio" dev="vda"/>
                        <serial>ed0a75bf-500a-46d7-be9f-8f8af8272a30</serial>
                        <boot order="1"/>
                        <driver cache="none" error_policy="stop" io="threads" name="qemu" type="raw"/>
                </disk>
                <channel type="spicevmc">
                        <target name="com.redhat.spice.0" type="virtio"/>
                </channel>
                <serial type="unix">
                        <source mode="bind" path="/var/run/ovirt-vmconsole-console/52a749ab-21fd-4793-9f4c-5589cc287bd3.sock"/>
                        <target port="0"/>
                </serial>
        </devices>
        <metadata>
                <ovirt:qos/>
        </metadata>
        <os>
                <type arch="x86_64" machine="pc-i440fx-rhel7.1.0">hvm</type>
                <smbios mode="sysinfo"/>
        </os>
        <sysinfo type="smbios">
                <system>
                        <entry name="manufacturer">oVirt</entry>
                        <entry name="product">oVirt Node</entry>
                        <entry name="version">7.2-7.el7</entry>
                        <entry name="serial">4C4C4544-0058-4410-8043-B5C04F4E5A31</entry>
                        <entry name="uuid">52a749ab-21fd-4793-9f4c-5589cc287bd3</entry>
                </system>
        </sysinfo>
        <clock adjustment="0" offset="variable">
                <timer name="rtc" tickpolicy="catchup"/>
                <timer name="pit" tickpolicy="delay"/>
                <timer name="hpet" present="no"/>
        </clock>
        <features>
                <acpi/>
        </features>
        <cpu match="exact">
                <model>Nehalem</model>
                <topology cores="1" sockets="16" threads="1"/>
                <numa>
                        <cell cpus="0,1" memory="4194304"/>
                </numa>
        </cpu>
        <numatune>
                <memory mode="interleave" nodeset="0"/>
        </numatune>
</domain>

Comment 5 zhe peng 2015-09-23 06:12:31 UTC
I can reproduce this with:
rhel7.2 libvirt-1.2.17-9.el7.x86_64
rhel7.1 libvirt-1.2.8-12.el7.x86_64
step:
1. prepare two host:rhel7.2 and rhel7.1 
2: prepare a guest with spice
 <graphics type='spice' port='5900' autoport='yes'>
      <listen type='network' network='default'/>
    </graphics>
3: do migration from rhel7.2->rhel7.1
get error:
error: XML error: graphics listen attribute $ip must match address attribute of first listen element (found none)

Comment 6 Jiri Denemark 2015-09-23 21:10:28 UTC
Patch sent upstream for review: https://www.redhat.com/archives/libvir-list/2015-September/msg00837.html

Comment 9 Jiri Denemark 2015-09-24 15:14:48 UTC
This is now fixed upstream by v1.2.19-132-gc0806dc:

commit c0806dc30bda562810b0d686e33c903862e3c8f1
Author: Jiri Denemark <jdenemar>
Date:   Wed Sep 23 14:48:06 2015 +0200

    domain: Fix migratable XML with graphics/@listen
    
    As of commit 6992994, we set graphics/@listen attribute according to the
    first listen child element even if that element is of type='network'.
    This was done for backward compatibility with applications which only
    support the original listen attribute. However, by doing so we broke
    migration to older libvirt which tried to check that the listen
    attribute matches one of the listen child elements but which did not
    take type='network' elements into account.
    
    We are not concerned about compatibility with old applications when
    formatting domain XML for migration for two reasons. The XML is consumed
    only by libvirtd and the IP address associated with type='network'
    listen address on the source host is just useless on the destination
    host. Thus, we can safely avoid propagating the type='network' IP
    address to graphics/@listen attribute when creating migratable XML.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1265111
    
    Signed-off-by: Jiri Denemark <jdenemar>

Comment 11 Fangge Jin 2015-09-25 05:22:16 UTC
Test pass on build 
rhel7.2 libvirt-1.2.17-11.el7_rc.6b1e830c.x86_64
rhel7.1 libvirt-1.2.8-16.el7_1.3.x86_64


1.Prepare two hosts: rhel7.2 and rhel7.1, both have network "default".

2.Prepare a running guest on rhel7.2 with spice graphic ,type='network':
...
    <graphics type='spice' autoport='yes'>
      <listen type='network' network='default'/>
    </graphics>
...

3.Migrate the guest to rhel7.1 host:
# virsh migrate test3 qemu+ssh://10.66.4.227/system --verbose --live
root.4.227's password: 
Migration: [100 %]

4.Check the guest dumpxml on rhel7.1 host:
...
    <graphics type='spice' port='5900' autoport='yes'>
      <listen type='network' address='192.168.122.1' network='default'/>
    </graphics>
...
And the guest graphic can be connected by remote-viewer:
remote-viewer spice://192.168.122.1:5900

5.Re-do step2~4 with with spice graphic ,type='address', also passed.

Comment 13 Fangge Jin 2015-09-25 10:37:30 UTC
I can reproduce on build libvirt-1.2.17-10.el7.x86_64

Verify pass on build libvirt-1.2.17-11.el7.x86_64

1.As step1~5 in comment11
2.Replace the graphic type with vnc, redo step1.

Comment 15 errata-xmlrpc 2015-11-19 06:54:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html