Bug 1186142

Summary: Fail to Migrate with Bridged network, eth + macvtap ,with different interface name on two hosts
Product: Red Hat Enterprise Linux 6 Reporter: Laine Stump <laine>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.6CC: dyuan, honzhang, jdenemar, jkachuck, jsvarova, lhuang, lmiksik, mjenner, mzhan, pzhang, rbalakri, rhodain, tlavigne, virt-bugs, xuzhang, yanyang
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.10.2-51.el6 Doc Type: Bug Fix
Doc Text:
The interface configuration of any libvirt domain which was of type='network' and referenced an "unmanaged" libvirt network had incorrect XML data for the interface transmitted during a migration, containing the "status" of the interface instead of the name of the network to use ("configuration"). As a consequence, the migration destination tried to set up the domain network interface using the status information from the source of the migration, and the migration thus failed. With this update, libvirt sends the configuration data for each device during migration rather than the status data, and the migration of a domain using interfaces of type='network' now succeeds.
Story Points: ---
Clone Of: 1177194
: 1203367 (view as bug list) Environment:
Last Closed: 2015-07-22 05:48:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1177194    
Bug Blocks: 1203367    

Description Laine Stump 2015-01-27 07:36:16 UTC
Bug 1064831 backported upstream commit 7d5bf484747979ce842fea9ae3ae673ab09bf935 to RHEL6. This was the patch that cause Bug 1177194 in RHEL7. I haven't tested it, but based on the above information, RHEL6 will have the same bug, beginning with RHEL6.6.

+++ This bug was initially created as a clone of Bug #1177194 +++

description of problem:
Fail to Migrate with Bridged network, eth + macvtap ,with different interface name on two hosts 

Version-Release number of selected component (if applicable):
libvirt-1.2.8-11.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
prepare two host .On both source and target host create a new network .
NOTE , source and target host has different interface name .

1.on source host .

1.1 define and start a new network
#cat vepa-network.xml

<network>
  <name>vepa-net</name>
  <forward dev='eno1' mode='vepa'>
    <interface dev='eno1'/>
  </forward>
</network>

# virsh net-define vepa-network.xml
# virsh net-start vepa-net
# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     no            yes
 vepa-net             active     no            yes


1.2 start a guest with the following interface using vepa-net:
......
<interface type='network'>
  <mac address='52:54:00:1b:6f:e5'/>
  <source network='vepa-net'/>
  <target dev='vnet0'/>
  <model type='virtio'/>
  <alias name='net0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
....

2. on the target host .

2.1 define and start a new network ,using the same network name as source host
#cat vepa-network.xml

<network>
  <name>vepa-net</name>  <==== note  same network name
  <forward dev='enp2s0' mode='vepa'>
    <interface dev='enp2s0'/>
  </forward>
</network>

# virsh net-define vepa-network.xml
# virsh net-start vepa-net
# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     no            yes
 vepa-net             active     no            yes


3. migrate

# virsh migrate --live r7 qemu+ssh://$ip/system
error: Unable to get index for interface eno1: No such device

Actual results:

Fail to migrate .
error: Unable to get index for interface eno1: No such device
      
Expected results:
Guest can migrate successfully .

additional info:
after start guest , dumpxml :
# virsh dumpxml r7 | grep interface -A 9
    <interface type='direct'>   <==== it change to direct
      <mac address='52:54:00:15:f5:12'/>
      <source network='vepa-net' dev='eno1' mode='vepa'/> <==and set the dev
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

this issue alao can produce when using private mode .

--- Additional comment from Pei Zhang on 2014-12-25 02:41:02 EST ---

additional info :
On RHEL7.0 . It can migrate successfully and the domain XML won't change between active and inactive status. 

version :
libvirt-1.1.1-29.el7_0.4.x86_64
qemu-kvm-rhev-1.5.3-60.el7_0.11.x86_64
kernel-3.10.0-123.el7.x86_64

domain XML :

active domain xml:

# virsh dumpxml r7 | grep interface -A 8
    <interface type='network'>   <=== It will not be changed to direct 
      <mac address='52:54:00:d9:99:2c'/>
      <source network='vepa-net'/>  <===  no extra dev info  
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

inactive domain xml :

# virsh dumpxml r7  --inactive | grep interface -A 8
    <interface type='network'>
      <mac address='52:54:00:d9:99:2c'/>
      <source network='vepa-net'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

--- Additional comment from Jiri Denemark on 2015-01-05 05:04:19 EST ---

Patch sent upstream for review: https://www.redhat.com/archives/libvir-list/2014-December/msg01013.html

--- Additional comment from Jiri Denemark on 2015-01-07 07:17:12 EST ---

Fixed upstream by v1.2.11-109-gdb19a4a:

commit db19a4a3c6f72e464bc3ad00e904d8df13887382
Author: Luyao Huang <lhuang>
Date:   Thu Dec 25 11:38:00 2014 +0800

    conf: Don't format actual network definition in migratable XML
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1177194
    
    When migrate a vm, we will generate a xml via qemuDomainDefFormatLive and
    pass this xml to target libvirtd. Libvirt will use the current network
    state in def->data.network.actual to generate the xml, this will make
    migrate failed when we set a network type guest interface use a macvtap
    network as a source in a vm then migrate vm to another host(which has the
    different macvtap network settings: different interface name, bridge name...)
    
    Add a flag check in virDomainNetDefFormat, if we set a VIR_DOMAIN_XML_MIGRATABLE
    flag when call virDomainNetDefFormat, we won't get the current vm interface
    state.
    
    Signed-off-by: Luyao Huang <lhuang>
    Signed-off-by: Jiri Denemark <jdenemar>

--- Additional comment from hongming on 2015-01-08 03:47:20 EST ---

Verify it as follows.The result is expected .Move its status to VERIFIED.

# rpm -q libvirt
libvirt-1.2.8-12.el7.x86_64

On source host
# virsh net-dumpxml vepa-net
<network connections='1'>
  <name>vepa-net</name>
  <uuid>78a038e7-dab9-4ce6-b63e-054031a22408</uuid>
  <forward dev='enp2s0' mode='vepa'>
    <interface dev='enp2s0' connections='1'/>
  </forward>
</network>

# virsh dumpxml r7a|grep interface -A6
    <interface type='direct'>
      <mac address='52:54:00:0a:e5:15'/>
      <source network='vepa-net' dev='enp2s0' mode='vepa'/>
      <target dev='macvtap0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>



# virsh migrate --live r7a qemu+ssh://$targetip/system --persistent
root.4.135's password: 


On target host
# virsh list |grep r7a
 48    r7a                            running


# virsh net-dumpxml vepa-net
<network connections='1'>
  <name>vepa-net</name>
  <uuid>db38469f-bb10-44e5-8ffe-384ddfec61bb</uuid>
  <forward dev='eno1' mode='vepa'>
    <interface dev='eno1' connections='1'/>
  </forward>
</network>


# virsh dumpxml r7a|grep interface -A6
    <interface type='direct'>
      <mac address='52:54:00:0a:e5:15'/>
      <source network='vepa-net' dev='eno1' mode='vepa'/>
      <target dev='macvtap1'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

# virsh dumpxml r7a --migratable|grep interface -A6
    <interface type='network'>
      <mac address='52:54:00:0a:e5:15'/>
      <source network='vepa-net'/>
      <target dev='macvtap1'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


The guest also can successfully migrated in the opposite direction.

Comment 4 Hu Jianwei 2015-03-09 14:23:03 UTC
I can reproduce it with below version.
[root@intel-5504-12-1 ~]# rpm -q libvirt
libvirt-0.10.2-46.el6.x86_64

[root@intel-5504-12-1 ~]# virsh migrate --live r65 qemu+ssh://10.66.82.225/system
The authenticity of host '10.66.82.225 (10.66.82.225)' can't be established.
RSA key fingerprint is 63:29:99:ca:cd:9e:7b:1f:0a:41:85:18:9e:e4:b1:f4.
Are you sure you want to continue connecting (yes/no)? yes
root.82.225's password: 
error: Unable to get index for interface eth0: No such device

Verified as below, the bug has been fixed.

[On the source]:
[root@intel-5504-12-1 libvirt]# rpm -q libvirt
libvirt-0.10.2-50.el6.x86_64

[root@intel-5504-12-1 libvirt]# virsh net-dumpxml vepa-net
<network connections='1'>
  <name>vepa-net</name>
  <uuid>9c2ff4c5-174c-0c26-a6aa-e6787c426680</uuid>
  <forward dev='eth0' mode='vepa'>
    <interface dev='eth0' connections='1'/>
  </forward>
</network>
[root@intel-5504-12-1 libvirt]# virsh dumpxml r65 | grep "/interface" -B6
    <interface type='direct'>
      <mac address='52:54:00:86:37:6f'/>
      <source dev='eth0' mode='vepa'/>
      <target dev='macvtap0'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
[root@intel-5504-12-1 libvirt]# virsh migrate --live r65 qemu+ssh://10.66.82.225/system --verbose
The authenticity of host '10.66.82.225 (10.66.82.225)' can't be established.
RSA key fingerprint is 63:29:99:ca:cd:9e:7b:1f:0a:41:85:18:9e:e4:b1:f4.
Are you sure you want to continue connecting (yes/no)? yes
root.82.225's password: 
Migration: [100 %]

[On the destination]:
[root@amd-6168-256-1 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     r65                            running
[root@amd-6168-256-1 ~]# virsh dumpxml r65 | grep "/interface" -B6
    <interface type='direct'>
      <mac address='52:54:00:86:37:6f'/>
      <source dev='hjw0' mode='vepa'/>
      <target dev='macvtap0'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
[root@amd-6168-256-1 ~]# virsh net-dumpxml vepa-net
<network connections='1'>
  <name>vepa-net</name>
  <uuid>556bb118-5724-c439-7200-19b3622c88ec</uuid>
  <forward dev='hjw0' mode='vepa'>
    <interface dev='hjw0' connections='1'/>
  </forward>
</network>

We can get expected results, changed to Verified.

Comment 10 Joseph Kachuck 2015-03-18 17:34:41 UTC
Hello,
This bug has been copied as 6.6 z-stream (EUS) bug #1203367

Thank You
Joe Kachuck

Comment 11 Laine Stump 2015-03-19 13:57:35 UTC
I just posted another patch which is useful for debugging the type of problems encountered in this bug:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2015-March/msg00615.html

Comment 12 Hu Jianwei 2015-03-23 06:34:42 UTC
I only verified the improved error message from comment 11 during hitting some errors.

[On old version]:

(gdb) info  b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00007fe34e263300 in virNetDevMacVLanCreate at util/virnetdevmacvlan.c:97
	breakpoint already hit 1 time
(gdb) 
152	    if (virNetlinkCommand(nl_msg, &recvbuf, &recvbuflen, 0, 0,
(gdb) 
157	    if (recvbuflen < NLMSG_LENGTH(0) || recvbuf == NULL)
(gdb) 
162	    switch (resp->nlmsg_type) {
(gdb) set resp->nlmsg_type = 0x2
(gdb) n
165	        if (resp->nlmsg_len < NLMSG_LENGTH(sizeof(*err)))
(gdb) 
168	        switch (err->error) {
(gdb) set err.error = 3
(gdb) n
178	            virReportSystemError(-err->error,
(gdb) c
Continuing.
C

[root@hongming ~]# virsh start rhel6.5
error: Failed to start domain rhel6.5
error: error creating macvtap type of interface: Unknown error 18446744073709551613


[On fixed version]:
we can get a more clear message from libvirtd:

[root@hongming libvirt]# rpm -q libvirt 
libvirt-0.10.2-51.el6.x86_64

153	    if (virNetlinkCommand(nl_msg, &recvbuf, &recvbuflen, 0, 0,
(gdb) 
158	    if (recvbuflen < NLMSG_LENGTH(0) || recvbuf == NULL)
(gdb) 
163	    switch (resp->nlmsg_type) {
(gdb) set resp->nlmsg_type = 0x2
(gdb) n
166	        if (resp->nlmsg_len < NLMSG_LENGTH(sizeof(*err)))
(gdb) 
169	        switch (err->error) {
(gdb) set err.error = 3
(gdb) n
179	            virReportSystemError(-err->error,
(gdb) c
Continuing.

[root@hongming libvirt]# virsh start rhel6.5
error: Failed to start domain rhel6.5
error: error creating macvtap interface macvtap1@eth0 (52:54:00:F5:C6:22): Unknown error 18446744073709551613

I think we get an expected results, changed to Verified.

Comment 13 Laine Stump 2015-04-05 16:00:25 UTC
Is it necessary to include in the Doc Text that the described problem only existed in RHEL6.6 libvirt, but not in any earlier versions?

Comment 15 errata-xmlrpc 2015-07-22 05:48:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1252.html