Bug 1321637

Summary:	libvirt fails to create a macvtap deivce if an attempted name was already created by some process other than libvirt
Product:	Red Hat Enterprise Linux 6	Reporter:	Laine Stump <laine>
Component:	libvirt	Assignee:	Laine Stump <laine>
Status:	CLOSED ERRATA	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.7	CC:	dyuan, jsuchane, laine, mzhan, rbalakri, salmy, sherold, shyu, virt-bugs, xuzhang, yalzhang
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-0.10.2-60.el6	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1321546	Environment:
Last Closed:	2016-05-10 19:26:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Laine Stump 2016-03-28 17:37:08 UTC

The following bug was introduced into the RHEL6 version of libvirt by the final patch in Bug 1276478. This means that there isn't yet a regression in a released build of libvirt, but there will be if this bug isn't resolved before release.

(I removed a proposed (incorrect) patch and related discussion from the transcript below, as it's irrelevant to getting the problem fixed)

+++ This bug was initially created as a clone of Bug #1321546 +++

Description of problem:

libvirt fail to generate correct macvtap name if there is already one 

Version-Release number of selected component (if applicable):

libvirt-1.3.2-1.el7.x86_64

How reproducible:

100%

Steps to Reproduce:

1. Create a macvtap device by ip command named macvtap1

# ip l add li enp2s0  macvtap1  type  macvtap
# ip -d l show macvtap1
6: macvtap1@enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500
    link/ether d2:d9:bb:d7:f6:5b brd ff:ff:ff:ff:ff:ff promiscuity 0 
    macvtap  mode vepa addrgenmode eui64 

2. Start a defined guest with interface type is direct

# virsh dumpxml r7 |grep "interface type" -A4
    <interface type='direct'>
      <mac address='52:54:00:0c:84:4d'/>
      <source dev='enp2s0' mode='vepa'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>

# time virsh start r7 
error: Failed to start domain r7
error: internal error: Too many unreserved macvtap devices in use


real	0m50.638s
user	0m0.013s
sys	0m0.008s


3. Check libvirtd log

# grep virNetDevMacVLanCreateWithVPortProfile /tmp/libvirtd.log |wc -l
8192


2016-03-28 08:35:30.890+0000: 28927: info : virNetDevMacVLanReserveID:166 : reserving device macvtap1
2016-03-28 08:35:30.902+0000: 28927: debug : virFileClose:103 : Closed fd 25
2016-03-28 08:35:30.902+0000: 28927: info : virNetDevMacVLanReleaseID:203 : releasing device macvtap1
2016-03-28 08:35:30.902+0000: 28927: info : virNetDevMacVLanCreateWithVPortProfile:1089 : Device macvtap1 wasn't reserved but already existed, skipping
2016-03-28 08:35:30.902+0000: 28927: info : virNetDevMacVLanReserveID:166 : reserving device macvtap1


Actual results:


Expected results:

guest should started successfully with correct macvtap device, like macvtap2 if there is already macvtap1(which create not by libvirt) existing on host. 

Additional info:

this is a regression of commit 370608b, which introduce bitmap of in-use macvtap devices.



--- Additional comment from Laine Stump on 2016-03-28 13:13:54 EDT ---


The proper fix is to call virBitmapNextClearBit() (inside virNetDevMacVLanReserveID()) with the most recently failed id (or -1) rather than with 0. This is what I've done in the following patch, now pushed upstream:


commit 5b5f12cffa86a1b4527bde620c2a86c5e96ad7c5
Author: Laine Stump <laine>
Date:   Mon Mar 28 10:14:04 2016 -0400

    util: avoid getting stuck on macvtapN name created outside libvirt

Comment 1 Laine Stump 2016-03-28 19:51:09 UTC

The upstream patch has been backported and posted on rhvirt-patches for inclusion in an upcoming build:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2016-March/msg00439.html

Comment 2 Laine Stump 2016-03-28 19:51:53 UTC

(Note that this bug was never in an officially release package, so no Doc Text is needed.)

Comment 3 Mike McCune 2016-03-28 22:54:17 UTC

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 4 Shanzhi Yu 2016-03-29 06:18:26 UTC

Reproduce this with libvirt-0.10.2-59.el6.x86_64.
Steps to reproduce:

1. add macvtap device named macvtap1
# ip link add link eth0 name macvtap1 type macvtap

# ip -d l show macvtap1
10: macvtap1@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
    link/ether 8a:58:b9:cc:28:a9 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    macvtap  mode vepa 

2. start a guest with direct type interface

# virsh dumpxml r7 |grep "interface type" -A4
    <interface type='direct'>
      <mac address='52:54:00:0c:84:4d'/>
      <source dev='eth0' mode='vepa'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

# time virsh start r7 
error: Failed to start domain r7
error: internal error Too many unreserved macvtap devices in use


real	0m0.806s
user	0m0.018s
sys	0m0.019s

Comment 7 yalzhang@redhat.com 2016-03-31 05:59:55 UTC

Verified the bug on libvirt-0.10.2-60.el6.x86_64 as below, the results is as expected.

1. create a macvtap device named macvtap1
# ip link add link eth0 name macvtap1 type macvtap

# ip -d l show macvtap1
37: macvtap1@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
    link/ether da:51:3a:91:7d:4b brd ff:ff:ff:ff:ff:ff promiscuity 0 
    macvtap  mode vepa 

2. start a guest with direct interface
# virsh dumpxml rh6 | grep /interface -B4  
 <interface type='direct'>
      <mac address='52:54:00:34:e5:52'/>
      <source dev='eth0' mode='vepa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

# time virsh start rh6
Domain rh6 started


real	0m0.578s
user	0m0.006s
sys	0m0.009s

# ip link show | grep macvtap
# ip link show | grep macvtap
37: macvtap1@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
39: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500

One more scenario:
1. create a macvtap device named macvtap0
# ip link add link eth0 name macvtap0 type macvtap
# ip link add link eth2 name macvtap3 type macvtap
# ip -d link show macvtap0
83: macvtap0@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
    link/ether 26:32:c2:53:6f:87 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    macvtap  mode vepa 
# ip -d link show macvtap3
84: macvtap3@eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
    link/ether e6:13:b1:d9:4d:85 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    macvtap  mode vepa 

2. start a guest with 4 direct interfaces, the source dev is eth0 and eth2.
# virsh dumpxml rh6 | grep /interface -B5
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:34:e5:52'/>
      <source dev='eth0' mode='vepa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:c9:29:2b'/>
      <source dev='eth0' mode='vepa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:6e:f4:07'/>
      <source dev='eth2' mode='vepa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:2d:21:34'/>
      <source dev='eth2' mode='vepa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>

# time virsh start rh6
Domain rh6 started
real	0m0.920s
user	0m0.007s
sys	0m0.008s

# ip link show | grep macvtap
83: macvtap0@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
84: macvtap3@eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
85: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500
86: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500
87: macvtap4@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500
88: macvtap5@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500

3. log in the guest to check the network, both 4 interfaces can get ip address. And the guest can ping outside.

Comment 8 yalzhang@redhat.com 2016-03-31 13:24:35 UTC

Add more scenarios, all the results is as expected on libvirt-0.10.2-60.el6.x86_64.
From the result, we can see that when libvirt generate a macvtap device, it will check the name from macvtap0, if macvtap0 exists, it will check macvtap1 and so on. 

1. Create macvtap1, macvtap3, macvtap5, then start a guest with direct interface, it will generate macvtap0. Start a 2nd guest, it will be macvtap2, the 3rd, macvtap4, the 4th, macvtap6, and so on.

2. Create 100+ macvtap devices, then start a guest with direct interface, check the time, it is OK.
# time virsh start r6new
Domain r6new started


real	0m0.595s
user	0m0.004s
sys	0m0.013s
# ip link show | grep macvtap
21: macvtap1@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
22: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500
23: macvtap5@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
31: macvtap0@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
....
134: macvtap104@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500 <=== libvirt create the macvtap104 when start the guest.

3. Use macvtap network
# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>b40c3968-d0c7-ad7c-c411-a8b63ff44620</uuid>
  <forward dev='eth0' mode='bridge'>
    <interface dev='eth0'/>
    <interface dev='eth2'/>
  </forward>
</network>
# ip link add link eth0 name macvtap0 type macvtap
# ip link add link eth0 name macvtap3 type macvtap
# ip link show | grep macvtap
135: macvtap0@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
136: macvtap3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
# virsh edit r6
...
   <interface type='network'>
      <source network='direct-macvtap'/>
    </interface>
...
# time virsh start r6
Domain r6 started


real	0m0.594s
user	0m0.010s
sys	0m0.006s

# ip link show | grep macvtap
135: macvtap0@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
136: macvtap3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
141: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500
# virsh attach-interface r6 network direct-macvtap
Interface attached successfully
# ip link show | grep macvtap
135: macvtap0@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
136: macvtap3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500
141: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500
142: macvtap2@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 500

As the test above, change the states to verified.

Comment 10 errata-xmlrpc 2016-05-10 19:26:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0738.html