Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1300843

Summary: Libvirt should not reduce the connections value on hostdev network after attaching interface failed
Product: Red Hat Enterprise Linux 6 Reporter: Laine Stump <laine>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.7CC: acathrow, dallan, dyuan, honzhang, jiahu, jishao, laine, mzhan, rbalakri, shyu, virt-bugs
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-0.10.2-57.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1020135 Environment:
Last Closed: 2016-05-10 19:26:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Laine Stump 2016-01-21 22:05:08 UTC 
+++ This bug was initially created as a clone of Bug #1020135 +++

Description of problem:
Libvirt should not reduce the connections value on hostdev network after attaching interface failed

Version-Release number of selected component (if applicable):
libvirt-1.1.1-9.el7.x86_64
qemu-kvm-rhev-1.5.3-9.el7.x86_64
kernel-3.10.0-33.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Add four VF interfaces to VF pool network "hostnet"
[root@sriov2 ~]# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x3'/>
  </forward>
</network>

2. Cold-plug two "hostnet" network interfaces to domain
[root@sriov2 ~]# virsh dumpxml r7 | grep interface -A10
    <interface type='network'>
      <mac address='52:54:00:8e:2b:e9'/>
      <source network='hostnet'/>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:01:97:b7'/>
      <source network='hostnet'/>
      <model type='rtl8139'/>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>

3. Start domain and check connections of "hostnet", the connections is 2.
[root@sriov2 ~]# virsh start r7
Domain r7 started

[root@sriov2 ~]#
[root@sriov2 ~]# virsh net-dumpxml hostnet
<network connections='2'>                         <=========it's 2.
  <name>hostnet</name>
...

4. Attach the rest of the VFs using attach-interface, check the connections of "hostnet", it's 4.
[root@sriov2 ~]# virsh attach-interface r7 network hostnet
Interface attached successfully
[root@sriov2 ~]# virsh net-dumpxml hostnet
<network connections='4'>
  <name>hostnet</name>
...

5. Due to there are only 4 VFs in "hostnet" network, we just cold-plug/hot-plug four times. 
[root@sriov2 ~]# virsh attach-interface r7 network hostnet
error: Failed to attach interface
error: internal error: network 'hostnet' requires exclusive access to interfaces, but none are available

[root@sriov2 ~]# virsh net-dumpxml hostnet
<network connections='3'>                                              <===== should be 4, do not reduce to 3.
  <name>hostnet</name>
...

6. Continue to exectute attach-interface more than three times, the value of connections lost in "hostnet" network.
[root@sriov2 ~]# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>                                             <===== should be 4, do not reduce to 0.                
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
...

Actual results:
See step 5 and step 6 for detail.

Expected results:
The connections value should be the same as before when failing to attach interface.

--- Additional comment from Laine Stump on 2013-10-18 06:14:38 EDT ---

I can see where the problem is, but need to think awhile about the cleanest way to fix it.

Explanation:

qemuDomainAttachNetDevice() calls networkAllocateActualDevice() to allocate a device from the pool. If that call fails, qemuDomainAttachNetDevice() goes to its cleanup: label, and ends up calling networkReleaseActualDevice(). The problem is that when networkReleaseActualDevice() sees that the netdev's "actual" pointer is NULL, it says "oh, so there is nothing to release!" and jumps down to its success: label, where the connections count for the network is decremented. Most of the time this is what should be done, but there are times when networkAllocateActualDevice() has failed but networkReleaseActualDevice() is still called as part of the error cleanup, and in that case the count *shouldn't* be decremented.

--- Additional comment from Laine Stump on 2013-11-06 06:49:34 EST ---

A fix has been pushed upstream:

commit b4e0299d4ff059c8707a760b7ec2063ccd57cc21
Author: Laine Stump <laine>
Date:   Mon Nov 4 17:01:17 2013 +0200

    network: fix connections count in case of allocate failure

--- Additional comment from hongming on 2013-11-13 03:53:26 EST ---

Verify it as follows. The result is expected. 

Version
libvirt-1.1.1-12.el7.x86_64

[root@sriov2 ~]# virsh net-start hostnet
Network hostnet started

[root@sriov2 ~]# virsh attach-interface r7 network hostnet
Interface attached successfully

[root@sriov2 ~]# virsh attach-interface r7 network hostnet
Interface attached successfully

[root@sriov2 ~]# virsh attach-interface r7 network hostnet
Interface attached successfully

[root@sriov2 ~]# virsh attach-interface r7 network hostnet
Interface attached successfully

[root@sriov2 ~]# virsh net-dumpxml hostnet
<network connections='4'>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x6'/>
  </forward>
</network>

[root@sriov2 ~]# virsh attach-interface r7 network hostnet
error: Failed to attach interface
error: internal error: network 'hostnet' requires exclusive access to interfaces, but none are available

[root@sriov2 ~]# virsh net-dumpxml hostnet
<network connections='4'>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x6'/>
  </forward>
</network>

[root@sriov2 ~]# virsh detach-interface r7 network --mac 52:54:00:78:02:07
Interface detached successfully

[root@sriov2 ~]# virsh net-dumpxml hostnet
<network connections='3'>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x6'/>
  </forward>
</network>

--- Additional comment from Jingjing Shao on 2016-01-18 04:18:09 EST ---

Hi Laine,
The bug can be reproduced rhel6.7 with libvirt-0.10.2-55.el6.x86_64
Is it need be fixed on rhel6.8 or not?

--- Additional comment from Laine Stump on 2016-01-18 10:47:23 EST ---

Yes, and as a matter of fact I had backported the patch locally last week as part of a scratch build, and it's on my list to clone the bug to RHEL6 this week.
Comment 4 Jingjing Shao 2016-02-18 06:01:18 UTC 
Verify it as follows. The result is expected. Move its status to VERIFIED.

[root@sriov1 Desktop]# rpm -q libvirt
libvirt-0.10.2-57.el6.x86_64

[root@sriov1 jishao]# virsh dumpxml r6| grep interface -A 10
    <interface type='network'>
      <mac address='52:54:00:46:34:01'/>
      <source network='hostnet'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:b2:d9:cc'/>
      <source network='hostnet'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:6c:b8:f3'/>
      <source network='hostnet'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>

[root@sriov1 jishao]# virsh start r6
Domain r6 started

[root@sriov1 jishao]# virsh net-dumpxml hostnet
<network connections='3'>
  <name>hostnet</name>
  <uuid>3e4658a7-164b-82a6-d91c-97570f0cbc09</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
  </forward>
</network>

[root@sriov1 jishao]# virsh attach-interface r6 network hostnet
Interface attached successfully

[root@sriov1 jishao]# virsh attach-interface r6 network hostnet
error: Failed to attach interface
error: internal error network 'hostnet' requires exclusive access to interfaces, but none are available

[root@sriov1 jishao]# virsh net-dumpxml hostnet
<network connections='4'>
  <name>hostnet</name>
  <uuid>3e4658a7-164b-82a6-d91c-97570f0cbc09</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
  </forward>
</network>

[root@sriov1 jishao]# virsh destroy r6

[root@sriov1 jishao]# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>3e4658a7-164b-82a6-d91c-97570f0cbc09</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
  </forward>
</network>

[root@sriov1 jishao]# virsh dumpxml r6| grep interface -A 10
[root@sriov1 jishao]#

[root@sriov1 jishao]# virsh attach-interface r6 network hostnet
Interface attached successfully

[root@sriov1 jishao]# virsh attach-interface r6 network hostnet
Interface attached successfully

[root@sriov1 jishao]# virsh attach-interface r6 network hostnet
Interface attached successfully

[root@sriov1 jishao]# virsh net-dumpxml hostnet
<network connections='3'>
  <name>hostnet</name>
  <uuid>3e4658a7-164b-82a6-d91c-97570f0cbc09</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
  </forward>
</network>

[root@sriov1 jishao]# virsh attach-interface r6 network hostnet
Interface attached successfully

[root@sriov1 jishao]# virsh net-dumpxml hostnet
<network connections='4'>
  <name>hostnet</name>
  <uuid>3e4658a7-164b-82a6-d91c-97570f0cbc09</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
  </forward>
</network>

[root@sriov1 jishao]# virsh attach-interface r6 network hostnet
error: Failed to attach interface
error: internal error network 'hostnet' requires exclusive access to interfaces, but none are available

[root@sriov1 jishao]# virsh net-dumpxml hostnet
<network connections='4'>
  <name>hostnet</name>
  <uuid>3e4658a7-164b-82a6-d91c-97570f0cbc09</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
  </forward>
</network>

[root@sriov1 jishao]# virsh detach-interface r6 hostdev --mac 52:54:00:d8:7e:e4
Interface detached successfully

[root@sriov1 jishao]# virsh detach-interface r6 hostdev --mac 52:54:00:23:39:ef
Interface detached successfully

[root@sriov1 jishao]# virsh net-dumpxml hostnet
<network connections='2'>
  <name>hostnet</name>
  <uuid>3e4658a7-164b-82a6-d91c-97570f0cbc09</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
  </forward>
</network>
Comment 6 errata-xmlrpc 2016-05-10 19:26:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0738.html