Bug 1744523

Summary:	Hotplug hostdev type interface from interface pool will fail as "Attempted double use of PCI Address"
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	yalzhang <yalzhang>
Component:	libvirt	Assignee:	Laine Stump <laine>
Status:	CLOSED ERRATA	QA Contact:	jiyan <jiyan>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.1	CC:	berrange, jdenemar, kchamart, laine, lmen, xuzhang
Target Milestone:	rc	Keywords:	Regression
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-5.6.0-8.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-02-04 18:28:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1585087

Description yalzhang@redhat.com 2019-08-22 10:34:19 UTC

Description of problem:
Hotplug hostdev type interface from interface pool will fail as "Attempted double use of PCI Address"

Version-Release number of selected component (if applicable):
libvirt-5.6.0-2.module+el8.1.0+4015+63576633.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Define and start a hostdev type network as below:
# virsh net-dumpxml passthrough1
<network>
  <name>passthrough1</name>
  <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
  </forward>
</network>

2. Start a vm, and hotplug an interface from above interface pool, it will fail as below:
# virsh attach-interface rhel  network passthrough1 
error: Failed to attach interface
error: XML error: Attempted double use of PCI Address 0000:09:01.0

# grep error /var/log/libvirt/libvirtd.log
2019-08-22 10:22:29.145+0000: 20376: error : virDomainPCIAddressReserveAddrInternal:854 : XML error: Attempted double use of PCI Address 0000:09:01.0

3. check the vm xml:
# virsh dumpxml rhel | grep "<address type='pci' domain='0x0000' bus='0x09'"
==> no output
# virsh dumpxml rhel | grep "index='9'"
    <controller type='pci' index='9' model='pcie-to-pci-bridge'>

Actual results:
Hotplug interface fail

Expected results:
Hotplug should not fail

Additional info:
1. if add "--model virtio", it will try to attach to pcie slot and report error as well:
# virsh attach-interface rhel  network passthrough1  --model virtio
error: Failed to attach interface
error: XML error: Attempted double use of PCI Address 0000:01:00.0

2. hotplug hostdev type interface succeed and plugged into pcie slot automatically. 
# virsh attach-interface rhel hostdev 0000:03:10:3 --managed 
Interface attached successfully
# virsh attach-interface rhel hostdev 0000:03:10:1 --managed  --model rtl8139
Interface attached successfully
# virsh dumpxml rhel | grep /interface -B8
   
# virsh dumpxml rhel | grep /interface -B9
    </controller>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:c2:29:20'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:e9:2b:36'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
      <model type='rtl8139'/>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </interface>
# virsh dumpxml rhel | egrep "index='8'|index='1'"
    <controller type='pci' index='1' model='pcie-root-port'>
    <controller type='pci' index='8' model='pcie-root-port'>

Comment 1 Daniel Berrangé 2019-09-12 17:27:07 UTC

I bisected upstream and hit this as the cause

commit 01ca4010d86584df502458d66f4a3956cc91ce37
Author: Wang Yechao <wang.yechao255.cn>
Date:   Mon Dec 17 19:30:34 2018 +0800

    qemu: Assign device addresses earlier in qemuDomainAttachNetDevice
    
    If code in the @actualType switch needs to have/know which PCI
    Address is being used, then we must assign it earlier. In particular
    a vhost-user device needs to call qemuDomainSupportsNicdev which
    requires an address to be defined.
    
    Signed-off-by: Wang Yechao <wang.yechao255.cn>
    Reviewed-by: John Ferlan <jferlan>


IIUC, what's happening is that qemuDomainAttachNetDevice reserves the PCI address. Since this is a hostdev NIC, we then call qemuDmoainAttachHostDevice, which also tries to reserve the PCI address, and thus fails.

Comment 3 Laine Stump 2019-10-19 06:21:42 UTC

The bug first appeared in libvirt v5.1.0

Posted a patch that fixes this upstream:

https://www.redhat.com/archives/libvir-list/2019-October/msg01291.html

Comment 4 Laine Stump 2019-11-07 16:38:49 UTC

Pushed upstream. Will be in upstream libvirt 5.10:

commit 47a7b8a96b6343d4af18ef80330f805ef031fe9b (HEAD -> master, origin/master, 
origin/HEAD)
Author: Laine Stump <laine>
Date:   Sat Oct 19 01:58:07 2019 -0400

    qemu: avoid double reservation of PCI address for interface type='hostdev'

Comment 7 jiyan 2019-11-25 10:17:36 UTC

Reproduce this bug on libvirt-5.6.0-7.module+el8.1.1+4483+2f45aaa2.x86_64.

Version:
libvirt-5.6.0-7.module+el8.1.1+4483+2f45aaa2.x86_64
qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f.x86_64
kernel-4.18.0-147.0.3.el8_1.x86_64

Steps:
# virsh net-list 
 Name           State    Autostart   Persistent
-------------------------------------------------
 passthrough1   active   no          yes

# virsh net-dumpxml passthrough1 
<network>
  <name>passthrough1</name>
  <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x5'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x7'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x5'/>
  </forward>
</network>

# virsh domstate test
shut off

# virsh start test
Domain test started

# virsh attach-interface test network passthrough1
error: Failed to attach interface
error: XML error: Attempted double use of PCI Address 0000:02:03.0

# virsh attach-interface test network passthrough1 --model virtio
error: Failed to attach interface
error: XML error: Attempted double use of PCI Address 0000:05:00.0

Verify this bug on libvirt-5.6.0-8.module+el8.1.1+4828+0f2d78eb.x86_64.

Steps:
# yum update libvirt* -y

# systemctl restart libvirtd

# rpm -qa libvirt
libvirt-5.6.0-8.module+el8.1.1+4828+0f2d78eb.x86_64

# virsh attach-interface test network passthrough1
Interface attached successfully

# virsh dumpxml test |grep "hostdev" -A7
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:34:9a:93'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/>
      </source>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </interface>

# virsh detach-interface --domain test --mac 52:54:00:34:9a:93 --type hostdev
Interface detached successfully

# virsh dumpxml test |grep "hostdev" -A7
No output

# virsh attach-interface test network passthrough1 --model virtio
Interface attached successfully

# virsh dumpxml test |grep "hostdev" -A7
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:40:c1:a8'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/>
      </source>
      <model type='virtio'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </interface>

Comment 8 jiyan 2019-11-25 10:28:50 UTC

During verifying this bug, I found another problem which is I can continue to define net with same name and uuid for many times.
Could you please whether this is expected and should I file a new bug to track it?

Version:
qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f.x86_64
libvirt-5.6.0-8.module+el8.1.1+4828+0f2d78eb.x86_64
kernel-4.18.0-147.0.3.el8_1.x86_64

Steps:
1. There is no other net here and prepare two xml with same name and uuid.
# virsh net-list --all
 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

# cat net.xml 
<network>
  <name>passthrough1</name>
  <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
      <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x1'/>
      <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x3'/>
      <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x5'/>
      <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x7'/>
      <address type="pci" domain='0x0000' bus='0x82' slot='0x11' function='0x1'/>
      <address type="pci" domain='0x0000' bus='0x82' slot='0x11' function='0x3'/>
      <address type="pci" domain='0x0000' bus='0x82' slot='0x11' function='0x5'/>
  </forward>
</network>

# cat net1.xml 
<network>
  <name>passthrough1</name>
  <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
      <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x1'/>
  </forward>
</network>

2. Define net from the first xml file
# virsh net-define net1.xml 
Network passthrough1 defined from net1.xml

# virsh net-start passthrough1 
Network passthrough1 started

# virsh net-list --all
 Name           State    Autostart   Persistent
-------------------------------------------------
 default        active   yes         yes
 passthrough1   active   no          yes

# virsh net-dumpxml passthrough1 
<network>
  <name>passthrough1</name>
  <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x5'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x7'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x5'/>
  </forward>
</network>

3. Define net from the second xml file
# virsh net-define net1.xml 
Network passthrough1 defined from net1.xml

# virsh net-start passthrough1 
error: Failed to start network passthrough1
error: Requested operation is not valid: network is already active

# virsh net-destroy passthrough1 
Network passthrough1 destroyed

# virsh net-start passthrough1 
Network passthrough1 started

# virsh net-list --all
 Name           State    Autostart   Persistent
-------------------------------------------------
 default        active   yes         yes
 passthrough1   active   no          yes

# virsh net-dumpxml passthrough1  (No need to undefine the firt definition, and i can see the following result.)
<network>
  <name>passthrough1</name>
  <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/>
  </forward>
</network>

Comment 9 Laine Stump 2019-11-25 19:29:51 UTC

(In reply to jiyan from comment #8)
> During verifying this bug, I found another problem which is I can continue
> to define net with same name and uuid for many times.
> Could you please whether this is expected and should I file a new bug to
> track it?

That is proper behavior. You're not "definite a new network", you are "re-defining the same network". This is the documented way to modify the network definition.

(if you were able to define another network with the same name and different uuid, or different name and same uuid, then *that* would be a bug)

Comment 10 Laine Stump 2019-11-25 19:30:26 UTC

s/definite/defining/

sigh.

Comment 12 errata-xmlrpc 2020-02-04 18:28:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0404