RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1045002 - failure to add new PCI passthrough device after restarting libvirtd due to "Duplicate ID"
Summary: failure to add new PCI passthrough device after restarting libvirtd due to "D...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-19 12:37 UTC by Laine Stump
Modified: 2014-06-18 01:01 UTC (History)
8 users (show)

Fixed In Version: libvirt-1.1.1-18.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-13 09:49:36 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Laine Stump 2013-12-19 12:37:19 UTC 
If a domain has an <interface type='hostdev'> or an <interface type='network'> where the network itself is a pool of hostdev devices, then libvirt will internally keep that device on both the interface list *and* the hostdev list for the domain. One of the places this comes in handy is when a new device is being added and libvirt wants to find a unique "alias" name for it - it just scans through the hostdev array and makes sure it picks a name that doesn't match the alias of any device in that array.

However, due to a bug in libvirt's domain XML parser, when libvirtd is restarted, if there is an <interface type='network'> with the network being a hostdev pool, the device will not be added to the reconstructed internal hostdev array, so its alias will not be found during a scan of the hostdev array, thus attempts to add a new hostdev (or <interface type='hostdev'> or <interface type='network'>) will result in this message:

internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device
Comment 1 Laine Stump 2013-12-19 15:32:03 UTC 
I posted a patch upstream to fix this:

https://www.redhat.com/archives/libvir-list/2013-December/msg01096.html
Comment 2 Xuesong Zhang 2013-12-20 10:07:16 UTC 
I can reproduce partly of this bug. I can't reproduce like the scenario 2, the step6 is also successfully, didn't meet the error.

hi, laine,
Would you please help to check the scenario 2? I can't reproduce this bug while the guest contains "<interface type='hostdev'>". 

builds:
libvirt-1.1.1-16.el7.x86_64
qemu-kvm-rhev-1.5.3-21.el7.x86_64
kernel-devel-3.10.0-64.el7.x86_64

steps:
Scenario1: <interface type='network'> where the network itself is a pool of hostdev devices

1. prepare one hostdev network
# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
  </forward>
</network>

2. prepare one xml like the following one:
# cat vf-vfpool.xml 
<interface type='network'>
 <source network='hostnet'/>
</interface>

3. add the following content to the guest xml:
<interface type='network'>
 <source network='hostnet'/>
</interface>

4. start the guest
# virsh start a
Domain a started

5. hot-plug the 2nd vf to the guest, it is successfully.
# virsh nodedev-detach pci_0000_03_10_0
Device pci_0000_03_10_0 detached

# virsh attach-device a vf-vfpool.xml 
Device attached successfully

6. restart the libvirtd service
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service


7. hot-plug the 3rd vf to the guest, it is failed.
# virsh nodedev-detach pci_0000_03_10_3
Device pci_0000_03_10_3 detached

[root@sriov1 xuzhang]# virsh attach-device a vf-vfpool.xml 
error: Failed to attach device from vf-vfpool.xml
error: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device



Scenario2: <interface type='hostdev'>
1. add the following content to the guest xml:
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
    </interface>

2. start the guest
# virsh start a
Domain a started

3. prepare the following 2 xml
# cat inter-hostdev-vfio.xml 
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
    </interface>

# cat inter-hostdev-vfio-2.xml 
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
      </source>
    </interface>

4. hot-plug 1st vf to the guest
# virsh attach-device a inter-hostdev-vfio.xml 
Device attached successfully

5. restart the libvirtd service
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

6. hot-plug the 2nd vf to the guest, it is also successfully.
# virsh attach-device a inter-hostdev-vfio-2.xml 
Device attached successfully
Comment 3 Laine Stump 2013-12-20 11:09:17 UTC 
(In reply to Zhang Xuesong from comment #2)
> I can reproduce partly of this bug. I can't reproduce like the scenario 2,
> the step6 is also successfully, didn't meet the error.
> 
> hi, laine,
> Would you please help to check the scenario 2? I can't reproduce this bug
> while the guest contains "<interface type='hostdev'>". 

That is correct. The "duplicate ID" error message after libvirtd restart only happens if the guest had an <interface type='network'> (where the network is a pool of hostdevs) prior to the restart. Existing <interface type='hostdev'> at libvirtd restart has always been handled correctly.

NB - you will also see a failure to hotplug a new <hostdev> or <interface type='hostdev'> after libvirtd restart, i.e. the type of the *existing* passthrough device is important, not the type of the new device you're hot-plugging.
Comment 4 Xuesong Zhang 2013-12-23 03:11:12 UTC 
(In reply to Laine Stump from comment #3)
> (In reply to Zhang Xuesong from comment #2)
> > I can reproduce partly of this bug. I can't reproduce like the scenario 2,
> > the step6 is also successfully, didn't meet the error.
> > 
> > hi, laine,
> > Would you please help to check the scenario 2? I can't reproduce this bug
> > while the guest contains "<interface type='hostdev'>". 
> 
> That is correct. The "duplicate ID" error message after libvirtd restart
> only happens if the guest had an <interface type='network'> (where the
> network is a pool of hostdevs) prior to the restart. Existing <interface
> type='hostdev'> at libvirtd restart has always been handled correctly.
> 
> NB - you will also see a failure to hotplug a new <hostdev> or <interface
> type='hostdev'> after libvirtd restart, i.e. the type of the *existing*
> passthrough device is important, not the type of the new device you're
> hot-plugging.

While I hot-plug one existing passthrough devcie to the guest, the error message in step 5 seems correct, is my scenario correct?

Here is the steps:
1. add the following xml to the guest
# cat inter-hostdev-vfio-0.xml 
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
    </interface>

2. start the guest
# virsh start a
Domain a started

3. hot-plug that VF to the guest again, the error message seems as expect.
# virsh attach-device a inter-hostdev-vfio-0.xml 
error: Failed to attach device from inter-hostdev-vfio-0.xml
error: Requested operation is not valid: PCI device 0000:03:10.0 is in use by domain a

4. restart the libvirtd
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

5. hot-plug the VF again, the error message seems as expect.
# virsh attach-device a inter-hostdev-vfio-0.xml 
error: Failed to attach device from inter-hostdev-vfio-0.xml
error: Requested operation is not valid: PCI device 0000:03:10.0 is in use by domain a
Comment 5 Laine Stump 2013-12-23 12:23:09 UTC 
(In reply to Zhang Xuesong from comment #4)

> While I hot-plug one existing passthrough devcie to the guest, the error
> message in step 5 seems correct, is my scenario correct?

Yes that's correct - restarting libvirtd does not detach the passthrough device from the guest, so it is still in use. However, that test doesn't exercise the bug that I reported here - that requires you to:

1) have an <interface type='network'> pointing to a network which is a pool of VFs for passthrough

2) restart libvirtd

3) attempt to hotplug any kind of hostdev:

  a) <hostdev>
  b) <interface type='hostdev'>
  c) <interface type='network'> with a network that is a pool of VFs

The test you've outlines in Comment 4 starts the domain with an <interface type='hostdev'>, so that doesn't test this bug.
Comment 6 Xuesong Zhang 2013-12-24 03:22:11 UTC 
(In reply to Laine Stump from comment #5)
> (In reply to Zhang Xuesong from comment #4)
> 
> > While I hot-plug one existing passthrough devcie to the guest, the error
> > message in step 5 seems correct, is my scenario correct?
> 
> Yes that's correct - restarting libvirtd does not detach the passthrough
> device from the guest, so it is still in use. However, that test doesn't
> exercise the bug that I reported here - that requires you to:
> 
> 1) have an <interface type='network'> pointing to a network which is a pool
> of VFs for passthrough
> 
> 2) restart libvirtd
> 
> 3) attempt to hotplug any kind of hostdev:
> 
>   a) <hostdev>
>   b) <interface type='hostdev'>
>   c) <interface type='network'> with a network that is a pool of VFs
> 
> The test you've outlines in Comment 4 starts the domain with an <interface
> type='hostdev'>, so that doesn't test this bug.

Thanks for your quick and details reply, yeah, I can reproduce this bug now.

Steps:
1. prepare one hostdev network
# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
  </forward>
</network>

2. add the following xml to guest.
<interface type='network'>
 <source network='hostnet'/>
</interface>

3. prepare the following 3 type vf for hot-plug.
# cat vf-vfpool.xml 
<interface type='network'>
 <source network='hostnet'/>
</interface>

# cat vf-hostdev-vfio.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address bus='0x03' slot='0x10' function='0x0'/>
  </source>
</hostdev>

# cat inter-hostdev-vfio-0.xml 
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
    </interface>

4. start the guest
# virsh start a
Domain a started

5. restart the libvirtd service
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

6. hot-plug the 3 type vf to the guest, it will report error.
# virsh nodedev-detach pci_0000_03_10_0
Device pci_0000_03_10_0 detached

# virsh attach-device a vf-vfpool.xml 
error: Failed to attach device from vf-vfpool.xml
error: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device

# virsh attach-device a vf-hostdev-vfio.xml 
error: Failed to attach device from vf-hostdev-vfio.xml
error: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device

# virsh attach-device a inter-hostdev-vfio-0.xml 
error: Failed to attach device from inter-hostdev-vfio-0.xml
error: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device
Comment 9 Jincheng Miao 2014-01-10 08:19:11 UTC 
in latest libvirt-1.1.1-18.el7.x86_64, the ID of interface assigned to guest will be set depend on the configuration after libvirtd restarted. 

This is the verification steps for three types of interface hot-plugged ("<interface type='network'>", "<hostdev/>", "<interface type='hostdev'>"):

# rpm -q libvirt
libvirt-1.1.1-18.el7.x86_64

1. prepare one hostdev network
# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
  </forward>
</network>

2. add a interface in guest xml, and start it
# virsh edit r7
    <interface type='network'>
      <source network='hostnet'/>
    </interface>

# virsh start r7

3. test for "<interface type='network'>"
# cat vf-vfpool.xml 
<interface type='network'>
  <source network='hostnet'/>
  <mac address='52:54:00:0e:09:6d'/>
</interface>

# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

# virsh attach-device r7 vf-vfpool.xml 
Device attached successfully

the new interface is hostdev1
# virsh dumpxml r7 | grep hostdev1
      <alias name='hostdev1'/>

# virsh detach-device r7 vf-vfpool.xml 
Device detached successfully

4. test for "<hostdev/>"
# cat vf-hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <mac address='52:54:00:0e:09:6d'/>
  <source>
    <address bus='0x03' slot='0x10' function='0x0'/>
  </source>
</hostdev>

# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

# virsh attach-device r7 vf-hostdev.xml 
Device attached successfully

# virsh dumpxml r7 | grep hostdev1
      <alias name='hostdev1'/>

# virsh detach-device r7 vf-hostdev.xml 
Device detached successfully


5. test for "<interface type='hostdev'>"
# cat inter-hostdev-vfio.xml 
<interface type='hostdev'>
  <driver name='vfio'/>
  <mac address='52:54:00:0e:09:6d'/>
  <source>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
  </source>
</interface>

# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

# virsh attach-device r7 inter-hostdev-vfio.xml
Device attached successfully

# virsh dumpxml r7 | grep hostdev1
      <alias name='hostdev1'/>

# virsh detach-device r7 inter-hostdev-vfio.xml
Device detached successfully

So, the "Duplicate ID" error is fixed in this patch, and I change the status to VERIFIED.
Comment 10 Laine Stump 2014-01-10 09:15:56 UTC 
(In reply to Jincheng Miao from comment #9)

> 4. test for "<hostdev/>"
> # cat vf-hostdev.xml 
> <hostdev mode='subsystem' type='pci' managed='yes'>
>   <driver name='vfio'/>
>   <mac address='52:54:00:0e:09:6d'/>
>   <source>
>     <address bus='0x03' slot='0x10' function='0x0'/>
>   </source>
> </hostdev>

It makes no difference to your test, but just fyi, the <mac> element is ignored in a <hostdev> device. That was the main reason for adding <interface type='hostdev'>.
Comment 11 Jincheng Miao 2014-01-13 03:44:09 UTC 
(In reply to Laine Stump from comment #10)
> (In reply to Jincheng Miao from comment #9)
> 
> > 4. test for "<hostdev/>"
> > # cat vf-hostdev.xml 
> > <hostdev mode='subsystem' type='pci' managed='yes'>
> >   <driver name='vfio'/>
> >   <mac address='52:54:00:0e:09:6d'/>
> >   <source>
> >     <address bus='0x03' slot='0x10' function='0x0'/>
> >   </source>
> > </hostdev>
> 
> It makes no difference to your test, but just fyi, the <mac> element is
> ignored in a <hostdev> device. That was the main reason for adding
> <interface type='hostdev'>.

Thanks Laine, I thought <hostdev> device would be affected as <interface type='hostdev'>, and I forgot that the mac of <hostdev> should be the same as passthroughed device.
Comment 12 Ludek Smid 2014-06-13 09:49:36 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.
Note You need to log in before you can comment on or make changes to this bug.