Bug 1045002

Summary:	failure to add new PCI passthrough device after restarting libvirtd due to "Duplicate ID"
Product:	Red Hat Enterprise Linux 7	Reporter:	Laine Stump <laine>
Component:	libvirt	Assignee:	Laine Stump <laine>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0	CC:	acathrow, dyuan, gsun, honzhang, jmiao, laine, mzhan, xuzhang
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-1.1.1-18.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-06-13 09:49:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Laine Stump 2013-12-19 12:37:19 UTC

If a domain has an <interface type='hostdev'> or an <interface type='network'> where the network itself is a pool of hostdev devices, then libvirt will internally keep that device on both the interface list *and* the hostdev list for the domain. One of the places this comes in handy is when a new device is being added and libvirt wants to find a unique "alias" name for it - it just scans through the hostdev array and makes sure it picks a name that doesn't match the alias of any device in that array.

However, due to a bug in libvirt's domain XML parser, when libvirtd is restarted, if there is an <interface type='network'> with the network being a hostdev pool, the device will not be added to the reconstructed internal hostdev array, so its alias will not be found during a scan of the hostdev array, thus attempts to add a new hostdev (or <interface type='hostdev'> or <interface type='network'>) will result in this message:

internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device

Comment 1 Laine Stump 2013-12-19 15:32:03 UTC

I posted a patch upstream to fix this:

https://www.redhat.com/archives/libvir-list/2013-December/msg01096.html

Comment 2 Xuesong Zhang 2013-12-20 10:07:16 UTC

I can reproduce partly of this bug. I can't reproduce like the scenario 2, the step6 is also successfully, didn't meet the error.

hi, laine,
Would you please help to check the scenario 2? I can't reproduce this bug while the guest contains "<interface type='hostdev'>". 

builds:
libvirt-1.1.1-16.el7.x86_64
qemu-kvm-rhev-1.5.3-21.el7.x86_64
kernel-devel-3.10.0-64.el7.x86_64

steps:
Scenario1: <interface type='network'> where the network itself is a pool of hostdev devices

1. prepare one hostdev network
# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
  </forward>
</network>

2. prepare one xml like the following one:
# cat vf-vfpool.xml 
<interface type='network'>
 <source network='hostnet'/>
</interface>

3. add the following content to the guest xml:
<interface type='network'>
 <source network='hostnet'/>
</interface>

4. start the guest
# virsh start a
Domain a started

5. hot-plug the 2nd vf to the guest, it is successfully.
# virsh nodedev-detach pci_0000_03_10_0
Device pci_0000_03_10_0 detached

# virsh attach-device a vf-vfpool.xml 
Device attached successfully

6. restart the libvirtd service
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service


7. hot-plug the 3rd vf to the guest, it is failed.
# virsh nodedev-detach pci_0000_03_10_3
Device pci_0000_03_10_3 detached

[root@sriov1 xuzhang]# virsh attach-device a vf-vfpool.xml 
error: Failed to attach device from vf-vfpool.xml
error: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device



Scenario2: <interface type='hostdev'>
1. add the following content to the guest xml:
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
    </interface>

2. start the guest
# virsh start a
Domain a started

3. prepare the following 2 xml
# cat inter-hostdev-vfio.xml 
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
    </interface>

# cat inter-hostdev-vfio-2.xml 
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
      </source>
    </interface>

4. hot-plug 1st vf to the guest
# virsh attach-device a inter-hostdev-vfio.xml 
Device attached successfully

5. restart the libvirtd service
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

6. hot-plug the 2nd vf to the guest, it is also successfully.
# virsh attach-device a inter-hostdev-vfio-2.xml 
Device attached successfully

Comment 3 Laine Stump 2013-12-20 11:09:17 UTC

(In reply to Zhang Xuesong from comment #2)
> I can reproduce partly of this bug. I can't reproduce like the scenario 2,
> the step6 is also successfully, didn't meet the error.
> 
> hi, laine,
> Would you please help to check the scenario 2? I can't reproduce this bug
> while the guest contains "<interface type='hostdev'>". 

That is correct. The "duplicate ID" error message after libvirtd restart only happens if the guest had an <interface type='network'> (where the network is a pool of hostdevs) prior to the restart. Existing <interface type='hostdev'> at libvirtd restart has always been handled correctly.

NB - you will also see a failure to hotplug a new <hostdev> or <interface type='hostdev'> after libvirtd restart, i.e. the type of the *existing* passthrough device is important, not the type of the new device you're hot-plugging.

Comment 4 Xuesong Zhang 2013-12-23 03:11:12 UTC

(In reply to Laine Stump from comment #3)
> (In reply to Zhang Xuesong from comment #2)
> > I can reproduce partly of this bug. I can't reproduce like the scenario 2,
> > the step6 is also successfully, didn't meet the error.
> > 
> > hi, laine,
> > Would you please help to check the scenario 2? I can't reproduce this bug
> > while the guest contains "<interface type='hostdev'>". 
> 
> That is correct. The "duplicate ID" error message after libvirtd restart
> only happens if the guest had an <interface type='network'> (where the
> network is a pool of hostdevs) prior to the restart. Existing <interface
> type='hostdev'> at libvirtd restart has always been handled correctly.
> 
> NB - you will also see a failure to hotplug a new <hostdev> or <interface
> type='hostdev'> after libvirtd restart, i.e. the type of the *existing*
> passthrough device is important, not the type of the new device you're
> hot-plugging.

While I hot-plug one existing passthrough devcie to the guest, the error message in step 5 seems correct, is my scenario correct?

Here is the steps:
1. add the following xml to the guest
# cat inter-hostdev-vfio-0.xml 
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
    </interface>

2. start the guest
# virsh start a
Domain a started

3. hot-plug that VF to the guest again, the error message seems as expect.
# virsh attach-device a inter-hostdev-vfio-0.xml 
error: Failed to attach device from inter-hostdev-vfio-0.xml
error: Requested operation is not valid: PCI device 0000:03:10.0 is in use by domain a

4. restart the libvirtd
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

5. hot-plug the VF again, the error message seems as expect.
# virsh attach-device a inter-hostdev-vfio-0.xml 
error: Failed to attach device from inter-hostdev-vfio-0.xml
error: Requested operation is not valid: PCI device 0000:03:10.0 is in use by domain a

Comment 5 Laine Stump 2013-12-23 12:23:09 UTC

(In reply to Zhang Xuesong from comment #4)

> While I hot-plug one existing passthrough devcie to the guest, the error
> message in step 5 seems correct, is my scenario correct?

Yes that's correct - restarting libvirtd does not detach the passthrough device from the guest, so it is still in use. However, that test doesn't exercise the bug that I reported here - that requires you to:

1) have an <interface type='network'> pointing to a network which is a pool of VFs for passthrough

2) restart libvirtd

3) attempt to hotplug any kind of hostdev:

  a) <hostdev>
  b) <interface type='hostdev'>
  c) <interface type='network'> with a network that is a pool of VFs

The test you've outlines in Comment 4 starts the domain with an <interface type='hostdev'>, so that doesn't test this bug.

Comment 6 Xuesong Zhang 2013-12-24 03:22:11 UTC

(In reply to Laine Stump from comment #5)
> (In reply to Zhang Xuesong from comment #4)
> 
> > While I hot-plug one existing passthrough devcie to the guest, the error
> > message in step 5 seems correct, is my scenario correct?
> 
> Yes that's correct - restarting libvirtd does not detach the passthrough
> device from the guest, so it is still in use. However, that test doesn't
> exercise the bug that I reported here - that requires you to:
> 
> 1) have an <interface type='network'> pointing to a network which is a pool
> of VFs for passthrough
> 
> 2) restart libvirtd
> 
> 3) attempt to hotplug any kind of hostdev:
> 
>   a) <hostdev>
>   b) <interface type='hostdev'>
>   c) <interface type='network'> with a network that is a pool of VFs
> 
> The test you've outlines in Comment 4 starts the domain with an <interface
> type='hostdev'>, so that doesn't test this bug.

Thanks for your quick and details reply, yeah, I can reproduce this bug now.

Steps:
1. prepare one hostdev network
# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
  </forward>
</network>

2. add the following xml to guest.
<interface type='network'>
 <source network='hostnet'/>
</interface>

3. prepare the following 3 type vf for hot-plug.
# cat vf-vfpool.xml 
<interface type='network'>
 <source network='hostnet'/>
</interface>

# cat vf-hostdev-vfio.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address bus='0x03' slot='0x10' function='0x0'/>
  </source>
</hostdev>

# cat inter-hostdev-vfio-0.xml 
<interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
    </interface>

4. start the guest
# virsh start a
Domain a started

5. restart the libvirtd service
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

6. hot-plug the 3 type vf to the guest, it will report error.
# virsh nodedev-detach pci_0000_03_10_0
Device pci_0000_03_10_0 detached

# virsh attach-device a vf-vfpool.xml 
error: Failed to attach device from vf-vfpool.xml
error: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device

# virsh attach-device a vf-hostdev-vfio.xml 
error: Failed to attach device from vf-hostdev-vfio.xml
error: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device

# virsh attach-device a inter-hostdev-vfio-0.xml 
error: Failed to attach device from inter-hostdev-vfio-0.xml
error: internal error: unable to execute QEMU command 'device_add': Duplicate ID 'hostdev0' for device

Comment 9 Jincheng Miao 2014-01-10 08:19:11 UTC

in latest libvirt-1.1.1-18.el7.x86_64, the ID of interface assigned to guest will be set depend on the configuration after libvirtd restarted. 

This is the verification steps for three types of interface hot-plugged ("<interface type='network'>", "<hostdev/>", "<interface type='hostdev'>"):

# rpm -q libvirt
libvirt-1.1.1-18.el7.x86_64

1. prepare one hostdev network
# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/>
  </forward>
</network>

2. add a interface in guest xml, and start it
# virsh edit r7
    <interface type='network'>
      <source network='hostnet'/>
    </interface>

# virsh start r7

3. test for "<interface type='network'>"
# cat vf-vfpool.xml 
<interface type='network'>
  <source network='hostnet'/>
  <mac address='52:54:00:0e:09:6d'/>
</interface>

# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

# virsh attach-device r7 vf-vfpool.xml 
Device attached successfully

the new interface is hostdev1
# virsh dumpxml r7 | grep hostdev1
      <alias name='hostdev1'/>

# virsh detach-device r7 vf-vfpool.xml 
Device detached successfully

4. test for "<hostdev/>"
# cat vf-hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <mac address='52:54:00:0e:09:6d'/>
  <source>
    <address bus='0x03' slot='0x10' function='0x0'/>
  </source>
</hostdev>

# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

# virsh attach-device r7 vf-hostdev.xml 
Device attached successfully

# virsh dumpxml r7 | grep hostdev1
      <alias name='hostdev1'/>

# virsh detach-device r7 vf-hostdev.xml 
Device detached successfully


5. test for "<interface type='hostdev'>"
# cat inter-hostdev-vfio.xml 
<interface type='hostdev'>
  <driver name='vfio'/>
  <mac address='52:54:00:0e:09:6d'/>
  <source>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
  </source>
</interface>

# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

# virsh attach-device r7 inter-hostdev-vfio.xml
Device attached successfully

# virsh dumpxml r7 | grep hostdev1
      <alias name='hostdev1'/>

# virsh detach-device r7 inter-hostdev-vfio.xml
Device detached successfully

So, the "Duplicate ID" error is fixed in this patch, and I change the status to VERIFIED.

Comment 10 Laine Stump 2014-01-10 09:15:56 UTC

(In reply to Jincheng Miao from comment #9)

> 4. test for "<hostdev/>"
> # cat vf-hostdev.xml 
> <hostdev mode='subsystem' type='pci' managed='yes'>
>   <driver name='vfio'/>
>   <mac address='52:54:00:0e:09:6d'/>
>   <source>
>     <address bus='0x03' slot='0x10' function='0x0'/>
>   </source>
> </hostdev>

It makes no difference to your test, but just fyi, the <mac> element is ignored in a <hostdev> device. That was the main reason for adding <interface type='hostdev'>.

Comment 11 Jincheng Miao 2014-01-13 03:44:09 UTC

(In reply to Laine Stump from comment #10)
> (In reply to Jincheng Miao from comment #9)
> 
> > 4. test for "<hostdev/>"
> > # cat vf-hostdev.xml 
> > <hostdev mode='subsystem' type='pci' managed='yes'>
> >   <driver name='vfio'/>
> >   <mac address='52:54:00:0e:09:6d'/>
> >   <source>
> >     <address bus='0x03' slot='0x10' function='0x0'/>
> >   </source>
> > </hostdev>
> 
> It makes no difference to your test, but just fyi, the <mac> element is
> ignored in a <hostdev> device. That was the main reason for adding
> <interface type='hostdev'>.

Thanks Laine, I thought <hostdev> device would be affected as <interface type='hostdev'>, and I forgot that the mac of <hostdev> should be the same as passthroughed device.

Comment 12 Ludek Smid 2014-06-13 09:49:36 UTC

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.