I guess it is more like a virt-manager bug.

(In reply to min zhan from comment #2)
> I guess it is more like a virt-manager bug.

But it also happens with virsh attach-device

(In reply to Alex Williamson from comment #3)
> (In reply to min zhan from comment #2)
> > I guess it is more like a virt-manager bug.
> 
> But it also happens with virsh attach-device

hi, Alex,

There is one related exist bug, please see the following comments. https://bugzilla.redhat.com/show_bug.cgi?id=1001738#c12

I think maybe your problem is caused by the same reason in that comments, it is that the default driver of nodedev detach is still "pci-stub", not "vfio", The developer is fixing this problem now, after that bug is fixed, I will try your scenario in this bug, and then add comments in this bug

(In reply to Alex Williamson from comment #3)
> (In reply to min zhan from comment #2)
> > I guess it is more like a virt-manager bug.
> 
> But it also happens with virsh attach-device

Yes, I just forgot to mention that we have similar libvirt bug 1001738 as mentioned in Comment 4. So I think this bug could be against virt-manager.

(In reply to Zhang Xuesong from comment #4)

> There is one related exist bug, please see the following comments.
> https://bugzilla.redhat.com/show_bug.cgi?id=1001738#c12
> 
> I think maybe your problem is caused by the same reason in that comments, it
> is that the default driver of nodedev detach is still "pci-stub", not
> "vfio", The developer is fixing this problem now, after that bug is fixed, I
> will try your scenario in this bug, and then add comments in this bug

When the xml contains "managed='yes'", nodedev-detach is not used - that's what the "managed='yes'" does.

(In reply to Laine Stump from comment #6)
> (In reply to Zhang Xuesong from comment #4)
> 
> > There is one related exist bug, please see the following comments.
> > https://bugzilla.redhat.com/show_bug.cgi?id=1001738#c12
> > 
> > I think maybe your problem is caused by the same reason in that comments, it
> > is that the default driver of nodedev detach is still "pci-stub", not
> > "vfio", The developer is fixing this problem now, after that bug is fixed, I
> > will try your scenario in this bug, and then add comments in this bug
> 
> When the xml contains "managed='yes'", nodedev-detach is not used - that's
> what the "managed='yes'" does.

Yeah, I know what the "managed='yes'" means. 
My meaning is that, while the "managed='yes'" is set, libvirt should call the nodedev-detach to change the PCI's driver to "VFIO" automatically, no need to manually change the PCI's driver as expected.
So, I guess the root cause of this maybe same with "https://bugzilla.redhat.com/show_bug.cgi?id=1001738#c12"
After the default driver of command "nodedev-detach" change to "VFIO", not "pci-stub", we will not meet the issue in this bug.
Is my guess right?

No. When managed='yes' is set, a lower level function, *not* virNodeDeviceDetachFlags(), is called, and given an explicit indicator of which driver to bind to. The default used by virNodeDeviceDetachFlags() is thus irrelevant in this case.

Whatever the problem is here, it is different than that described in https://bugzilla.redhat.com/show_bug.cgi?id=1001738#c12 (and more directly in Bug 1035188, which I filed shortly after adding that comment).

(In reply to Laine Stump from comment #8)
> No. When managed='yes' is set, a lower level function, *not*
> virNodeDeviceDetachFlags(), is called, and given an explicit indicator of
> which driver to bind to. The default used by virNodeDeviceDetachFlags() is
> thus irrelevant in this case.
> 
> Whatever the problem is here, it is different than that described in
> https://bugzilla.redhat.com/show_bug.cgi?id=1001738#c12 (and more directly
> in Bug 1035188, which I filed shortly after adding that comment).

While hot-plug the vf from vfpool, also need to change the pci driver to vfio manually. This scenario also need to be fixed.

Steps:
1. define and start the hostdev type network.
# virsh net-dumpxml hostnet2
<network>
  <name>hostnet2</name>
  <uuid>d70eb6fa-9fca-4124-84d3-a054666c3236</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <pf dev='ens1f0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
  </forward>
</network>

2. prepare one xml like following one for the hot-plug
# cat vf-vfpool2.xml 
<interface type='network'>
 <source network='hostnet2'/>
</interface>

3. hot-plug the vf of vfpool to the guest, it will report error.
# virsh attach-device a vf-vfpool2.xml 
error: Failed to attach device from vf-vfpool2.xml
error: unable to set user and group to '107:107' on '/dev/vfio/22': No such file or directory

4. after changing the pci driver to vfio manually, the hot-plug will be successfully.
# virsh nodedev-dumpxml pci_0000_03_10_0|grep driver -A1
  <driver>
    <name>igbvf</name>
  </driver>
  <capability type='pci'>
# virsh nodedev-detach pci_0000_03_10_0 --driver vfio
Device pci_0000_03_10_0 detached

# virsh nodedev-dumpxml pci_0000_03_10_0|grep driver -A1
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
# virsh attach-device a vf-vfpool2.xml 
Device attached successfully

What was causing the failure was that the memlock limit of the process was being set to the guest memory size in KiB rather than bytes. I've fixed that problem locally and now the device is properly attached.

However, the error message that was being logged was an artifact of the error recovery cleanup, *not* the actual error, so now I have to hunt down why that is happening and fix it so that the next time there is a problem during device attach it will be easier to find the source of the problem.

I've also noticed that if you restart libvirtd while there is a domain that has a type='hostdev' interface, the hostdev array will not properly get an entry for that hybrid hostdev, so all further attempts to attach a new hostdev will fail due to a "duplicate alias" error (because it doesn't know about the existing hostdevN device, the new device is given the same alias).

I've posted 3 patches upstream to remedy:

1) the original problem with memlock limit

2) obscuring of the real error message with bad error recover (which was only present in Comment 10 of this BZ, and was caused by one of the patches committed to the RHEL7 branch to fix Bug 1025108 (upstream to fix Bug 1016511)

3) failure to add new hostdevs after a restart of libvirtd if there were existing devices of the type <interface type='network'> where the network was a pool of hostdevs.


https://www.redhat.com/archives/libvir-list/2013-December/msg01093.html

I forgot to mention that item (3) in the above list has a separate BZ filed for it: Bug 1045002. I made it a separate bug because it also exists in earlier releases, while item (1) only goes back as far as VFIO support, and (2) hasn't been in any upstream release (and has only been in a scratch build of RHEL7)

# rpm -q libvirt
libvirt-1.1.1-18.el7.x86_64

1. test for hot-plug device
For managed host device:
# cat vf-hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address bus='0x03' slot='0x10' function='0x0'/>
  </source>
</hostdev>

# virsh attach-device r7 vf-hostdev.xml 
Device attached successfully

# virsh detach-device r7 vf-hostdev.xml 
Device detached successfully

For un-managed host device, a error occurs:
# cat vf-hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='no'>
  <driver name='vfio'/>
  <source>
    <address bus='0x03' slot='0x10' function='0x0'/>
  </source>
</hostdev>

# virsh attach-device r7 vf-hostdev.xml 
error: Failed to attach device from vf-hostdev.xml
error: Path '/dev/vfio/22' is not accessible: No such file or directory

And in virt-manager, this device pci_0000_03_10_0 can be hot-plugged into guest.

2. test for memlock of hot-plug
For the memlock part, since the patch aims to hard_limit, I test it like this:
set the hard_limit to 2G(2097152 KiB) for the guest:
# virsh dumpxml r7
<domain type='kvm' id='11'>
  <name>r7</name>
  <uuid>c80b0546-3b9b-46c7-bba8-ca98693d1f1d</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>2097152</hard_limit>
  </memtune>
...
</domain>

check the RLIMIT_MEMLOCK and VmLck of this guest:
# cat /proc/`pidof qemu-kvm`/limits | grep locked
Max locked memory         65536                65536                bytes     
# cat /proc/`pidof qemu-kvm`/status | grep VmLck
VmLck:	       0 kB

hot-plug the pci_0000_03_10_0
# virsh attach-device r7 vf-hostdev.xml 
Device attached successfully

re-check the RLIMIT_MEMLOCK and VmLck of this guest:
# cat /proc/`pidof qemu-kvm`/limits | grep locked
Max locked memory         2147483648           2147483648           bytes     
# cat /proc/`pidof qemu-kvm`/status | grep VmLck
VmLck:	 1179656 kB

The value of RLIMIT_MEMLOCK is 2147483648 bytes (2097152 * 1024), the patch works.

hot-unplug it:
# virsh detach-device r7 vf-hostdev.xml 
Device detached successfully

# cat /proc/`pidof qemu-kvm`/limits | grep locked
Max locked memory         2147483648           2147483648           bytes     
# cat /proc/`pidof qemu-kvm`/status | grep VmLck
VmLck:	       0 kB

after hot-unplug this device, the locked memory will be back to 0kB, so the locked memory limits of hot-plug is fixed.

3.
The error "unable to set user and group to '107:107' on '/dev/vfio/22'" comment 10 said is tested in Bug 1045002.

As above, hot-plug in libvirt-1.1.1-18.el7.x86_64 is working well, and locked memory limit, which is set by hot-plug, is correct.

So I change the status to VERIFIED.

Sorry, I have rechecked this bug on a non-SRIOV network card. It seems have some problem, could you help to review, Laine?

For 82579LM Gigabit network card, there are two problems:
1. When NetworkManager is running, it's easily to trigger the error "internal error: unable to execute QEMU command 'device_add': Device initialization failed.".
# modprobe vfio_pci

# echo 1 > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

# virsh start r7
Domain r7 started

# cat hostdev.xml
<hostdev mode="subsystem" type="pci" managed="yes">
  <driver name='vfio'/>
  <source>
    <address domain="0x0000" bus="0x00" slot="0x19" function="0x0"/>
  </source>
</hostdev>

# virsh attach-device r7 hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add': Device initialization failed.

(The attachments are /var/log/message and libvirtd.log)

But if I shutdown NetworkManager I can always hot-plug this network card.

2. Sometimes the e1000e driver will not be restored after detach-device
I try to debug it, but when I attach libvirtd in gdb, this error will not happen. So I think there is a race condition.

Please move this BZ back to VERIFIED, and file a new BZ for the issue in Comment 17 - it's pretty certain that it's a different issue.

In the description, note any difference in behavior if you manually detach the device from the host driver with "virsh nodedev-detach" and use "managed='no'" instead of "managed='yes'". You can file it against libvirt if you like, but please cc alex.williamson and dcbw so that vfio and NM people are aware.

The fact that it sometimes fails makes me wonder if NM is maybe doing something that causes a rebind of the device to the host driver, maybe because it's still active from NM's point of view or something, but we really should discuss further in a separate BZ.

(In reply to Laine Stump from comment #20)
> Please move this BZ back to VERIFIED, and file a new BZ for the issue in
> Comment 17 - it's pretty certain that it's a different issue.

Sorry about my misleading, I recheck the attach-device failing, it caused by selinux deny. The related bug is bug 1046598 . And this deny does not always ocurr.

Since the bug is aiming to hot-plug by using vfio driver, and it works in libvirt, I change the status to VERIFIED.

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.