RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1409957 - hot-plugged VF can‘t be used again after reset SRIOV-NIC's kernel mod
Summary: hot-plugged VF can‘t be used again after reset SRIOV-NIC's kernel mod
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Jingjing Shao
URL:
Whiteboard:
: 1402951 (view as bug list)
Depends On:
Blocks: 1420851
TreeView+ depends on / blocked
 
Reported: 2017-01-04 02:13 UTC by Han Han
Modified: 2022-10-31 02:28 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-15 21:09:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
libvirtd log (7.64 MB, text/plain)
2017-01-04 02:13 UTC, Han Han
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1376907 1 None None None 2021-01-20 06:05:38 UTC
Red Hat Bugzilla 1402951 0 unspecified CLOSED libvirt nodedev doesn't rescan the pci_devices 2021-02-22 00:41:40 UTC

Internal Links: 1376907 1402951

Description Han Han 2017-01-04 02:13:24 UTC
Created attachment 1237047 [details]
libvirtd log

Description of problem:
As subject

Version-Release number of selected component (if applicable):
kernel-3.10.0-514.6.1.el7.x86_64
libvirt-2.0.0-10.el7_3.3.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.2.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Reload SRIOV kernel mod and create VF
# modprobe -r ixgbe
# modprobe ixgbe max_vfs=2

2. Prepare an VM and hot-plug the VF
# DOM=V
# virsh start $DOM && sleep 20
Domain V started

# virsh attach-device V hostdev.xml
Device attached successfully

3. Reset kernel mod and try to attach/detach the VF
# modprobe -r ixgbe
# modprobe ixgbe max_vfs=2
# virsh detach-device $DOM hostdev.xml
error: Failed to detach device from hostdev.xml
error: operation failed: no device matching mac address 02:24:6b:89:bc:e0 found

# virsh attach-device $DOM hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: Not detaching active device 0000:86:10.1

Actual results:
As step3

Expected results:
VF coulde be attached after resetting the kernel mod.


Additional info:
After restart libvirtd, VF could be attached in step3.

Comment 1 Han Han 2017-01-04 02:15:24 UTC
The hostdev.xml file:
<interface type='hostdev' managed='yes'>
<mac address='02:24:6b:89:bc:e0'/>
<source>
<address type='pci' domain='0x0000' bus='0x86' slot='0x10' function='0x1'/>
</source>
</interface>

Comment 2 Jingjing Shao 2017-01-04 07:04:56 UTC
Additional info:

After the steps3 in the Description,

4, check the dumpxml of Domain V
#virsh dumpxml V | grep hostdev -A6

5, check the nodedev-dumpxml of VF
# virsh nodedev-dumpxml pci_0000_86_10_1
<device>
  <name>pci_0000_86_10_1</name>
  <path>/sys/devices/pci0000:80/0000:80:01.0/0000:86:10.1</path>
  <parent>pci_0000_80_01_0</parent>
  <driver>
    <name>ixgbevf</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>134</bus>
    <slot>16</slot>
    <function>1</function>
    <product id='0x10ed'>82599 Ethernet Controller Virtual Function</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='phys_function'>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x1'/>
    </capability>
    <iommuGroup number='83'>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x1'/>
    </iommuGroup>
    <numa node='1'/>
    <pci-express>
      <link validity='cap' port='0' width='0'/>
      <link validity='sta' width='0'/>
    </pci-express>
  </capability>
</device>


6, shutddown the domain
# virsh destroy V
Domain V destroyed

7.detach the VF from host, then get error

# virsh nodedev-detach pci_0000_86_10_1 
error: Failed to detach device pci_0000_86_10_1
error: Requested operation is not valid: PCI device 0000:86:10.1 is in use by driver QEMU, domain V

8. restart libvirtd 
service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service



After restart libvirtd, all the things turn to be right

9.# virsh nodedev-detach pci_0000_86_10_1
Device pci_0000_86_10_1 detached

# virsh nodedev-reattach pci_0000_86_10_1
Device pci_0000_86_10_1 re-attached


10.# virsh start V
Domain V started

# virsh attach-device V interface.xml
Device attached successfully

Comment 3 Jiri Denemark 2017-01-04 08:15:43 UTC
Is this a regression introduced by libvirt-2.0.0-10.el7_3.3.x86_64 or can you reproduce even with libvirt-2.0.0-10.el7_3.2.x86_64 or libvirt-2.0.0-10.el7.x86_64?

Comment 4 Han Han 2017-01-04 10:02:36 UTC
Hi Jiri,
The bug can be reproduced on libvirt-2.0.0-10.el7_3.2.x86_64 and libvirt-2.0.0-10.el7.x86_64 as commment0 and comment2.
It is not a regression on RHEL7.3.

Comment 6 Jaroslav Suchanek 2017-01-06 13:40:43 UTC
Dup (or just different result) of bug 1402951?

Comment 7 Han Han 2017-01-10 02:36:41 UTC
The bug can be reproduced on libvirt-2.5.0-1.el7.x86_64 as comment0 and commment2

Comment 10 Laine Stump 2017-12-04 18:11:39 UTC
*** Bug 1402951 has been marked as a duplicate of this bug. ***

Comment 12 lizhengui 2020-01-10 02:39:00 UTC
Has this bug been fixed or is it not considered a bug?

Comment 13 Han Han 2020-01-10 03:28:14 UTC
(In reply to lizhengui from comment #12)
> Has this bug been fixed or is it not considered a bug?

Sorry, I didn't track the bug any more. Could you please share your bug reproducing version?

Comment 14 lizhengui 2020-01-10 03:34:13 UTC
My bug reproducing version libvirt 3.2.0. I think the lastest libvirt also has the bug.

Comment 15 Han Han 2020-01-10 03:53:01 UTC
(In reply to lizhengui from comment #14)
> My bug reproducing version libvirt 3.2.0. I think the lastest libvirt also
> has the bug.

And what's your qemu and kernel version?

Comment 16 Han Han 2020-01-10 03:54:15 UTC
Lili, could you please have a test on latest RHEL7 and RHEL8?

Comment 17 lizhengui 2020-01-10 06:01:02 UTC
(In reply to Han Han from comment #15)
> (In reply to lizhengui from comment #14)
> > My bug reproducing version libvirt 3.2.0. I think the lastest libvirt also
> > has the bug.
> 
> And what's your qemu and kernel version?

QEMU emulator version 2.8.1.1

linux-lAtuOc:~ # uname -r
3.10.0-862.14.1.6_48.x86_64

Comment 18 Han Han 2020-01-10 06:26:49 UTC
Assign to jiyan. Please help to test that on latest RHEL7 and RHEL8

Comment 19 lizhengui 2020-01-11 02:38:01 UTC
(In reply to Han Han from comment #18)
> Assign to jiyan. Please help to test that on latest RHEL7 and RHEL8


I think this is a bug of libvirt. If a link down occurs on the pass-through device, it will trigger the kernel to notify qemu to delete the device. After receiving the delete event of qemu, libvirt will do virHostdevReAttachPCIDevices. In the virHostdevGetPCIHostDeviceList function, because of the device has not been linked up, so the virPCIDeviceNew function fails to exit for the device's config file cannot be found. Therefore, the device will not be removed from mgr->activePCIHostdevs.The next time the virtual machine starts, it fails because the device is still in mgr->activePCIHostdevs, although this time the device has been re-linked up.

Comment 20 lizhengui 2020-01-11 02:48:26 UTC
(In reply to lizhengui from comment #19)
> (In reply to Han Han from comment #18)
> > Assign to jiyan. Please help to test that on latest RHEL7 and RHEL8
> 
> 
> I think this is a bug of libvirt. If a link down occurs on the pass-through
> device, it will trigger the kernel to notify qemu to delete the device.
> After receiving the delete event of qemu, libvirt will do
> virHostdevReAttachPCIDevices. In the virHostdevGetPCIHostDeviceList
> function, because of the device has not been linked up, so the
> virPCIDeviceNew function fails to exit for the device's config file cannot
> be found. Therefore, the device will not be removed from
> mgr->activePCIHostdevs.The next time the virtual machine starts, it fails
> because the device is still in mgr->activePCIHostdevs, although this time
> the device has been re-linked up.

(In reply to Han Han from comment #18)
> Assign to jiyan. Please help to test that on latest RHEL7 and RHEL8

add libvirt info:
2020-01-08T11:48:08.798015+08:00|info|libvirtd[54674]|[59786]|qemuProcessEventHandler[5025]|: vm=0x7eff28191d90, event=2
2020-01-08T11:48:08.798106+08:00|err|libvirtd[54674]|[59786]|virPCIDeviceNew[2001]|: Device 0000:b8:00.0 not found: could not access /sys/bus/pci/devices/0000:b8:00.0/config: No such file or directory
2020-01-08T11:48:08.798193+08:00|err|libvirtd[54674]|[59786]|virHostdevReAttachPCIDevices[981]|: Failed to allocate PCI device list: Device 0000:b8:00.0 not found: could not access /sys/bus/pci/devices/0000:b8:00.0/config: No such file or directory


Note You need to log in before you can comment on or make changes to this bug.