Bug 1019387

Summary: Nodedev-detach fails for devices without driver
Product: Red Hat Enterprise Linux 7 Reporter: Dave Allan <dallan>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: acathrow, dyuan, honzhang, jdenemar, jiahu, vlastimil.holer, xuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.1.1-12.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1018897 Environment:
Last Closed: 2014-06-13 10:24:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dave Allan 2013-10-15 15:28:36 UTC 
+++ This bug was initially created as a clone of Bug #1018897 +++

Description of problem:
When device is not managed by any driver (real or stub), sysfs device entries doesn't contain "driver" symlink. Domain with PCI device in managed mode fails to start due to missing "driver" symlink on sysfs. Libvirt nodedev-detach fails as well.

Version-Release number of selected component (if applicable):
1.1.3

How reproducible:

Steps to Reproduce:
1. find device without driver
2. virsh nodedev-detach pci_...

Actual results:
Device must be manually assigned to stub driver.

Expected results:
Device is assigned to stub driver automatically.

Additional info:
# ls -a /sys/bus/pci/devices/0000:03:00.0/
.			  enable	 msi_bus    resource2
..			  firmware_node  numa_node  resource2_wc
broken_parity_status	  index		 power	    subsystem
class			  irq		 remove     subsystem_device
config			  label		 rescan     subsystem_vendor
consistent_dma_mask_bits  local_cpulist  reset	    uevent
device			  local_cpus	 resource   vendor
dma_mask_bits		  modalias	 resource0  vpd

# virsh create deployment.0
setlocale: No such file or directory
error: Failed to create domain from deployment.0
error: internal error: Invalid device 0000:03:00.0 driver file /sys/bus/pci/devices/0000:03:00.0/driver is not a symlink

# virsh nodedev-detach pci_0000_03_00_0
setlocale: No such file or directory
error: Failed to detach device pci_0000_03_00_0
error: internal error: Invalid device 0000:03:00.0 driver file /sys/bus/pci/devices/0000:03:00.0/driver is not a symlink

# modprobe pci_stub ids=15b3:1003
# virsh nodedev-detach pci_0000_03_00_0
Device pci_0000_03_00_0 detached
Comment 2 Michal Privoznik 2013-11-04 16:36:24 UTC 
Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-November/msg00123.html
Comment 4 Hu Jianwei 2013-11-12 02:29:34 UTC 
I can reproduce it with libvirt-1.1.1-11.el7.x86_64, and can't reproduce it with libvirt-1.1.1-12.el7.x86_64.

Version:
libvirt-1.1.1-12.el7.x86_64
qemu-kvm-1.5.3-10.el7.x86_64
kernel-3.10.0-33.el7.x86_64

1. Removed the driver of PCI device.
[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v| grep Kernel
	Kernel driver in use: igb
[root@sriov2 ~]# 
[root@sriov2 ~]# rmmod igb

[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Quad Port Server Adapter
	...
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

[root@sriov2 ~]# 

2.Create one domain with this PCI device assigned.
[root@sriov2 ~]# cat test.xml | grep -aA8 hostdev
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address bus='0x0e' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
    <controller type='usb' index='0'>
...

[root@sriov2 ~]# virsh create test.xml 
Domain test created from test.xml

[root@sriov2 ~]# virsh dumpxml test| grep -aA8 hostdev 
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
...
[root@sriov2 ~]# ls -a /sys/bus/pci/devices/0000\:0e\:00.0/
.                     config                    dma_mask_bits  irq            msi_bus    remove    resource0  sriov_numvfs      subsystem_vendor
..                    consistent_dma_mask_bits  driver         local_cpulist  msi_irqs   rescan    resource1  sriov_totalvfs    uevent
broken_parity_status  d3cold_allowed            enable         local_cpus     numa_node  reset     resource2  subsystem         vendor
class                 device                    iommu_group    modalias       power      resource  resource3  subsystem_device

[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v | grep Kernel
	Kernel driver in use: pci-stub
[root@sriov2 ~]# 
[root@sriov2 ~]# ls -a /sys/bus/pci/devices/0000\:0e\:00.0/driver/
.  ..  0000:0e:00.0  0000:0e:00.1  0000:10:00.0  0000:10:00.1  bind  new_id  remove_id  uevent  unbind
[root@sriov2 ~]#  
[root@sriov2 ~]# virsh destroy test
Domain test destroyed

3. Detach the PCI device using virsh cmd and check its driver.
[root@sriov2 ~]# virsh nodedev-detach pci_0000_0e_00_0
Device pci_0000_0e_00_0 detached

[root@sriov2 ~]# ls -a /sys/bus/pci/devices/0000\:0e\:00.0/
.                     config                    dma_mask_bits  irq            msi_bus    rescan     resource1     sriov_totalvfs    uevent
..                    consistent_dma_mask_bits  driver         local_cpulist  numa_node  reset      resource2     subsystem         vendor
broken_parity_status  d3cold_allowed            enable         local_cpus     power      resource   resource3     subsystem_device
class                 device                    iommu_group    modalias       remove     resource0  sriov_numvfs  subsystem_vendor

[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v| grep Kernel
	Kernel driver in use: pci-stub
[root@sriov2 ~]#

4. Reattach the PCI device, check the driver again, the driver was disappeared.
[root@sriov2 ~]# virsh nodedev-reattach pci_0000_0e_00_0
Device pci_0000_0e_00_0 re-attached

[root@sriov2 ~]# ls -a /sys/bus/pci/devices/0000\:0e\:00.0/driver/
ls: cannot access /sys/bus/pci/devices/0000:0e:00.0/driver/: No such file or directory

[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Quad Port Server Adapter
	...
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

[root@sriov2 ~]#

5. Reload the PCI driver on host, and check it again after detaching.
[root@sriov2 ~]# modprobe igb
[root@sriov2 ~]# ls -a /sys/bus/pci/devices/0000\:0e\:00.0/driver/
.  ..  0000:0e:00.0  bind  module  new_id  remove_id  uevent  unbind
[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v| grep Kernel
	Kernel driver in use: igb
[root@sriov2 ~]#

[root@sriov2 ~]# virsh nodedev-detach pci_0000_0e_00_0
Device pci_0000_0e_00_0 detached

[root@sriov2 ~]# ls -a /sys/bus/pci/devices/0000\:0e\:00.0/driver/
.  ..  0000:0e:00.0  0000:0e:00.1  0000:10:00.0  0000:10:00.1  bind  new_id  remove_id  uevent  unbind
[root@sriov2 ~]#
[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v| grep Kernel
	Kernel driver in use: pci-stub
[root@sriov2 ~]# 
[root@sriov2 ~]# virsh nodedev-reattach pci_0000_0e_00_0
Device pci_0000_0e_00_0 re-attached

[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v| grep Kernel
	Kernel driver in use: igb
[root@sriov2 ~]# 
[root@sriov2 ~]# virsh create test.xml 
Domain test created from test.xml

[root@sriov2 ~]# virsh dumpxml test| grep -aA8 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
...
[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v| grep Kernel
	Kernel driver in use: pci-stub
[root@sriov2 ~]# 
[root@sriov2 ~]# virsh destroy test
Domain test destroyed

[root@sriov2 ~]# lspci -s 0000:0e:00.0 -v| grep Kernel
	Kernel driver in use: igb
[root@sriov2 ~]# 

We can get expected results, changed to verified.
Comment 5 Hu Jianwei 2014-02-19 02:29:55 UTC 
According to bug 1035188, libvirt now defaults to using vfio for device assignment, so update the new test results.

Version:
libvirt-1.1.1-23.el7.x86_64
qemu-kvm-rhev-1.5.3-48.el7.x86_64
kernel-3.10.0-88.el7.x86_64

1. Removed the driver of PCI device.
[root@sriov1 ~]# lspci -s 0000:03:00.0 -v| grep Kernel
	Kernel driver in use: igb
[root@sriov1 ~]# rmmod igb

[root@sriov1 ~]# 
[root@sriov1 ~]# 
[root@sriov1 ~]# lspci -s 0000:03:00.0 -v
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
	Physical Slot: 1
	Flags: fast devsel, IRQ 28
...
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

2.Create one domain with this PCI device assigned.
[root@sriov1 ~]# cat test.xml | grep -aA8 hostdev
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
...
[root@sriov1 ~]# virsh create test.xml 
Domain test created from test.xml

[root@sriov1 ~]# virsh dumpxml test| grep -aA8 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>                            <==== the driver is vfio.
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
...

[root@sriov1 ~]# ls -a /sys/bus/pci/devices/0000\:03\:00.0/
.                     config                    dma_mask_bits  iommu_group    modalias   power   resource   resource3       subsystem         vendor
..                    consistent_dma_mask_bits  driver         irq            msi_bus    remove  resource0  rom             subsystem_device
broken_parity_status  d3cold_allowed            enabled        local_cpulist  msi_irqs   rescan  resource1  sriov_numvfs    subsystem_vendor
class                 device                    firmware_node  local_cpus     numa_node  reset   resource2  sriov_totalvfs  uevent

[root@sriov1 ~]# lspci -s 0000:03:00.0 -v | grep Kernel
	Kernel driver in use: vfio-pci                                    <==== the kernel driver is vfio-pci, not pci-stub.

[root@sriov1 ~]# ls -a /sys/bus/pci/devices/0000\:03\:00.0/driver/
.  ..  0000:03:00.0  0000:03:00.1  bind  module  new_id  remove_id  uevent  unbind

[root@sriov1 ~]# virsh destroy test
Domain test destroyed

3. Detach the PCI device using virsh cmd and check its driver.
[root@sriov1 ~]# virsh nodedev-detach pci_0000_03_00_0
Device pci_0000_03_00_0 detached

[root@sriov1 ~]# ls -a /sys/bus/pci/devices/0000\:03\:00.0/
.                     config                    dma_mask_bits  iommu_group    modalias   remove    resource0  rom             subsystem_device
..                    consistent_dma_mask_bits  driver         irq            msi_bus    rescan    resource1  sriov_numvfs    subsystem_vendor
broken_parity_status  d3cold_allowed            enabled        local_cpulist  numa_node  reset     resource2  sriov_totalvfs  uevent
class                 device                    firmware_node  local_cpus     power      resource  resource3  subsystem       vendor

[root@sriov1 ~]# lspci -s 0000:03:00.0 -v| grep Kernel
	Kernel driver in use: vfio-pci

4. Reattach the PCI device, check the driver again, the driver was disappeared.
[root@sriov1 ~]# virsh nodedev-reattach pci_0000_03_00_0
Device pci_0000_03_00_0 re-attached

[root@sriov1 ~]# ls -a /sys/bus/pci/devices/0000\:03\:00.0/driver/
ls: cannot access /sys/bus/pci/devices/0000:03:00.0/driver/: No such file or directory
[root@sriov1 ~]# lspci -s 0000:03:00.0 -v
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
	Physical Slot: 1
	Flags: fast devsel, IRQ 28
...
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

5. Reload the PCI driver on host, and check it again after detaching.
[root@sriov1 ~]# modprobe igb
[root@sriov1 ~]# ls -a /sys/bus/pci/devices/0000\:03\:00.0/driver/
.  ..  0000:03:00.0  bind  module  new_id  remove_id  uevent  unbind

[root@sriov1 ~]# lspci -s 0000:03:00.0 -v| grep Kernel
	Kernel driver in use: igb

[root@sriov1 ~]# virsh nodedev-detach pci_0000_03_00_0
Device pci_0000_03_00_0 detached

[root@sriov1 ~]# ls -a /sys/bus/pci/devices/0000\:03\:00.0/driver/
.  ..  0000:03:00.0  0000:03:00.1  bind  module  new_id  remove_id  uevent  unbind
[root@sriov1 ~]# lspci -s 0000:03:00.0 -v| grep Kernel
	Kernel driver in use: vfio-pci

[root@sriov1 ~]# virsh nodedev-reattach pci_0000_03_00_0
Device pci_0000_03_00_0 re-attached

[root@sriov1 ~]# lspci -s 0000:03:00.0 -v| grep Kernel
	Kernel driver in use: igb
[root@sriov1 ~]# virsh create test.xml
Domain test created from test.xml

[root@sriov1 ~]# virsh dumpxml test| grep -aA8 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
...

[root@sriov1 ~]# lspci -s 0000:03:00.0 -v| grep Kernel
	Kernel driver in use: vfio-pci
[root@sriov1 ~]# virsh destroy test
Domain test destroyed

[root@sriov1 ~]# lspci -s 0000:03:00.0 -v| grep Kernel
	Kernel driver in use: igb

All related results are expected.
Comment 6 Ludek Smid 2014-06-13 10:24:40 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.