Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1269446 - unable to detach virtual function from the host
Summary: unable to detach virtual function from the host
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-10-07 11:31 UTC by Siddharth Moghe
Modified: 2015-10-17 05:13 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-17 05:13:22 UTC
Target Upstream Version:


Attachments (Terms of Use)
dmesg (113.57 KB, text/plain)
2015-10-16 13:35 UTC, Siddharth Moghe
no flags Details
/etc/default/grub (234 bytes, text/plain)
2015-10-16 13:36 UTC, Siddharth Moghe
no flags Details
grub2-mkconifg -o /boot/grub2/grub.cfg (3.90 KB, text/plain)
2015-10-16 13:37 UTC, Siddharth Moghe
no flags Details

Description Siddharth Moghe 2015-10-07 11:31:18 UTC
Description of problem:
Unable to detach VF from the host.


Version-Release number of selected component (if applicable):

Kernel - Linux rhe7.1-hp-2 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

machine:
Handle 0x0009, DMI type 1, 27 bytes
System Information
	Manufacturer: HP
	Product Name: ProLiant DL380 Gen9
	Version: Not Specified
	Serial Number: SGH517XJX2
	UUID: 30393137-3436-4753-4835-3137584A5832
	Wake-up Type: Power Switch
	SKU Number: 719064-B21
	Family: ProLiant

Handle 0x00BD, DMI type 32, 11 bytes
System Boot Information
	Status: No errors detected


Libvirt:
libvirt-gobject-0.1.7-3.el7.x86_64
libvirt-daemon-driver-interface-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-kvm-1.2.8-16.el7_1.4.x86_64
libvirt-client-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-storage-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-lxc-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-config-network-1.2.8-16.el7_1.4.x86_64
libvirt-gconfig-0.1.7-3.el7.x86_64
libvirt-daemon-driver-network-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-secret-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-config-nwfilter-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-1.2.8-16.el7_1.4.x86_64
libvirt-python-1.2.8-7.el7_1.1.x86_64
libvirt-glib-0.1.7-3.el7.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
libvirt-1.2.8-16.el7_1.4.x86_64

Kernel:
kernel-3.10.0-229.el7.x86_64
kernel-headers-3.10.0-229.14.1.el7.x86_64
abrt-addon-kerneloops-2.1.11-19.el7.x86_64
kernel-devel-3.10.0-229.14.1.el7.x86_64
kernel-tools-3.10.0-229.el7.x86_64
kernel-tools-libs-3.10.0-229.el7.x86_64


Qemu:
qemu-img-1.5.3-86.el7.x86_64
ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch
qemu-guest-agent-2.1.0-4.el7.x86_64
qemu-kvm-1.5.3-86.el7.x86_64
qemu-kvm-common-1.5.3-86.el7.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64



How reproducible: every time


Steps to Reproduce:
1. Enabled intel_iommu=on in /etc/default/grub
2. echo 4 > /sys/bus/pci/devices/0000\:05\:00.0/sriov_numvfs
3. virsh nodedev-list | grep pci_0000_05
4. virsh nodedev-detach pci_0000_05_10_0

Actual results:
error: Failed to detach device pci_0000_05_10_0
error: Operation not supported: neither VFIO nor KVM device assignment is currently supported on this system

If connected with virt-manager without detaching - error 

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1393, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 966, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices



Expected results:
The VF is supposed to detach from the Host manually and with virt-manager



Additional info:

Comment 3 Laine Stump 2015-10-07 15:31:42 UTC
1) What is the output of these two commands:

  * "ls -l /dev/vfio/vfio"
  * "ls -l /sys/kernel/iommu_groups/*"

2) does this change if you run "modprobe -i vfio" ?

Bug 1044595, which was resolved in time for RHEL7.0 was supposed to make vfio loaded and available by default, yet the error message you're reporting occurs only if either /sys/kernel/iommu_groups is missing or empty, or /dev/vfio/vfio doesn't exist, which I think shouldn't be the case.

Comment 4 Laine Stump 2015-10-07 15:35:27 UTC
Another suggestion from someone with a much better memory than me: check the BIOS configuration of the machine to make sure that VT-d is enabled there.

Comment 5 Siddharth Moghe 2015-10-07 16:19:45 UTC
VT-d is enabled by default on the HP gen9 machines


[root@rhe7 Desktop]# ls -l /dev/vfio/vfio
crw-------. 1 root root 10, 196 Oct  7 21:46 /dev/vfio/vfio


[root@rhe7 Desktop]# ls -l /sys/kernel/iommu_groups/
total 0

Comment 6 Laine Stump 2015-10-07 17:10:34 UTC
This is exactly the symptoms displayed when intel_iommu=on is missing from the kernel commandline.

In the bug description you say that you've added that to /etc/default/grub, but have you run grub2-mkconfig (setting the appropriate output file) so that it actually gets added to the commandline used by grub during boot, and verified after booting that the commandline used for boot contained the option (this will be in the first several lines of the dmesg output)?

Comment 7 Siddharth Moghe 2015-10-08 05:59:29 UTC
no, i had not run grub2-mkconfig 

but now even after i ran it, i dont see any change.

Comment 8 Alex Williamson 2015-10-08 14:44:25 UTC
Please attach dmesg from the system

Comment 9 Siddharth Moghe 2015-10-16 13:35:20 UTC
Created attachment 1083652 [details]
dmesg

Comment 10 Siddharth Moghe 2015-10-16 13:36:08 UTC
Created attachment 1083653 [details]
/etc/default/grub

Comment 11 Siddharth Moghe 2015-10-16 13:37:23 UTC
Created attachment 1083654 [details]
grub2-mkconifg -o /boot/grub2/grub.cfg

Comment 12 Alex Williamson 2015-10-16 14:16:10 UTC
The kernel commandline in dmesg indicates intel_iommu=on is not being enabled:

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/rhel-root ro nomodeset crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_US.UTF-8

Since your system is using UEFI, the output of grub2-mkconfig should be to  /etc/grub2-efi.cfg.  Please:

1) Run: grub2-mkconifg -o /etc/grub2-efi.cfg
2) Reboot

And let us know if the problem is resolved.

Comment 13 Siddharth Moghe 2015-10-17 02:15:04 UTC
wow, did not realize this even when i was seeing the OS not installing in GUI in the first place. I will try that, in addition to also checking if it works on legacy boot mode with the previous configurations.

Comment 14 Siddharth Moghe 2015-10-17 05:13:22 UTC
thanks, this is working. Closing the bug.


Note You need to log in before you can comment on or make changes to this bug.