Bug 1269446 - unable to detach virtual function from the host
unable to detach virtual function from the host
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.1
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: Laine Stump
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-07 07:31 EDT by Siddharth Moghe
Modified: 2015-10-17 01:13 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-17 01:13:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dmesg (113.57 KB, text/plain)
2015-10-16 09:35 EDT, Siddharth Moghe
no flags Details
/etc/default/grub (234 bytes, text/plain)
2015-10-16 09:36 EDT, Siddharth Moghe
no flags Details
grub2-mkconifg -o /boot/grub2/grub.cfg (3.90 KB, text/plain)
2015-10-16 09:37 EDT, Siddharth Moghe
no flags Details

  None (edit)
Description Siddharth Moghe 2015-10-07 07:31:18 EDT
Description of problem:
Unable to detach VF from the host.


Version-Release number of selected component (if applicable):

Kernel - Linux rhe7.1-hp-2 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

machine:
Handle 0x0009, DMI type 1, 27 bytes
System Information
	Manufacturer: HP
	Product Name: ProLiant DL380 Gen9
	Version: Not Specified
	Serial Number: SGH517XJX2
	UUID: 30393137-3436-4753-4835-3137584A5832
	Wake-up Type: Power Switch
	SKU Number: 719064-B21
	Family: ProLiant

Handle 0x00BD, DMI type 32, 11 bytes
System Boot Information
	Status: No errors detected


Libvirt:
libvirt-gobject-0.1.7-3.el7.x86_64
libvirt-daemon-driver-interface-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-kvm-1.2.8-16.el7_1.4.x86_64
libvirt-client-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-storage-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-lxc-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-config-network-1.2.8-16.el7_1.4.x86_64
libvirt-gconfig-0.1.7-3.el7.x86_64
libvirt-daemon-driver-network-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-secret-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-config-nwfilter-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-1.2.8-16.el7_1.4.x86_64
libvirt-python-1.2.8-7.el7_1.1.x86_64
libvirt-glib-0.1.7-3.el7.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
libvirt-1.2.8-16.el7_1.4.x86_64

Kernel:
kernel-3.10.0-229.el7.x86_64
kernel-headers-3.10.0-229.14.1.el7.x86_64
abrt-addon-kerneloops-2.1.11-19.el7.x86_64
kernel-devel-3.10.0-229.14.1.el7.x86_64
kernel-tools-3.10.0-229.el7.x86_64
kernel-tools-libs-3.10.0-229.el7.x86_64


Qemu:
qemu-img-1.5.3-86.el7.x86_64
ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch
qemu-guest-agent-2.1.0-4.el7.x86_64
qemu-kvm-1.5.3-86.el7.x86_64
qemu-kvm-common-1.5.3-86.el7.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64



How reproducible: every time


Steps to Reproduce:
1. Enabled intel_iommu=on in /etc/default/grub
2. echo 4 > /sys/bus/pci/devices/0000\:05\:00.0/sriov_numvfs
3. virsh nodedev-list | grep pci_0000_05
4. virsh nodedev-detach pci_0000_05_10_0

Actual results:
error: Failed to detach device pci_0000_05_10_0
error: Operation not supported: neither VFIO nor KVM device assignment is currently supported on this system

If connected with virt-manager without detaching - error 

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1393, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 966, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices



Expected results:
The VF is supposed to detach from the Host manually and with virt-manager



Additional info:
Comment 3 Laine Stump 2015-10-07 11:31:42 EDT
1) What is the output of these two commands:

  * "ls -l /dev/vfio/vfio"
  * "ls -l /sys/kernel/iommu_groups/*"

2) does this change if you run "modprobe -i vfio" ?

Bug 1044595, which was resolved in time for RHEL7.0 was supposed to make vfio loaded and available by default, yet the error message you're reporting occurs only if either /sys/kernel/iommu_groups is missing or empty, or /dev/vfio/vfio doesn't exist, which I think shouldn't be the case.
Comment 4 Laine Stump 2015-10-07 11:35:27 EDT
Another suggestion from someone with a much better memory than me: check the BIOS configuration of the machine to make sure that VT-d is enabled there.
Comment 5 Siddharth Moghe 2015-10-07 12:19:45 EDT
VT-d is enabled by default on the HP gen9 machines


[root@rhe7 Desktop]# ls -l /dev/vfio/vfio
crw-------. 1 root root 10, 196 Oct  7 21:46 /dev/vfio/vfio


[root@rhe7 Desktop]# ls -l /sys/kernel/iommu_groups/
total 0
Comment 6 Laine Stump 2015-10-07 13:10:34 EDT
This is exactly the symptoms displayed when intel_iommu=on is missing from the kernel commandline.

In the bug description you say that you've added that to /etc/default/grub, but have you run grub2-mkconfig (setting the appropriate output file) so that it actually gets added to the commandline used by grub during boot, and verified after booting that the commandline used for boot contained the option (this will be in the first several lines of the dmesg output)?
Comment 7 Siddharth Moghe 2015-10-08 01:59:29 EDT
no, i had not run grub2-mkconfig 

but now even after i ran it, i dont see any change.
Comment 8 Alex Williamson 2015-10-08 10:44:25 EDT
Please attach dmesg from the system
Comment 9 Siddharth Moghe 2015-10-16 09:35 EDT
Created attachment 1083652 [details]
dmesg
Comment 10 Siddharth Moghe 2015-10-16 09:36 EDT
Created attachment 1083653 [details]
/etc/default/grub
Comment 11 Siddharth Moghe 2015-10-16 09:37 EDT
Created attachment 1083654 [details]
grub2-mkconifg -o /boot/grub2/grub.cfg
Comment 12 Alex Williamson 2015-10-16 10:16:10 EDT
The kernel commandline in dmesg indicates intel_iommu=on is not being enabled:

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/rhel-root ro nomodeset crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_US.UTF-8

Since your system is using UEFI, the output of grub2-mkconfig should be to  /etc/grub2-efi.cfg.  Please:

1) Run: grub2-mkconifg -o /etc/grub2-efi.cfg
2) Reboot

And let us know if the problem is resolved.
Comment 13 Siddharth Moghe 2015-10-16 22:15:04 EDT
wow, did not realize this even when i was seeing the OS not installing in GUI in the first place. I will try that, in addition to also checking if it works on legacy boot mode with the previous configurations.
Comment 14 Siddharth Moghe 2015-10-17 01:13:22 EDT
thanks, this is working. Closing the bug.

Note You need to log in before you can comment on or make changes to this bug.