Description of problem: PCI device assignment fails to pass through host. Version-Release number of component: libvirt-client.x86_64 4.1.0-5.fc28 How reproducible: That PCI device assigned using virt-manager. $ lspci | grep -i network 03:00.0 Network controller: Qualcomm Atheros AR9285 Wireless Network Adapter (PCI-Express) (rev 01)) Specifications related to the CPU used are enabled in BIOS; checked with: $ cat /proc/cpuinfo | grep -i vmx Specifications activated in the kernel (/etc/default/grub): (...) GRUB_CMDLINE_LINUX="(...) intel_iommu=on iommu=pt". Grub config file was regenerated: grub2-mkconfig -o /etc/grub2.cfg. System was rebooted. $ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.18.7-200.fc28.x86_64 (...) intel_iommu=on iommu=pt Element iommuGroup is missing from that output: # virsh nodedev-dumpxml pci_0000_03_00_0 <device> <name>pci_0000_03_00_0</name> <path>/sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0</path> <parent>pci_0000_00_1c_1</parent> <driver> <name>ath9k</name> </driver> <capability type='pci'> <domain>0</domain> <bus>3</bus> <slot>0</slot> <function>0</function> <product id='0x002b'>AR9285 Wireless Network Adapter (PCI-Express)</product> <vendor id='0x168c'>Qualcomm Atheros</vendor> <pci-express> <link validity='cap' port='0' speed='2.5' width='1'/> <link validity='sta' speed='2.5' width='1'/> </pci-express> </capability> </device> Actual results: error message: 'Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 82, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1508, in startup self._backend.create() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices' Expected results: involved device to be allowed to pass through host to guest. Additional info: source documentation at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/virtualization_deployment_and_administration_guide/index#sect-Guest_virtual_machine_device_configuration-PCI_devices $ dnf list installed | grep libvirt (...) libvirt-client.x86_64 4.1.0-5.fc28 @updates libvirt-daemon.x86_64 4.1.0-5.fc28 @updates Component libvirt not installed along with 'dnf install @virtualization'.
Can you post the output from the virt-host-validate tool?
All features related to LXC returned status PASS. All features related to QEMU returned status PASS except –as output–: $ virt-host-validate QEMU: Checking for device assignment IOMMU support : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
(In reply to ricky.tigg from comment #2) > All features related to LXC returned status PASS. > All features related to QEMU returned status PASS except –as output–: > > $ virt-host-validate > QEMU: Checking for device assignment IOMMU support > : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not > supported by this hardware platform) That's why the IOMMU stuff is missing in your output. If you look under /sys/kernel/iommu_groups, I also assume you don't have anything in there, right? Check whether your platform actually supports Intel VT-d and if so, whether you have in enabled in BIOS.
Right. I had enabled before that report feature UEFI boot; others required features are all allowed. Complete output contains four occurrences of that partial output: $ cat /proc/cpuinfo flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat flush_l1d
So, did you enable IOMMU in BIOS? How did that go?
Created attachment 1486638 [details] Features status in host's motherboard BIOS The motherboard BIOS of my host does not provide any feature explicitly mentioned as 'IOMMU'. The following features are enabled (see attachment): - Intel Virtualization Technology (aka Intel VT-x); - VT-d (aka IOMMU technology). Nothing particular to notice, when rebooting for changes to take effect.
Hmm, that's weird, that seems like it should be enough. Time for more information, does dropping the iommu=pt part help?
Dropping that specification did not bring change.
Following command have been executed: $ rpm -Va --nofiles --nodigest $ dnf clean packages 0 files removed $ sudo rpm --rebuilddb $ sudo rpm -Va | egrep -i "efi|grub" .M....... c /boot/efi/EFI/fedora/grub.cfg .M....... c /boot/efi/EFI/fedora/grubenv .M....... c /etc/default/grub .M....... c /boot/grub2/grub.cfg .M....... c /boot/efi/EFI/fedora/grub.cfg .M....... c /boot/efi/EFI/fedora/grubenv
Components updated: libvirt-client.x86_64 6.6.0, virt-manager.noarch 3.1.0 Network: NAT; Attempting to add a PCI device, wireless controller, produces the message, here partial, "libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices".
Hi, in this specific case updating libvirt or virt-manager will not help. As the command virt-host-validate states there is no IOMMU support from kernel side. Based on your comment 6 you have IOMMU (vt-d) enabled but still it is not enabled inside the host OS. Can you please provide what CPU and motherboard do you have or try searching any info about your HW whether it fully supports IOMMU or not.
$ cat /proc/cpuinfo | sed -n '3,5p; 20,21p' cpu family : 6 model : 23 model name : Intel(R) Core(TM)2 Duo CPU P9600 @ 2.53GHz flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm pti tpr_shadow vnmi flexpriority vpid dtherm ida vmx flags : vnmi flexpriority tsc_offset vtpr vapic # dmidecode | grep -A3 '^System Information' System Information Manufacturer: Dell Inc. Product Name: Latitude E4300 Version: Not Specified On this model, different than the previous, at last IOMMU was added to the kernel. $ dmesg | grep -E 'DMAR|IOMMU' [ 0.027705] ACPI: DMAR 0x00000000DD060400 0000F8 (v01 DELL M09 27D90A0D ASL 00000061) [ 0.098141] DMAR: IOMMU enabled [ 0.184116] DMAR: Host address width 36 [ 0.184117] DMAR: DRHD base: 0x000000fed10000 flags: 0x0 [ 0.184122] DMAR: dmar0: reg_base_addr fed10000 ver 1:0 cap c9008020e30260 ecap 1000 [ 0.184123] DMAR: DRHD base: 0x000000fed11000 flags: 0x0 [ 0.184127] DMAR: dmar1: reg_base_addr fed11000 ver 1:0 cap c0000020630260 ecap 1000 [ 0.184128] DMAR: DRHD base: 0x000000fed13000 flags: 0x1 [ 0.184131] DMAR: dmar2: reg_base_addr fed13000 ver 1:0 cap c9008020630260 ecap 1000 [ 0.184132] DMAR: RMRR base: 0x000000dd7e7000 end: 0x000000dd7fffff [ 0.184134] DMAR: RMRR base: 0x000000ddc00000 end: 0x000000dfffffff [ 0.746135] pci 0000:00:00.0: DMAR: Disabling IOMMU for graphics on this chipset [ 0.746137] pci 0000:00:00.0: DMAR: Forcing write-buffer flush capability [ 1.436046] DMAR: No ATSR found [ 1.436110] DMAR: dmar0: Using Register based invalidation [ 1.436137] DMAR: dmar2: Using Register based invalidation [ 1.458485] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 484.973839] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform [ 783.693583] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform [ 852.491911] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform A new message is produced which makes obsolete the previous. Error starting domain: internal error: process exited while connecting to monitor: 2020-10-30T10:48:17.648591Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-10-30T10:48:17.648624Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-10-30T10:48:17.691122Z qemu-system-x86_64: -device vfio-pci,host=0000:0c:00.0,id=hostdev0,bus=pci.7,addr=0x0: vfio 0000:0c:00.0: failed to setup container for group 4: Failed to set iommu for container: Operation not permitted Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/object/domain.py", line 1330, in startup self._backend.create() File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1234, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirt.libvirtError: internal error: process exited while connecting to monitor: 2020-10-30T10:48:17.648591Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-10-30T10:48:17.648624Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-10-30T10:48:17.691122Z qemu-system-x86_64: -device vfio-pci,host=0000:0c:00.0,id=hostdev0,bus=pci.7,addr=0x0: vfio 0000:0c:00.0: failed to setup container for group 4: Failed to set iommu for container: Operation not permitted
Hi, so it looks like your HW has IOMMU without interrupt remapping, to solve this issue you need to enable unsafe interrupts using this command: echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf You can read more about it here <https://pve.proxmox.com/wiki/Pci_passthrough> section "IOMMU Interrupt Remapping".
Getting closer. No error message while guest is being powering-on. I noticed that the process making the PCI device available for use on guest must be a redirection from host because the Wi-FI radio there is then turned-off and no longer available as soon as guestt is being powered-on. Host side: $ rfkill list all | sed -n '1,3p;7,9p' 0: dell-wifi: Wireless LAN Soft blocked: yes Hard blocked: no 2: dell-wwan: Wireless WAN Soft blocked: yes Hard blocked: no Guest side: $ rfkill list all 0: phy0: Wireless LAN Soft blocked: no Hard blocked: yes A "no" is expected at "Hard blocked:" Wi-Fi radio service cannot be turned-on be the operation attempted via gnome-control-center or 'nmcli r wifi on'. 'systemctl status wpa_supplicant' reports service being loaded and active, running. 'nmcli r wifi' reports "disabled'".
Thank you for reporting this issue to the libvirt project. Unfortunately we have been unable to resolve this issue due to insufficient maintainer capacity and it will now be closed. This is not a reflection on the possible validity of the issue, merely the lack of resources to investigate and address it, for which we apologise. If you none the less feel the issue is still important, you may choose to report it again at the new project issue tracker https://gitlab.com/libvirt/libvirt/-/issues The project also welcomes contribution from anyone who believes they can provide a solution.