Bug 1830091

Summary:	VM does not start with PCI passthrough
Product:	[Fedora] Fedora	Reporter:	zabra
Component:	libvirt	Assignee:	Libvirt Maintainers <libvirt-maint>
Status:	CLOSED WORKSFORME	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	32	CC:	agedosier, alex.williamson, berrange, clalancette, crobinso, itamar, jforbes, laine, libvirt-maint, veillard, virt-maint
Target Milestone:	---	Keywords:	Regression
Target Release:	---
Hardware:	x86_64
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-05-05 13:05:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description zabra 2020-04-30 20:02:54 UTC

Description of problem:

I have a VM that uses PCI passthrough to pass a GPU to a windows guest.
Everything worked fine until I decided to update Fedora from version 31 to 32.
After that I cannot start the VM anymore.
I traced it down to qemu quitting with an error:

2020-04-30T19:30:03.900479Z qemu-system-x86_64: VFIO_MAP_DMA: -14
2020-04-30T19:30:03.900499Z qemu-system-x86_64: vfio_dma_map(0x5581f874a870, 0x812000000, 0x2000000, 0x7fb567ffd000) = -14 (Bad address)
qemu: hardware error: vfio: DMA mapping failed, unable to continue


Version-Release number of selected component (if applicable): 6.1.0-2.fc32

Version of qemu: 4.2.0-7.fc32


How reproducible: Always


Steps to Reproduce:
1. Setup vfio-pci kernel module to bind to the device
2. Add the PCI device to the VM
3. Start VM

Actual results: VM does not boot


Expected results: VM should boot


Additional info:

AMD Ryzen 3700X
Gigabyte x570 Aorus Pro

PCI-device to pass through: NVIDIA GeForce 1060

Comment 1 zabra 2020-05-01 08:22:46 UTC

Works with qemu 5.0.0 compiled from source, so I guess something changed in qemu from Fedora 32 repository that causes it to break.

Comment 2 Cole Robinson 2020-05-01 13:23:17 UTC

Thanks for the report. My GPU passthrough VM boots fine on Fedora 32 with stock virt packages. I also tested with qemu 5.0.0-rc3 from virt-preview repo and my VM booted fine.

Can you provide: rpm -q qemu-system-x86 libvirt-daemon

Comment 3 zabra 2020-05-01 13:38:20 UTC

Sure:

rpm -q qemu-system-x86 libvirt-daemon

qemu-system-x86-4.2.0-7.fc32.x86_64
libvirt-daemon-6.1.0-2.fc32.x86_64

Comment 4 Cole Robinson 2020-05-01 13:42:56 UTC

aw have you seen this error before:

2020-04-30T19:30:03.900479Z qemu-system-x86_64: VFIO_MAP_DMA: -14
2020-04-30T19:30:03.900499Z qemu-system-x86_64: vfio_dma_map(0x5581f874a870, 0x812000000, 0x2000000, 0x7fb567ffd000) = -14 (Bad address)
qemu: hardware error: vfio: DMA mapping failed, unable to continue

Comment 5 zabra 2020-05-01 14:02:43 UTC

When I disable SELinux I can also also start the VM with the package provided qemu.

To clarify here are the exact steps that I did:
* first notice that it does not work with package provided qemu
* compiled 5.0.0
* got a different error when starting: "Error starting domain: internal error: process exited while connecting to monitor: libvirt:  error : cannot execute binary /usr/local/bin/qemu-system-x86_64: Permission denied"
* disabled SELinux
* everything worked fine
* assumed a problem in the package provided version
* tried to go back to package provided qemu and see if I can reproduce it with SELinux turned off
* could not reproduce, everything worked fine now
* enabled SELinux, rebooted, relabelling took place
* tried to start again: did not work again

Comment 6 zabra 2020-05-02 07:25:16 UTC

OK I think I know now what the problem is. I have a shared memory device that is used for looking glass that seems to be the problem:

type=AVC msg=audit(1588404111.774:675): avc:  denied  { getattr } for  pid=21842 comm="qemu-system-x86" path="/dev/shm/looking-glass" dev="tmpfs" ino=4467 scontext=system_u:system_r:svirt_t:s0:c74,c515 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=1

Somehow SELinux is now blocking the access, but that seems to be a problem on my side then.

You can close the issue. Thanks!

Comment 7 Cole Robinson 2020-05-05 13:05:34 UTC

Thanks for following up, closing

If you are using libvirt though, and not using the qemu-commandline passthrough feature, maybe libvirt should be labeling that path for you, so selinux 'just works'. So it could be a libvirt bug. I don't know anything about looking-glass though

Comment 8 zabra 2020-05-09 19:20:07 UTC

Hm fair point, I'll lay out the things that I did so you or the person in charge to make the decision can judge.

I have a shmem entry in the libvirt config of my VM:

$ virsh edit win10

...
   <shmem name='looking-glass'>
      <model type='ivshmem-plain'/>
      <size unit='M'>32</size>
      <address type='pci' domain='0x0000' bus='0x0b' slot='0x01' function='0x0'/>
    </shmem>
...


Also I have a file in /etc/tmpfiles.d for changing the user

$ cat /etc/tmpfiles.d/10-looking-glass.conf 
f	/dev/shm/looking-glass	0660	fb	kvm	-


After the VM is up I start a program from the terminal as user fb that does r/w on the /dev.