Bug 1514538 - libvirt + qemu + hugepages won't start with SElinux enabled
Summary: libvirt + qemu + hugepages won't start with SElinux enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: selinux-policy
Version: 27
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Lukas Vrabec
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-17 17:40 UTC by Patrick O'Callaghan
Modified: 2018-06-21 04:48 UTC (History)
15 users (show)

Fixed In Version: selinux-policy-3.13.1-283.34.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-05 22:27:34 UTC


Attachments (Terms of Use)
Log of 'restorecon -Rv /' and 'ausearch -m AVC -ts today' (13.25 KB, text/plain)
2017-11-20 16:28 UTC, Patrick O'Callaghan
no flags Details
Dump of xml file (5.79 KB, text/plain)
2017-11-22 15:40 UTC, Patrick O'Callaghan
no flags Details
Copy of /etc/libvirt/qemu.conf (26.87 KB, text/plain)
2017-11-22 15:41 UTC, Patrick O'Callaghan
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1592688 None CLOSED SELinux is preventing /usr/libexec/qemu-kvm from map access 2019-06-11 02:27:24 UTC

Internal Links: 1592688

Description Patrick O'Callaghan 2017-11-17 17:40:12 UTC
Description of problem:
Since upgrading to F27 yesterday I keep getting a pop-up error on trying to start my Win10 VM, which worked correctly before the upgrade. The message says "unable to map backing store for guest RAM: Permission denied". I assume this is related to hugepages, which I use. The journal entry appears to be this:

libvirtd[8610]: 2017-11-17T17:11:41.105867Z qemu-system-x86_64: unable to map backing store for guest RAM: Permission denied

There doesn't appear to be an AVC denial record, but if I turn off SElinux, start the VM, then turn SElinux back on, it works.

Version-Release number of selected component (if applicable):
selinux-policy-3.13.1-283.14.fc27.noarch
qemu-block-rbd-2.10.1-1.fc27.x86_64
qemu-system-x86-core-2.10.1-1.fc27.x86_64
qemu-block-nfs-2.10.1-1.fc27.x86_64
ipxe-roms-qemu-20161108-2.gitb991c67.fc26.noarch
qemu-common-2.10.1-1.fc27.x86_64
qemu-system-x86-2.10.1-1.fc27.x86_64
libvirt-daemon-driver-qemu-3.7.0-2.fc27.x86_64
qemu-block-dmg-2.10.1-1.fc27.x86_64
qemu-kvm-2.10.1-1.fc27.x86_64
qemu-guest-agent-2.10.1-1.fc27.x86_64
qemu-block-ssh-2.10.1-1.fc27.x86_64
qemu-block-iscsi-2.10.1-1.fc27.x86_64
qemu-img-2.10.1-1.fc27.x86_64
qemu-block-curl-2.10.1-1.fc27.x86_64
qemu-block-gluster-2.10.1-1.fc27.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Using virt-manager, attempt to start a VM
2. Get pop-up error with permissions problem
3. Disbale SElinux and try again, with success

Actual results:
VM fails to start. Nothing shows in journalctl.

Expected results:
VM starts correctly

Additional info:
Using hugepages and pinned CPUs. The exact pop-up message is as follows:

Error starting domain: internal error: qemu unexpectedly closed the monitor: 2017-11-17T17:37:18.995524Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/5 (label charserial0)
2017-11-17T17:37:19.008982Z qemu-system-x86_64: unable to map backing store for guest RAM: Permission denied

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 82, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1505, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1062, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: qemu unexpectedly closed the monitor: 2017-11-17T17:37:18.995524Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/5 (label charserial0)
2017-11-17T17:37:19.008982Z qemu-system-x86_64: unable to map backing store for guest RAM: Permission denied

Comment 1 Lukas Vrabec 2017-11-20 11:49:12 UTC
Hi, 

Could you do following: 

1. relabel whole system after upgrade: 
# restorecon -Rv /

2. Try to reproduce the issue
3. Attach output of:
# ausearch -m AVC -ts today 

Thanks,
Lukas.

Comment 2 Patrick O'Callaghan 2017-11-20 16:28:14 UTC
Created attachment 1355948 [details]
Log of 'restorecon -Rv /' and 'ausearch -m AVC -ts today'

As requested.

Comment 3 Lukas Vrabec 2017-11-20 16:36:54 UTC
It looks your issue is fixed, Am I right?

Comment 4 Patrick O'Callaghan 2017-11-20 16:46:36 UTC
(In reply to Lukas Vrabec from comment #3)
> It looks your issue is fixed, Am I right?

No, the behaviour is exactly the same. I can only start the VM by turning off setenforce (and can then turn it on again as soon as it starts to boot).

Comment 5 Ed Greshko 2017-11-22 00:36:38 UTC
Have you tried "semodule -D" to remove dontaudits from policy first?  Many times having don't audit turned on will prevent AVC's from being generated.

Comment 6 Patrick O'Callaghan 2017-11-22 14:57:49 UTC
(In reply to Ed Greshko from comment #5)
> Have you tried "semodule -D" to remove dontaudits from policy first?  Many
> times having don't audit turned on will prevent AVC's from being generated.

Yes ("semodule -DB"). It had no effect, i.e. the error still exists but there is still no AVC.

Comment 7 Cole Robinson 2017-11-22 15:05:13 UTC
Reopening since reporter says the issue persists.

Lukas, any idea for more debugging steps? Do the steps above for re-enabling dontaudit rules look correct? Are there any dontaudit rules for hugetlbfs?

Patrick, can you also provide the full VM XML (sudo virsh dumpxml $vmname)? Where is your hugepages mount, /dev/hugepages? Any custom hugepages config in /etc/libvirt/qemu.conf?

Comment 8 Patrick O'Callaghan 2017-11-22 15:38:06 UTC
(In reply to Cole Robinson from comment #7)
> Reopening since reporter says the issue persists.
> 
> Lukas, any idea for more debugging steps? Do the steps above for re-enabling
> dontaudit rules look correct? Are there any dontaudit rules for hugetlbfs?
> 
> Patrick, can you also provide the full VM XML (sudo virsh dumpxml $vmname)?
> Where is your hugepages mount, /dev/hugepages? Any custom hugepages config
> in /etc/libvirt/qemu.conf?

OK, I'll send that, plus /etc/libvirt/qemu.conf. The only customisation is some evdev stuff. Hugepages is mounted on /dev/hugepages.

Comment 9 Patrick O'Callaghan 2017-11-22 15:40:18 UTC
Created attachment 1357630 [details]
Dump of xml file

Comment 10 Patrick O'Callaghan 2017-11-22 15:41:19 UTC
Created attachment 1357631 [details]
Copy of /etc/libvirt/qemu.conf

Comment 11 Lukas Vrabec 2017-12-11 11:54:24 UTC
Patrick, 

Could you please run:
# semodule -DB

Then reproduce the issue and then add output of:

# ausearch -m AVC, USER_AVC -ts today

Thanks,
Lukas.

Comment 12 Patrick O'Callaghan 2017-12-11 12:26:00 UTC
I think I already did this, see Comment 6. Trying again:

$ sudo semodule -DB
$ sudo ausearch -m AVC, USER_AVC -ts today
USER_AVC is an unsupported option

[I'm not an expert in ausearch so if that wasn't what you meant literally then please explain.]

Just for completeness:

$ sudo ausearch -m AVC -ts today
<no matches>

Comment 13 Nicolas Pöhlmann 2018-01-18 05:43:32 UTC
(In reply to Patrick O'Callaghan from comment #12)
> I think I already did this, see Comment 6. Trying again:
> 
> $ sudo semodule -DB
> $ sudo ausearch -m AVC, USER_AVC -ts today
> USER_AVC is an unsupported option

Patrick, the command from Lukas had a litte typo in it, that's why it failed. You had to write the ausearch command like this:

# ausearch -m AVC,USER_AVC -ts today

But anyway, we ran into the same problem too. After trying to activate hugepages under F27 libvirt/qemu creates the error message "unable to map backing store for guest RAM: Permission denied" as SELinux is preventing the KVM group to map the RAM. Without using hugepages the VM is starting correctly.

We tested the same configuration on another machine under F26 and it's full working without any SELinux alarms. So this must be a problem under F27 only.

Comment 14 Patrick O'Callaghan 2018-01-18 12:01:21 UTC
(In reply to Nicolas Pöhlmann from comment #13)
> (In reply to Patrick O'Callaghan from comment #12)
> > I think I already did this, see Comment 6. Trying again:
> > 
> > $ sudo semodule -DB
> > $ sudo ausearch -m AVC, USER_AVC -ts today
> > USER_AVC is an unsupported option
> 
> Patrick, the command from Lukas had a litte typo in it, that's why it
> failed. You had to write the ausearch command like this:
> 
> # ausearch -m AVC,USER_AVC -ts today

I tried again using that suggestion but I still don't get any output from ausearch (after shutting down the VM and attempting to restart it with SElinux enabled).

> But anyway, we ran into the same problem too. After trying to activate
> hugepages under F27 libvirt/qemu creates the error message "unable to map
> backing store for guest RAM: Permission denied" as SELinux is preventing the
> KVM group to map the RAM. Without using hugepages the VM is starting
> correctly.
> 
> We tested the same configuration on another machine under F26 and it's full
> working without any SELinux alarms. So this must be a problem under F27 only.

Exactly. I said the same thing in my original report. This failure did not occur with F26.

Thanks for the confirmation in any case. At least it's not just me :-)

Comment 15 Clarke Wixon 2018-01-23 14:34:27 UTC
It's not just you!

I have exactly the same issue, and it started immediately following my upgrade from F26 to F27, with no other changes to my virt setup.

Just chiming in here to try to give this issue some additional momentum.

Comment 16 Patrick O'Callaghan 2018-01-23 16:30:46 UTC
(In reply to Clarke Wixon from comment #15)
> It's not just you!
> 
> I have exactly the same issue, and it started immediately following my
> upgrade from F26 to F27, with no other changes to my virt setup.
> 
> Just chiming in here to try to give this issue some additional momentum.

Thanks, I appreciate it.

Comment 17 Patrick O'Callaghan 2018-03-29 10:29:46 UTC
I note that F28beta has been announced. Can we be assured that this bug will have been fixed by release date? I don't see it on the blocker list but I've not been checking test releases.

Comment 18 Cole Robinson 2018-04-17 19:54:44 UTC
I'm running f28, just tried this, and it still seems to be busted. Here's the AVC that popped up:

type=AVC msg=audit(1523994495.285:412): avc:  denied  { map } for  pid=3718 comm="qemu-system-x86" path=2F6465762F6875676570616765732F6C6962766972742F71656D752F332D6632372F71656D755F6261636B5F6D656D2E70632E72616D2E7A5764414135202864656C6574656429 dev="hugetlbfs" ino=61749 scontext=system_u:system_r:svirt_t:s0:c350,c761 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file permissive=0


Hopefully that explains things?

Comment 19 Patrick O'Callaghan 2018-04-17 21:45:20 UTC
(In reply to Cole Robinson from comment #18)
> I'm running f28, just tried this, and it still seems to be busted. Here's
> the AVC that popped up:
> 
> type=AVC msg=audit(1523994495.285:412): avc:  denied  { map } for  pid=3718
> comm="qemu-system-x86"
> path=2F6465762F6875676570616765732F6C6962766972742F71656D752F332D6632372F7165
> 6D755F6261636B5F6D656D2E70632E72616D2E7A5764414135202864656C6574656429
> dev="hugetlbfs" ino=61749 scontext=system_u:system_r:svirt_t:s0:c350,c761
> tcontext=system_u:object_r:svirt_image_t:s0 tclass=file permissive=0
> 
> 
> Hopefully that explains things?

Disappointed to hear that. I'd have said this should be a blocker for F28 but it appears not to be.

Comment 20 Cole Robinson 2018-04-17 22:05:24 UTC
(In reply to Patrick O'Callaghan from comment #19)
> 
> Disappointed to hear that. I'd have said this should be a blocker for F28
> but it appears not to be.

This type of bug won't meet the Fedora blocker criteria. But now that selinux devs have an AVC hopefully it's a simple fix on their side

Comment 21 Fedora Update System 2018-04-29 13:19:37 UTC
selinux-policy-3.13.1-283.34.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-aa26da1777

Comment 22 Patrick O'Callaghan 2018-04-29 14:06:07 UTC
(In reply to Fedora Update System from comment #21)
> selinux-policy-3.13.1-283.34.fc27 has been submitted as an update to Fedora
> 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-aa26da1777

Excellent. This appears to fix the bug. After rebooting I started my VM and got no error message.

Comment 23 Fedora Update System 2018-04-30 14:17:44 UTC
selinux-policy-3.13.1-283.34.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-aa26da1777

Comment 24 Fedora Update System 2018-05-05 22:27:34 UTC
selinux-policy-3.13.1-283.34.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.