Bug 1210263
| Summary: | Need SELinux policy updates for InfiniBand: RDMA migration fails with SELinux | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Dr. David Alan Gilbert <dgilbert> | |
| Component: | selinux-policy | Assignee: | Lukas Vrabec <lvrabec> | |
| Status: | CLOSED ERRATA | QA Contact: | Marek Haicman <mhaicman> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 7.2 | CC: | dgilbert, dledford, dyuan, fjin, jdenemar, lvrabec, mgrepl, mhaicman, mmalik, mzhan, plautrba, pvrabec, rbalakri, ssekidde, xuzhang, zpeng | |
| Target Milestone: | rc | |||
| Target Release: | --- | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | selinux-policy-3.13.1-86.el7 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1331802 (view as bug list) | Environment: | ||
| Last Closed: | 2016-11-04 02:18:12 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1331802 | |||
Is this something that svirt_t processes should be able to access (perhaps guarded with a new sebool) or should libvirt somehow relabel the file? I don't know enough about Infiniband and RDMA, but relabeling the file seems wrong to me because QEMU may not be the only user of RDMA. Mirek Grepl confirmed a new sebool would be a better solution; moving to selinux-policy. I am also not sure about /dev/infiniband purpose. I need to do some research. Should we label FD? (In reply to Miroslav Grepl from comment #4) > I am also not sure about /dev/infiniband purpose. I need to do some research. > > Should we label FD? Infiniband is hardware that provides an RDMA transport between hosts. As for the meaning of each of the files in /dev/inifiniband I don't know; dledford is a good contact for the details of how it's supposed to work. All of the infiniband files are basically command pipes. You use the file to send commands to kernel modules. The permissions on the files are intentionally. Those with world writable permissions accept commands that any user is allowed to send. Those with root only write permissions are files that accept privileged commands. For the purposes of live migration, it would make sense that qemu would have access to the world writable files. The root only writable files are not needed by qemu/kvm and would present a security risk if it were allowed to access them freely. It would probably be best if we split the infiniband SElinux type into two types infiniband_user_t and infiniband_mgmt_t. The world writable files would be infiniband_user_t and access should be freely allowed from other selinux contexts. The infiniband_mgmt_t should be limited to certain applications (I can provide a list if needed). Moving to libvirt to have a discussion about possible labeling. The labeling sounds OK to me, but IIUC the labeling will be static and not managed by libvirt. And libvirt will not even need to know what the labeling is. That said it's really up to infiniband and selinux-policy guys to discuss and implement the right labeling. (In reply to Doug Ledford from comment #9) > All of the infiniband files are basically command pipes. You use the file > to send commands to kernel modules. The permissions on the files are > intentionally. Those with world writable permissions accept commands that > any user is allowed to send. Those with root only write permissions are > files that accept privileged commands. For the purposes of live migration, > it would make sense that qemu would have access to the world writable files. > The root only writable files are not needed by qemu/kvm and would present a > security risk if it were allowed to access them freely. > > It would probably be best if we split the infiniband SElinux type into two > types infiniband_user_t and infiniband_mgmt_t. Could you define which devices should have user and which mgmt type? [root@virtlab414 ~]# ls -lZ /dev/infiniband/ crw-------. root root system_u:object_r:infiniband_device_t:s0 issm0 crw-------. root root system_u:object_r:infiniband_device_t:s0 issm1 crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 rdma_cm crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 ucm0 crw-------. root root system_u:object_r:infiniband_device_t:s0 umad0 crw-------. root root system_u:object_r:infiniband_device_t:s0 umad1 crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 uverbs0 The world writable files > would be infiniband_user_t and access should be freely allowed from other > selinux contexts. The infiniband_mgmt_t should be limited to certain > applications (I can provide a list if needed). Could you provide a list? Thank you. Hi dledford, Could you check comment12 ? Thank you. I already answered which files should have which SELinux types: world writeable (mode 666) files should have normal IB type root writeable (mode 600) files should have IB mgmt type Apps that will need IB mgmt type (at a minimum, I might have missed some): From the opensm package: /usr/sbin/opensm /usr/sbin/osmtest from ibutils: /usr/bin/* from ibacm: /usr/bin/* /usr/sbin/* from infiniband-diags: /usr/sbin/* from srptools: /usr/sbin/srp_daemon /usr/sbin/ibsrpdm from perftest: unknown, will need to test it and see what breaks without having mgmt type and decide if we want to give mgmt type or keep the breakage from mvapich2: ditto perftest opa-basic-tools: unknown, will have to test and make recommendations opa-address-resolution: won't need a change, it will get it's type from ibacm as it is just a plugin library that ibacm uses opa-fm: will need testing to confirm, but I suspect it needs /usr/sbin/*, /usr/lib/opa-fm/bin/*, /usr/lib/opa-fm/runtime/* That's the rough list. It might need additional tweaks based on testing, and even though I think I have all of the umad using apps installed at the moment, I could have missed some. I split the devices as was mentioned in comment14. Can I make scratch builds for testing and collecting some AVCs? Attaching scratch builds: http://download.eng.brq.redhat.com/scratch/lvrabec/scratch_selinux_policy_infiniband/ Could somebody test it and collect AVC denials? Thank you. Hi Lukas,
It still seems to be unhappy with your packages;
(again this is from the destination side):
type=AVC msg=audit(1467889442.717:292): avc: denied { read write } for pid=3385 comm="qemu-kvm" name="rdma_cm" dev="devtmpfs" ino=18626 scontext=system_u:system_r:svirt_t:s0:c333,c758 tcontext=system_u:object_r:infiniband_device_t:s0 tclass=chr_file
type=AVC msg=audit(1467889442.717:293): avc: denied { read write } for pid=3385 comm="qemu-kvm" name="rdma_cm" dev="devtmpfs" ino=18626 scontext=system_u:system_r:svirt_t:s0:c333,c758 tcontext=system_u:object_r:infiniband_device_t:s0 tclass=chr_file
qemu-kvm-rhev-2.6.0-9.el7.x86_64
libvirt-1.3.5-1.el7.x86_64
selinux-policy-3.13.1-82.el7.5.noarch
selinux-policy-targeted-3.13.1-82.el7.5.noarch
[root@virtlab403 ~]# ls -lZ /dev/infiniband/
crw-------. root root system_u:object_r:infiniband_mgmt_device_t:s0 issm0
crw-------. root root system_u:object_r:infiniband_mgmt_device_t:s0 issm1
crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 rdma_cm
crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 ucm0
crw-------. root root system_u:object_r:infiniband_mgmt_device_t:s0 umad0
crw-------. root root system_u:object_r:infiniband_mgmt_device_t:s0 umad1
crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 uverbs0
Hi, These AVCs are expected, it's easy fix. I see fixed labeling here. That's good. I'm going to fix these AVCs asap. Thank you for testing. hi, Is there any other issues with this fix selinux build? Does thie bug can be tested and verified now. We are verifying one libvirt testonly bug 1331802, which is depend on this selinux bug. If no other issues for this selinux bug, we'd like to change our libvirt bug to verify status. Thanks. Following is the verify result and steps for your reference: https://bugzilla.redhat.com/show_bug.cgi?id=1331802#c2 Functional testing verified the version selinux-policy-3.13.1-86.el7 has necessary rules. Moving to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2283.html |
Description of problem: In SELinux enforcing mode, an RDMA migration fails with: librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm in permissive we see in audit.log (on the destination): type=VIRT_RESOURCE msg=audit(1428573029.576:1228): pid=1840 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm resrc=cgroup reason=allow vm="rhel71on6" uuid=756bdee2-1f44-494b-91c4-a91c0f4d9cca cgroup="/sys/fs/cgroup/devices/machine.slice/machine-qemu\x2drhel71on6.scope/" class=major category=pty maj=88 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=AVC msg=audit(1428573030.167:1229): avc: denied { read write } for pid=17643 comm="qemu-kvm" name="rdma_cm" dev="devtmpfs" ino=27794 scontext=system_u:system_r:svirt_t:s0:c481,c874 tcontext=system_u:object_r:infiniband_device_t:s0 tclass=chr_file type=AVC msg=audit(1428573030.167:1229): avc: denied { open } for pid=17643 comm="qemu-kvm" path="/dev/infiniband/rdma_cm" dev="devtmpfs" ino=27794 scontext=system_u:system_r:svirt_t:s0:c481,c874 tcontext=system_u:object_r:infiniband_device_t:s0 tclass=chr_file type=SYSCALL msg=audit(1428573030.167:1229): arch=c000003e syscall=2 success=yes exit=29 a0=7f9581ab7c5c a1=80002 a2=7f958a08bf90 a3=7fffaf8a3060 items=0 ppid=1 pid=17643 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c481,c874 key=(null) [root@virtlab414 ~]# ls -lZ /dev/infiniband/ crw-------. root root system_u:object_r:infiniband_device_t:s0 issm0 crw-------. root root system_u:object_r:infiniband_device_t:s0 issm1 crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 rdma_cm crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 ucm0 crw-------. root root system_u:object_r:infiniband_device_t:s0 umad0 crw-------. root root system_u:object_r:infiniband_device_t:s0 umad1 crw-rw-rw-. root root system_u:object_r:infiniband_device_t:s0 uverbs0 Version-Release number of selected component (if applicable): [root@virtlab414 ~]# rpm -qa | grep libvirt libvirt-daemon-driver-interface-1.2.8-16.el7_1.2.x86_64 libvirt-client-1.2.8-16.el7_1.2.x86_64 libvirt-daemon-driver-secret-1.2.8-16.el7_1.2.x86_64 libvirt-daemon-driver-qemu-1.2.8-16.el7_1.2.x86_64 libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.2.x86_64 libvirt-daemon-driver-network-1.2.8-16.el7_1.2.x86_64 libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.2.x86_64 libvirt-daemon-1.2.8-16.el7_1.2.x86_64 libvirt-daemon-kvm-1.2.8-16.el7_1.2.x86_64 libvirt-daemon-driver-storage-1.2.8-16.el7_1.2.x86_64 [root@virtlab414 ~]# rpm -qa | grep qemu qemu-guest-agent-2.1.0-4.el7.x86_64 qemu-kvm-rhev-2.1.2-23.el7_1.1.x86_64 libvirt-daemon-driver-qemu-1.2.8-16.el7_1.2.x86_64 qemu-kvm-common-rhev-2.1.2-23.el7_1.1.x86_64 qemu-kvm-rhev-debuginfo-2.1.2-23.el7_1.1.x86_64 qemu-kvm-tools-rhev-2.1.2-23.el7_1.1.x86_64 qemu-img-rhev-2.1.2-23.el7_1.1.x86_64 ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch [root@virtlab414 ~]# rpm -qa | grep policy selinux-policy-targeted-3.13.1-23.el7.noarch selinux-policy-3.13.1-23.el7.noarch policycoreutils-2.2.5-15.el7.x86_64 (This is a 7 upgraded to 7.1 host) How reproducible: 100% Steps to Reproduce: 1. Take two hosts with RDMA cards 2. Start a VM on one 3. virsh migrate --live --migrateuri rdma://ibpair rhel71on6 qemu+ssh://ibpair/system (where ibpair is the name of the other host) Actual results: error: internal error: early end of file from monitor: possible problem: librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm RDMA ERROR: could not create rdma event channel 2015-04-09T09:43:28.047845Z qemu-kvm: -incoming rdma:[::]:49152: RDMA ERROR: could not create rdma event channel Expected results: A happy migration. Additional info: