Bug 1249685 - python-cffi should not require execmem when selinux is enabled
python-cffi should not require execmem when selinux is enabled
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: python-cffi (Show other bugs)
7.1
Unspecified Unspecified
high Severity high
: pre-dev-freeze
: 7.3
Assigned To: Nathaniel McCallum
Namita Soman
: Automation
Depends On: 1297581
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-03 10:50 EDT by wes hayutin
Modified: 2017-08-11 07:55 EDT (History)
13 users (show)

See Also:
Fixed In Version: python-cffi-1.6.0-4.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-11 07:55:29 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Reproducer script. Need to set SELinux bool deny_execmem to "on" before running (438 bytes, text/plain)
2015-08-11 07:35 EDT, Javier Peña
no flags Details
cat /proc/mounts from affected systems (4.95 KB, text/plain)
2015-08-13 09:23 EDT, Javier Peña
no flags Details
console output with verification steps (2.49 KB, text/plain)
2016-08-24 12:03 EDT, Kaleem
no flags Details

  None (edit)
Description wes hayutin 2015-08-03 10:50:45 EDT
Description of problem:

Full logs https://prod-rdojenkins.rhcloud.com/view/RDO-Liberty-Delorean-Trunk/job/khaleesi-rdo-liberty-delorean-centos-7.0-aio-packstack-neutron-ml2-vxlan-rabbitmq-enforcing-tempest-rpm-minimal/12/artifact/khaleesi/collected_files/rdo-pksk-8hcmb-rhos-ci-12-controller.tar.bz2



Denials on Nova api

type=AVC msg=audit(1438557558.744:717): avc:  denied  { execmem } for  pid=6120 comm="nova-api" scontext=system_u:system_r:nova_api_t:s0 tcontext=system_u:system_r:nova_api_t:s0 tclass=process
type=SYSCALL msg=audit(1438557558.744:717): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=7 a3=22 items=0 ppid=1133 pid=6120 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-api" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_api_t:s0 key=(null)
type=ANOM_ABEND msg=audit(1438557558.747:718): auid=4294967295 uid=162 gid=162 ses=4294967295 subj=system_u:system_r:nova_api_t:s0 pid=6120 comm="nova-api" reason="memory violation" sig=11
type=AVC msg=audit(1438557560.333:719): avc:  denied  { execmem } for  pid=6121 comm="nova-api" scontext=system_u:system_r:nova_api_t:s0 tcontext=system_u:system_r:nova_api_t:s0 tclass=process
type=SYSCALL msg=audit(1438557560.333:719): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=7 a3=22 items=0 ppid=1133 pid=6121 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-api" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_api_t:s0 key=(null)
type=ANOM_ABEND msg=audit(1438557560.335:720): auid=4294967295 uid=162 gid=162 ses=4294967295 subj=system_u:system_r:nova_api_t:s0 pid=6121 comm="nova-api" reason="memory violation" sig=11
Comment 1 Alan Pevec 2015-08-03 12:00:03 EDT
libffi is doing execmem, need to trace down which python library is triggering it:
Aug  2 23:19:20 rdo-pksk-8hcmb-rhos-ci-12-controller kernel: nova-api[6121]: segfault at c ip 00007f31016dd945 sp 00007ffdd16bf038 error 6 in libffi.so.6.0.1[7f31016d8000+7000]
Comment 2 Javier Peña 2015-08-11 07:31:01 EDT
I have spent some time looking into this. I can reproduce the issue consistently by trying to upload a SSH key via Nova, using "nova keypair-add test".

Tracing the Nova source, I see it breaks in the call at https://github.com/openstack/nova/blob/1dde750e019176f020a060a79febabd8b71e6391/nova/crypto.py#L139-L140 . This is calling a method from cryptography, which internally uses python-cffi, which in turn uses libffi.
Comment 3 Javier Peña 2015-08-11 07:35:35 EDT
Created attachment 1061452 [details]
Reproducer script. Need to set SELinux bool deny_execmem to "on" before running

After some more testing, I have created a simple reproducer. When running on a system with default SELinux settings (deny_execmem to off) it works fine, but setting deny_execmem to on breaks it. It seems the Nova context does not allow execmem, so it gets the AVC that leads to the subsequent errors.

Is this a libffi bug (because it should not use execmem) or we need to add some SELinux rules?
Comment 4 Javier Peña 2015-08-13 04:42:15 EDT
Adding the libffi maintainer to shed some light on this. Is libffi expected to use execmem?
Comment 5 Anthony Green 2015-08-13 07:19:57 EDT
libffi is supposed to check whether or not selinux is enabled.  If it is, it goes through a code path that, while slower, does not require execmem.  What is the host OS version?
Comment 6 Javier Peña 2015-08-13 07:47:32 EDT
I have been able to reproduce it in Fedora 21 (with libffi 3.1-7), CentOS 7 and RHEL 7 (libffi 3.0.13-11). Happy to provide additional debugging if it proves useful.
Comment 7 Anthony Green 2015-08-13 08:02:51 EDT
On any of these systems where it is failing, could you please send me the results of:

ls -l /selinux

and 

cat /proc/mounts
Comment 8 Javier Peña 2015-08-13 09:23:00 EDT
Created attachment 1062542 [details]
cat /proc/mounts from affected systems

This is the output from a Fedora 21 and a RHEL 7 system. In both cases, the selinuxfs file system is mounted under /sys/fs/selinux
Comment 9 Anthony Green 2015-08-13 09:48:23 EDT
As a quick sanity check I ran the libffi testsuite on RHEL7 in Enforcing mode and came up with nothing.  I should update libffi to test for /sys/fs/selinux in addition to /selinux, but that shouldn't be the problem, because we should be catching that selinux is enabled through the /proc/mounts output.

So, the next step would be for me to reproduce the exact error you see. I see a script attached to this ticket, but I don't know what I need to install to make it work.  Could you please give me instructions, assuming I'm starting with a clean base RHEL7 installation?

Thanks!
Comment 11 Javier Peña 2015-08-13 10:11:00 EDT
From a clean RHEL7 install, you can run the following commands to reproduce it using the script:

# yum -y install http://rdoproject.org/repos/openstack-kilo/rdo-release-kilo.rpm
# yum -y install python-cryptography python-paramiko
# setsebool deny_execmem 1
# <run reproducer script>
Comment 13 Anthony Green 2015-08-13 10:59:37 EDT
I am able to reproduce this on RHEL7 and can look into it a bit more today...

Program received signal SIGSEGV, Segmentation fault.
ffi_prep_closure_loc (closure=0x0, cif=0x1f02a40, 
    fun=0x7fffea500db0 <invoke_callback>, user_data=0x1fc2e10, codeloc=0x0)
    at ../src/x86/ffi64.c:552
552	    = (unsigned long) codeloc;
(gdb) where
#0  ffi_prep_closure_loc (closure=0x0, cif=0x1f02a40, 
    fun=0x7fffea500db0 <invoke_callback>, user_data=0x1fc2e10, codeloc=0x0)
    at ../src/x86/ffi64.c:552
#1  0x00007fffea50121e in b_callback ()
   from /usr/lib64/python2.7/site-packages/_cffi_backend.so
#2  0x00007ffff7af5b74 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#3  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#4  0x00007ffff7af583f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#5  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#6  0x00007ffff7af583f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#7  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#8  0x00007ffff7af583f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#10 0x00007ffff7a84088 in function_call () from /lib64/libpython2.7.so.1.0
Comment 14 Alan Pevec 2015-08-17 12:17:48 EDT
Shall we move this to RHEL/libffi ?
Comment 15 Alan Pevec 2015-08-19 11:06:24 EDT
Possibly related issue reported upstream https://bugs.launchpad.net/nova/+bug/1483408
Comment 16 David Moreau Simard 2015-09-14 17:32:53 EDT
Any update ? Hate to bump this issue but I'm also running into segfaults attempting to get nova-api liberty to start on CentOS7 with:
- python-cffi-1.1.2-3.el7.x86_64
- python-cryptography-1.0-1.el7.x86_64

This is currently a blocker for liberty.
Comment 17 Lon Hohberger 2015-09-15 17:21:30 EDT
By default, deny_execmem is disabled; enabling it in and then saying "fix it globally" is probably not the right answer - I'm pretty sure libffi requires execmem to be off.

A quick workaround would be to allow this AVC to pass:

type=AVC msg=audit(1438557560.333:719): avc:  denied  { execmem } for  pid=6121 comm="nova-api" scontext=system_u:system_r:nova_api_t:s0 tcontext=system_u:system_r:nova_api_t:s0 tclass=process

It looks like all of the various, architecture-specific implementations of ffi_prep_closure_loc() all do various things that require execmem and/or use trampolines.

Oof. This'll take some thought.
Comment 18 Lon Hohberger 2015-09-15 17:45:45 EDT
Anthony, 

It looks like libffi builds trampoline structures and then executes them later in the x86/ffi64.c both in 3.0.13 and current upstream.  I think that's execmem, by definition?

https://github.com/atgreen/libffi/blob/master/src/x86/ffi64.c#L677

If I don't misunderstand, then there's nothing else to do but allow the AVC for nova.
Comment 19 Javier Peña 2015-09-16 03:30:19 EDT
(In reply to Lon Hohberger from comment #17)
> By default, deny_execmem is disabled; enabling it in and then saying "fix it
> globally" is probably not the right answer - I'm pretty sure libffi requires
> execmem to be off.
> 

Lon,

deny_execmem is off, and even then nova-api crashes with this error. I just mentioned it because it was the quickest way to reproduce the issue with a small script.

nova-api is running in a context that denies execmem, and according to comment #5, libffi should be able to detect SELinux and use a code path that does not use execmem. IMHO this is still a libffi bug. It should be possible to add SELinux rules to allow nova-api (and httpd/keystone-all as well, according to the latest tests) to use execmem, but it would just be a matter of time until another confined service uses libffi and we get the same issue.
Comment 20 Andrew Haley 2015-09-16 08:13:26 EDT
This similar to https://bugzilla.redhat.com/show_bug.cgi?id=531233

The workaround for no_execmem described in comment 5 works by mapping the same region of memory twice, once rw and once x.  This is done by creating a temporary file.

The problem is that this dual mapping does not survive fork(), so it is not a solution which works in all cases.  If a process forks and then tries to use the closure in the child process it will crash.  Python uses fork() in this way, so it has to be allowed execmem.
Comment 21 Andrew Haley 2015-09-16 09:33:05 EDT
(In reply to Andrew Haley from comment #20)

> The problem is that this dual mapping does not survive fork(), so it is not
> a solution which works in all cases.  If a process forks and then tries to
> use the closure in the child process it will crash.  Python uses fork() in
> this way, so it has to be allowed execmem.

I'm sorry, I got that wrong.  The problem is not that the mapping for the callback is destroyed after the fork, it's that the child process destroys the mapping.  Unfortunately it is shared, so it is also destroyed in the parent.  To make this work properly we'd have to prevent the mapping from being passed to the child or prevent the child from closing everything.
Comment 22 Alan Pevec 2015-09-16 10:33:11 EDT
> deny_execmem is off, and even then nova-api crashes with this error.

IIUC exec* booleans (was allow_exec* previously) are for unconfined domains and nova is confined hence boolean doesn't have an effect, cf. http://danwalsh.livejournal.com/6117.html
Comment 23 Alan Pevec 2015-09-16 10:35:18 EDT
Also note relevant quote from Dan's article:
"execmem

There might be some reasonable uses for execmem, but Ulrich describes how programs could program in a safer manner to not require this access. Again if you find an application requiring this priv you should report it immediately to the upstream maintainers and to the selinux-policy. Since there is no way to get around this one other than setting allow_execstack to on."
Comment 24 Andrew Haley 2015-09-16 10:41:18 EDT
(In reply to Alan Pevec from comment #23)
> Also note relevant quote from Dan's article:
> "execmem
> 
> There might be some reasonable uses for execmem, but Ulrich describes how
> programs could program in a safer manner to not require this access.

We already use this trick, but it does not work in all cases.
Comment 25 David Moreau Simard 2015-09-16 11:22:03 EDT
Relevant bit of information I forgot to mention - we didn't seem to bump into this until we updated the python-cryptography and python-cffi packages to their current version:
- python-cffi-1.1.2-3.el7.x86_64
- python-cryptography-1.0-1.el7.x86_64

We had to update these packages in order to bring a fix [1] for an upstream bug [2].

We were most likely using the kilo version of these packages when we were not encountering this issue. I know for a fact the version for python-cryptography was 0.8.2 - looking at the repos, this would mean something like:
- python-cffi-0.8.6-1.el7.x86_64
- python-cryptography-0.8.2-1.el7.x86_64

[1]: https://github.com/openstack/requirements/commit/61b24799ff850f6de98bf8e868fbfb249cac5ea1
[2]: https://bugs.launchpad.net/nova/+bug/1481084
Comment 26 David Moreau Simard 2015-09-16 12:51:49 EDT
For the keystone AVC:

type=AVC msg=audit(1442422246.845:32049): avc:  denied  { execmem } for  pid=19729 comm="httpd" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=process
type=SYSCALL msg=audit(1442422246.845:32049): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=7 a3=22 items=0 ppid=19357 pid=19729 auid=4294967295 uid=163 gid=163 euid=163 suid=163 fsuid=163 egid=163 sgid=163 fsgid=163 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=ANOM_ABEND msg=audit(1442422246.845:32050): auid=4294967295 uid=163 gid=163 ses=4294967295 subj=system_u:system_r:httpd_t:s0 pid=19729 comm="httpd" reason="memory violation" sig=11
Comment 27 Lon Hohberger 2015-09-16 13:34:58 EDT
Thanks, David.

I think I found a bug in python-cffi - it's passing prot_exec to libffi, which is the victim here.
Comment 28 Lon Hohberger 2015-09-16 13:54:36 EDT
I have a potential fix.  I'll post it here after I build.
Comment 29 Lon Hohberger 2015-09-17 11:44:52 EDT
(In reply to Andrew Haley from comment #20)
> This similar to https://bugzilla.redhat.com/show_bug.cgi?id=531233
> 
> The workaround for no_execmem described in comment 5 works by mapping the
> same region of memory twice, once rw and once x.  This is done by creating a
> temporary file.

I made an attempted naive fix which simply adds the selinux check to python-cffi, which didn't work (obviously).

The issue is in cffi - the 1.1.2 version vs 0.8.6 adds code which does mmap using PROT_EXEC and doesn't check for selinux.  It also doesn't do the split-rw/x trick.  HOWEVER, libffi does.

It appears to be an issue in python-cffi, not libffi.

I also haven't looked outside cffi's malloc_closure.h to see where and how the memory is used, but testing a partial rewrite with cffi's allocation routines to simply wrap libffi's allocation routines (and then using ffi_prep_closure_loc instead of ffi_prep_closure) half-worked: some things worked perfectly even in enforcing, some things crashed (weird).
Comment 30 Lon Hohberger 2015-09-17 11:46:39 EDT
Oh - where my second fix crashed is interesting, it didn't crash in libffi or cffi, but in python-cryptography (it's a little difficult to get useful stack traces of python processes).
Comment 32 Lon Hohberger 2015-09-17 13:20:22 EDT
For now, I'll add a nova_execmem boolean and turn that on along with httpd_execmem in %post, it'll only be present in openstack-selinux until the cffi bug is worked out.

Not ideal, but better than doing 'setenforce 0' by a large margin.
Comment 33 Alan Pevec 2015-09-18 05:48:31 EDT
Moving to RHEL/python-cffi based on comment 29
Comment 37 Nathaniel McCallum 2015-09-18 11:42:49 EDT
python-cffi (all the versions I checked) has this problem in c/malloc_closure.h.

Upstream python-cffi applied a fix for grsecurity. Perhaps a similar patch will work for selinux? https://bitbucket.org/cffi/cffi/commits/c7edb1e84eb3

However, this code is copy/paste from upstream python (the cffi module) and exists in all versions of python I checked.
https://hg.python.org/cpython/file/tip/Modules/_ctypes/malloc_closure.c
https://hg.python.org/cpython/file/v2.7.10/Modules/_ctypes/malloc_closure.c

So, I'd like to propose that we solve this in upstream python and then port this patch to python-cffi.

If everyone agrees, then we should fork this bug and assign one to python and make the python-cffi bug depend on the python bug. Thoughts?
Comment 40 Lon Hohberger 2015-09-18 11:59:10 EDT
openstack-selinux workaround:
 * Add boolean: nova_use_execmem
 * Enable boolean: nova_use_execmem  (for nova)
 * Enable boolean: httpd_execmem     (for keystone)
Comment 42 Bruno Bompastor 2015-11-16 05:25:01 EST
Hi,

nova-network has the same problem.
Can you add the boolean as well for nova-network?

module nova_network 1.0;

require {
	type nova_network_t;
	class process execmem;
}

#============= nova_network_t ==============
allow nova_network_t self:process execmem;

Thanks,

Bruno Bompastor.
Comment 43 Christian Heimes 2015-11-16 08:06:04 EST
The problem is caused by the current cffi's callback API. It converts a Python callback into a C function pointer. libffi creates dynamic code to create a closure. The latest version of python-cryptography uses callbacks in a couple of places like the os.urandom() CPRNG engine. The new code gets called right at import time.

I've talked to cffi devs and we explored possible solutions. Armin Rigo found a better way and is working on a new callback API for cffi 1.4. https://bitbucket.org/cffi/cffi/branch/static-callback#chg-doc/source/using.rst  I'll contribute patches to python-cryptography as soon as possible.
Comment 44 Nathaniel McCallum 2016-01-11 18:05:06 EST
Patches are now available upstream.
Comment 45 Martin Kosek 2016-05-31 05:40:55 EDT
If this problem is fixed as part of python-cffi rebase (Bug 1297585), this bug should be moved also to ON_QA (and QA teams should agree who would test it).
Comment 49 Kaleem 2016-08-24 12:03:04 EDT
Verified. Christian provided steps to verify this.

Version:
========
[root@dhcp207-129 ~]# rpm -q python-cffi
python-cffi-1.6.0-5.el7.x86_64
[root@dhcp207-129 ~]# 

Please find the attached file for verification steps's console output.
Comment 50 Kaleem 2016-08-24 12:03 EDT
Created attachment 1193708 [details]
console output with verification steps

Note You need to log in before you can comment on or make changes to this bug.