libffi is doing execmem, need to trace down which python library is triggering it:
Aug  2 23:19:20 rdo-pksk-8hcmb-rhos-ci-12-controller kernel: nova-api[6121]: segfault at c ip 00007f31016dd945 sp 00007ffdd16bf038 error 6 in libffi.so.6.0.1[7f31016d8000+7000]

I have spent some time looking into this. I can reproduce the issue consistently by trying to upload a SSH key via Nova, using "nova keypair-add test".

Tracing the Nova source, I see it breaks in the call at https://github.com/openstack/nova/blob/1dde750e019176f020a060a79febabd8b71e6391/nova/crypto.py#L139-L140 . This is calling a method from cryptography, which internally uses python-cffi, which in turn uses libffi.

Created attachment 1061452 [details]
Reproducer script. Need to set SELinux bool deny_execmem to "on" before running

After some more testing, I have created a simple reproducer. When running on a system with default SELinux settings (deny_execmem to off) it works fine, but setting deny_execmem to on breaks it. It seems the Nova context does not allow execmem, so it gets the AVC that leads to the subsequent errors.

Is this a libffi bug (because it should not use execmem) or we need to add some SELinux rules?

Adding the libffi maintainer to shed some light on this. Is libffi expected to use execmem?

libffi is supposed to check whether or not selinux is enabled.  If it is, it goes through a code path that, while slower, does not require execmem.  What is the host OS version?

I have been able to reproduce it in Fedora 21 (with libffi 3.1-7), CentOS 7 and RHEL 7 (libffi 3.0.13-11). Happy to provide additional debugging if it proves useful.

On any of these systems where it is failing, could you please send me the results of:

ls -l /selinux

and 

cat /proc/mounts

Created attachment 1062542 [details]
cat /proc/mounts from affected systems

This is the output from a Fedora 21 and a RHEL 7 system. In both cases, the selinuxfs file system is mounted under /sys/fs/selinux

As a quick sanity check I ran the libffi testsuite on RHEL7 in Enforcing mode and came up with nothing.  I should update libffi to test for /sys/fs/selinux in addition to /selinux, but that shouldn't be the problem, because we should be catching that selinux is enabled through the /proc/mounts output.

So, the next step would be for me to reproduce the exact error you see. I see a script attached to this ticket, but I don't know what I need to install to make it work.  Could you please give me instructions, assuming I'm starting with a clean base RHEL7 installation?

Thanks!

From a clean RHEL7 install, you can run the following commands to reproduce it using the script:

# yum -y install http://rdoproject.org/repos/openstack-kilo/rdo-release-kilo.rpm
# yum -y install python-cryptography python-paramiko
# setsebool deny_execmem 1
# <run reproducer script>

I am able to reproduce this on RHEL7 and can look into it a bit more today...

Program received signal SIGSEGV, Segmentation fault.
ffi_prep_closure_loc (closure=0x0, cif=0x1f02a40, 
    fun=0x7fffea500db0 <invoke_callback>, user_data=0x1fc2e10, codeloc=0x0)
    at ../src/x86/ffi64.c:552
552	    = (unsigned long) codeloc;
(gdb) where
#0  ffi_prep_closure_loc (closure=0x0, cif=0x1f02a40, 
    fun=0x7fffea500db0 <invoke_callback>, user_data=0x1fc2e10, codeloc=0x0)
    at ../src/x86/ffi64.c:552
#1  0x00007fffea50121e in b_callback ()
   from /usr/lib64/python2.7/site-packages/_cffi_backend.so
#2  0x00007ffff7af5b74 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#3  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#4  0x00007ffff7af583f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#5  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#6  0x00007ffff7af583f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#7  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#8  0x00007ffff7af583f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#10 0x00007ffff7a84088 in function_call () from /lib64/libpython2.7.so.1.0

Shall we move this to RHEL/libffi ?

Possibly related issue reported upstream https://bugs.launchpad.net/nova/+bug/1483408

Any update ? Hate to bump this issue but I'm also running into segfaults attempting to get nova-api liberty to start on CentOS7 with:
- python-cffi-1.1.2-3.el7.x86_64
- python-cryptography-1.0-1.el7.x86_64

This is currently a blocker for liberty.

By default, deny_execmem is disabled; enabling it in and then saying "fix it globally" is probably not the right answer - I'm pretty sure libffi requires execmem to be off.

A quick workaround would be to allow this AVC to pass:

type=AVC msg=audit(1438557560.333:719): avc:  denied  { execmem } for  pid=6121 comm="nova-api" scontext=system_u:system_r:nova_api_t:s0 tcontext=system_u:system_r:nova_api_t:s0 tclass=process

It looks like all of the various, architecture-specific implementations of ffi_prep_closure_loc() all do various things that require execmem and/or use trampolines.

Oof. This'll take some thought.

Anthony, 

It looks like libffi builds trampoline structures and then executes them later in the x86/ffi64.c both in 3.0.13 and current upstream.  I think that's execmem, by definition?

https://github.com/atgreen/libffi/blob/master/src/x86/ffi64.c#L677

If I don't misunderstand, then there's nothing else to do but allow the AVC for nova.

(In reply to Lon Hohberger from comment #17)
> By default, deny_execmem is disabled; enabling it in and then saying "fix it
> globally" is probably not the right answer - I'm pretty sure libffi requires
> execmem to be off.
> 

Lon,

deny_execmem is off, and even then nova-api crashes with this error. I just mentioned it because it was the quickest way to reproduce the issue with a small script.

nova-api is running in a context that denies execmem, and according to comment #5, libffi should be able to detect SELinux and use a code path that does not use execmem. IMHO this is still a libffi bug. It should be possible to add SELinux rules to allow nova-api (and httpd/keystone-all as well, according to the latest tests) to use execmem, but it would just be a matter of time until another confined service uses libffi and we get the same issue.

This similar to https://bugzilla.redhat.com/show_bug.cgi?id=531233

The workaround for no_execmem described in comment 5 works by mapping the same region of memory twice, once rw and once x.  This is done by creating a temporary file.

The problem is that this dual mapping does not survive fork(), so it is not a solution which works in all cases.  If a process forks and then tries to use the closure in the child process it will crash.  Python uses fork() in this way, so it has to be allowed execmem.

(In reply to Andrew Haley from comment #20)

> The problem is that this dual mapping does not survive fork(), so it is not
> a solution which works in all cases.  If a process forks and then tries to
> use the closure in the child process it will crash.  Python uses fork() in
> this way, so it has to be allowed execmem.

I'm sorry, I got that wrong.  The problem is not that the mapping for the callback is destroyed after the fork, it's that the child process destroys the mapping.  Unfortunately it is shared, so it is also destroyed in the parent.  To make this work properly we'd have to prevent the mapping from being passed to the child or prevent the child from closing everything.

> deny_execmem is off, and even then nova-api crashes with this error.

IIUC exec* booleans (was allow_exec* previously) are for unconfined domains and nova is confined hence boolean doesn't have an effect, cf. http://danwalsh.livejournal.com/6117.html

Also note relevant quote from Dan's article:
"execmem

There might be some reasonable uses for execmem, but Ulrich describes how programs could program in a safer manner to not require this access. Again if you find an application requiring this priv you should report it immediately to the upstream maintainers and to the selinux-policy. Since there is no way to get around this one other than setting allow_execstack to on."

(In reply to Alan Pevec from comment #23)
> Also note relevant quote from Dan's article:
> "execmem
> 
> There might be some reasonable uses for execmem, but Ulrich describes how
> programs could program in a safer manner to not require this access.

We already use this trick, but it does not work in all cases.

Relevant bit of information I forgot to mention - we didn't seem to bump into this until we updated the python-cryptography and python-cffi packages to their current version:
- python-cffi-1.1.2-3.el7.x86_64
- python-cryptography-1.0-1.el7.x86_64

We had to update these packages in order to bring a fix [1] for an upstream bug [2].

We were most likely using the kilo version of these packages when we were not encountering this issue. I know for a fact the version for python-cryptography was 0.8.2 - looking at the repos, this would mean something like:
- python-cffi-0.8.6-1.el7.x86_64
- python-cryptography-0.8.2-1.el7.x86_64

[1]: https://github.com/openstack/requirements/commit/61b24799ff850f6de98bf8e868fbfb249cac5ea1
[2]: https://bugs.launchpad.net/nova/+bug/1481084

For the keystone AVC:

type=AVC msg=audit(1442422246.845:32049): avc:  denied  { execmem } for  pid=19729 comm="httpd" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=process
type=SYSCALL msg=audit(1442422246.845:32049): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=7 a3=22 items=0 ppid=19357 pid=19729 auid=4294967295 uid=163 gid=163 euid=163 suid=163 fsuid=163 egid=163 sgid=163 fsgid=163 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=ANOM_ABEND msg=audit(1442422246.845:32050): auid=4294967295 uid=163 gid=163 ses=4294967295 subj=system_u:system_r:httpd_t:s0 pid=19729 comm="httpd" reason="memory violation" sig=11

Thanks, David.

I think I found a bug in python-cffi - it's passing prot_exec to libffi, which is the victim here.

I have a potential fix.  I'll post it here after I build.

(In reply to Andrew Haley from comment #20)
> This similar to https://bugzilla.redhat.com/show_bug.cgi?id=531233
> 
> The workaround for no_execmem described in comment 5 works by mapping the
> same region of memory twice, once rw and once x.  This is done by creating a
> temporary file.

I made an attempted naive fix which simply adds the selinux check to python-cffi, which didn't work (obviously).

The issue is in cffi - the 1.1.2 version vs 0.8.6 adds code which does mmap using PROT_EXEC and doesn't check for selinux.  It also doesn't do the split-rw/x trick.  HOWEVER, libffi does.

It appears to be an issue in python-cffi, not libffi.

I also haven't looked outside cffi's malloc_closure.h to see where and how the memory is used, but testing a partial rewrite with cffi's allocation routines to simply wrap libffi's allocation routines (and then using ffi_prep_closure_loc instead of ffi_prep_closure) half-worked: some things worked perfectly even in enforcing, some things crashed (weird).

Oh - where my second fix crashed is interesting, it didn't crash in libffi or cffi, but in python-cryptography (it's a little difficult to get useful stack traces of python processes).

For now, I'll add a nova_execmem boolean and turn that on along with httpd_execmem in %post, it'll only be present in openstack-selinux until the cffi bug is worked out.

Not ideal, but better than doing 'setenforce 0' by a large margin.

Moving to RHEL/python-cffi based on comment 29

python-cffi (all the versions I checked) has this problem in c/malloc_closure.h.

Upstream python-cffi applied a fix for grsecurity. Perhaps a similar patch will work for selinux? https://bitbucket.org/cffi/cffi/commits/c7edb1e84eb3

However, this code is copy/paste from upstream python (the cffi module) and exists in all versions of python I checked.
https://hg.python.org/cpython/file/tip/Modules/_ctypes/malloc_closure.c
https://hg.python.org/cpython/file/v2.7.10/Modules/_ctypes/malloc_closure.c

So, I'd like to propose that we solve this in upstream python and then port this patch to python-cffi.

If everyone agrees, then we should fork this bug and assign one to python and make the python-cffi bug depend on the python bug. Thoughts?

Hi,

here are openstack-selinux workarounds:

https://people.redhat.com/lhh/openstack-selinux.spec
https://people.redhat.com/lhh/openstack-selinux-0.6.39-1.el7ost.src.rpm

https://people.redhat.com/lhh/openstack-selinux-0.6.39-1.el7.noarch.rpm

openstack-selinux workaround:
 * Add boolean: nova_use_execmem
 * Enable boolean: nova_use_execmem  (for nova)
 * Enable boolean: httpd_execmem     (for keystone)

Hi,

nova-network has the same problem.
Can you add the boolean as well for nova-network?

module nova_network 1.0;

require {
	type nova_network_t;
	class process execmem;
}

#============= nova_network_t ==============
allow nova_network_t self:process execmem;

Thanks,

Bruno Bompastor.

The problem is caused by the current cffi's callback API. It converts a Python callback into a C function pointer. libffi creates dynamic code to create a closure. The latest version of python-cryptography uses callbacks in a couple of places like the os.urandom() CPRNG engine. The new code gets called right at import time.

I've talked to cffi devs and we explored possible solutions. Armin Rigo found a better way and is working on a new callback API for cffi 1.4. https://bitbucket.org/cffi/cffi/branch/static-callback#chg-doc/source/using.rst  I'll contribute patches to python-cryptography as soon as possible.

Patches are now available upstream.

If this problem is fixed as part of python-cffi rebase (Bug 1297585), this bug should be moved also to ON_QA (and QA teams should agree who would test it).

Verified. Christian provided steps to verify this.

Version:
========
[root@dhcp207-129 ~]# rpm -q python-cffi
python-cffi-1.6.0-5.el7.x86_64
[root@dhcp207-129 ~]# 

Please find the attached file for verification steps's console output.

Created attachment 1193708 [details]
console output with verification steps