After upgrading my FreeIPA server machine from Fedora 21 to Fedora 23, I started getting SELinux denials for 'execmem'. I'll put the 'sealert -a' output below. I don't know what exactly is causing this, but filing against FreeIPA for triage. I ran 'execstack' over all .so(.*) files on the system and everything in /usr/bin and /usr/sbin but didn't find any marked with an X. Not sure how else to investigate this. found 2 alerts in /var/log/audit/audit.log -------------------------------------------------------------------------------- SELinux is preventing /usr/sbin/httpd from using the execmem access on a process. ***** Plugin catchall_boolean (89.3 confidence) suggests ****************** If you want to allow httpd to execmem Then you must tell SELinux about this by enabling the 'httpd_execmem' boolean. Do setsebool -P httpd_execmem 1 ***** Plugin catchall (11.6 confidence) suggests ************************** If you believe that httpd should be allowed execmem access on processes labeled httpd_t by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep httpd /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:httpd_t:s0 Target Context system_u:system_r:httpd_t:s0 Target Objects Unknown [ process ] Source httpd Source Path /usr/sbin/httpd Port <Unknown> Host <Unknown> Source RPM Packages httpd-2.4.16-1.fc23.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-152.fc23.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name id.happyassassin.net Platform Linux id.happyassassin.net 4.2.3-300.fc23.x86_64 #1 SMP Mon Oct 5 15:42:54 UTC 2015 x86_64 x86_64 Alert Count 346 First Seen 2015-11-01 16:29:22 PST Last Seen 2015-11-01 17:02:56 PST Local ID a90ca5d0-bcff-482c-9b23-7447f5a19c9a Raw Audit Messages type=AVC msg=audit(1446426176.247:514): avc: denied { execmem } for pid=1453 comm="httpd" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=process permissive=0 type=SYSCALL msg=audit(1446426176.247:514): arch=x86_64 syscall=mmap success=no exit=EACCES a0=0 a1=1000 a2=7 a3=22 items=0 ppid=1107 pid=1453 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm=httpd exe=/usr/sbin/httpd subj=system_u:system_r:httpd_t:s0 key=(null) Hash: httpd,httpd_t,httpd_t,process,execmem -------------------------------------------------------------------------------- SELinux is preventing /usr/bin/python2.7 from using the execmem access on a process. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that python2.7 should be allowed execmem access on processes labeled certmonger_t by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep dogtag-ipa-ca-r /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp Additional Information: Source Context system_u:system_r:certmonger_t:s0 Target Context system_u:system_r:certmonger_t:s0 Target Objects Unknown [ process ] Source dogtag-ipa-ca-r Source Path /usr/bin/python2.7 Port <Unknown> Host <Unknown> Source RPM Packages python-2.7.10-8.fc23.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-152.fc23.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name id.happyassassin.net Platform Linux id.happyassassin.net 4.2.3-300.fc23.x86_64 #1 SMP Mon Oct 5 15:42:54 UTC 2015 x86_64 x86_64 Alert Count 24 First Seen 2015-11-01 16:01:17 PST Last Seen 2015-11-01 17:02:24 PST Local ID c799c53d-9acd-4637-b11d-02e19c4ede64 Raw Audit Messages type=AVC msg=audit(1446426144.701:447): avc: denied { execmem } for pid=853 comm="dogtag-ipa-ca-r" scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:certmonger_t:s0 tclass=process permissive=0 type=SYSCALL msg=audit(1446426144.701:447): arch=x86_64 syscall=mmap success=no exit=EACCES a0=0 a1=1000 a2=7 a3=22 items=0 ppid=496 pid=853 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=dogtag-ipa-ca-r exe=/usr/bin/python2.7 subj=system_u:system_r:certmonger_t:s0 key=(null) Hash: dogtag-ipa-ca-r,certmonger_t,certmonger_t,process,execmem
Honza, have you see this?
Note this was on my real server, which was installed as like F19 or something and upgraded since then, and this was after I upgraded to F23, encountered https://bugzilla.redhat.com/show_bug.cgi?id=1274905 , and had to fix that up manually. So that could be involved, I guess. I'll try and see if I can find any errors later.
F19? That makes it a bit harder to create a reproducer for your problem. :) I've done a new installation of FreeIPA on a fresh Fedora 23 box. The installation and some cert operations worked without triggering any SELinux warnings. I can't pin-out the exact operation that has triggered the SELinux error. Do you remember if the error coincides with the upgrade or with an operation after the update, e.g. cert renewal?
The denials definitely started with the F23 upgrade. It looks like the dogtag denials came only when dealing with the initial upgrade issues. The httpd ones seem to happen constantly, exactly two every 6.something seconds.
Ah, I didn't realize that both alerts are really separate issues. I thought they might be related. Do you see any errors or messages in /var/log/httpd/error_log that are related to the execmem alert for httpd?
Petr: I haven't seen this.
Well, yeah, there are some very obvious errors. If I turn the execmem boolean back off (the default setting) and restart httpd, I get a stream of execmem denials, and a stream of segfaults in httpd child processes: [Thu Nov 05 09:20:31.662167 2015] [core:notice] [pid 26778] AH00052: child pid 27463 exit signal Segmentation fault (11) [Thu Nov 05 09:20:31.662817 2015] [core:notice] [pid 26778] AH00052: child pid 27464 exit signal Segmentation fault (11) [Thu Nov 05 09:20:32.664291 2015] [core:notice] [pid 26778] AH00052: child pid 27471 exit signal Segmentation fault (11) I'm fairly sure those are attempts to spawn the ipa wsgi process, so that's the thing that's really causing the problem. I also noticed that another set of the dogtag execmem denials appeared this morning, when my system ran automatic updates and updated pki-core and a bunch of other stuff. There were 8 denials all looking like this: time->Thu Nov 5 04:53:37 2015 type=PROCTITLE msg=audit(1446728017.425:1042): proctitle=2F7573722F62696E2F707974686F6E32002D45002F7573722F6C6962657865632F636572746D6F6E6765722F646F677461672D6970612D63612D72656E65772D6167656E742D7375626D6974 type=SYSCALL msg=audit(1446728017.425:1042): arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=7 a3=22 items=0 ppid=12778 pid=12833 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dogtag-ipa-ca-r" exe="/usr/bin/python2.7" subj=system_u:system_r:certmonger_t:s0 key=(null) type=AVC msg=audit(1446728017.425:1042): avc: denied { execmem } for pid=12833 comm="dogtag-ipa-ca-r" scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:system_r:certmonger_t:s0 tclass=process permissive=0 and right after the denials, the dogtag process actually segfaults, just like the ipa wsgi processes are segfaulting: Nov 05 04:53:37 id.happyassassin.net kernel: dogtag-ipa-ca-r[12830]: segfault at 0 ip 00007f7b4e46790b sp 00007ffdd78c7ec8 error 6 in libffi.so.6.0.2[7f7b4e462000+7000]
I've a wild guess. FreeIPA 4.2 has a new feature called KDC Proxy. Since I integrated the feature myself and also worked on the python-kdcproxy package, I'm familiar with its internal details. I've designed the KDC Proxy to run as a second mod_wsgi daemon process. The second daemon isolates the proxy from the mod_wsgi daemon processes, that serve FreeIPA's web interface. Since there are only two mod_wsgi daemon processes in FreeIPA and you haven't complained about problems with the web interface, it's logical to assume that the kdcproxy daemon is the culprit. Plus python-kdcproxy package uses ctypes to interact with libkrb5.so and ctypes doesn't play well with SELinux's execmem protection. You can put it to a test. Please run these commands as root: # ipa-ldap-updater /usr/share/ipa/kdcproxy-disable.uldif # systemctl restart httpd.service After the restart the symlink /etc/httpd/conf.d/ipa-kdc-proxy.conf should be gone and Apache should no longer spawn mod_wsgi daemons for kdcproxy. If you don't see SELinux violations or segfaults anymore, then you have successfully pin-pointed the offending process.
The segfault in libffi.so could be related to either a ctypes problem or similar. ctypes uses the foreign function interface library (libffi) to load shared libraries and execute functions. I guess the segfaults are caused by execmem protection.
Yes, the crashes always follow the execmem denials (and with the httpd execmem boolean set to on, the IPA wsgi processes don't crash). I have not actually checked if the web interface works when I have the httpd execmem boolean set to off. As soon as I saw the denials, I filed the bug, and set the execmem boolean on to verify that it stopped the denials. I didn't check at any point whether the web interface was working when the denials were happening. I just checked, and indeed, the web interface does not work when the boolean is set to off and the denials are happening and the wsgi processes are crashing. I just get a page saying "Unknown Error". When I set the boolean to on, the web interface works.
Jan: could you run this: getsebool httpd_execmem and just check you don't have it set to 'on'? You may have set it in the past and forgotten about it...
This is from a new clean F23 VM after running ipa-server-install: # getsebool httpd_execmem httpd_execmem --> off I don't have any other F23 system, sorry.
Ok it is being caused by libffi. So is # ipa-ldap-updater /usr/share/ipa/kdcproxy-disable.uldif # systemctl restart httpd.service a solution here?
I doubt it - since that theory was based on the idea the web UI keeps working, which it doesn't - but I'll check. ... Indeed, that does *not* stop the AVCs.
If we're suspecting libffi here, note it has some special handling in this file: https://github.com/ffi/ffi/blob/master/ext/ffi_c/libffi/src/closures.c#L133-L164 which is intended to detect whether SELinux is enabled and avoid doing the kind of things that trigger these AVCs. I guess it's possible something in there has started going wrong (though the package has only had automated rebuilds)? The other possibility I guess is this patch pbrobinson added to fix some aarch64 issues: http://pkgs.fedoraproject.org/cgit/libffi.git/tree/libffi-aarch64-rhbz1174037.patch I guess I can try a build with the SELinux detection cleverness bypassed and the value just hardcoded to 1, and also one with pbrobinson's patch reverted...
Neither of those helps, unfortunately.
Aha. So I've narrowed it down a bit. If I downgrade python-cffi to 0.8.6 - the version that was in F21 - and downgrade python-cryptography to 0.9 and rebuild it against python-cffi (1.0 will not build against cffi 0.8.6), the denials go away. So it seems likely that this is something between python-cffi 0.8.6 and 1.1.2. I did try updating to cffi 1.3.0, but that doesn't help.
As for why Jan doesn't see it on a fresh install, a theory - freeipa seems to use cffi only via python-cryptography, perhaps my upgraded system has a somewhat different crypto config to a fresh install and is hitting a crypto codepath that a fresh install does not hit (some cipher or something that is not used on a fresh install, perhaps?)
Huh, interesting thing: the cffi downgrade is not actually needed! Just downgrading python-cryptography to 0.9 seems to be sufficient. So the problem is actually between python-cryptography 0.9 and 1.0, it seems.
So the python-cryptography folks are willing to help with this but they need a traceback, and no matter how many dodgy Google search results I follow I cannot for the life of me persuade apache to generate backtraces for the crashing wsgi child processes; anyone got any hot tips on that? In the meantime, I can say that it crashes exactly at this line in ipalib/plugable.py , Api.finalize() : self.__do_if_not_done('load_plugins') a debug log message added before that line gets printed, one added after that line does not.
OK, so narrowing it down further, I turned the 'importing plugin module %s' log in import_plugins() into an 'info' rather than a 'debug' and now I can see that, every time, it crashes when loading the ipalib.plugins.vault plugin: [Fri Nov 06 14:26:55.040989 2015] [wsgi:error] [pid 9152] ipa: INFO: importing plugin module ipalib.plugins.vault [Fri Nov 06 14:26:55.041308 2015] [wsgi:error] [pid 9153] ipa: INFO: importing plugin module ipalib.plugins.vault [Fri Nov 06 14:26:55.600933 2015] [core:notice] [pid 9101] AH00052: child pid 9152 exit signal Segmentation fault (11) [Fri Nov 06 14:26:55.601495 2015] [core:notice] [pid 9101] AH00052: child pid 9153 exit signal Segmentation fault (11) which certainly makes sense, as we're in the crypto bits there. Still can't get a traceback out of httpd, grr.
So hmm. On my production box, either of these is sufficient to cause the problem, run in the system_u:system_r:httpd_t:s0 context via runcon: ---- from ipaserver.install import cainstance, certs ---- ---- from ipalib import api api.bootstrap(context='server', log=None) import ipalib.plugins.vault ---- but neither reproduces on a clean install. Will keep poking, and see if I can get a backtrace from the reproducers on my prod box.
Created attachment 1090812 [details] (C) backtrace of one of the reproducers Well, I managed to get a backtrace from one of the reproducers on my prod machine; here it is. Hope someone can get something out of this.
My reproducer is now down to: import pki Which kinda implies we're in /etc/pki somewhere, but lord knows where. I'll keep trying to narrow it down.
woohoo, alright! I have a reproducer on a clean install: setsebool -P deny_execmem on dnf install python-urllib3 dnf install python-ndg_httpsclient python -c 'import urllib3.contrib.pyopenssl' The first denies execmem for all processes (by default stuff you run as a regular user is allowed to execmem, but httpd isn't). And ndg_httpsclient is important. The crash actually seems to happen when importing requests, which has this block: try: from .packages.urllib3.contrib import pyopenssl pyopenssl.inject_into_urllib3() except ImportError: pass if python-ndg_httpsclient isn't installed, trying to import urllib3.contrib.pyopenssl gives you this: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 48, in <module> from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT ImportError: No module named ndg.httpsclient.ssl_peer_verification which, as we can see, requests will just silently accept. So that's why you don't see the issue on a clean install, even a clean FreeIPA deployment: python-ndg_httpsclient isn't a dependency of freeipa so it won't be pulled in. From the yum history on my test machine, it looks like it used to be a dependency of python-urllib3, but no longer is (I guess someone initially added it to make sure SNI worked, then it got taken out to reduce deps or something).
So ultimately we wind up with this as a minimal reproducer: setsebool -P deny_execmem on dnf install python-cryptography python -c 'from cryptography.hazmat.bindings.openssl.binding import Binding' python-cryptography devs reckon this is most likely a cffi or ffi bug, so setting back to python-cffi for now.
Filed an issue with upstream python-cffi: https://bitbucket.org/cffi/cffi/issues/231
Upstream says #1256106 is similar.
You are making progress, awesome! Your analysis makes sense. The Python client library for Dogtag PKI uses python-requests to talk to Dogtags' REST API. python-requests is build on top of urllib3 which itself can either use PyOpenSSL or Python's ssl module for TLS/SSL. PyOpenSSL has been rewritten on top of python-cryptography, which uses cffi. urllib3 tries to use PyOpenSSL because Python's ssl module for Python < 2.7.9 does neither support SNI nor has builtin support for CN/SAN hostname matching. When python-ndg_httpsclient is not installed, requests is forced to fall back to Python's ssl module. Here is a slightly better reproducer with a stack trace. setsebool -P deny_execmem on dnf install gdb python-cryptography dnf debuginfo-install python python-cryptography python-cffi # gdb python GNU gdb (GDB) Fedora 7.10-29.fc23 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from python...Reading symbols from /usr/lib/debug/usr/bin/python2.7.debug...done. done. (gdb) run Starting program: /usr/bin/python [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Python 2.7.10 (default, Sep 8 2015, 17:20:17) [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from cryptography.hazmat.bindings.openssl.binding import Binding Program received signal SIGSEGV, Segmentation fault. ffi_prep_closure_loc (closure=closure@entry=0x0, cif=0x7ffff7e63b20, fun=fun@entry= 0x7fffecf408c0 <invoke_callback>, user_data=user_data@entry=0x7ffff7e3f6e0, codeloc=codeloc@entry=0x0) at ../src/x86/ffi64.c:550 550 tramp[0] = 0xbb49; /* mov <code>, %r11 */ Missing separate debuginfos, use: dnf debuginfo-install krb5-libs-1.13.2-12.fc23.x86_64 (gdb) bt #0 ffi_prep_closure_loc (closure=closure@entry=0x0, cif=0x7ffff7e63b20, fun=fun@entry= 0x7fffecf408c0 <invoke_callback>, user_data=user_data@entry=0x7ffff7e3f6e0, codeloc=codeloc@entry=0x0) at ../src/x86/ffi64.c:550 #1 0x00007fffecd28b48 in ffi_prep_closure (closure=closure@entry=0x0, cif=<optimized out>, fun=fun@entry= 0x7fffecf408c0 <invoke_callback>, user_data=user_data@entry=0x7ffff7e3f6e0) at ../src/prep_cif.c:242 #2 0x00007fffecf40d88 in b_callback (self=self@entry=0x0, args=args@entry=(<_cffi_backend.CTypeDescr at remote 0x7ffff7ee68c0>, <function at remote 0x7ffff7e5e8c0>, -1)) at c/_cffi_backend.c:4858 #3 0x00007fffecf410fb in _ffi_callback_decorator ( outer_args=(<_cffi_backend.CTypeDescr at remote 0x7ffff7ee68c0>, <function at remote 0x7ffff7e5e8c0>, -1), fn=<optimized out>) at c/ffi_obj.c:696 #4 0x00007ffff7af5934 in call_function (oparg=<optimized out>, pp_stack=0x7fffffffdba0) at /usr/src/debug/Python-2.7.10/Python/ceval.c:4100 #5 PyEval_EvalFrameEx ( f=f@entry=Frame 0x7ffff7e7c750, for file /usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py, line 15, in <module> (), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.10/Python/ceval.c:2755 #6 0x00007ffff7af66b4 in PyEval_EvalCodeEx (co=co@entry=0x7ffff7e94eb0, The same issue has been reported a couple of months ago, https://bugzilla.redhat.com/show_bug.cgi?id=1249685
I already provided the updated backtrace in the upstream bug...
I'm sorry. I didn't noticed that you have attached a back trace to the upstream bug.
I've been discussing fixes for the problem with upstream: https://github.com/pyca/cryptography/issues/2477 https://bitbucket.org/cffi/cffi/issues/232/static-callbacks
https://bitbucket.org/cffi/cffi/commits/7b096a521f52 seems to be the upstream fix for this; what's our next move downstream?
I'm waiting for an official release of cffi 1.4.0 to get https://github.com/pyca/cryptography/pull/2488 and https://github.com/pyca/cryptography/pull/2481 into python-cryptography 1.2.
python-cffi-1.4.2-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-49b6510089
python-cffi-1.4.2-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-49b6510089
python-cffi-1.4.2-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
python-cffi 1.4 alone does not fix the issue. A new version of python-cryptography is required, too.
This message is a reminder that Fedora 23 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 23. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '23'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 23 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
F23 is now on python-cryptography-1.5.3-3.fc23 , and I confirmed this is fixed. From an original F23 Workstation live image, I can reproduce with the #c27 reproducer after installing python-cryptography from 'fedora' repo; after updating python-cryptography (which also updates python-ffi) from 'updates' repo, the reproducer no longer crashes.