Description of problem: We have encountered a strange execmem erorr and we suppose that it might be a bug in Fedora python. Our python application executed via httpd fails on requests.get(...) line. Then we get AVC avc: denied { execmem } for pid=1167 comm="httpd" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=process permissive=0 Running `setsebool -P deny_execmem 0` _solves_ the issue, but should it even happen? There is a full stacktrace [root@copr-fe-dev sysconfig][STG]# coredumpctl dump /usr/sbin/httpd PID: 4597 (httpd) UID: 992 (copr-fe) GID: 989 (copr-fe) Signal: 11 (SEGV) Timestamp: Wed 2018-01-17 21:12:06 UTC (2min 42s ago) Command Line: other -DFOREGROUND Executable: /usr/sbin/httpd Control Group: /system.slice/httpd.service Unit: httpd.service Slice: system.slice Boot ID: 9b03c7d47efc4b21ba3a4f10994456ee Machine ID: 90d9fbf8f1a544b79aabd3b78776fde5 Hostname: copr-fe-dev.cloud.fedoraproject.org Storage: /var/lib/systemd/coredump/core.httpd.992.9b03c7d47efc4b21ba3a4f10994456ee.4597.1516223526000000.lz4 Message: Process 4597 (httpd) of user 992 dumped core. Stack trace of thread 4597: #0 0x00007f8a7529836b __poll (libc.so.6) #1 0x00007f8a759b9435 apr_poll (libapr-1.so.0) #2 0x00007f8a679f2282 wsgi_start_process (mod_wsgi_python3.so) #3 0x00007f8a679f456b wsgi_manage_process (mod_wsgi_python3.so) #4 0x00007f8a759b42ab apr_proc_other_child_alert (libapr-1.so.0) #5 0x00007f8a6b03e083 event_run (mod_mpm_event.so) #6 0x0000559644a45fce ap_run_mpm (httpd) #7 0x0000559644a3e914 main (httpd) #8 0x00007f8a751ae00a __libc_start_main (libc.so.6) #9 0x0000559644a3ea1a _start (httpd) Stack trace of thread 4602: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4606: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4607: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4601: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4608: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4610: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4600: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4613: #0 0x00007f8a752a4a86 epoll_pwait (libc.so.6) #1 0x00007f8a759b86e9 impl_pollset_poll (libapr-1.so.0) #2 0x00007f8a679ef7e0 wsgi_daemon_thread (mod_wsgi_python3.so) #3 0x00007f8a7577b61b start_thread (libpthread.so.0) #4 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4598: #0 0x00007f8a7529a2a3 __select (libc.so.6) #1 0x00007f8a759bd7e5 apr_sleep (libapr-1.so.0) #2 0x00007f8a679e4290 wsgi_monitor_thread (mod_wsgi_python3.so) #3 0x00007f8a7577b61b start_thread (libpthread.so.0) #4 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4614: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4605: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4604: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4609: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4611: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4612: #0 0x00007f8a75781c4b pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f8a679efd3d wsgi_daemon_thread (mod_wsgi_python3.so) #2 0x00007f8a7577b61b start_thread (libpthread.so.0) #3 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4599: #0 0x00007f8a7529a2a3 __select (libc.so.6) #1 0x00007f8a759bd7e5 apr_sleep (libapr-1.so.0) #2 0x00007f8a679e6aeb wsgi_deadlock_thread (mod_wsgi_python3.so) #3 0x00007f8a7577b61b start_thread (libpthread.so.0) #4 0x00007f8a752a491f __clone (libc.so.6) Stack trace of thread 4603: #0 0x00007f8a751c4957 kill (libc.so.6) #1 0x00007f8a75786a80 __restore_rt (libpthread.so.0) #2 0x00007f8a4ff578db ffi_prep_closure_loc (libffi.so.6) #3 0x00007f8a4a64e76d b_callback (_cffi_backend.cpython-36m-x86_64-linux-gnu.so) #4 0x00007f8a4a65195f ffi_callback (_cffi_backend.cpython-36m-x86_64-linux-gnu.so) #5 0x00007f8a675f7273 _PyCFunction_FastCallDict (libpython3.6m.so.1.0) #6 0x00007f8a675fe27a call_function (libpython3.6m.so.1.0) #7 0x00007f8a67627eca _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #8 0x00007f8a6758e456 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #9 0x00007f8a6758ee3c _PyFunction_FastCallDict (libpython3.6m.so.1.0) #10 0x00007f8a6758f3de _PyObject_FastCallDict (libpython3.6m.so.1.0) #11 0x00007f8a67599261 _PyObject_Call_Prepend (libpython3.6m.so.1.0) #12 0x00007f8a6758f7fb PyObject_Call (libpython3.6m.so.1.0) #13 0x00007f8a675f6499 slot_tp_init (libpython3.6m.so.1.0) #14 0x00007f8a675f771e type_call (libpython3.6m.so.1.0) #15 0x00007f8a6758f204 _PyObject_FastCallDict (libpython3.6m.so.1.0) #16 0x00007f8a675fe3ad call_function (libpython3.6m.so.1.0) #17 0x00007f8a67627eca _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #18 0x00007f8a675c0bea fast_function (libpython3.6m.so.1.0) #19 0x00007f8a675fe33e call_function (libpython3.6m.so.1.0) #20 0x00007f8a67627eca _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #21 0x00007f8a6758ed9a _PyFunction_FastCallDict (libpython3.6m.so.1.0) #22 0x00007f8a6758f3de _PyObject_FastCallDict (libpython3.6m.so.1.0) #23 0x00007f8a675b4c1c PyObject_CallFunctionObjArgs (libpython3.6m.so.1.0) #24 0x00007f8a675b4c7f property_descr_set (libpython3.6m.so.1.0) #25 0x00007f8a67620cac _PyObject_GenericSetAttrWithDict (libpython3.6m.so.1.0) #26 0x00007f8a67621155 PyObject_SetAttr (libpython3.6m.so.1.0) #27 0x00007f8a676287bb _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #28 0x00007f8a6758e0b3 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #29 0x00007f8a675c0e11 fast_function (libpython3.6m.so.1.0) #30 0x00007f8a675fe33e call_function (libpython3.6m.so.1.0) #31 0x00007f8a67628dbf _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #32 0x00007f8a675c0bea fast_function (libpython3.6m.so.1.0) #33 0x00007f8a675fe33e call_function (libpython3.6m.so.1.0) #34 0x00007f8a67627eca _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #35 0x00007f8a6758e255 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #36 0x00007f8a675c0e11 fast_function (libpython3.6m.so.1.0) #37 0x00007f8a675fe33e call_function (libpython3.6m.so.1.0) #38 0x00007f8a67627eca _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #39 0x00007f8a6758e0b3 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #40 0x00007f8a675c0e11 fast_function (libpython3.6m.so.1.0) #41 0x00007f8a675fe33e call_function (libpython3.6m.so.1.0) #42 0x00007f8a67628dbf _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #43 0x00007f8a6758e0b3 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #44 0x00007f8a675c0e11 fast_function (libpython3.6m.so.1.0) #45 0x00007f8a675fe33e call_function (libpython3.6m.so.1.0) #46 0x00007f8a67628dbf _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #47 0x00007f8a6758e0b3 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #48 0x00007f8a6758f08d _PyFunction_FastCallDict (libpython3.6m.so.1.0) #49 0x00007f8a6758f3de _PyObject_FastCallDict (libpython3.6m.so.1.0) #50 0x00007f8a67599261 _PyObject_Call_Prepend (libpython3.6m.so.1.0) #51 0x00007f8a6758f7fb PyObject_Call (libpython3.6m.so.1.0) #52 0x00007f8a676298b7 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #53 0x00007f8a6758e0b3 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #54 0x00007f8a6758f08d _PyFunction_FastCallDict (libpython3.6m.so.1.0) #55 0x00007f8a6758f3de _PyObject_FastCallDict (libpython3.6m.so.1.0) #56 0x00007f8a67599261 _PyObject_Call_Prepend (libpython3.6m.so.1.0) #57 0x00007f8a6758f7fb PyObject_Call (libpython3.6m.so.1.0) #58 0x00007f8a676298b7 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #59 0x00007f8a6758e0b3 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #60 0x00007f8a6758f08d _PyFunction_FastCallDict (libpython3.6m.so.1.0) #61 0x00007f8a6758f3de _PyObject_FastCallDict (libpython3.6m.so.1.0) #62 0x00007f8a67599261 _PyObject_Call_Prepend (libpython3.6m.so.1.0) #63 0x00007f8a6758f7fb PyObject_Call (libpython3.6m.so.1.0) Refusing to dump core to tty (use shell redirection or specify --output). Is this a bug in Fedora python3? Thank you Version-Release number of selected component (if applicable): python3-3.6.3-2 httpd-2.4.29-1
Actually, setsebool -P httpd_execmem 1 solves the issue. I was debugging with Jakub yesterday. But there is some interesting info on deny_execmem in Selinux docs (man authconfig_execmem): If you want to deny user domains applications to map a memory region as both executable and writable, this is dangerous and the executable should be reported in bugzilla, you must turn on the deny_execmem boolean. Enabled by default.
dmesg shows this: [1181555.186369] httpd[6338]: segfault at 0 ip 00007f466301f8db sp 00007f4678578bc8 error 6 in libffi.so.6.0.2[7f466301a000+7000]
After some more investigation, I think this should reassigned to mod_wsgi component because that's the one likely using libffi.so. The question here is if setting `setsebool -P httpd_execmem 1` is really the correct step (it seems to be quite unusual for this to be required).
Sorry, I have unset assignee by accident.
That's because of the wrong component.
Any progress here?
I thought libffi tried to use a tempfile for this? (see e.g. bug 522731 and bug 488396).
Hit that again on a different system. r = requests.post(target_uri, json=result_json, timeout=20) in a python3 script seqfaults. This is really a weird bug. Could anyone, please, take a look at this?
Btw. this time the script is invoked from logrotate: type=AVC msg=audit(1519960698.810:3807202): avc: denied { execmem } for pid=25029 comm="copr_log_hitcou" scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tclass=process permissive=0 type=ANOM_ABEND msg=audit(1519960698.883:3807203): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 pid=25029 comm="copr_log_hitcou" exe="/usr/bin/python3.6" sig=11 res=1 So maybe this isn't only mod_wsgi related? Can someone finally address this?
Can you give me a full reproducer for Comment 8?
(In reply to Miro Hrončok from comment #10) > Can you give me a full reproducer for Comment 8? Sure... root@coprbox /home/clime $ cat > /usr/bin/mypython3 <<EOF #!/bin/sh /usr/bin/python3 EOF root@coprbox /home/clime $ chcon -t logrotate_exec_t /usr/bin/mypython3 root@coprbox /home/clime $ chmod +x /usr/bin/mypython3 root@coprbox /home/clime $ runcon system_u:system_r:logrotate_t:s0 /usr/bin/mypython3 Python 3.6.4 (default, Feb 1 2018, 11:06:09) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> requests.get('https://fedoraproject.org') /usr/bin/mypython3: line 2: 10151 Segmentation fault (core dumped) /usr/bin/python3 On Fedora27 system.
Thanks. Will look at it next week.
I cannot reproduce this with python3-3.6.4-9: $ sudo -i [sudo] heslo pro churchyard: # cat > /usr/bin/mypython3 <<EOF > #!/bin/sh > /usr/bin/python3 > EOF $ sudo chcon -t logrotate_exec_t /usr/bin/mypython3 $ sudo chmod +x /usr/bin/mypython3 $ sudo runcon system_u:system_r:logrotate_t:s0 /usr/bin/mypython3 Python 3.6.4 (default, Mar 13 2018, 18:18:20) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> requests.get('https://fedoraproject.org') Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection raise err File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) PermissionError: [Errno 13] Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request self._validate_conn(conn) File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn conn.connect() File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect conn = self._new_conn() File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fb7ecc20b70>: Failed to establish a new connection: [Errno 13] Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 440, in send timeout=timeout File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='fedoraproject.org', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb7ecc20b70>: Failed to establish a new connection: [Errno 13] Permission denied',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.6/site-packages/requests/api.py", line 72, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3.6/site-packages/requests/api.py", line 58, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 508, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 508, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='fedoraproject.org', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb7ecc20b70>: Failed to establish a new connection: [Errno 13] Permission denied',)) >>> Error in atexit._run_exitfuncs: PermissionError: [Errno 13] Permission denied $ rpm -q python3 python3-3.6.4-9.fc27.x86_64
Set: /usr/sbin/setsebool -P nis_enabled 1 to see the segfault in the reproducer.
I can reproduce segfault taking steps from comment #11 and #14.
I'm sorry, I really won't have time for this. I think I ignored it long enough, reassigning to default.
> I can reproduce segfault taking steps from comment #11 and #14. I combined steps of these commands, but I'm unable to reproduce the issue on an up to date Fedora 28 with python3-3.6.6-1.fc28.x86_64 and python3-requests-2.18.4-4.fc28.noarch.
I failed to reproduce the bug on Fedora 27 using: * python3-3.6.6-1.fc27.x86_64 * python3-requests-2.18.4-2.fc27.noarch I used this script: --- set -x -e PROGRAM=/usr/bin/mypython3 rm -f $PROGRAM cat > $PROGRAM <<EOF #!/bin/sh /usr/bin/python3 EOF chcon -t logrotate_exec_t $PROGRAM chmod +x $PROGRAM /usr/sbin/setsebool -P nis_enabled 1 echo "in the REPL, type:" echo ">>> import requests" echo ">>> requests.get('https://fedoraproject.org')" runcon system_u:system_r:logrotate_t:s0 $PROGRAM --- Output: --- [vstinner@fedora27 ~]$ sudo ./selinux_python27_bug.sh + PROGRAM=/usr/bin/mypython3 + rm -f /usr/bin/mypython3 + cat + chcon -t logrotate_exec_t /usr/bin/mypython3 + chmod +x /usr/bin/mypython3 + /usr/sbin/setsebool -P nis_enabled 1 + echo 'in the REPL, type:' in the REPL, type: + echo '>>> import requests' >>> import requests + echo '>>> requests.get('\''https://fedoraproject.org'\'')' >>> requests.get('https://fedoraproject.org') + runcon system_u:system_r:logrotate_t:s0 /usr/bin/mypython3 Python 3.6.6 (default, Jul 19 2018, 16:29:00) >>> import requests >>> requests.get('https://fedoraproject.org') <Response [200]> >>> ^D Error in atexit._run_exitfuncs: PermissionError: [Errno 13] Permission denied ---
> Actually, setsebool -P httpd_execmem 1 solves the issue. I was debugging with Jakub yesterday. It seems like the issue is related to libffi and SELinux (execmem option). cffi got a fix for execmem last year: bug #1249685.
Summary of my findings. To me, the most obvious explanation is that libffi tries to create a memory page in write+execute mode, SELinux blocks the allocation of the memory page, libffi writes into a memory page which has the memory address... 0 (NULL pointer) and so the process does crash. Sources: (*) Running `setsebool -P deny_execmem 0` _solves_ the issue, (...) (*) "httpd[6338]: segfault at 0 ip 00007f466301f8db sp 00007f4678578bc8 error 6 in libffi.so.6.0.2" <= the code causing the crash belongs to libffi and the code tries to access memory at the address 0 (NULL pointer) (*) The only thread which calls libffi does something related to signals ("__restore_rt" is "sigreturn()"), it's likely the code handling SIGSEGV signal (segmentation fault) Stack trace of thread 4603: #0 0x00007f8a751c4957 kill (libc.so.6) #1 0x00007f8a75786a80 __restore_rt (libpthread.so.0) #2 0x00007f8a4ff578db ffi_prep_closure_loc (libffi.so.6) #3 0x00007f8a4a64e76d b_callback (_cffi_backend.cpython-36m-x86_64-linux-gnu.so) #4 0x00007f8a4a65195f ffi_callback (_cffi_backend.cpython-36m-x86_64-linux-gnu.so) Problem: it's unclear to me which code uses libffi. In Python, there are two main modules using libffi: cffi and ctypes. It seems like cryptography uses cffi. I don't know the link between the bug and cryptography. The reproducer script uses the Python "requests" module which uses urllib3. I'm not sure how urllib3 uses libffi: it should be related to TLS to access HTTPS. The script uses a HTTPS url: "requests.get('https://fedoraproject.org')". I read that urllib3 can use PyOpenSSL if it's installed. I don't know if people who reproduced the bug had python3-pyopenssl installed or not. If someone can still reproduce the bug, I would need the following information: * Full SELinux configuration, results of commands: * getenforce * getsebool deny_execmem * getsebool nis_enabled * (I don't know if there is a better way to dump the current full SELinux config) * Version of the following packages, rpm -q: * python3
Oops, I submitted a partial comment, sorry :-( * Version of the following packages, rpm -q: * python3 * libffi * python3-requests * python3-cryptography * python3-cffi * Architecture: output of "uname -a" command I probably forgot some information, but these info should already help me :-)
Created attachment 1495650 [details] Script collecting info and supposed to reproduce the crash I wrote selinux_python27_bug.sh script which should be run as root: it dumps information and it supposed to reproduced the crash, but I'm unable to reproduce the crash using this script. Note: I removed -P from "/usr/sbin/setsebool -P nis_enabled 1" to be able to recover the previous SELinux configuratio at next reboot.
Mikolaj Izdebski reproduce the bug. Output of my script: ++ PROGRAM=/usr/bin/mypython3 ++ rm -f /usr/bin/mypython3 ++ cat ++ chcon -t logrotate_exec_t /usr/bin/mypython3 ++ chmod +x /usr/bin/mypython3 ++ /usr/sbin/setsebool nis_enabled 1 ++ echo '=== Info ===' === Info === ++ getenforce Enforcing ++ getsebool deny_execmem deny_execmem --> off ++ getsebool nis_enabled nis_enabled --> on ++ rpm -q python3 python3-3.6.6-1.fc27.x86_64 ++ rpm -q libffi libffi-3.1-14.fc27.x86_64 ++ rpm -q python3-requests python3-requests-2.18.4-1.fc27.noarch ++ rpm -q python3-cryptography python3-cryptography-2.3-1.fc27.x86_64 ++ rpm -q python3-cffi python3-cffi-1.10.0-3.fc27.x86_64 ++ rpm -q python3-pyOpenSSL python3-pyOpenSSL-17.2.0-2.fc27.noarch ++ runcon system_u:system_r:logrotate_t:s0 /usr/bin/mypython3 uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:logrotate_t:s0 system_u:system_r:logrotate_t:s0 === Info === in the REPL, type: import requests requests.get('https://fedoraproject.org') exit() Python 3.6.6 (default, Jul 19 2018, 16:29:00) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> requests.get('https://fedoraproject.org') /usr/bin/mypython3: line 12: 17881 Segmentation fault (core dumped) /usr/bin/python3
Ok, the bug is in pyOpenSSL which Mikolaj identified the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1567862 I can finally reproduce the bug if I downgrade requests and have PyOpenSSL installed: sudo dnf install python3-requests-2.18.4-1.fc27.noarch python3-pyOpenSSL Workaround with python3-requests-2.18.4-1.fc27.noarch: manually remove python3-pyOpenSSL. So it's now clear that python3-pyOpenSSL triggers the bug. python3-requests-2.18.4-2.fc27 doesn't reproduce the bug since it has been modified to never use pyOpenSSL. Notes: * I reproduce the bug with deny_execmem=on and deny_execmem=off: deny_execmem has no impact. * With "/usr/sbin/setsebool nis_enabled 0", requests fails with "PermissionError: [Errno 13] Permission denied" on sock.connect(sa) in create_connection() Python traceback generated by faulthandler: >>> faulthandler.enable() >>> import requests; requests.get('https://fedoraproject.org') Fatal Python error: Segmentation fault Current thread 0x00007f3d743e5540 (most recent call first): File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 247 in __init__ File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 971 in set_verify File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 400 in verify_mode File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 274 in create_urllib3_context File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 315 in connect File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850 in _validate_conn File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346 in _make_request File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601 in urlopen File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 440 in send File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 618 in send File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 508 in request File "/usr/lib/python3.6/site-packages/requests/api.py", line 58 in request File "/usr/lib/python3.6/site-packages/requests/api.py", line 72 in get File "<stdin>", line 1 in <module> Extract of OpenSSL/SSL.py:247: class _VerifyHelper(_CallbackExceptionHelper): ... def __init__(self, callback): ... self.callback = _ffi.callback( "int (*)(int, X509_STORE_CTX *)", wrapper) # <~~~~~~ LINE 247 ~~~~~ Logs when the bug occurs (journalctl): oct. 19 16:25:04 fedora27 audit[2135]: AVC avc: denied { execmem } for pid=2135 comm="python3" scontext=system_u:system_r:logrotate_t:s0 tcontext=system_u:system_r:logrotate_t:s0 tclass=process permissive=0 oct. 19 16:25:04 fedora27 audit[2135]: ANOM_ABEND auid=1000 uid=0 gid=0 ses=11 subj=system_u:system_r:logrotate_t:s0 pid=2135 comm="python3" exe="/usr/bin/python3.6" sig=11 res=1 oct. 19 16:25:04 fedora27 kernel: python3[2135]: segfault at 0 ip 00007fd4a4deb8db sp 00007ffd3b4d0208 error 6 in libffi.so.6.0.2[7fd4a4de6000+7000] oct. 19 16:25:04 fedora27 kernel: Code: e8 1a bd ff ff 66 2e 0f 1f 84 00 00 00 00 00 8b 06 83 e8 01 83 f8 04 77 56 b8 49 bb ff ff 41 b9 49 ba ff ff 41 ba ff e3 ff ff <66> 89 07 48 8b 05 f3 16 20 00 66 44 89 4f 0a 4c 89 47 0c 48 89 47 oct. 19 16:25:04 fedora27 systemd[1]: Started Process Core Dump (PID 2140/UID 0). oct. 19 16:25:04 fedora27 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@8-2140-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' oct. 19 16:25:05 fedora27 sudo[2115]: pam_unix(sudo:session): session closed for user root oct. 19 16:25:05 fedora27 audit[2115]: USER_END pid=2115 uid=0 auid=1000 ses=11 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success' oct. 19 16:25:05 fedora27 audit[2115]: CRED_DISP pid=2115 uid=0 auid=1000 ses=11 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success' oct. 19 16:25:05 fedora27 systemd-coredump[2141]: Process 2135 (python3) of user 0 dumped core. Stack trace of thread 2135: #0 0x00007fd4a4deb8db ffi_prep_closure_loc (libffi.so.6) #1 0x00007fd4a345576d b_callback (_cffi_backend.cpython-36m-x86_64-linux-gnu.so) #2 0x00007fd4a345895f ffi_callback (_cffi_backend.cpython-36m-x86_64-linux-gnu.so) #3 0x00007fd4a9018273 _PyCFunction_FastCallDict (libpython3.6m.so.1.0) ... oct. 19 16:25:05 fedora27 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@8-2140-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
The crash can be simplified to: Python 3.6.3 (default, Oct 9 2017, 12:07:10) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from cryptography.hazmat.bindings.openssl.binding import Binding >>> binding = Binding() >>> binding.init_static_locks() >>> ffi = binding.ffi >>> def func(): pass ... >>> callback = ffi.callback("int (*)(void *)", func) /usr/bin/mypython3 : ligne 11 : 2781 Erreur de segmentation (core dumped)/usr/bin/python3 So the crash comes from crytography and is related to SELinux.
I installed debug symbols: sudo dnf debuginfo-install python3-cryptography More complete traceback: >>> callback = ffi.callback("int (*)(void *)", func) Stack trace of thread 3109: #0 0x00007f08b06e8060 raise (libpthread.so.0) #1 0x00007f08b06e81c0 __restore_rt (libpthread.so.0) #2 0x00007f08ae2a88db ffi_prep_closure_loc (libffi.so.6) #3 0x00007f08ae4bf76d b_callback (_cffi_backend.cpython-36m-x86_64-linux-gnu.so) #4 0x00007f08ae4c295f ffi_callback (_cffi_backend.cpython-36m-x86_64-linux-gnu.so) #5 0x00007f08b0a7a273 _PyCFunction_FastCallDict (libpython3.6m.so.1.0) ... ffi_callback() calls ffi_prep_closure_loc() which emits the SELinux audit warning and then causes a segfault at NULL.
On the IRC channel #cryptography-dev, Alex Gaynor explained me that pyOpenSSL uses the "raw cffi" binding of OpenSSL from cryptography. The problem is that pyOpenSSL doesn't use libffi properly and so is incompatible with SELinux deny_execmem=on. I change again the component to pyOpenSSL. Links: * https://github.com/pyca/cryptography/issues/2477 * https://bitbucket.org/cffi/cffi/issues/231/writeable-memory-execution-execmem-with * https://bitbucket.org/cffi/cffi/issues/232/static-callbacks * https://bugzilla.redhat.com/show_bug.cgi?id=1249685#c37 : nova worked around the issue by allowing execmem
Links between all components: Apache -> mod_wsgi -> python3.6 -> requests: requests.get() -> urllib3: SSL create_urllib3_context() -> pyOpenSSL -> cryptography -> cffi: cffi_obj.callback(...) -> libffi: ffi_prep_closure_loc() -> SELinux blocks a memory operation in ffi_prep_closure_loc(), then ffi_prep_closure_loc() causes a segfault on accessing memory at address 0 (NULL pointer)
Simple reproducer using directly the cryptography "raw (unsafe) cffi binding" of OpenSSL to mimick what pyOpenSSL does at OpenSSL/SSL.py:247: class _VerifyHelper(_CallbackExceptionHelper): ... def __init__(self, callback): ... self.callback = _ffi.callback( "int (*)(int, X509_STORE_CTX *)", wrapper) # <~~~~~~ LINE 247 ~~~~~ [vstinner@fedora27 ~]$ sudo setsebool deny_execmem on # system wide change [vstinner@fedora27 ~]$ cat crash.py from cryptography.hazmat.bindings.openssl.binding import Binding binding = Binding() binding.init_static_locks() ffi = binding.ffi def func(): pass callback = ffi.callback("int (*)(void *)", func) [vstinner@fedora27 ~]$ python3 crash.py Erreur de segmentation (core dumped)
Victor CCed me on this bug. PyOpenSSL still uses python-cffi's old callback style. A couple of years ago I worked with cffi and cryptography upstream to fix the issue. Armin Rigo implemented https://cffi.readthedocs.io/en/latest/using.html#extern-python-new-style-callbacks and I patched python-cryptography. See #1277224 for the full story. I decided against changing PyOpenSSL, because it would have required a major redesign of PyOpenSSL. Neither upstream nor I were willing to invest the necessary amount of time. libffi has a hack to work around execmem restriction. However this hack is really dangerous in combination with fork(). Simply speaking, the workaround mmap() the same physical memory location into two virtual memory location using MAP_SHARED and a shared file descriptor. One mmap is writeable, the other is executable. Sine it's a MAP_SHARED mmap, the region is shared between parent and all child processes. Why is execmem an issue in Fedora 27? python-requests uses PyOpenSSL by default. The package calls urllib3.contrib.pyopenssl.inject_into_urllib3() to monkey patch its code to replace the ssl stdlib module with PyOpenSSL. This was required for very old versions of Python that did not support SNI. In #1567862 I requested to remove the patch from python-requests. For Fedora 27, you have three options to work around the issue 1) modify sys.modules like I did for FreeIPA, https://github.com/freeipa/freeipa/blob/master/install/share/wsgi.py#L32 2) call urllib3.contrib.pyopenssl.extract_from_urllib3() to undo the monkey patching 3) Allow execmem for httpd
But the bug 1567862 indicates that the PyOpenSSL should not be used by urllib3 and requests by default anymore. I am confused.
Victor, when you reproduced this, what version of python-requests and python-urllib3 was installed?
Tomas: This bug is for F27. python-requests is patched since F28.
Ah, now I see in comment 26 that Victor had to downgrade requests and urllib3 and that the original reproducer is fixed. I do not think this is reasonably fixable in pyOpenSSL in Fedora 27.
See my comment #26: "sudo dnf install python3-requests-2.18.4-1.fc27.noarch python3-pyOpenSSL" I had to downgrade python3-requests to reproduce the bug: * python3-requests-2.18.4-1.fc27.noarch * python3-urllib3-1.22-6.fc27.noarch Using the latest version of python3-requests (2.18.4-2.fc27), requests.get() does no longer crash. So Python requests.get() can be used in Apache with execmem blocked in Fedora 27. That's why I reassigned the bug to pyOpenSSL. (Apache, requests, cryptography and urllib3 are now fine in Fedora 27.)
"Using the latest version of python3-requests (2.18.4-2.fc27), requests.get() does no longer crash." Jeremy Cline removed pyopenssl import and pyopenssl.inject_into_urllib3() call from the Fedora 27 packages in this package version: https://src.fedoraproject.org/rpms/python-requests/c/3f626829ac1ae29a9665cfbf749926a4f1014e86?branch=f27
Oh I see now. So the original issue with requests.get() is already fixed, however pyOpenSSL still contains code that can crash.
I'd like to rephrase "can crash". PyOpenSSL uses a cffi and libffi feature that requires dynamic machine code creation. Since the feature requires writeable and executable memory pages, the callback trampoline is not compatible with SELinux's execmem protection. It would take a major effort to redesign the callbacks to use python-cffi's new callback system. I'm pretty sure that upstream has no capacity to invest that much time into both python-cryptography and PyOpenSSL to redesign and rewrite all callbacks. PyOpenSSL is pretty much in maintenance-only mode.
Then I am inclined to close it as NOTABUG. We are not going to invest into PyOpenSSL development either.