Hide Forgot
Description of problem: We are seeing a weird issue on RHEL-6/F15/and F14 as well. We have apache running with mod_python and mod_wsgi. Everything is fine until we reloads (SIGHUPS) httpd and causes segmentation faults. Investigating further we discovered that if we remove abrt-addon-python everything works fine. But if we have this package installed and apache reloads we see segmentation faults in error log crashing child pids. this might not even be an issue with abrt and it might just be mod_python or even mod_wsgi. But filing under this so at least we can get some right eyes to look at it. Please feel free to reassign to appropriate component if anything obvious stands out. Version-Release number of selected component (if applicable): abrt-addon-python-1.1.13-4.el6.x86_64 mod_python-3.3.1-14.el6.1.x86_64 mod_wsgi-3.2-1.el6.x86_64 httpd-2.2.15-5.el6.x86_64 How reproducible: always Steps to Reproduce: 1. Have an apache server running with mod_python and mod_wsgi loaded 2. yum install abrt-addon-python if its not already installed 3. /etc/init.d/httpd reload 4. see /var/log/httpd/error_log for seg faults Additional info: Here are my steps and results at each step: # rpm -q abrt-addon-python abrt-addon-python-1.1.13-4.el6.x86_64 # sudo /etc/init.d/httpd reload Reloading httpd: [OK] #tail -f /var/log/httpd/error_log [Fri Jun 24 09:53:06 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Fri Jun 24 09:53:06 2011] [notice] mod_python: using mutex_directory /tmp [Fri Jun 24 09:53:06 2011] [warn] mod_wsgi: Compiled for Python/2.6.2. [Fri Jun 24 09:53:06 2011] [warn] mod_wsgi: Runtime using Python/2.6.5. [Fri Jun 24 09:53:06 2011] [notice] Apache/2.2.15 (Unix) DAV/2 mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 configured -- resuming normal operations [Fri Jun 24 09:53:07 2011] [notice] child pid 2124 exit signal Segmentation fault (11) [Fri Jun 24 09:53:08 2011] [notice] child pid 2148 exit signal Segmentation fault (11) [Fri Jun 24 09:53:09 2011] [notice] child pid 2154 exit signal Segmentation fault (11) [Fri Jun 24 09:53:10 2011] [notice] child pid 2160 exit signal Segmentation fault (11) [Fri Jun 24 09:53:11 2011] [notice] child pid 2167 exit signal Segmentation fault (11) [Fri Jun 24 09:53:12 2011] [notice] child pid 2173 exit signal Segmentation fault (11) [Fri Jun 24 09:53:13 2011] [notice] child pid 2180 exit signal Segmentation fault (11) [Fri Jun 24 09:53:14 2011] [notice] child pid 2186 exit signal Segmentation fault (11) [Fri Jun 24 09:53:15 2011] [notice] child pid 2192 exit signal Segmentation fault (11) # sudo yum remove abrt-addon-python Loaded plugins: pulp-profile-update, rhnplugin Setting up Remove Process Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them. --> Running transaction check ---> Package abrt-addon-python.x86_64 0:1.1.13-4.el6 set to be erased --> Processing Dependency: abrt-addon-python for package: abrt-cli-1.1.13-4.el6.x86_64 --> Running transaction check ---> Package abrt-cli.x86_64 0:1.1.13-4.el6 set to be erased --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================ Package Arch Version Repository Size ============================================================================================================================ Removing: abrt-addon-python x86_64 1.1.13-4.el6 @anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0 21 k Removing for dependencies: abrt-cli x86_64 1.1.13-4.el6 @anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0 70 k Transaction Summary ============================================================================================================================ Remove 2 Package(s) Reinstall 0 Package(s) Downgrade 0 Package(s) Is this ok [y/N]: y Downloading Packages: Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Erasing : abrt-cli-1.1.13-4.el6.x86_64 1/2 Erasing : abrt-addon-python-1.1.13-4.el6.x86_64 2/2 Removed: abrt-addon-python.x86_64 0:1.1.13-4.el6 Dependency Removed: abrt-cli.x86_64 0:1.1.13-4.el6 Complete! # sudo /etc/init.d/httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] # /etc/init.d/httpd reload Reloading httpd: # tail -f /var/log/httpd/error_log [Fri Jun 24 09:53:49 2011] [warn] The Alias directive in /etc/httpd/conf.d/pulp.conf at line 53 will probably never match because it overlaps an earlier Alias. httpd: Could not reliably determine the server's fully qualified domain name, using pulp-qe-rhel6.usersys.redhat.com for ServerName [Fri Jun 24 09:53:49 2011] [notice] Digest: generating secret for digest authentication ... [Fri Jun 24 09:53:49 2011] [notice] Digest: done [Fri Jun 24 09:53:49 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Fri Jun 24 09:53:49 2011] [notice] mod_python: using mutex_directory /tmp [Fri Jun 24 09:53:49 2011] [warn] mod_wsgi: Compiled for Python/2.6.2. [Fri Jun 24 09:53:49 2011] [warn] mod_wsgi: Runtime using Python/2.6.5. [Fri Jun 24 09:53:49 2011] [notice] Apache/2.2.15 (Unix) DAV/2 mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 configured -- resuming normal operations [Fri Jun 24 09:53:49 2011] [error] 2011-06-24 09:53:49,648 pulp.server.async:INFO: Task reply handler, started. As you can see when we remove abrt-addon-python all is well.
Created attachment 509776 [details] apache error logs
What does the backtrace look like? # echo CoreDumpDirectory /tmp > /etc/httpd/conf.d/coredump.conf
Hmm I dont see any core dump files, this is what I did: # echo CoreDumpDirectory /tmp/httpd_dump > /etc/httpd/conf.d/coredump.conf # mkdir -p /tmp/httpd_dump # chmod 0777 /tmp/httpd_dump/ # /etc/init.d/httpd restart # /etc/init.d/httpd reload # echo $? 0 # ls -l /tmp/httpd_dump/ total 0 did i miss anything?
Created attachment 509809 [details] httpd core dumps attached is the core dump while reload was causing segfaults.
Created attachment 509814 [details] gdb bt output gdb bt output on core.9985 dump
Created attachment 510307 [details] core dumps with debuginfo turned on from httpd/mod_python /mod_wsgi
Do you have still that problem with last snap rhel6 and abrt/libreport?
I assume you are using pulp (backtrace says "Core was generated by `(wsgi:pulp) '."). As for RHEL, I am only able to reproduce this issue on 6.0, after installing pulp from the custom repo http://repos.fedorapeople.org/repos/pulp/pulp. The error seems to be caused by python not being able to parse /usr/lib/python2.6/site-packages/abrt.pth when called from WSGI: /etc/httpd/conf.d/pulp.conf WSGIImportScript /srv/pulp/bootstrap.wsgi process-group=pulp application-group=pulp In 6.1 and latest RHEL6 snapshot the segfault does not happen for me. Instead of this I can see some KeyError exceptions in httpd log (not sure whether this is related or not): [Fri Aug 26 14:07:39 2011] [error] Exception KeyError: KeyError(139996824549344,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored On F15 the error still occurs but the output it is much more verbose: [Fri Aug 26 10:03:23 2011] [notice] SIGHUP received. Attempting to restart [Fri Aug 26 10:03:24 2011] [notice] Digest: generating secret for digest authentication ... [Fri Aug 26 10:03:24 2011] [notice] Digest: done [Fri Aug 26 10:03:24 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Fri Aug 26 10:03:24 2011] [notice] mod_python: using mutex_directory /tmp [Fri Aug 26 10:03:24 2011] [notice] SSL FIPS mode disabled [Fri Aug 26 10:03:24 2011] [notice] Apache/2.2.19 (Unix) DAV/2 PHP/5.3.6 mod_python/3.3.1 Python/2.7.1 mod_ssl/2.2.19 OpenSSL/1.0.0d-fips mod_wsgi/3.2 mod_perl/2.0.4 Perl/v5.12.4 confi gured -- resuming normal operations *** glibc detected *** (wsgi:pulp) : double free or corruption (!prev): 0x00007f3a7e3eb270 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x3d24a7703a)[0x7f3a7acdc03a] /usr/lib64/libpython2.7.so.1.0(+0x3d27259526)[0x7f3a6f210526] /usr/lib64/libpython2.7.so.1.0(+0x3d2726c432)[0x7f3a6f223432] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x53a)[0x7f3a6f297e0a] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5556)[0x7f3a6f296a86] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f3a6f298125] /usr/lib64/libpython2.7.so.1.0(+0x3d2726daa3)[0x7f3a6f224aa3] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3] /usr/lib64/libpython2.7.so.1.0(+0x3d27257b6f)[0x7f3a6f20eb6f] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1340)[0x7f3a6f292870] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f3a6f298125] /usr/lib64/libpython2.7.so.1.0(+0x3d2726daa3)[0x7f3a6f224aa3] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3] /usr/lib64/libpython2.7.so.1.0(+0x3d27257b6f)[0x7f3a6f20eb6f] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3] /usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f3a6f290f47] /usr/lib64/libpython2.7.so.1.0(PyInstance_New+0x6e)[0x7f3a6f20f8de] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3b73)[0x7f3a6f2950a3] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f3a6f298125] /usr/lib64/libpython2.7.so.1.0(+0x3d2726d9ac)[0x7f3a6f2249ac] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3] /usr/lib64/libpython2.7.so.1.0(+0x3d27257b6f)[0x7f3a6f20eb6f] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3] /usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f3a6f290f47] /usr/lib64/libpython2.7.so.1.0(PyInstance_New+0x6e)[0x7f3a6f20f8de] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3b73)[0x7f3a6f2950a3] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f3a6f298125] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f3a6f298252] /usr/lib64/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xc2)[0x7f3a6f2a84e2] /etc/httpd/modules/mod_wsgi.so(+0xd883)[0x7f3a6ed71883] /etc/httpd/modules/mod_wsgi.so(+0x121e6)[0x7f3a6ed761e6] /etc/httpd/modules/mod_wsgi.so(+0x12921)[0x7f3a6ed76921] /etc/httpd/modules/mod_wsgi.so(+0x199fb)[0x7f3a6ed7d9fb] (wsgi:pulp) (ap_run_post_config+0x61)[0x7f3a7c9771b1] (wsgi:pulp) (main+0xc55)[0x7f3a7c962695] /lib64/libc.so.6(__libc_start_main+0xed)[0x7f3a7ac8639d] (wsgi:pulp) (+0x17771)[0x7f3a7c962771] ======= Memory map: ======== ... memory map ... [Fri Aug 26 10:03:43 2011] [notice] child pid 2630 exit signal Segmentation fault (11) [Fri Aug 26 10:03:44 2011] [notice] child pid 2636 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Fri Aug 26 10:03:45 2011] [notice] child pid 2642 exit signal Aborted (6) [Fri Aug 26 10:03:46 2011] [notice] child pid 2648 exit signal Segmentation fault (11) [Fri Aug 26 10:03:47 2011] [notice] child pid 2654 exit signal Segmentation fault (11) [Fri Aug 26 10:03:48 2011] [notice] child pid 2660 exit signal Segmentation fault (11) [Fri Aug 26 10:03:49 2011] [notice] child pid 2666 exit signal Segmentation fault (11) [Fri Aug 26 10:03:50 2011] [notice] child pid 2672 exit signal Segmentation fault (11) Fatal Python error: GC object already tracked Fatal Python error: GC object already tracked [Fri Aug 26 10:03:51 2011] [notice] child pid 2678 exit signal Aborted (6) [Fri Aug 26 10:03:52 2011] [notice] child pid 2684 exit signal Segmentation fault (11) Anyway this bug does not seem to be related to ABRT and should be reassigned to either mod_wsgi or python.
Reassigning to mod_wsgi because it's on the bottom of the traceback, and it doesn't seem to handle /usr/lib/python2.6/site-packages/abrt.pth.
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. If you would like it considered as an exception in the current release, please ask your support representative.