Bug 716465

Summary:

installing abrt-addon-python causes apache to crash

Product:

Red Hat Enterprise Linux 6

Reporter:

Pradeep Kilambi <pkilambi>

Component:

mod_wsgi

Assignee:

Joe Orton <jorton>

Status:

CLOSED WONTFIX

QA Contact:

BaseOS QE Security Team <qe-baseos-security>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

6.3

CC:

ahecox, dvlasenk, gavin, jorton, kklic, mnowak, mtoman, npajkovs, ovasik, syeghiay, tsanders

Target Milestone:

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2011-12-16 15:24:00 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Attachments:

Description	Flags
apache error logs	none
httpd core dumps	none
gdb bt output	none
core dumps with debuginfo turned on from httpd/mod_python /mod_wsgi	none

Description Pradeep Kilambi 2011-06-24 14:20:58 UTC

Description of problem:

We are seeing a weird issue on RHEL-6/F15/and F14 as well. We have apache running with mod_python and mod_wsgi. Everything is fine until we reloads (SIGHUPS) httpd and causes segmentation faults. 

Investigating further we discovered that if we remove  abrt-addon-python everything works fine. But if we have this package installed and apache reloads we see segmentation faults in error log crashing child pids.

this might not even be an issue with abrt and it might just be mod_python or even mod_wsgi. But filing under this so at least we can get some right eyes to look at it. Please feel free to reassign to appropriate component if anything obvious stands out.

Version-Release number of selected component (if applicable):
abrt-addon-python-1.1.13-4.el6.x86_64
mod_python-3.3.1-14.el6.1.x86_64
mod_wsgi-3.2-1.el6.x86_64
httpd-2.2.15-5.el6.x86_64


How reproducible:
always

Steps to Reproduce:
1. Have an apache server running with mod_python and mod_wsgi loaded
2. yum install abrt-addon-python if its not already installed
3. /etc/init.d/httpd reload
4. see /var/log/httpd/error_log for seg faults

Additional info:
Here are my steps and results at each step:

# rpm -q abrt-addon-python
abrt-addon-python-1.1.13-4.el6.x86_64

# sudo /etc/init.d/httpd reload
Reloading httpd:                           [OK]

#tail -f /var/log/httpd/error_log
[Fri Jun 24 09:53:06 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Fri Jun 24 09:53:06 2011] [notice] mod_python: using mutex_directory /tmp 
[Fri Jun 24 09:53:06 2011] [warn] mod_wsgi: Compiled for Python/2.6.2.
[Fri Jun 24 09:53:06 2011] [warn] mod_wsgi: Runtime using Python/2.6.5.
[Fri Jun 24 09:53:06 2011] [notice] Apache/2.2.15 (Unix) DAV/2 mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 configured -- resuming normal operations
[Fri Jun 24 09:53:07 2011] [notice] child pid 2124 exit signal Segmentation fault (11)
[Fri Jun 24 09:53:08 2011] [notice] child pid 2148 exit signal Segmentation fault (11)
[Fri Jun 24 09:53:09 2011] [notice] child pid 2154 exit signal Segmentation fault (11)
[Fri Jun 24 09:53:10 2011] [notice] child pid 2160 exit signal Segmentation fault (11)
[Fri Jun 24 09:53:11 2011] [notice] child pid 2167 exit signal Segmentation fault (11)
[Fri Jun 24 09:53:12 2011] [notice] child pid 2173 exit signal Segmentation fault (11)
[Fri Jun 24 09:53:13 2011] [notice] child pid 2180 exit signal Segmentation fault (11)
[Fri Jun 24 09:53:14 2011] [notice] child pid 2186 exit signal Segmentation fault (11)
[Fri Jun 24 09:53:15 2011] [notice] child pid 2192 exit signal Segmentation fault (11)

# sudo yum remove abrt-addon-python
Loaded plugins: pulp-profile-update, rhnplugin
Setting up Remove Process
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them.
--> Running transaction check
---> Package abrt-addon-python.x86_64 0:1.1.13-4.el6 set to be erased
--> Processing Dependency: abrt-addon-python for package: abrt-cli-1.1.13-4.el6.x86_64
--> Running transaction check
---> Package abrt-cli.x86_64 0:1.1.13-4.el6 set to be erased
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================
 Package                Arch        Version              Repository                                                    Size
============================================================================================================================
Removing:
 abrt-addon-python      x86_64      1.1.13-4.el6         @anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0       21 k
Removing for dependencies:
 abrt-cli               x86_64      1.1.13-4.el6         @anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0       70 k

Transaction Summary
============================================================================================================================
Remove        2 Package(s)
Reinstall     0 Package(s)
Downgrade     0 Package(s)

Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing        : abrt-cli-1.1.13-4.el6.x86_64                                                                         1/2 
  Erasing        : abrt-addon-python-1.1.13-4.el6.x86_64                                                                2/2 


Removed:
  abrt-addon-python.x86_64 0:1.1.13-4.el6                                                                                   

Dependency Removed:
  abrt-cli.x86_64 0:1.1.13-4.el6                                                                                            

Complete!

# sudo /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

# /etc/init.d/httpd reload
Reloading httpd: 

# tail -f /var/log/httpd/error_log
[Fri Jun 24 09:53:49 2011] [warn] The Alias directive in /etc/httpd/conf.d/pulp.conf at line 53 will probably never match because it overlaps an earlier Alias.
httpd: Could not reliably determine the server's fully qualified domain name, using pulp-qe-rhel6.usersys.redhat.com for ServerName
[Fri Jun 24 09:53:49 2011] [notice] Digest: generating secret for digest authentication ...
[Fri Jun 24 09:53:49 2011] [notice] Digest: done
[Fri Jun 24 09:53:49 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Fri Jun 24 09:53:49 2011] [notice] mod_python: using mutex_directory /tmp 
[Fri Jun 24 09:53:49 2011] [warn] mod_wsgi: Compiled for Python/2.6.2.
[Fri Jun 24 09:53:49 2011] [warn] mod_wsgi: Runtime using Python/2.6.5.
[Fri Jun 24 09:53:49 2011] [notice] Apache/2.2.15 (Unix) DAV/2 mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 configured -- resuming normal operations
[Fri Jun 24 09:53:49 2011] [error] 2011-06-24 09:53:49,648 pulp.server.async:INFO: Task reply handler, started.

As you can see  when we remove abrt-addon-python all is well.

Comment 2 Pradeep Kilambi 2011-06-24 14:23:08 UTC

Created attachment 509776 [details]
apache error logs

Comment 3 Joe Orton 2011-06-24 15:01:07 UTC

What does the backtrace look like?

# echo CoreDumpDirectory /tmp > /etc/httpd/conf.d/coredump.conf

Comment 4 Pradeep Kilambi 2011-06-24 16:21:43 UTC

Hmm I dont see any core dump files, this is what I did:

# echo CoreDumpDirectory /tmp/httpd_dump > /etc/httpd/conf.d/coredump.conf

# mkdir -p /tmp/httpd_dump

# chmod 0777 /tmp/httpd_dump/

# /etc/init.d/httpd restart

# /etc/init.d/httpd reload

# echo $?
0

# ls -l /tmp/httpd_dump/
total 0

did i miss anything?

Comment 5 Pradeep Kilambi 2011-06-24 16:40:33 UTC

Created attachment 509809 [details]
httpd core dumps

attached is the core dump while reload was causing segfaults.

Comment 6 Pradeep Kilambi 2011-06-24 16:53:08 UTC

Created attachment 509814 [details]
gdb bt output

gdb bt output  on core.9985 dump

Comment 7 Pradeep Kilambi 2011-06-28 15:13:58 UTC

Created attachment 510307 [details]
core dumps with debuginfo turned on from httpd/mod_python /mod_wsgi

Comment 8 Nikola Pajkovsky 2011-08-24 13:29:39 UTC

Do you have still that problem with last snap rhel6 and abrt/libreport?

Comment 9 Michal Toman 2011-08-26 13:19:05 UTC

I assume you are using pulp (backtrace says "Core was generated by `(wsgi:pulp)    '."). As for RHEL, I am only able to reproduce this issue on 6.0, after installing pulp from the custom repo http://repos.fedorapeople.org/repos/pulp/pulp. The error seems to be caused by python not being able to parse /usr/lib/python2.6/site-packages/abrt.pth when called from WSGI: /etc/httpd/conf.d/pulp.conf
WSGIImportScript /srv/pulp/bootstrap.wsgi process-group=pulp application-group=pulp

In 6.1 and latest RHEL6 snapshot the segfault does not happen for me. Instead of this I can see some KeyError exceptions in httpd log (not sure whether this is related or not):
[Fri Aug 26 14:07:39 2011] [error] Exception KeyError: KeyError(139996824549344,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored

On F15 the error still occurs but the output it is much more verbose:

[Fri Aug 26 10:03:23 2011] [notice] SIGHUP received.  Attempting to restart
[Fri Aug 26 10:03:24 2011] [notice] Digest: generating secret for digest authentication ...
[Fri Aug 26 10:03:24 2011] [notice] Digest: done
[Fri Aug 26 10:03:24 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Fri Aug 26 10:03:24 2011] [notice] mod_python: using mutex_directory /tmp 
[Fri Aug 26 10:03:24 2011] [notice] SSL FIPS mode disabled
[Fri Aug 26 10:03:24 2011] [notice] Apache/2.2.19 (Unix) DAV/2 PHP/5.3.6 mod_python/3.3.1 Python/2.7.1 mod_ssl/2.2.19 OpenSSL/1.0.0d-fips mod_wsgi/3.2 mod_perl/2.0.4 Perl/v5.12.4 confi
gured -- resuming normal operations
*** glibc detected *** (wsgi:pulp)    : double free or corruption (!prev): 0x00007f3a7e3eb270 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x3d24a7703a)[0x7f3a7acdc03a]
/usr/lib64/libpython2.7.so.1.0(+0x3d27259526)[0x7f3a6f210526]
/usr/lib64/libpython2.7.so.1.0(+0x3d2726c432)[0x7f3a6f223432]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x53a)[0x7f3a6f297e0a]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5556)[0x7f3a6f296a86]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f3a6f298125]
/usr/lib64/libpython2.7.so.1.0(+0x3d2726daa3)[0x7f3a6f224aa3]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3]
/usr/lib64/libpython2.7.so.1.0(+0x3d27257b6f)[0x7f3a6f20eb6f]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1340)[0x7f3a6f292870]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f3a6f298125]
/usr/lib64/libpython2.7.so.1.0(+0x3d2726daa3)[0x7f3a6f224aa3]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3]
/usr/lib64/libpython2.7.so.1.0(+0x3d27257b6f)[0x7f3a6f20eb6f]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3]
/usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f3a6f290f47]
/usr/lib64/libpython2.7.so.1.0(PyInstance_New+0x6e)[0x7f3a6f20f8de]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3b73)[0x7f3a6f2950a3]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f3a6f298125]
/usr/lib64/libpython2.7.so.1.0(+0x3d2726d9ac)[0x7f3a6f2249ac]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3]
/usr/lib64/libpython2.7.so.1.0(+0x3d27257b6f)[0x7f3a6f20eb6f]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3]
/usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f3a6f290f47]
/usr/lib64/libpython2.7.so.1.0(PyInstance_New+0x6e)[0x7f3a6f20f8de]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f3a6f1fffe3]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3b73)[0x7f3a6f2950a3]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855)[0x7f3a6f298125]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f3a6f298252]
/usr/lib64/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xc2)[0x7f3a6f2a84e2]
/etc/httpd/modules/mod_wsgi.so(+0xd883)[0x7f3a6ed71883]
/etc/httpd/modules/mod_wsgi.so(+0x121e6)[0x7f3a6ed761e6]
/etc/httpd/modules/mod_wsgi.so(+0x12921)[0x7f3a6ed76921]
/etc/httpd/modules/mod_wsgi.so(+0x199fb)[0x7f3a6ed7d9fb]
(wsgi:pulp)    (ap_run_post_config+0x61)[0x7f3a7c9771b1]
(wsgi:pulp)    (main+0xc55)[0x7f3a7c962695]
/lib64/libc.so.6(__libc_start_main+0xed)[0x7f3a7ac8639d]
(wsgi:pulp)    (+0x17771)[0x7f3a7c962771]
======= Memory map: ========
... memory map ...
[Fri Aug 26 10:03:43 2011] [notice] child pid 2630 exit signal Segmentation fault (11)
[Fri Aug 26 10:03:44 2011] [notice] child pid 2636 exit signal Segmentation fault (11)
Fatal Python error: ceval: tstate mix-up
[Fri Aug 26 10:03:45 2011] [notice] child pid 2642 exit signal Aborted (6)
[Fri Aug 26 10:03:46 2011] [notice] child pid 2648 exit signal Segmentation fault (11)
[Fri Aug 26 10:03:47 2011] [notice] child pid 2654 exit signal Segmentation fault (11)
[Fri Aug 26 10:03:48 2011] [notice] child pid 2660 exit signal Segmentation fault (11)
[Fri Aug 26 10:03:49 2011] [notice] child pid 2666 exit signal Segmentation fault (11)
[Fri Aug 26 10:03:50 2011] [notice] child pid 2672 exit signal Segmentation fault (11)
Fatal Python error: GC object already tracked
Fatal Python error: GC object already tracked
[Fri Aug 26 10:03:51 2011] [notice] child pid 2678 exit signal Aborted (6)
[Fri Aug 26 10:03:52 2011] [notice] child pid 2684 exit signal Segmentation fault (11)

Anyway this bug does not seem to be related to ABRT and should be reassigned to either mod_wsgi or python.

Comment 10 Karel Klíč 2011-08-26 13:35:41 UTC

Reassigning to mod_wsgi because it's on the bottom of the traceback, and it doesn't seem to handle /usr/lib/python2.6/site-packages/abrt.pth.

Comment 13 RHEL Program Management 2011-08-26 17:48:13 UTC

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.