Bouncing httpd service produce this line in /var/log/messages Apr 13 14:58:42 pulp-qe kernel: httpd[28051]: segfault at 58 ip 00007fbdd056aca5 sp 00007fbdb0df90b0 error 4 in libpython2.6.so.1.0[7fbdd0492000+16f000] but everything seems to work even with the message. This problem does not happen consistently, but I have noticed it on both f13 and f14.
Seeing a crash in http around Sunday(4/17) @ 3:36am on a blade in boston we setup on Friday(4/15). This seems to reoccur on other machines. I think Preethi's box has seen crashes around this time before as well. We had synced rhel6/f14 i386/x86_64 content. cron-20110417: Apr 17 03:36:01 sat-perf-03 anacron[18351]: Job `cron.daily' started Apr 17 03:36:01 sat-perf-03 run-parts(/etc/cron.daily)[18500]: starting logrotate messages-20110417: Apr 15 11:24:16 sat-perf-03 kernel: httpd[13787]: segfault at 58 ip 00007f127107b2cd sp 00007f1251f2c060 error 4 in libpython2.6.so.1.0.#prelink#.upDJtQ (deleted)[7f1270f9f000+170000] Apr 15 11:24:16 sat-perf-03 qpidd[12331]: 2011-04-15 11:24:16 notice Shut down Apr 15 11:24:16 sat-perf-03 qpidd[4563]: 2011-04-15 11:24:16 notice Listening on TCP port 5672 Apr 15 11:24:16 sat-perf-03 qpidd[4563]: 2011-04-15 11:24:16 notice Broker running Apr 15 11:28:33 sat-perf-03 python: abrt: detected unhandled Python exception in /usr/bin/pulp-admin Apr 15 11:28:33 sat-perf-03 abrtd: dumpsocket: New client connected Apr 15 11:28:33 sat-perf-03 abrtd: dumpsocket: Saved Python crash dump of pid 4645 to /var/spool/abrt/pyhook-1302881313-4645 Apr 15 11:28:33 sat-perf-03 abrtd: Directory 'pyhook-1302881313-4645' creation detected Apr 15 11:28:33 sat-perf-03 abrtd: New crash /var/spool/abrt/pyhook-1302881313-4645, processing Apr 15 11:28:35 sat-perf-03 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team Apr 15 11:28:36 sat-perf-03 abrtd: dumpsocket: Socket client disconnected Apr 15 11:34:16 sat-perf-03 qpidd[4563]: 2011-04-15 11:34:16 notice Shut down Apr 15 11:34:16 sat-perf-03 qpidd[4919]: 2011-04-15 11:34:16 notice Listening on TCP port 5672 Apr 15 11:34:16 sat-perf-03 qpidd[4919]: 2011-04-15 11:34:16 notice Broker running Apr 15 11:36:30 sat-perf-03 qpidd[4919]: 2011-04-15 11:36:30 notice Shut down Apr 15 11:36:30 sat-perf-03 qpidd[5043]: 2011-04-15 11:36:30 notice Listening on TCP port 5672 Apr 15 11:36:30 sat-perf-03 qpidd[5043]: 2011-04-15 11:36:30 notice Broker running Apr 15 11:51:50 sat-perf-03 kernel: usb 1-1: USB disconnect, address 2 Apr 15 13:04:06 sat-perf-03 python: abrt: detected unhandled Python exception in /usr/share/candlepin/cpsetup Apr 15 13:04:06 sat-perf-03 abrtd: dumpsocket: New client connected Apr 15 13:04:06 sat-perf-03 abrtd: dumpsocket: Saved Python crash dump of pid 5561 to /var/spool/abrt/pyhook-1302887046-5561 Apr 15 13:04:06 sat-perf-03 abrtd: dumpsocket: Socket client disconnected Apr 15 13:04:06 sat-perf-03 abrtd: Directory 'pyhook-1302887046-5561' creation detected Apr 15 13:04:06 sat-perf-03 abrtd: Crash is in database already (dup of /var/spool/abrt/pyhook-1302879763-12781) Apr 15 13:04:06 sat-perf-03 abrtd: Deleting crash pyhook-1302887046-5561 (dup of pyhook-1302879763-12781), sending dbus signal Apr 15 13:20:52 sat-perf-03 qpidd[5043]: 2011-04-15 13:20:52 notice Shut down Apr 15 13:20:52 sat-perf-03 qpidd[6410]: 2011-04-15 13:20:52 notice Listening on TCP port 5672 Apr 15 13:20:52 sat-perf-03 qpidd[6410]: 2011-04-15 13:20:52 notice Broker running Apr 15 13:33:26 sat-perf-03 kernel: httpd[6482]: segfault at 8 ip 00007f00cf9d9484 sp 00007f00ab9f6fe0 error 4 in libpython2.6.so.1.0[7f00cf947000+170000] Apr 15 13:33:26 sat-perf-03 qpidd[6410]: 2011-04-15 13:33:26 notice Shut down Apr 15 13:34:08 sat-perf-03 qpidd[6644]: 2011-04-15 13:34:08 notice Listening on TCP port 5672 Apr 15 13:34:08 sat-perf-03 qpidd[6644]: 2011-04-15 13:34:08 notice Broker running Apr 17 03:36:02 sat-perf-03 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1499" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'restart'. Apr 17 03:36:02 sat-perf-03 kernel: Kernel logging (proc) stopped. httpd/error_log-20110417: [Sun Apr 17 03:36:02 2011] [notice] SIGHUP received. Attempting to restart httpd/error_log [Sun Apr 17 03:36:02 2011] [notice] Digest: generating secret for digest authentication ... [Sun Apr 17 03:36:02 2011] [notice] Digest: done [Sun Apr 17 03:36:02 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Sun Apr 17 03:36:02 2011] [notice] mod_python: using mutex_directory /tmp [Sun Apr 17 03:36:02 2011] [warn] mod_wsgi: Compiled for Python/2.6.2. [Sun Apr 17 03:36:02 2011] [warn] mod_wsgi: Runtime using Python/2.6.5. [Sun Apr 17 03:36:02 2011] [notice] Apache/2.2.15 (Unix) DAV/2 mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 configured -- resuming normal operations Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:03 2011] [notice] child pid 18518 exit signal Segmentation fault (11) [Sun Apr 17 03:36:04 2011] [notice] child pid 18737 exit signal Segmentation fault (11) [Sun Apr 17 03:36:05 2011] [notice] child pid 18743 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:06 2011] [notice] child pid 18750 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:07 2011] [notice] child pid 18756 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:08 2011] [notice] child pid 18762 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:09 2011] [notice] child pid 18768 exit signal Segmentation fault (11) [Sun Apr 17 03:36:10 2011] [notice] child pid 18775 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:11 2011] [notice] child pid 18781 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:12 2011] [notice] child pid 18787 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:13 2011] [notice] child pid 18793 exit signal Aborted (6) ) : Objects/classobject.c:2361: instancemethod_dealloc: Assertion `g->gc.gc_refs != (-2)' failed. [Sun Apr 17 03:36:14 2011] [notice] child pid 18799 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:15 2011] [notice] child pid 18805 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:16 2011] [notice] child pid 18811 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:17 2011] [notice] child pid 18817 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:18 2011] [notice] child pid 18823 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:19 2011] [notice] child pid 18829 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:20 2011] [notice] child pid 18835 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:21 2011] [notice] child pid 18841 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up Fatal Python error: PyEval_SaveThread: NULL tstate [Sun Apr 17 03:36:22 2011] [notice] child pid 18847 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:23 2011] [notice] child pid 18853 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:24 2011] [notice] child pid 18859 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:25 2011] [notice] child pid 18865 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:26 2011] [notice] child pid 18871 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:27 2011] [notice] child pid 18877 exit signal Segmentation fault (11) *** glibc detected *** (wsgi:pulp) : malloc(): memory corruption: 0x00007f934c000ad0 *** Fatal Python error: GC object already tracked [Sun Apr 17 03:36:28 2011] [notice] child pid 18883 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:29 2011] [notice] child pid 18889 exit signal Segmentation fault (11) *** glibc detected *** (wsgi:pulp) : double free or corruption (!prev): 0x00007f9368fd45d0 *** [Sun Apr 17 03:36:29 2011] [error] mod_wsgi (pid=18895): Target WSGI script '/srv/pulp/bootstrap.wsgi' cannot be loaded as Python module. [Sun Apr 17 03:36:29 2011] [error] mod_wsgi (pid=18895): Exception occurred processing WSGI script '/srv/pulp/bootstrap.wsgi'. ======= Backtrace: ========= /lib64/libc.so.6(+0x353bc75676)[0x7f936581f676] /lib64/libc.so.6(+0x353bc7aac0)[0x7f9365824ac0] /lib64/libc.so.6(realloc+0xe5)[0x7f9365824cb5] /usr/lib64/libpython2.6.so.1.0(_PyObject_GC_Resize+0x39)[0x7f935c55c5f9] /usr/lib64/libpython2.6.so.1.0(PyFrame_New+0x33d)[0x7f935c4c0a7d] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x85)[0x7f935c532d25] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x501e)[0x7f935c53161e] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5f2d)[0x7f935c53252d] /usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x904)[0x7f935c5335a4] /usr/lib64/libpython2.6.so.1.0(+0x353ec6e9f0)[0x7f935c4c19f0] /usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f935c496e13] /usr/lib64/libpython2.6.so.1.0(+0x353ec592ef)[0x7f935c4ac2ef] /usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f935c496e13] /usr/lib64/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f935c52bac3] /usr/lib64/libpython2.6.so.1.0(+0x353ed0b47a)[0x7f935c55e47a] /lib64/libpthread.so.0(+0x353c0077e1)[0x7f9365b307e1] /lib64/libc.so.6(clone+0x6d)[0x7f936588b8ed] ======= Memory map: ======== 7f934c000000-7f934c021000 rw-p 00000000 00:00 0 7f934c021000-7f9350000000 ---p 00000000 00:00 0 7f935143a000-7f935143b000 ---p 00000000 00:00 0 7f935143b000-7f9351e3b000 rw-p 00000000 00:00 0 7f9351e3b000-7f9351e43000 r-xp 00000000 fd:00 2889478 /usr/lib64/python2.6/site-packages/bson/_cbson.so 7f9351e43000-7f9352043000 ---p 00008000 fd:00 2889478 /usr/lib64/python2.6/site-packages/bson/_cbson.so 7f9352043000-7f9352044000 rw-p 00008000 fd:00 2889478 /usr/lib64/python2.6/site-packages/bson/_cbson.so 7f9352044000-7f93520c4000 r-xp 00000000 fd:00 2886960 /usr/lib64/python2.6/lib-dynload/unicodedata.so 7f93520c4000-7f93522c3000 ---p 00080000 fd:00 2886960 /usr/lib64/python2.6/lib-dynload/unicodedata.so 7f93522c3000-7f93522d4000 rw-p 0007f000 fd:00 2886960 /usr/lib64/python2.6/lib-dynload/unicodedata.so 7f93522d4000-7f93522d8000 r-xp 00000000 fd:00 2889410 /usr/lib64/python2.6/lib-dynload/termios.so 7f93522d8000-7f93524d8000 ---p 00004000 fd:00 2889410 /usr/lib64/python2.6/lib-dynload/termios.so 7f93524d8000-7f93524da000 rw-p 00004000 fd:00 2889410 /usr/lib64/python2.6/lib-dynload/termios.so <snip> 674ab000-7f93674af000 rw-p 00052000 fd:00 2501806 /usr/sbin/httpd 7f93674af000-7f93674b2000 rw-p 00000000 00:00 0 7f9368184000-7f9369010000 rw-p 00000000 00:00 0 [heap] 7fffe1634000-7fffe1671000 rw-p 00000000 00:00 0 [stack] 7fffe16d9000-7fffe16da000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] [Sun Apr 17 03:36:30 2011] [notice] child pid 18895 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:31 2011] [notice] child pid 18901 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:32 2011] [notice] child pid 18907 exit signal Aborted (6) *** glibc detected *** (wsgi:pulp) : malloc(): memory corruption: 0x00007f9368fa5ae0 *** Fatal Python error: GC object already tracked [Sun Apr 17 03:36:33 2011] [notice] child pid 18913 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:34 2011] [notice] child pid 18919 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:35 2011] [notice] child pid 18926 exit signal Segmentation fault (11) [Sun Apr 17 03:36:36 2011] [notice] child pid 18932 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:37 2011] [notice] child pid 18938 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:38 2011] [notice] child pid 18944 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:39 2011] [notice] child pid 18951 exit signal Segmentation fault (11) Fatal Python error: GC object already tracked [Sun Apr 17 03:36:40 2011] [notice] child pid 18957 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:41 2011] [notice] child pid 18963 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up Fatal Python error: PyEval_SaveThread: NULL tstate [Sun Apr 17 03:36:42 2011] [notice] child pid 18969 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:43 2011] [notice] child pid 18975 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:44 2011] [notice] child pid 18981 exit signal Segmentation fault (11) Fatal Python error: GC object already tracked [Sun Apr 17 03:36:45 2011] [notice] child pid 18987 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:46 2011] [notice] child pid 18994 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:47 2011] [notice] child pid 19000 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:48 2011] [notice] child pid 19006 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:49 2011] [notice] child pid 19012 exit signal Segmentation fault (11) [Sun Apr 17 03:36:50 2011] [notice] child pid 19018 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:51 2011] [notice] child pid 19024 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:52 2011] [notice] child pid 19030 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:53 2011] [notice] child pid 19036 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:54 2011] [notice] child pid 19042 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:55 2011] [notice] child pid 19048 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:56 2011] [notice] child pid 19054 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:57 2011] [notice] child pid 19060 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun Apr 17 03:36:58 2011] [notice] child pid 19066 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up :
Created attachment 492927 [details] Debug logs from crash 4/17@3:36am
Investigating as part of sprint 23.
Preethi sees another httpd crash on a Sunday around 3am. At 3:33am a SIGHUP is sent to apache, on restart there are multiple crashes. From apache error logs on 5/1 [Sun May 01 03:33:01 2011] [notice] SIGHUP received. Attempting to restart [Sun May 01 03:33:01 2011] [notice] Digest: generating secret for digest authentication ... [Sun May 01 03:33:01 2011] [notice] Digest: done [Sun May 01 03:33:01 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Sun May 01 03:33:01 2011] [notice] mod_python: using mutex_directory /tmp [Sun May 01 03:33:01 2011] [notice] SSL FIPS mode disabled [Sun May 01 03:33:01 2011] [notice] Apache/2.2.17 (Unix) DAV/2 mod_python/3.3.1 Python/2.7 mod_ssl/2.2.17 OpenSSL/1.0.0c-fips mod_wsgi/3.2 configured -- resuming normal operations ) : /builddir/build/BUILD/Python-2.7/Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. [Sun May 01 03:33:02 2011] [notice] child pid 31255 exit signal Aborted (6) ... ... ... [Sun May 01 03:33:34 2011] [notice] child pid 1427 exit signal Segmentation fault (11) *** glibc detected *** (wsgi:pulp) : double free or corruption (!prev): 0x00007f7d2e3d4da0 *** ) : /builddir/build/BUILD/Python-2.7/Objects/obmalloc.c:779: PyObject_Malloc: Assertion `bp != ((void *)0)' failed. [Sun May 01 03:33:35 2011] [notice] child pid 1441 exit signal Aborted (6) Fatal Python error: ceval: tstate mix-up [Sun May 01 03:33:37 2011] [notice] child pid 1457 exit signal Aborted (6) [Sun May 01 03:33:38 2011] [notice] child pid 1467 exit signal Segmentation fault (11) ... ... ... ) : /builddir/build/BUILD/Python-2.7/Modules/gcmodule.c:326: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. [Sun May 01 03:35:15 2011] [notice] child pid 2547 exit signal Aborted (6) [Sun May 01 03:35:16 2011] [notice] child pid 2553 exit signal Segmentation fault (11) [Sun May 01 03:35:17 2011] [notice] child pid 2559 exit signal Segmentation fault (11) *** glibc detected *** (wsgi:pulp) : realloc(): invalid pointer: 0x00007f7d21ec98a0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x3586a75526)[0x7f7d2b637526] /lib64/libc.so.6(realloc+0x32e)[0x7f7d2b63c90e] /usr/lib64/libpython2.7.so.1.0(_PyString_Resize+0x4d)[0x7f7d21ba6f8d] /usr/lib64/libpython2.7.so.1.0(PyString_FromFormatV+0x387)[0x7f7d21bae237] /usr/lib64/libpython2.7.so.1.0(PyErr_Format+0x90)[0x7f7d21c06aa0] /usr/lib64/libpython2.7.so.1.0(+0x3593e458ac)[0x7f7d21b578ac] /usr/lib64/libpython2.7.so.1.0(PyMapping_Size+0x34)[0x7f7d21b5ad04] /usr/lib64/libpython2.7.so.1.0(_PyObject_LengthHint+0x23)[0x7f7d21b5bb33] /usr/lib64/libpython2.7.so.1.0(+0x3593e77b34)[0x7f7d21b89b34] /usr/lib64/libpython2.7.so.1.0(PySequence_List+0x4a)[0x7f7d21b5ac1a] /usr/lib64/libpython2.7.so.1.0(PySequence_Fast+0x3c)[0x7f7d21b5c26c] /usr/lib64/python2.7/lib-dynload/selectmodule.so(+0x2ceb)[0x7f7d1f13fceb] /usr/lib64/python2.7/lib-dynload/selectmodule.so(+0x301c)[0x7f7d1f14001c] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x560b)[0x7f7d21bfb65b] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88d)[0x7f7d21bfd04d] /usr/lib64/libpython2.7.so.1.0(+0x3593e71d6b)[0x7f7d21b83d6b] /usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x53)[0x7f7d21b5afc3] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3f49)[0x7f7d21bf9f99] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x66cd)[0x7f7d21bfc71d]
On my Fedora 14 x86_64 developer box running code from 4/29/11 [Sun May 01 03:34:06 2011] [notice] SIGHUP received. Attempting to restart [Sun May 01 03:34:06 2011] [notice] Digest: generating secret for digest authentication ... [Sun May 01 03:34:06 2011] [notice] Digest: done [Sun May 01 03:34:06 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Sun May 01 03:34:06 2011] [notice] mod_python: using mutex_directory /tmp [Sun May 01 03:34:06 2011] [notice] SSL FIPS mode disabled [Sun May 01 03:34:06 2011] [notice] Apache/2.2.17 (Unix) DAV/2 mod_python/3.3.1 Python/2.7 mod_ssl/2.2.17 OpenSSL/1.0.0d-fips mod_wsgi/3.2 configured -- resuming normal operations [Sun May 01 03:34:07 2011] [notice] child pid 13940 exit signal Segmentation fault (11) [Sun May 01 03:34:08 2011] [notice] child pid 13984 exit signal Segmentation fault (11) [Sun May 01 03:34:09 2011] [notice] child pid 13990 exit signal Segmentation fault (11) [Sun May 01 03:34:10 2011] [notice] child pid 13996 exit signal Segmentation fault (11) [Sun May 01 03:34:11 2011] [notice] child pid 14002 exit signal Segmentation fault (11) [Sun May 01 03:34:12 2011] [notice] child pid 14008 exit signal Segmentation fault (11) [Sun May 01 03:34:13 2011] [notice] child pid 14026 exit signal Segmentation fault (11) [Sun May 01 03:34:14 2011] [notice] child pid 14032 exit signal Segmentation fault (11) *** glibc detected *** (wsgi:pulp) : malloc(): smallbin double linked list corrupted: 0x00007f29ed39a550 *** [Mon May 02 07:44:20 2011] [notice] caught SIGTERM, shutting down [Mon May 02 07:44:20 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Mon May 02 07:44:20 2011] [notice] SSL FIPS mode disabled [Mon May 02 07:44:22 2011] [notice] Digest: generating secret for digest authentication ... [Mon May 02 07:44:22 2011] [notice] Digest: done [Mon May 02 07:44:22 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Mon May 02 07:44:22 2011] [notice] mod_python: using mutex_directory /tmp [Mon May 02 07:44:22 2011] [notice] SSL FIPS mode disabled [Mon May 02 07:44:22 2011] [notice] Apache/2.2.17 (Unix) DAV/2 mod_python/3.3.1 Python/2.7 mod_ssl/2.2.17 OpenSSL/1.0.0d-fips mod_wsgi/3.2 configured -- resuming normal operations
To simulate the messages we've seen in the log you can run: /etc/init.d/httpd reload Shows crashes with below ] Apache/2.2.17 (Unix) DAV/2 mod_python/3.3.1 Python/2.7 mod_ssl/2.2.17 OpenSSL/1.0.0d-fips mod_wsgi/3.2 configured -- resuming normal operations Fatal Python error: ceval: tstate mix-up [Mon May 02 13:00:13 2011] [notice] child pid 16810 exit signal Aborted (6) [Mon May 02 13:00:14 2011] [notice] child pid 16825 exit signal Segmentation fault (11) [Mon May 02 13:00:15 2011] [notice] child pid 16831 exit signal Segmentation fault (11) [Mon May 02 13:00:16 2011] [notice] child pid 16837 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Mon May 02 13:00:17 2011] [notice] child pid 16843 exit signal Aborted (6) *** glibc detected *** (wsgi:pulp) : malloc(): smallbin double linked list corrupted: 0x00007ffe6af06ad0 *** pulp-admin repo list hangs if run after the reload.
I saw similar issue on rhel6. https://bugzilla.redhat.com/show_bug.cgi?id=701221 Im my case, system was idle from last 2 days. After two days, when I tried to access the pulp-server with "pulp-admin repo list" command, it simply thrown an "Internal Server error". I saw similar messages of "segmentation fault" in error_log file of apache. But when I restart the httpd daemon, server came up and everything start working.
*** Bug 701221 has been marked as a duplicate of this bug. ***
Seen Thur 3:37am on Preethi's box from CR 181. [Thu May 26 03:37:21 2011] [notice] SIGHUP received. Attempting to restart [Thu May 26 03:37:21 2011] [notice] Digest: generating secret for digest authentication ... [Thu May 26 03:37:21 2011] [notice] Digest: done [Thu May 26 03:37:21 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Thu May 26 03:37:21 2011] [notice] mod_python: using mutex_directory /tmp [Thu May 26 03:37:21 2011] [notice] SSL FIPS mode disabled [Thu May 26 03:37:21 2011] [notice] Apache/2.2.17 (Unix) DAV/2 mod_python/3.3.1 Python/2.7 mod_ssl/2.2.17 OpenSSL/1.0.0c-fips mod_wsgi/3.2 configured -- resuming normal operations Fatal Python error: GC object already tracked [Thu May 26 03:37:23 2011] [notice] child pid 28077 exit signal Aborted (6) *** glibc detected *** (wsgi:pulp) : free(): corrupted unsorted chunks: 0x00007ff058f105e0 *** ) : /builddir/build/BUILD/Python-2.7/Objects/frameobject.c:671: PyFrame_New: Assertion `f->f_code == code' failed. *** glibc detected *** (wsgi:pulp) : malloc(): memory corruption: 0x00007ff059004530 *** [Thu May 26 03:37:24 2011] [notice] child pid 28092 exit signal Aborted (6) *** glibc detected *** (wsgi:pulp) : double free or corruption (!prev): 0x00007ff059743f00 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x3586a76d63)[0x7ff056657d63] /lib64/libc.so.6(+0x3586a78de0)[0x7ff056659de0] /lib64/libc.so.6(realloc+0xe6)[0x7ff05665b6c6] /usr/lib64/libpython2.7.so.1.0(_PyObject_GC_Resize+0x39)[0x7ff04cc4c4a9] /usr/lib64/libpython2.7.so.1.0(PyFrame_New+0x339)[0x7ff04cba1df9] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5984)[0x7ff04cc1a9d4] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x66cd)[0x7ff04cc1b71d] ======= Memory map: ======== 7ff03c000000-7ff03c021000 rw-p 00000000 00:00 0 7ff03c021000-7ff040000000 ---p 00000000 00:00 0
My Fedora 14 Box, X86_64, crashed Sunday night at 3:14AM. Here is the pertinent stuff in /var/log/httpd/error_log [Sun Jun 12 03:14:04 2011] [notice] child pid 1535 exit signal Aborted (6) [Sun Jun 12 03:14:05 2011] [notice] child pid 1543 exit signal Segmentation fault (11) [Sun Jun 12 03:14:06 2011] [notice] child pid 1556 exit signal Segmentation fault (11) [Sun Jun 12 03:14:07 2011] [notice] child pid 1579 exit signal Segmentation fault (11) *** glibc detected *** (wsgi:pulp) : free(): corrupted unsorted chunks: 0x00007febfde9cbd0 *** [Sun Jun 12 03:14:08 2011] [notice] child pid 1589 exit signal Segmentation fault (11) Fatal Python error: ceval: tstate mix-up [Sun Jun 12 03:14:09 2011] [notice] child pid 1649 exit signal Aborted (6) [Sun Jun 12 03:14:10 2011] [notice] child pid 1656 exit signal Segmentation fault (11) *** glibc detected *** (wsgi:pulp) : double free or corruption (!prev): 0x00007febfeab0eb0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x3e24276d63)[0x7febfad57d63] /lib64/libc.so.6(+0x3e24278de0)[0x7febfad59de0] /lib64/libc.so.6(realloc+0xe6)[0x7febfad5b6c6] /usr/lib64/libpython2.7.so.1.0(_PyObject_GC_Resize+0x39)[0x7febf134b4a9] /usr/lib64/libpython2.7.so.1.0(PyFrame_New+0x339)[0x7febf12a0df9] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5984)[0x7febf13199d4] /usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x66cd)[0x7febf131a71d]
This is a RHUI blocker. The RHUI filed bug is 713738
Todd found this link which describes a similar issue seen during 'logrotate' running. http://www.canfield.com/content/subversion-moddav-and-runaway-memory Short term fix they used was to change logrotate to 'restart' httpd instead of issuing a 'reload'
I've committed the change to remove the dependency on mod_python. Repo auth is now handled by mod_wsgi, and mod_python is no longer needed. Since it's gone, we'll no longer see these segfaults. commit: bca15251be91d1ecd8b29edbd6e34591fbb8a142
Build 0.206, CR14.
moving to verified I haven't seen this problem in a while and I am on build 232 [root@preethi ~]# rpm -q pulp pulp-0.0.232-1.fc15.noarch
Pulp v1.0 is released Closed Current Release.