Bug 711095
| Summary: | Python garbage collector assertion failure running pq_execute (cursorObject refcount issue) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Red Hat Case Diagnostics <case-diagnostics> | ||||||||||
| Component: | python-psycopg2 | Assignee: | Tom Lane <tgl> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | qe-baseos-daemons | ||||||||||
| Severity: | high | Docs Contact: | |||||||||||
| Priority: | high | ||||||||||||
| Version: | 6.0 | CC: | azelinka, cww, daniele.varrazzo, dmalcolm, hhorak, jscotka, julian, maor.conforti, psklenar, rdassen, syeghiay, ychavan | ||||||||||
| Target Milestone: | rc | Keywords: | ZStream | ||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | x86_64 | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | abrt_hash:f8a305d69951574f208e836a452f63fe1020d300 | ||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
| Doc Text: |
Previously, if a second thread in a single application triggered the Python garbage collection while a copy operation was in progress, the copy operation terminated unexpectedly. With this update, appropriate object reference count adjustments have been added to the code, and this bug no longer occurs.
|
Story Points: | --- | ||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2013-02-21 08:41:25 UTC | Type: | --- | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Bug Depends On: | |||||||||||||
| Bug Blocks: | 720306 | ||||||||||||
| Attachments: |
|
||||||||||||
Created attachment 503247 [details]
File: backtrace
Thanks for filing this bug report. How often is this problem occurring?
I believe the crash is happening due to a bug in python-psycopg2. I am about to attach a proposed patch to that package which I believe will fix it. I don't have a minimal reproducer yet.
Technical analysis follows:
Based on reviewing the coredump, the crash is happening in thread #1, during one of python's periodic invocations of its garbage collector. The assertion failure is that an object has a lower reference count than expected (based on the reference-owning objects pointing to it).
This object is a StringIO object, containing a series of lines containing dates and numbers (I'm deliberately not including it in a public bz comment, in case it contains private information).
Meanwhile thread #4 is within python-psycopg2's pq_execute function executing a query. Specifically, it is within the function "psyco_curs_copy_from" in psycopg2-2.0.13/psycopg/cursor_type.c
I notice that within thread #4, pcurs->copyfile contains the textual data seen in the crash in thread #1. What's happened is that the StringIO instance containing the query has too low a reference count.
Further, pyscopg2's cursorObject participates in the Python garbage collector: psycopg2-2.0.13/psycopg/cursor_type.c: cursor_traverse has:
Py_VISIT(self->copyfile);
It acquires PyObject *file via:
if (!PyArg_ParseTupleAndKeywords(args, kwargs,
"O&s|ss" CONV_CODE_PY_SSIZE_T "O", kwlist,
_psyco_curs_has_read_check, &file, &table_name, &sep, &null, &bufsize,
&columns))
i.e. from "O& (object) [converter, anything]" with converter "_psyco_curs_has_read_check":
Looking at _psyco_curs_has_read_check, it has this code and comment:
/* It's OK to store a borrowed reference, because it is only held for
* the duration of psyco_curs_copy_from. */
*((PyObject**)var) = o;
return 1;
This is thus storing a borrowed reference to the StringIO object into the psyco's cursorObject without INCREFing it. This reference is is visible to the garbage collector.
The comment in the code above is incorrect: within the call to pq_execute, there is an invocation of the Py_BEGIN_ALLOW_THREADS macro. This allows the GIL to transition to another thread, and it seems that this is what happened in this case (within the coredump, that thread is waiting at the subsequent Py_END_ALLOW_THREADS macro, waiting to reacquire the GIL).
When the GC runs in the other thread, it detects that the refcount is too low, and bails out with an assertion failure. (I believe that if assertions were disabled, it could try to free up the memory, so we'd be seeing a different kind of crash)
I believe the fix will be to add INCREF/DECREF pairs around the call to pq_execute within psyco_curs_copy_from, and I'll try to write a patch that does so. I don't have a minimal reproducer yet.
I checked that latest upstream release of the code (2.4.1), and I believe this bug is still present there.
Created attachment 503316 [details]
Proposed patch
I'm not really qualified to evaluate this. I'll see if I can get a comment from upstream. Created attachment 503327 [details]
Minimal reproducer
With the attached script, I'm reliably able to reproduce the error:
python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
on my test box (with the same versions as in the report).
(In reply to comment #14) > Created attachment 503327 [details] > With the attached script, I'm reliably able to reproduce the error: Having said that, getting it to happen may be somewhat timing dependent (this is a bad interaction between two threads). I had to tweak the delay in the loop in the GCThread (currently 0.1) before I could get it to occur, this was with the postgres server on the same host as the script, so YMMV. The patch attached in comment #11 fixes the crash seen by running comment #14, so I believe we have a fix for this issue. (In reply to comment #14) > Created attachment 503327 [details] > Minimal reproducer > > With the attached script, I'm reliably able to reproduce the error: > python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' > failed. > > on my test box (with the same versions as in the report). I've also now stepped through the crash in gdb, and this reproducer is indeed crashing python in the same way as in the original coredump. (gdb) t a a py-bt Thread 2 (Thread 0x7fffefd1a700 (LWP 9729)): #9 file 'test.py', in 'run' #12 file '/usr/lib64/python2.6/threading.py', in '__bootstrap_inner' #15 file '/usr/lib64/python2.6/threading.py', in '__bootstrap' Thread 1 (Thread 0x7ffff7fed700 (LWP 9727)): #7 file 'test.py', in '<module>' Thread 2 is garbage-collecting (and crashing with that assertion failure) Thread 1 is waiting to reacquire the GIL, within pq_execute, within psyco_curs_copy_from. For reference, the upstream bug Tom opened appears to be here: http://psycopg.lighthouseapp.com/projects/62710/tickets/58 Note to psycopg2 upstream folks: to reproduce the C level assertion failure in gcmodule.c, you need a build of Python with assertions enabled. Typically for an upstream build of python, this means configuring --with-pydebug. Thank you for the report and the patch. I have verified the bug and the fix. Note that the functions copy_to and copy_expert were affected by the same bug: I have fixed them as well. The devel branch of my repos (https://github.com/dvarrazzo/psycopg) contains the patches. They will be released in the upcoming release 2.4.2. Daniele: Thanks for the info. Your upstream fix appears to be: https://github.com/dvarrazzo/psycopg/commit/e9a485e30bf74c598536417819f196476e0dc406 Looking there, I think that commit may introduce a bug within psyco_curs_copy_expert, within the error handling paths. Specifically, the "file" variable is set using PyArg_ParseTupleAndKeywords with code "O", which gives a borrowed reference, not modifying the refcount. If an error occurs, and one of the first two "goto exit;" clauses is run, then my reading of that code is that you have a Py_XDECREF(file), without a matching Py_INCREF(file), so "file" loses a reference, which may lead to it being prematirely deallocated. Am I reading that correctly? I think the Py_INCREF(file); within in that function needs to be moved higher up, to before the first "goto exit" e.g. to directly below this comment: /* Any failure from here forward should 'goto exit' rather than 'return NULL' directly. */ The changes to the other two functions (copy_from/copy_to) look correct to me. [FWIW, I'm working on a static analysis tool to try to make it easier to deal with this kind of thing in the future, see: https://fedoraproject.org/wiki/Features/StaticAnalysisOfCPythonExtensions But that code isn't ready yet] (In reply to comment #24) > Looking there, I think that commit may introduce a bug within > psyco_curs_copy_expert, within the error handling paths. Uhm... right. I'd implemented a pattern typical of an object owned by the function, didn't spot that file was coming from the parsed args. I've modified copy_expert to make it more similar to copy_from/to. https://github.com/dvarrazzo/psycopg/commit/816b5dda33a8914532da160bded65a6f3afb9c9b Thank you again! > [FWIW, I'm working on a static analysis tool to try to make it easier to deal > with this kind of thing in the future I would *love* to use such a tool! (In reply to comment #25) > (In reply to comment #24) > > > Looking there, I think that commit may introduce a bug within > > psyco_curs_copy_expert, within the error handling paths. > > Uhm... right. I'd implemented a pattern typical of an object owned by the > function, didn't spot that file was coming from the parsed args. > > I've modified copy_expert to make it more similar to copy_from/to. > > https://github.com/dvarrazzo/psycopg/commit/816b5dda33a8914532da160bded65a6f3afb9c9b FWIW, I doublechecked that new implementation of psyco_curs_copy_expert, and it looks correct to me also. Thanks! Question: Do you have an idea why I see this error only on with the RHEL6 installation of Python? We've tried it on our own build of Python v2.6.5 + Psycopg2 v.2.0.13 (same versions as what comes with RHEL6), on the same RHEL6 machine, and this error does not happen. (It also didn't happen on CentOS 5.5) Any ideas? (In reply to comment #27) These assertions are typically disabled when building python, but, for better or worse, the build flags we use mean that they are enabled in RHEL 6's python build. There was some discussion within bug 614680 about changing this in RHEL 6.1. As noted in that bug, doing so could well turn errors of this kind into segfaults rather than asserts, hence in that bug I chose to keep the extra error-checking they provide, whilst improving the error messages (in python-2.6.6-5.el6) so that they indicate which object has reference-count issues. Whether or not the assertion is protecting against a segfault varies from case to case. My (tentative) reading of this case is that (for this specific case) the assertion is overzealous. The StringIO instance has an ob_refcnt of 2: presumably 2 frame's worth of dictionaries of local variables (the code is too optimized to directly verify this). The third reference is the one within the cursor_object within psyco_curs_copy_from. If those other two references were garbage-collected away by the other thread, then the cursorObject could have a borrowed reference to the StringIO, but the StringIO could have been deallocated from under it. However, I don't believe this would have been a problem, for two reasons: - assuming that the other two referrers are frame locals() dictionaries, then these will outlive the call to psyco_curs_copy_from - upon reacquisition of the GIL, psyco_curs_copy_from makes no further use of the value, immediately setting it back to NULL. Having said that, the benign nature here is merely due to the specifics of the user code and of psycopg2 in this case: invariants are being violated, but it happens to not matter. There are plenty of other situations in which similar bugs could lead to crashes of /usr/bin/python, and such crashes would be significantly less predictable and more obscure. There is perhaps an argument for reassessing bug 614680 (e.g. turning off the asserts, or making this a command-line flag or somesuch), but for now, the fix is to patch python-psycopg2 (using the patches referred to in comment #24 and comment #25). Hopefully the above makes sense. (In reply to comment #29) > There is perhaps an argument for reassessing bug 614680 (e.g. turning off the > asserts, or making this a command-line flag or somesuch), but for now, the fix > is to patch python-psycopg2 (using the patches referred to in comment #24 and > comment #25). I quite agree that turning off the assertions would be an inappropriate "fix". Created attachment 503578 [details] Backport of the two upstream patches to the 2.0.13 in RHEL6 Candidate patch: the attached patch is a combination of: https://github.com/dvarrazzo/psycopg/commit/e9a485e30bf74c598536417819f196476e0dc406 followed by: https://github.com/dvarrazzo/psycopg/commit/679af4a97528c140ac8e9cd85598f717b3f5cfb9 fixed up to apply cleanly to the 2.0.13 within RHEL. [It does not contain the change to tests/test_copy.py from 679af4a97528c140ac8e9cd85598f717b3f5cfb9 as that entire file only appeared after 2.0.13] As a smoketest, I have rebuilt RHEL6's python-psycopg2-2.0.13-2.el6 adding this patch, and tested with the psycopg2-2.0.13/scripts/ticket58.py (see the prepped source tree). Each of the three test cases within that script ("copy_from", "copy_to", and "copy_expert") fail with this error: python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. Aborted (core dumped) when run against the old package (with a suitably-configured postgres DB, see the comment in that file). All of them successfully run to completion with the patched version of the package. I have merely smoketested the proposed test cases. I have not tested other functionality of the package. BTW, I've pushed psycopg 2.4.2 into Fedora rawhide, so this is fixed over there. This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. If you would like it considered as an exception in the current release, please ask your support representative.
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Previously, if a second thread in a single application triggered the Python garbage collection while a copy operation was in progress, the copy operation terminated unexpectedly. With this update, appropriate object reference count adjustments have been added to the code, and this bug no longer occurs.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0327.html |
This description generated by Andreas from an ABRT report architecture: x86_64 cmdline: /usr/bin/python manage.py runconcurrentserver 0.0.0.0:8000 comment: The Django process dies about a minute after starting serving pages to browser. component: python executable: /usr/bin/python kernel: 2.6.32-71.14.1_KM7.el6.x86_64 package: python-2.6.5-3.el6_0.2 reason: Process /usr/bin/python was killed by signal 6 (SIGABRT) release: Red Hat Enterprise Linux Server release 6.0 (Santiago) reproduce: 1. 2. 3. time: 1307369154 uid: 0 backtrace: [New Thread 1215] [New Thread 1187] [New Thread 1192] [New Thread 1197] [New Thread 1194] [New Thread 1191] [New Thread 1186] Core was generated by `/usr/bin/python manage.py runconcurrentserver 0.0.0.0:8000'. Program terminated with signal 6, Aborted. #0 0x0000003b620329a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); Thread 7 (Thread 1186): #0 0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82 No locals. #1 0x00007f1528008219 in ?? () from /usr/lib64/python2.6/lib-dynload/timemodule.so No symbol table info available. #2 0x0000003b65cdeb01 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #3 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #4 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #5 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #6 0x0000003b65cde61e in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #7 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #8 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #9 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #10 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #11 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #12 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #13 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #14 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #15 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #16 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #17 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #18 0x0000003b65cde61e in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #19 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #20 0x0000003b65ce06a2 in PyEval_EvalCode () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #21 0x0000003b65cfb6ec in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #22 0x0000003b65cfb7c0 in PyRun_FileExFlags () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #23 0x0000003b65cfcc5c in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #24 0x0000003b65d092bd in Py_Main () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #25 0x0000003b6201ec5d in __libc_start_main (main=0x400710 <main>, argc=4, ubp_av=0x7fffcb045808, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fffcb0457f8) at libc-start.c:226 result = <value optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 5746914283774943270, 4195872, 140736599447552, 0, 0, -5746960227838320602, 5744306228104142886}, mask_was_saved = 0}}, priv = {pad = { 0x0, 0x0, 0x400730, 0x7fffcb045808}, data = {prev = 0x0, cleanup = 0x0, canceltype = 4196144}}} not_first_call = <value optimized out> #26 0x0000000000400649 in _start () No symbol table info available. Thread 6 (Thread 1191): #0 0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82 No locals. #1 0x00007f1528008219 in ?? () from /usr/lib64/python2.6/lib-dynload/timemodule.so No symbol table info available. #2 0x0000003b65cdeb01 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #3 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #4 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #5 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #6 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #7 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #8 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #9 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #10 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #11 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #12 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #13 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #14 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #15 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #16 0x000000374f8077e1 in start_thread (arg=0x7f151d76f700) at pthread_create.c:301 __res = <value optimized out> pd = 0x7f151d76f700 now = <value optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728665376512, 5746914283774943270, 139728677952336, 139728665377216, 0, 3, -5626369248591470554, 5742013758861352998}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <value optimized out> pagesize_m1 = <value optimized out> sp = <value optimized out> freesize = <value optimized out> #17 0x0000003b620e18ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 No locals. Thread 5 (Thread 1194): #0 0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82 No locals. #1 0x00007f1528008219 in ?? () from /usr/lib64/python2.6/lib-dynload/timemodule.so No symbol table info available. #2 0x0000003b65cdeb01 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #3 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #4 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #5 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #6 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #7 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #8 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #9 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #10 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #11 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #12 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #13 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #14 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #15 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #16 0x000000374f8077e1 in start_thread (arg=0x7f150ffff700) at pthread_create.c:301 __res = <value optimized out> pd = 0x7f150ffff700 now = <value optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728439473920, 5746914283774943270, 139728677952336, 139728439474624, 0, 3, -5626337440063676378, 5742013758861352998}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <value optimized out> pagesize_m1 = <value optimized out> sp = <value optimized out> freesize = <value optimized out> #17 0x0000003b620e18ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 No locals. Thread 4 (Thread 1197): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86 No locals. #1 0x0000003b65d06ec8 in PyThread_acquire_lock () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #2 0x0000003b65ce0704 in PyEval_RestoreThread () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #3 0x00007f151f20d288 in ?? () from /usr/lib64/python2.6/site-packages/psycopg2/_psycopg.so No symbol table info available. #4 0x00007f151f211d0a in ?? () from /usr/lib64/python2.6/site-packages/psycopg2/_psycopg.so No symbol table info available. #5 0x0000003b65cdf90b in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #6 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #7 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #8 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #9 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #10 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #11 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #12 0x0000003b65cde61e in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #13 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #14 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #15 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #16 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #17 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #18 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #19 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #20 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #21 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #22 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #23 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #24 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #25 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #26 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #27 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #28 0x000000374f8077e1 in start_thread (arg=0x7f150ebfd700) at pthread_create.c:301 __res = <value optimized out> pd = 0x7f150ebfd700 now = <value optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728418494208, 5746914283774943270, 139728677952336, 139728418494912, 0, 3, -5626340189916487642, 5742013758861352998}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <value optimized out> pagesize_m1 = <value optimized out> sp = <value optimized out> freesize = <value optimized out> #29 0x0000003b620e18ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 No locals. Thread 3 (Thread 1192): #0 0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82 No locals. #1 0x00007f1528008219 in ?? () from /usr/lib64/python2.6/lib-dynload/timemodule.so No symbol table info available. #2 0x0000003b65cdeb01 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #3 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #4 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #5 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #6 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #7 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #8 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #9 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #10 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #11 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #12 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #13 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #14 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #15 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #16 0x000000374f8077e1 in start_thread (arg=0x7f151cd6e700) at pthread_create.c:301 __res = <value optimized out> pd = 0x7f151cd6e700 now = <value optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728654886656, 5746914283774943270, 139728677952336, 139728654887360, 0, 3, -5626370624591618010, 5742013758861352998}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <value optimized out> pagesize_m1 = <value optimized out> sp = <value optimized out> freesize = <value optimized out> #17 0x0000003b620e18ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 No locals. Thread 2 (Thread 1187): #0 0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82 No locals. #1 0x00007f1527e03404 in ?? () from /usr/lib64/python2.6/lib-dynload/selectmodule.so No symbol table info available. #2 0x0000003b65cdeb01 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #3 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #4 0x0000003b65cde61e in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #5 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #6 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #7 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #8 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #9 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #10 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #11 0x000000374f8077e1 in start_thread (arg=0x7f151edae700) at pthread_create.c:301 __res = <value optimized out> pd = 0x7f151edae700 now = <value optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728688703232, 5746914283774943270, 140736599442448, 139728688703936, 0, 3, -5626375125717344218, 5742013758861352998}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <value optimized out> pagesize_m1 = <value optimized out> sp = <value optimized out> freesize = <value optimized out> #12 0x0000003b620e18ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 No locals. Thread 1 (Thread 1215): #0 0x0000003b620329a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 resultvar = 0 pid = <value optimized out> selftid = 1215 #1 0x0000003b62034185 in abort () at abort.c:92 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x3b65d47cc5, sa_sigaction = 0x3b65d47cc5}, sa_mask = {__val = {255048558270, 139728677980016, 277, 139728677980256, 255047654286, 206158430232, 139728677980272, 139728677980048, 255047561544, 206158430256, 139728677980296, 139728440452944, 117, 117, 0, 140736599455871}}, sa_flags = 1645481933, sa_restorer = 0x3b65d47ca0} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x0000003b6202b935 in __assert_fail ( assertion=0x3b65d47cc5 "gc->gc.gc_refs != 0", file=<value optimized out>, line=277, function=<value optimized out>) at assert.c:81 buf = 0x7f15100ee750 "python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.\n" #3 0x0000003b65d0971e in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #4 0x0000003b65c96e5e in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #5 0x0000003b65d09e7f in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #6 0x0000003b65d0a8da in _PyObject_GC_Malloc () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #7 0x0000003b65d0a94d in _PyObject_GC_New () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #8 0x0000003b65c74f5c in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #9 0x0000003b65c44e0e in PyObject_GetIter () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #10 0x0000003b65cd9ee7 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #11 0x0000003b65c6387e in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #12 0x0000003b65cd9f4d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #13 0x0000003b65c6387e in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #14 0x0000003b65cd9f4d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #15 0x0000003b65c6387e in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #16 0x0000003b65c745eb in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #17 0x0000003b65c74870 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #18 0x0000003b65c9b318 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #19 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #20 0x0000003b65cdd8b1 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #21 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #22 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #23 0x0000003b65cde61e in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #24 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #25 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #26 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #27 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #28 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #29 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #30 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #31 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #32 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #33 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #34 0x0000003b65c9cc44 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #35 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #36 0x0000003b65cdd8b1 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #37 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #38 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #39 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #40 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #41 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #42 0x0000003b65c9cc44 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #43 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #44 0x0000003b65cdd8b1 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #45 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #46 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #47 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #48 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #49 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #50 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #51 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #52 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #53 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #54 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #55 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #56 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #57 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #58 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #59 0x0000003b65c58a46 in PyInstance_New () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #60 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #61 0x0000003b65cdd8b1 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #62 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #63 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #64 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #65 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #66 0x0000003b65cdd3f2 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #67 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #68 0x0000003b65cdf52d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #69 0x0000003b65ce05a4 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #70 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #71 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #72 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #73 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #74 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #75 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0 No symbol table info available. #76 0x000000374f8077e1 in start_thread (arg=0x7f151e378700) at pthread_create.c:301 __res = <value optimized out> pd = 0x7f151e378700 now = <value optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728677996288, 5746914283774943270, 139728688697296, 139728677996992, 0, 3, -5626376400248889306, 5742013758861352998}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <value optimized out> pagesize_m1 = <value optimized out> sp = <value optimized out> freesize = <value optimized out> #77 0x0000003b620e18ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 No locals. From To Syms Read Shared Object Library 0x0000003b65c3c490 0x0000003b65d1ef08 Yes (*) /usr/lib64/libpython2.6.so.1.0 0x000000374f805640 0x000000374f810ec8 Yes /lib64/libpthread-2.12.so 0x0000003b61c00de0 0x0000003b61c01998 Yes /lib64/libdl-2.12.so 0x0000003b64000e10 0x0000003b64001688 Yes /lib64/libutil-2.12.so 0x0000003b62c03ea0 0x0000003b62c43fa8 Yes /lib64/libm-2.12.so 0x0000003b6201e9a0 0x0000003b6212bbe0 Yes /lib64/libc-2.12.so 0x0000003b61800af0 0x0000003b61818854 Yes /lib64/ld-2.12.so 0x00007f152820b9d0 0x00007f152820bfe8 Yes (*) /usr/lib64/python2.6/lib-dynload/syslog.so 0x00007f15280078d0 0x00007f1528008898 Yes (*) /usr/lib64/python2.6/lib-dynload/timemodule.so 0x00007f1527e01be0 0x00007f1527e036a8 Yes (*) /usr/lib64/python2.6/lib-dynload/selectmodule.so 0x00007f1527bfcd10 0x00007f1527bfda18 Yes (*) /usr/lib64/python2.6/lib-dynload/fcntlmodule.so 0x00007f15279f5820 0x00007f15279f87a8 Yes (*) /usr/lib64/python2.6/lib-dynload/_struct.so 0x00007f15277ae0c0 0x00007f15277aff78 Yes (*) /usr/lib64/python2.6/lib-dynload/binascii.so 0x00007f15275a9aa0 0x00007f15275aabd8 Yes (*) /usr/lib64/python2.6/lib-dynload/cStringIO.so 0x00007f152739c110 0x00007f15273a1898 Yes (*) /usr/lib64/python2.6/lib-dynload/_socketmodule.so 0x00007f1527193f70 0x00007f1527196958 Yes (*) /usr/lib64/python2.6/lib-dynload/_ssl.so 0x000000374bc14500 0x000000374bc45f68 Yes (*) /usr/lib64/libssl.so.10 0x000000374885c900 0x0000003748920d08 Yes (*) /usr/lib64/libcrypto.so.10 0x000000374fc07d80 0x000000374fc30af8 Yes (*) /lib64/libgssapi_krb5.so.2 0x000000374b81a1c0 0x000000374b88cb78 Yes (*) /lib64/libkrb5.so.3 0x00000037494013b0 0x0000003749401f88 Yes (*) /lib64/libcom_err.so.2 0x000000374a404290 0x000000374a41bb28 Yes (*) /lib64/libk5crypto.so.3 0x0000003b62801ef0 0x0000003b6280d1a8 Yes (*) /lib64/libz.so.1 0x00000037498027a0 0x0000003749807898 Yes (*) /lib64/libkrb5support.so.0 0x000000374cc00bc0 0x000000374cc011a8 Yes (*) /lib64/libkeyutils.so.1 0x0000003b644038c0 0x0000003b64412558 Yes /lib64/libresolv-2.12.so 0x0000003b63005610 0x0000003b63015708 Yes (*) /lib64/libselinux.so.1 0x00007f15210ba600 0x00007f15210bcdf8 Yes (*) /usr/lib64/python2.6/lib-dynload/stropmodule.so 0x00007f1520e75ec0 0x00007f1520e76938 Yes (*) /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so 0x00007f1520c71a20 0x00007f1520c72fd8 Yes (*) /usr/lib64/python2.6/lib-dynload/_localemodule.so 0x00007f1520a69eb0 0x00007f1520a6ba28 Yes (*) /usr/lib64/python2.6/lib-dynload/operator.so 0x00007f1520820140 0x00007f1520822ae8 Yes (*) /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so 0x00007f152060e750 0x00007f1520616b68 Yes (*) /usr/lib64/python2.6/lib-dynload/datetime.so 0x00007f152025a080 0x00007f152025c208 Yes (*) /usr/lib64/python2.6/lib-dynload/mathmodule.so 0x00007f15200561c0 0x00007f1520056ee8 Yes (*) /usr/lib64/python2.6/lib-dynload/_randommodule.so 0x00007f151fe45880 0x00007f151fe50658 Yes (*) /usr/lib64/python2.6/lib-dynload/cPickle.so 0x00007f151fc37bb0 0x00007f151fc3c668 Yes (*) /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so 0x00007f151fa32540 0x00007f151fa33078 Yes (*) /usr/lib64/python2.6/lib-dynload/_hashlib.so 0x00007f151f8299f0 0x00007f151f82d378 Yes (*) /usr/lib64/python2.6/lib-dynload/arraymodule.so 0x00007f151f625b70 0x00007f151f6261c8 Yes (*) /usr/lib64/python2.6/lib-dynload/_bisectmodule.so 0x00007f151f4238d0 0x00007f151f423ba8 Yes (*) /usr/lib64/python2.6/lib-dynload/_weakref.so 0x00007f151f20a8c0 0x00007f151f217698 Yes (*) /usr/lib64/python2.6/site-packages/psycopg2/_psycopg.so 0x0000003b63408230 0x0000003b6341d7e8 Yes (*) /usr/lib64/libpq.so.5 0x000000374e400c00 0x000000374e4059a8 Yes /lib64/libcrypt-2.12.so 0x00007f151efc4390 0x00007f151efef878 Yes (*) /usr/lib64/libldap_r-2.4.so.2 0x000000374ec030c0 0x000000374ec3d838 Yes (*) /lib64/libfreebl3.so 0x0000003b670033b0 0x0000003b6700a318 Yes (*) /usr/lib64/liblber-2.4.so.2 0x000000374d8046a0 0x000000374d814408 Yes (*) /usr/lib64/libsasl2.so.2 0x00007f151edb1500 0x00007f151edb1e58 Yes (*) /usr/lib64/python2.6/lib-dynload/termios.so 0x00007f151d771c20 0x00007f151d776148 Yes (*) /usr/lib64/python2.6/site-packages/simplejson/_speedups.so 0x00007f151c165580 0x00007f151c16a7f8 Yes (*) /usr/lib64/python2.6/lib-dynload/pyexpat.so 0x0000003b65803b70 0x0000003b6581ca08 Yes (*) /lib64/libexpat.so.1 0x00007fffcb1ff700 0x00007fffcb1ffaa0 Yes /lib/modules/2.6.32-71.14.1_KM7.el6.x86_64/vdso/vdso.so (*): Shared library is missing debugging information. $1 = 0x7f15100ee750 "python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.\n" No symbol "__glib_assert_msg" in current context. rax 0x0 0 rbx 0x7fffcb04787f 140736599455871 rcx 0xffffffffffffffff -1 rdx 0x6 6 rsi 0x4bf 1215 rdi 0x4a2 1186 rbp 0x3b62140fcd 0x3b62140fcd rsp 0x7f151e374738 0x7f151e374738 r8 0x7f151e378700 139728677996288 r9 0x206e6f6974726573 2336927755367769459 r10 0x8 8 r11 0x202 514 r12 0x3b65d47cc5 255111494853 r13 0x3b65d481ec 255111496172 r14 0x3b62140fcd 255048552397 r15 0x115 277 rip 0x3b620329a5 0x3b620329a5 <raise+53> eflags 0x202 [ IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 Dump of assembler code for function raise: 0x0000003b62032970 <+0>: mov %fs:0x2d4,%eax 0x0000003b62032978 <+8>: mov %fs:0x2d0,%esi 0x0000003b62032980 <+16>: test %esi,%esi 0x0000003b62032982 <+18>: jne 0x3b620329b0 <raise+64> 0x0000003b62032984 <+20>: mov $0xba,%eax 0x0000003b62032989 <+25>: syscall 0x0000003b6203298b <+27>: mov %eax,%esi 0x0000003b6203298d <+29>: mov %eax,%fs:0x2d0 0x0000003b62032995 <+37>: movslq %edi,%rdx 0x0000003b62032998 <+40>: movslq %esi,%rsi 0x0000003b6203299b <+43>: movslq %eax,%rdi 0x0000003b6203299e <+46>: mov $0xea,%eax 0x0000003b620329a3 <+51>: syscall => 0x0000003b620329a5 <+53>: cmp $0xfffffffffffff000,%rax 0x0000003b620329ab <+59>: ja 0x3b620329bf <raise+79> 0x0000003b620329ad <+61>: repz retq 0x0000003b620329af <+63>: nop 0x0000003b620329b0 <+64>: test %eax,%eax 0x0000003b620329b2 <+66>: jg 0x3b62032995 <raise+37> 0x0000003b620329b4 <+68>: test $0x7fffffff,%eax 0x0000003b620329b9 <+73>: jne 0x3b620329cf <raise+95> 0x0000003b620329bb <+75>: mov %esi,%eax 0x0000003b620329bd <+77>: jmp 0x3b62032995 <raise+37> 0x0000003b620329bf <+79>: mov 0x3465da(%rip),%rdx # 0x3b62378fa0 0x0000003b620329c6 <+86>: neg %eax 0x0000003b620329c8 <+88>: mov %eax,%fs:(%rdx) 0x0000003b620329cb <+91>: or $0xffffffffffffffff,%eax 0x0000003b620329ce <+94>: retq 0x0000003b620329cf <+95>: neg %eax 0x0000003b620329d1 <+97>: jmp 0x3b62032995 <raise+37> End of assembler dump.