This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 711095 - Python garbage collector assertion failure running pq_execute (cursorObject refcount issue)
Python garbage collector assertion failure running pq_execute (cursorObject r...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: python-psycopg2 (Show other bugs)
6.0
x86_64 Unspecified
high Severity high
: rc
: ---
Assigned To: Tom Lane
qe-baseos-daemons
abrt_hash:f8a305d69951574f208e836a452...
: ZStream
Depends On:
Blocks: 720306
  Show dependency treegraph
 
Reported: 2011-06-06 10:19 EDT by Red Hat Case Diagnostics
Modified: 2013-12-11 11:48 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, if a second thread in a single application triggered the Python garbage collection while a copy operation was in progress, the copy operation terminated unexpectedly. With this update, appropriate object reference count adjustments have been added to the code, and this bug no longer occurs.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-21 03:41:25 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
File: backtrace (34.42 KB, text/plain)
2011-06-06 10:20 EDT, Red Hat Case Diagnostics
no flags Details
Proposed patch (1.39 KB, patch)
2011-06-06 15:58 EDT, Dave Malcolm
no flags Details | Diff
Minimal reproducer (1.13 KB, text/plain)
2011-06-06 17:03 EDT, Dave Malcolm
no flags Details
Backport of the two upstream patches to the 2.0.13 in RHEL6 (5.98 KB, patch)
2011-06-07 17:08 EDT, Dave Malcolm
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Legacy) 57425 None None None Never
Red Hat Knowledge Base (Legacy) 63134 None None None Never

  None (edit)
Description Red Hat Case Diagnostics 2011-06-06 10:19:45 EDT
This description generated by Andreas from an ABRT report
architecture: x86_64
cmdline: /usr/bin/python manage.py runconcurrentserver 0.0.0.0:8000
comment: The Django process dies about a minute after starting serving pages to browser.
component: python
executable: /usr/bin/python
kernel: 2.6.32-71.14.1_KM7.el6.x86_64
package: python-2.6.5-3.el6_0.2
reason: Process /usr/bin/python was killed by signal 6 (SIGABRT)
release: Red Hat Enterprise Linux Server release 6.0 (Santiago)
reproduce: 1. 
2. 
3. 

time: 1307369154
uid: 0
backtrace: [New Thread 1215]
[New Thread 1187]
[New Thread 1192]
[New Thread 1197]
[New Thread 1194]
[New Thread 1191]
[New Thread 1186]
Core was generated by `/usr/bin/python manage.py runconcurrentserver 0.0.0.0:8000'.
Program terminated with signal 6, Aborted.
#0  0x0000003b620329a5 in raise (sig=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);

Thread 7 (Thread 1186):
#0  0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82
No locals.
#1  0x00007f1528008219 in ?? ()
   from /usr/lib64/python2.6/lib-dynload/timemodule.so
No symbol table info available.
#2  0x0000003b65cdeb01 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#3  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#4  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#5  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#6  0x0000003b65cde61e in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#7  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#8  0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#9  0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#10 0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#11 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#12 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#13 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#14 0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#15 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#16 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#17 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#18 0x0000003b65cde61e in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#19 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#20 0x0000003b65ce06a2 in PyEval_EvalCode ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#21 0x0000003b65cfb6ec in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#22 0x0000003b65cfb7c0 in PyRun_FileExFlags ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#23 0x0000003b65cfcc5c in PyRun_SimpleFileExFlags ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#24 0x0000003b65d092bd in Py_Main () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#25 0x0000003b6201ec5d in __libc_start_main (main=0x400710 <main>, argc=4, 
    ubp_av=0x7fffcb045808, init=<value optimized out>, 
    fini=<value optimized out>, rtld_fini=<value optimized out>, 
    stack_end=0x7fffcb0457f8) at libc-start.c:226
        result = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 5746914283774943270, 
                4195872, 140736599447552, 0, 0, -5746960227838320602, 
                5744306228104142886}, mask_was_saved = 0}}, priv = {pad = {
              0x0, 0x0, 0x400730, 0x7fffcb045808}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 4196144}}}
        not_first_call = <value optimized out>
#26 0x0000000000400649 in _start ()
No symbol table info available.

Thread 6 (Thread 1191):
#0  0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82
No locals.
#1  0x00007f1528008219 in ?? ()
   from /usr/lib64/python2.6/lib-dynload/timemodule.so
No symbol table info available.
#2  0x0000003b65cdeb01 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#3  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#4  0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#5  0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#6  0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#7  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#8  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#9  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#10 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#11 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#12 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#13 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#14 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#15 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#16 0x000000374f8077e1 in start_thread (arg=0x7f151d76f700)
    at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f151d76f700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728665376512, 
                5746914283774943270, 139728677952336, 139728665377216, 0, 3, 
                -5626369248591470554, 5742013758861352998}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#17 0x0000003b620e18ed in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.

Thread 5 (Thread 1194):
#0  0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82
No locals.
#1  0x00007f1528008219 in ?? ()
   from /usr/lib64/python2.6/lib-dynload/timemodule.so
No symbol table info available.
#2  0x0000003b65cdeb01 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#3  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#4  0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#5  0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#6  0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#7  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#8  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#9  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#10 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#11 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#12 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#13 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#14 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#15 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#16 0x000000374f8077e1 in start_thread (arg=0x7f150ffff700)
    at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f150ffff700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728439473920, 
                5746914283774943270, 139728677952336, 139728439474624, 0, 3, 
                -5626337440063676378, 5742013758861352998}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#17 0x0000003b620e18ed in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.

Thread 4 (Thread 1197):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
No locals.
#1  0x0000003b65d06ec8 in PyThread_acquire_lock ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#2  0x0000003b65ce0704 in PyEval_RestoreThread ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#3  0x00007f151f20d288 in ?? ()
   from /usr/lib64/python2.6/site-packages/psycopg2/_psycopg.so
No symbol table info available.
#4  0x00007f151f211d0a in ?? ()
   from /usr/lib64/python2.6/site-packages/psycopg2/_psycopg.so
No symbol table info available.
#5  0x0000003b65cdf90b in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#6  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#7  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#8  0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#9  0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#10 0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#11 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#12 0x0000003b65cde61e in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#13 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#14 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#15 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#16 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#17 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#18 0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#19 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#20 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#21 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#22 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#23 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#24 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#25 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#26 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#27 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#28 0x000000374f8077e1 in start_thread (arg=0x7f150ebfd700)
    at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f150ebfd700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728418494208, 
                5746914283774943270, 139728677952336, 139728418494912, 0, 3, 
                -5626340189916487642, 5742013758861352998}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#29 0x0000003b620e18ed in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.

Thread 3 (Thread 1192):
#0  0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82
No locals.
#1  0x00007f1528008219 in ?? ()
   from /usr/lib64/python2.6/lib-dynload/timemodule.so
No symbol table info available.
#2  0x0000003b65cdeb01 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#3  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#4  0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#5  0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#6  0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#7  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#8  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#9  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#10 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#11 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#12 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#13 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#14 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#15 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#16 0x000000374f8077e1 in start_thread (arg=0x7f151cd6e700)
    at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f151cd6e700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728654886656, 
                5746914283774943270, 139728677952336, 139728654887360, 0, 3, 
                -5626370624591618010, 5742013758861352998}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#17 0x0000003b620e18ed in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.

Thread 2 (Thread 1187):
#0  0x0000003b620da443 in select () at ../sysdeps/unix/syscall-template.S:82
No locals.
#1  0x00007f1527e03404 in ?? ()
   from /usr/lib64/python2.6/lib-dynload/selectmodule.so
No symbol table info available.
#2  0x0000003b65cdeb01 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#3  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#4  0x0000003b65cde61e in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#5  0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#6  0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#7  0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#8  0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#9  0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#10 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#11 0x000000374f8077e1 in start_thread (arg=0x7f151edae700)
    at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f151edae700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728688703232, 
                5746914283774943270, 140736599442448, 139728688703936, 0, 3, 
                -5626375125717344218, 5742013758861352998}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#12 0x0000003b620e18ed in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.

Thread 1 (Thread 1215):
#0  0x0000003b620329a5 in raise (sig=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
        resultvar = 0
        pid = <value optimized out>
        selftid = 1215
#1  0x0000003b62034185 in abort () at abort.c:92
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x3b65d47cc5, 
            sa_sigaction = 0x3b65d47cc5}, sa_mask = {__val = {255048558270, 
              139728677980016, 277, 139728677980256, 255047654286, 
              206158430232, 139728677980272, 139728677980048, 255047561544, 
              206158430256, 139728677980296, 139728440452944, 117, 117, 0, 
              140736599455871}}, sa_flags = 1645481933, 
          sa_restorer = 0x3b65d47ca0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x0000003b6202b935 in __assert_fail (
    assertion=0x3b65d47cc5 "gc->gc.gc_refs != 0", file=<value optimized out>, 
    line=277, function=<value optimized out>) at assert.c:81
        buf = 0x7f15100ee750 "python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.\n"
#3  0x0000003b65d0971e in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#4  0x0000003b65c96e5e in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#5  0x0000003b65d09e7f in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#6  0x0000003b65d0a8da in _PyObject_GC_Malloc ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#7  0x0000003b65d0a94d in _PyObject_GC_New ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#8  0x0000003b65c74f5c in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#9  0x0000003b65c44e0e in PyObject_GetIter ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#10 0x0000003b65cd9ee7 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#11 0x0000003b65c6387e in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#12 0x0000003b65cd9f4d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#13 0x0000003b65c6387e in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#14 0x0000003b65cd9f4d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#15 0x0000003b65c6387e in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#16 0x0000003b65c745eb in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#17 0x0000003b65c74870 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#18 0x0000003b65c9b318 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#19 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#20 0x0000003b65cdd8b1 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#21 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#22 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#23 0x0000003b65cde61e in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#24 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#25 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#26 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#27 0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#28 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#29 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#30 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#31 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#32 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#33 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#34 0x0000003b65c9cc44 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#35 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#36 0x0000003b65cdd8b1 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#37 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#38 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#39 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#40 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#41 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#42 0x0000003b65c9cc44 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#43 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#44 0x0000003b65cdd8b1 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#45 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#46 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#47 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#48 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#49 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#50 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#51 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#52 0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#53 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#54 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#55 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#56 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#57 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#58 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#59 0x0000003b65c58a46 in PyInstance_New ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#60 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#61 0x0000003b65cdd8b1 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#62 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#63 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#64 0x0000003b65c6eaed in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#65 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#66 0x0000003b65cdd3f2 in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#67 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#68 0x0000003b65cdf52d in PyEval_EvalFrameEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#69 0x0000003b65ce05a4 in PyEval_EvalCodeEx ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#70 0x0000003b65c6e9f0 in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#71 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#72 0x0000003b65c592ef in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#73 0x0000003b65c43e13 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#74 0x0000003b65cd8ac3 in PyEval_CallObjectWithKeywords ()
   from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#75 0x0000003b65d0b47a in ?? () from /usr/lib64/libpython2.6.so.1.0
No symbol table info available.
#76 0x000000374f8077e1 in start_thread (arg=0x7f151e378700)
    at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f151e378700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139728677996288, 
                5746914283774943270, 139728688697296, 139728677996992, 0, 3, 
                -5626376400248889306, 5742013758861352998}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#77 0x0000003b620e18ed in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
No locals.
From                To                  Syms Read   Shared Object Library
0x0000003b65c3c490  0x0000003b65d1ef08  Yes (*)     /usr/lib64/libpython2.6.so.1.0
0x000000374f805640  0x000000374f810ec8  Yes         /lib64/libpthread-2.12.so
0x0000003b61c00de0  0x0000003b61c01998  Yes         /lib64/libdl-2.12.so
0x0000003b64000e10  0x0000003b64001688  Yes         /lib64/libutil-2.12.so
0x0000003b62c03ea0  0x0000003b62c43fa8  Yes         /lib64/libm-2.12.so
0x0000003b6201e9a0  0x0000003b6212bbe0  Yes         /lib64/libc-2.12.so
0x0000003b61800af0  0x0000003b61818854  Yes         /lib64/ld-2.12.so
0x00007f152820b9d0  0x00007f152820bfe8  Yes (*)     /usr/lib64/python2.6/lib-dynload/syslog.so
0x00007f15280078d0  0x00007f1528008898  Yes (*)     /usr/lib64/python2.6/lib-dynload/timemodule.so
0x00007f1527e01be0  0x00007f1527e036a8  Yes (*)     /usr/lib64/python2.6/lib-dynload/selectmodule.so
0x00007f1527bfcd10  0x00007f1527bfda18  Yes (*)     /usr/lib64/python2.6/lib-dynload/fcntlmodule.so
0x00007f15279f5820  0x00007f15279f87a8  Yes (*)     /usr/lib64/python2.6/lib-dynload/_struct.so
0x00007f15277ae0c0  0x00007f15277aff78  Yes (*)     /usr/lib64/python2.6/lib-dynload/binascii.so
0x00007f15275a9aa0  0x00007f15275aabd8  Yes (*)     /usr/lib64/python2.6/lib-dynload/cStringIO.so
0x00007f152739c110  0x00007f15273a1898  Yes (*)     /usr/lib64/python2.6/lib-dynload/_socketmodule.so
0x00007f1527193f70  0x00007f1527196958  Yes (*)     /usr/lib64/python2.6/lib-dynload/_ssl.so
0x000000374bc14500  0x000000374bc45f68  Yes (*)     /usr/lib64/libssl.so.10
0x000000374885c900  0x0000003748920d08  Yes (*)     /usr/lib64/libcrypto.so.10
0x000000374fc07d80  0x000000374fc30af8  Yes (*)     /lib64/libgssapi_krb5.so.2
0x000000374b81a1c0  0x000000374b88cb78  Yes (*)     /lib64/libkrb5.so.3
0x00000037494013b0  0x0000003749401f88  Yes (*)     /lib64/libcom_err.so.2
0x000000374a404290  0x000000374a41bb28  Yes (*)     /lib64/libk5crypto.so.3
0x0000003b62801ef0  0x0000003b6280d1a8  Yes (*)     /lib64/libz.so.1
0x00000037498027a0  0x0000003749807898  Yes (*)     /lib64/libkrb5support.so.0
0x000000374cc00bc0  0x000000374cc011a8  Yes (*)     /lib64/libkeyutils.so.1
0x0000003b644038c0  0x0000003b64412558  Yes         /lib64/libresolv-2.12.so
0x0000003b63005610  0x0000003b63015708  Yes (*)     /lib64/libselinux.so.1
0x00007f15210ba600  0x00007f15210bcdf8  Yes (*)     /usr/lib64/python2.6/lib-dynload/stropmodule.so
0x00007f1520e75ec0  0x00007f1520e76938  Yes (*)     /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so
0x00007f1520c71a20  0x00007f1520c72fd8  Yes (*)     /usr/lib64/python2.6/lib-dynload/_localemodule.so
0x00007f1520a69eb0  0x00007f1520a6ba28  Yes (*)     /usr/lib64/python2.6/lib-dynload/operator.so
0x00007f1520820140  0x00007f1520822ae8  Yes (*)     /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so
0x00007f152060e750  0x00007f1520616b68  Yes (*)     /usr/lib64/python2.6/lib-dynload/datetime.so
0x00007f152025a080  0x00007f152025c208  Yes (*)     /usr/lib64/python2.6/lib-dynload/mathmodule.so
0x00007f15200561c0  0x00007f1520056ee8  Yes (*)     /usr/lib64/python2.6/lib-dynload/_randommodule.so
0x00007f151fe45880  0x00007f151fe50658  Yes (*)     /usr/lib64/python2.6/lib-dynload/cPickle.so
0x00007f151fc37bb0  0x00007f151fc3c668  Yes (*)     /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so
0x00007f151fa32540  0x00007f151fa33078  Yes (*)     /usr/lib64/python2.6/lib-dynload/_hashlib.so
0x00007f151f8299f0  0x00007f151f82d378  Yes (*)     /usr/lib64/python2.6/lib-dynload/arraymodule.so
0x00007f151f625b70  0x00007f151f6261c8  Yes (*)     /usr/lib64/python2.6/lib-dynload/_bisectmodule.so
0x00007f151f4238d0  0x00007f151f423ba8  Yes (*)     /usr/lib64/python2.6/lib-dynload/_weakref.so
0x00007f151f20a8c0  0x00007f151f217698  Yes (*)     /usr/lib64/python2.6/site-packages/psycopg2/_psycopg.so
0x0000003b63408230  0x0000003b6341d7e8  Yes (*)     /usr/lib64/libpq.so.5
0x000000374e400c00  0x000000374e4059a8  Yes         /lib64/libcrypt-2.12.so
0x00007f151efc4390  0x00007f151efef878  Yes (*)     /usr/lib64/libldap_r-2.4.so.2
0x000000374ec030c0  0x000000374ec3d838  Yes (*)     /lib64/libfreebl3.so
0x0000003b670033b0  0x0000003b6700a318  Yes (*)     /usr/lib64/liblber-2.4.so.2
0x000000374d8046a0  0x000000374d814408  Yes (*)     /usr/lib64/libsasl2.so.2
0x00007f151edb1500  0x00007f151edb1e58  Yes (*)     /usr/lib64/python2.6/lib-dynload/termios.so
0x00007f151d771c20  0x00007f151d776148  Yes (*)     /usr/lib64/python2.6/site-packages/simplejson/_speedups.so
0x00007f151c165580  0x00007f151c16a7f8  Yes (*)     /usr/lib64/python2.6/lib-dynload/pyexpat.so
0x0000003b65803b70  0x0000003b6581ca08  Yes (*)     /lib64/libexpat.so.1
0x00007fffcb1ff700  0x00007fffcb1ffaa0  Yes         /lib/modules/2.6.32-71.14.1_KM7.el6.x86_64/vdso/vdso.so
(*): Shared library is missing debugging information.
$1 = 0x7f15100ee750 "python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.\n"
No symbol "__glib_assert_msg" in current context.
rax            0x0	0
rbx            0x7fffcb04787f	140736599455871
rcx            0xffffffffffffffff	-1
rdx            0x6	6
rsi            0x4bf	1215
rdi            0x4a2	1186
rbp            0x3b62140fcd	0x3b62140fcd
rsp            0x7f151e374738	0x7f151e374738
r8             0x7f151e378700	139728677996288
r9             0x206e6f6974726573	2336927755367769459
r10            0x8	8
r11            0x202	514
r12            0x3b65d47cc5	255111494853
r13            0x3b65d481ec	255111496172
r14            0x3b62140fcd	255048552397
r15            0x115	277
rip            0x3b620329a5	0x3b620329a5 <raise+53>
eflags         0x202	[ IF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
Dump of assembler code for function raise:
   0x0000003b62032970 <+0>:	mov    %fs:0x2d4,%eax
   0x0000003b62032978 <+8>:	mov    %fs:0x2d0,%esi
   0x0000003b62032980 <+16>:	test   %esi,%esi
   0x0000003b62032982 <+18>:	jne    0x3b620329b0 <raise+64>
   0x0000003b62032984 <+20>:	mov    $0xba,%eax
   0x0000003b62032989 <+25>:	syscall 
   0x0000003b6203298b <+27>:	mov    %eax,%esi
   0x0000003b6203298d <+29>:	mov    %eax,%fs:0x2d0
   0x0000003b62032995 <+37>:	movslq %edi,%rdx
   0x0000003b62032998 <+40>:	movslq %esi,%rsi
   0x0000003b6203299b <+43>:	movslq %eax,%rdi
   0x0000003b6203299e <+46>:	mov    $0xea,%eax
   0x0000003b620329a3 <+51>:	syscall 
=> 0x0000003b620329a5 <+53>:	cmp    $0xfffffffffffff000,%rax
   0x0000003b620329ab <+59>:	ja     0x3b620329bf <raise+79>
   0x0000003b620329ad <+61>:	repz retq 
   0x0000003b620329af <+63>:	nop
   0x0000003b620329b0 <+64>:	test   %eax,%eax
   0x0000003b620329b2 <+66>:	jg     0x3b62032995 <raise+37>
   0x0000003b620329b4 <+68>:	test   $0x7fffffff,%eax
   0x0000003b620329b9 <+73>:	jne    0x3b620329cf <raise+95>
   0x0000003b620329bb <+75>:	mov    %esi,%eax
   0x0000003b620329bd <+77>:	jmp    0x3b62032995 <raise+37>
   0x0000003b620329bf <+79>:	mov    0x3465da(%rip),%rdx        # 0x3b62378fa0
   0x0000003b620329c6 <+86>:	neg    %eax
   0x0000003b620329c8 <+88>:	mov    %eax,%fs:(%rdx)
   0x0000003b620329cb <+91>:	or     $0xffffffffffffffff,%eax
   0x0000003b620329ce <+94>:	retq   
   0x0000003b620329cf <+95>:	neg    %eax
   0x0000003b620329d1 <+97>:	jmp    0x3b62032995 <raise+37>
End of assembler dump.
Comment 2 Red Hat Case Diagnostics 2011-06-06 10:20:21 EDT
Created attachment 503247 [details]
File: backtrace
Comment 8 Dave Malcolm 2011-06-06 15:40:32 EDT
Thanks for filing this bug report.  How often is this problem occurring?

I believe the crash is happening due to a bug in python-psycopg2.  I am about to attach a proposed patch to that package which I believe will fix it.  I don't have a minimal reproducer yet.

Technical analysis follows:

Based on reviewing the coredump, the crash is happening in thread #1, during one of python's periodic invocations of its garbage collector.  The assertion failure is that an object has a lower reference count than expected (based on the reference-owning objects pointing to it).

This object is a StringIO object, containing a series of lines containing dates and numbers (I'm deliberately not including it in a public bz comment, in case it contains private information).

Meanwhile thread #4 is within python-psycopg2's pq_execute function executing a query. Specifically, it is within the function "psyco_curs_copy_from" in psycopg2-2.0.13/psycopg/cursor_type.c

I notice that within thread #4, pcurs->copyfile contains the textual data seen in the crash in thread #1.  What's happened is that the StringIO instance containing the query has too low a reference count.

Further, pyscopg2's cursorObject participates in the Python garbage collector:  psycopg2-2.0.13/psycopg/cursor_type.c: cursor_traverse has:
    Py_VISIT(self->copyfile);

It acquires PyObject *file via:

    if (!PyArg_ParseTupleAndKeywords(args, kwargs,
        "O&s|ss" CONV_CODE_PY_SSIZE_T "O", kwlist,
        _psyco_curs_has_read_check, &file, &table_name, &sep, &null, &bufsize,
	&columns))

i.e. from "O& (object) [converter, anything]" with converter "_psyco_curs_has_read_check":

Looking at _psyco_curs_has_read_check, it has this code and comment:
        /* It's OK to store a borrowed reference, because it is only held for 
         * the duration of psyco_curs_copy_from. */
        *((PyObject**)var) = o;
        return 1;

This is thus storing a borrowed reference to the StringIO object into the psyco's cursorObject without INCREFing it.  This reference is is visible to the garbage collector.

The comment in the code above is incorrect: within the call to pq_execute, there is an invocation of the Py_BEGIN_ALLOW_THREADS macro.  This allows the GIL to transition to another thread, and it seems that this is what happened in this case  (within the coredump, that thread is waiting at the subsequent Py_END_ALLOW_THREADS macro, waiting to reacquire the GIL).

When the GC runs in the other thread, it detects that the refcount is too low, and bails out with an assertion failure.  (I believe that if assertions were disabled, it could try to free up the memory, so we'd be seeing a different kind of crash)

I believe the fix will be to add INCREF/DECREF pairs around the call to pq_execute within psyco_curs_copy_from, and I'll try to write a patch that does so.  I don't have a minimal reproducer yet.

I checked that latest upstream release of the code (2.4.1), and I believe this bug is still present there.
Comment 11 Dave Malcolm 2011-06-06 15:58:46 EDT
Created attachment 503316 [details]
Proposed patch
Comment 13 Tom Lane 2011-06-06 16:22:40 EDT
I'm not really qualified to evaluate this.  I'll see if I can get a comment from upstream.
Comment 14 Dave Malcolm 2011-06-06 17:03:24 EDT
Created attachment 503327 [details]
Minimal reproducer

With the attached script, I'm reliably able to reproduce the error:
  python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.

on my test box (with the same versions as in the report).
Comment 15 Dave Malcolm 2011-06-06 17:05:38 EDT
(In reply to comment #14)
> Created attachment 503327 [details]

> With the attached script, I'm reliably able to reproduce the error:

Having said that, getting it to happen may be somewhat timing dependent (this is a bad interaction between two threads).  I had to tweak the delay in the loop in the GCThread (currently 0.1) before I could get it to occur, this was with the postgres server on the same host as the script, so YMMV.
Comment 16 Dave Malcolm 2011-06-06 17:17:33 EDT
The patch attached in comment #11 fixes the crash seen by running comment #14, so I believe we have a fix for this issue.
Comment 17 Dave Malcolm 2011-06-06 17:26:06 EDT
(In reply to comment #14)
> Created attachment 503327 [details]
> Minimal reproducer
> 
> With the attached script, I'm reliably able to reproduce the error:
>   python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0'
> failed.
> 
> on my test box (with the same versions as in the report).

I've also now stepped through the crash in gdb, and this reproducer is indeed crashing python in the same way as in the original coredump.

  (gdb) t a a py-bt
  
  Thread 2 (Thread 0x7fffefd1a700 (LWP 9729)):
  #9 file 'test.py', in 'run'
  #12 file '/usr/lib64/python2.6/threading.py', in '__bootstrap_inner'
  #15 file '/usr/lib64/python2.6/threading.py', in '__bootstrap'

  Thread 1 (Thread 0x7ffff7fed700 (LWP 9727)):
  #7 file 'test.py', in '<module>'

Thread 2 is garbage-collecting (and crashing with that assertion failure)

Thread 1 is waiting to reacquire the GIL, within pq_execute, within psyco_curs_copy_from.
Comment 21 Dave Malcolm 2011-06-06 19:04:40 EDT
For reference, the upstream bug Tom opened appears to be here:
http://psycopg.lighthouseapp.com/projects/62710/tickets/58
Comment 22 Dave Malcolm 2011-06-06 19:06:53 EDT
Note to psycopg2 upstream folks: to reproduce the C level assertion failure in gcmodule.c, you need a build of Python with assertions enabled.  Typically for an upstream build of python, this means configuring --with-pydebug.
Comment 23 Daniele Varrazzo 2011-06-06 19:30:51 EDT
Thank you for the report and the patch. I have verified the bug and the fix.

Note that the functions copy_to and copy_expert were affected by the same bug: I have fixed them as well.

The devel branch of my repos (https://github.com/dvarrazzo/psycopg) contains the patches. They will be released in the upcoming release 2.4.2.
Comment 24 Dave Malcolm 2011-06-06 19:52:19 EDT
Daniele:  Thanks for the info.  Your upstream fix appears to be:
  https://github.com/dvarrazzo/psycopg/commit/e9a485e30bf74c598536417819f196476e0dc406

Looking there, I think that commit may introduce a bug within psyco_curs_copy_expert, within the error handling paths.

Specifically, the "file" variable is set using PyArg_ParseTupleAndKeywords with code "O", which gives a borrowed reference, not modifying the refcount.

If an error occurs, and one of the first two "goto exit;" clauses is run, then my reading of that code is that you have a Py_XDECREF(file), without a matching Py_INCREF(file), so "file" loses a reference, which may lead to it being prematirely deallocated.

Am I reading that correctly?  I think the
   Py_INCREF(file);
within in that function needs to be moved higher up, to before the first "goto exit" e.g. to directly below this comment:
   /* Any failure from here forward should 'goto exit' rather than
       'return NULL' directly. */

The changes to the other two functions (copy_from/copy_to) look correct to me.

[FWIW, I'm working on a static analysis tool to try to make it easier to deal with this kind of thing in the future, see:
https://fedoraproject.org/wiki/Features/StaticAnalysisOfCPythonExtensions
But that code isn't ready yet]
Comment 25 Daniele Varrazzo 2011-06-06 20:32:22 EDT
(In reply to comment #24)

> Looking there, I think that commit may introduce a bug within
> psyco_curs_copy_expert, within the error handling paths.

Uhm... right. I'd implemented a pattern typical of an object owned by the function, didn't spot that file was coming from the parsed args.

I've modified copy_expert to make it more similar to copy_from/to.

https://github.com/dvarrazzo/psycopg/commit/816b5dda33a8914532da160bded65a6f3afb9c9b

Thank you again!


> [FWIW, I'm working on a static analysis tool to try to make it easier to deal
> with this kind of thing in the future

I would *love* to use such a tool!
Comment 26 Dave Malcolm 2011-06-06 20:46:05 EDT
(In reply to comment #25)
> (In reply to comment #24)
> 
> > Looking there, I think that commit may introduce a bug within
> > psyco_curs_copy_expert, within the error handling paths.
> 
> Uhm... right. I'd implemented a pattern typical of an object owned by the
> function, didn't spot that file was coming from the parsed args.
> 
> I've modified copy_expert to make it more similar to copy_from/to.
> 
> https://github.com/dvarrazzo/psycopg/commit/816b5dda33a8914532da160bded65a6f3afb9c9b

FWIW, I doublechecked that new implementation of psyco_curs_copy_expert, and it looks correct to me also.

Thanks!
Comment 27 Maor 2011-06-07 04:27:42 EDT
Question:
Do you have an idea why I see this error only on with the RHEL6 installation of Python?
We've tried it on our own build of Python v2.6.5 + Psycopg2 v.2.0.13 (same versions as what comes with RHEL6), on the same RHEL6 machine, and this error does not happen.
(It also didn't happen on CentOS 5.5)

Any ideas?
Comment 29 Dave Malcolm 2011-06-07 07:45:47 EDT
(In reply to comment #27)
These assertions are typically disabled when building python, but, for better or worse, the build flags we use mean that they are enabled in RHEL 6's python build.

There was some discussion within bug 614680 about changing this in RHEL 6.1. As noted in that bug, doing so could well turn errors of this kind into segfaults rather than asserts, hence in that bug I chose to keep the extra error-checking they provide, whilst improving the error messages (in python-2.6.6-5.el6) so that they indicate which object has reference-count issues.

Whether or not the assertion is protecting against a segfault varies from case to case.

My (tentative) reading of this case is that (for this specific case) the assertion is overzealous.  The StringIO instance has an ob_refcnt of 2: presumably 2 frame's worth of dictionaries of local variables (the code is too optimized to directly verify this).  The third reference is the one within the cursor_object within psyco_curs_copy_from.  If those other two references were garbage-collected away by the other thread, then the cursorObject could have a borrowed reference to the StringIO, but the StringIO could have been deallocated from under it.  However, I don't believe this would have been a problem, for two reasons:
- assuming that the other two referrers are frame locals() dictionaries, then these will outlive the call to psyco_curs_copy_from
- upon reacquisition of the GIL, psyco_curs_copy_from makes no further use of the value, immediately setting it back to NULL.

Having said that, the benign nature here is merely due to the specifics of the user code and of psycopg2 in this case: invariants are being violated, but it happens to not matter.  There are plenty of other situations in which similar bugs could lead to crashes of /usr/bin/python, and such crashes would be significantly less predictable and more obscure. 

There is perhaps an argument for reassessing bug 614680 (e.g. turning off the asserts, or making this a command-line flag or somesuch), but for now, the fix is to patch python-psycopg2 (using the patches referred to in comment #24 and comment #25).

Hopefully the above makes sense.
Comment 32 Tom Lane 2011-06-07 10:07:21 EDT
(In reply to comment #29)
> There is perhaps an argument for reassessing bug 614680 (e.g. turning off the
> asserts, or making this a command-line flag or somesuch), but for now, the fix
> is to patch python-psycopg2 (using the patches referred to in comment #24 and
> comment #25).

I quite agree that turning off the assertions would be an inappropriate "fix".
Comment 34 Dave Malcolm 2011-06-07 17:08:33 EDT
Created attachment 503578 [details]
Backport of the two upstream patches to the 2.0.13 in RHEL6


Candidate patch: the attached patch is a combination of:
  https://github.com/dvarrazzo/psycopg/commit/e9a485e30bf74c598536417819f196476e0dc406
followed by:
  https://github.com/dvarrazzo/psycopg/commit/679af4a97528c140ac8e9cd85598f717b3f5cfb9
fixed up to apply cleanly to the 2.0.13 within RHEL.

[It does not contain the change to tests/test_copy.py from 679af4a97528c140ac8e9cd85598f717b3f5cfb9 as that entire file only appeared after 2.0.13]

As a smoketest, I have rebuilt RHEL6's python-psycopg2-2.0.13-2.el6 adding this patch, and tested with the psycopg2-2.0.13/scripts/ticket58.py (see the prepped source tree).

Each of the three test cases within that script ("copy_from", "copy_to", and "copy_expert") fail with this error:
  python: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
  Aborted (core dumped)
when run against the old package (with a suitably-configured postgres DB, see the comment in that file).

All of them successfully run to completion with the patched version of the package.

I have merely smoketested the proposed test cases.  I have not tested other functionality of the package.
Comment 45 Tom Lane 2011-06-18 17:31:20 EDT
BTW, I've pushed psycopg 2.4.2 into Fedora rawhide, so this is fixed over there.
Comment 46 RHEL Product and Program Management 2011-07-05 19:51:08 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 54 Tomas Capek 2011-07-14 06:31:10 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, if a second thread in a single application triggered the Python garbage collection while a copy operation was in progress, the copy operation terminated unexpectedly. With this update, appropriate object reference count adjustments have been added to the code, and this bug no longer occurs.
Comment 60 errata-xmlrpc 2013-02-21 03:41:25 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0327.html

Note You need to log in before you can comment on or make changes to this bug.