RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 854692 - python rarely aborts on exit with 'Fatal Python error: GC object already tracked'
Summary: python rarely aborts on exit with 'Fatal Python error: GC object already trac...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: python
Version: 6.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Dave Malcolm
QA Contact: BaseOS QE - Apps
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-09-05 14:59 UTC by Frantisek Reznicek
Modified: 2015-11-16 01:14 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-10-11 16:06:20 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Frantisek Reznicek 2012-09-05 14:59:28 UTC 
Description of problem:

Python rarely aborts on exit with 'Fatal Python error: GC object already tracked'.

Running sync test involving MRG/M broker and python client (python-qpid) leads sometimes in python abort during shutdown. Details:

  #0  0x00007feb35a188a5 in raise () from /lib64/libc.so.6
  #1  0x00007feb35a1a085 in abort () from /lib64/libc.so.6
  #2  0x00007feb3671b24e in Py_FatalError (msg=<value optimized out>) at Python/pythonrun.c:1682
  #3  0x00007feb366b7cbe in PyTuple_New (size=<value optimized out>) at Objects/tupleobject.c:82
  #4  0x00007feb366fea89 in load_args (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:3964
  #5  call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:3792
  #6  PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2453
  #7  0x00007feb366ffccf in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:3880
  #8  call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:3815
  ...
  #30 0x00007feb3671bc90 in PyRun_FileExFlags (fp=0xa77180, filename=0x7ffffe0c15e3 "./sync3.py", start=<value optimized out>, ... at Python/pythonrun.c:1344
  #31 0x00007feb3671d17c in PyRun_SimpleFileExFlags (fp=0xa77180, filename=0x7ffffe0c15e3 "./sync3.py", closeit=1, flags=0x7ffffe0bf350)
      at Python/pythonrun.c:948
  #32 0x00007feb36729c32 in Py_Main (argc=<value optimized out>, argv=<value optimized out>) at Modules/main.c:618
  #33 0x00007feb35a04cdd in __libc_start_main () from /lib64/libc.so.6
  #34 0x0000000000400649 in _start ()

This situation was seen twice on RHEL6.3 x86_64 VM machine.


Version-Release number of selected component (if applicable):
python-2.6.6-29.el6_2.2.x86_64
python-qpid-0.14-10.el6_3.noarch
python-qpid-qmf-0.14-13.el6_3.x86_64
qpid-cpp-client-0.14-20.el6_3.x86_64
qpid-cpp-client-devel-0.14-20.el6_3.x86_64
qpid-cpp-client-devel-docs-0.14-20.el6_3.noarch
qpid-cpp-client-rdma-0.14-20.el6_3.x86_64
qpid-cpp-client-ssl-0.14-20.el6_3.x86_64
qpid-cpp-debuginfo-0.14-20.el6_3.x86_64
qpid-cpp-server-0.14-20.el6_3.x86_64
qpid-cpp-server-cluster-0.14-20.el6_3.x86_64
qpid-cpp-server-devel-0.14-20.el6_3.x86_64
qpid-cpp-server-rdma-0.14-20.el6_3.x86_64
qpid-cpp-server-ssl-0.14-20.el6_3.x86_64
qpid-cpp-server-store-0.14-20.el6_3.x86_64
qpid-cpp-server-xml-0.14-20.el6_3.x86_64


How reproducible:
very difficult

Steps to Reproduce:
0. # four machines needed eventually adjust the test
1. service qpidd start  on host A
2. ./sync3.py A A B C D on every machine (A, B, C, D)
3. echo $?
  
Actual results:
Python aborts during shutdown.

Expected results:
Python should not abort during shutdown.

Additional info:
Comment 3 Dave Malcolm 2012-09-05 15:53:57 UTC 
Do you have a more minimal reproducer?

What is the frequency with which this behavior occurs?

What shared libraries are loaded into the python process when it happens? (see /proc/PID/maps)

Does the behavior happen if you run the reproducer under valgrind?
Comment 4 Dave Malcolm 2012-09-05 15:56:17 UTC 
Looking at transcript_dump.log within attachment 610083 [details], thread 1 is showing the Py_FatalError; thread 2 is waiting on a select.

Thread 1 is building a tuple to contain the arguments for a function call.

tupleobject.c:82 is
    _PyObject_GC_TRACK(op);

which is this macro defined in objimpl.h:

  /* Tell the GC to track this object.  NB: While the object is tracked the
   * collector it must be safe to call the ob_traverse method. */
  #define _PyObject_GC_TRACK(o) do { \
      PyGC_Head *g = _Py_AS_GC(o); \
      if (g->gc.gc_refs != _PyGC_REFS_UNTRACKED) \
          Py_FatalError("GC object already tracked"); \
      g->gc.gc_refs = _PyGC_REFS_REACHABLE; \
      g->gc.gc_next = _PyGC_generation0; \
      g->gc.gc_prev = _PyGC_generation0->gc.gc_prev; \
      g->gc.gc_prev->gc.gc_next = g; \
      _PyGC_generation0->gc.gc_prev = g; \
      } while (0);

so clearly we're hitting this line:
   Py_FatalError("GC object already tracked");

Note that:
  #define _PyGC_REFS_UNTRACKED                    (-2)

The tuple in question either came (a) from the free list, or (b) was freshly allocated...

For (a): the tuple storage space was added to the free list by tupledealloc, which calls PyObject_GC_UnTrack(op) before cleaning up the object; this is defined in Modules/gcmodule.c as :

  void
  PyObject_GC_UnTrack(void *op)
  {
      /* Obscure:  the Py_TRASHCAN mechanism requires that we be able to
       * call PyObject_GC_UnTrack twice on an object.
       */
      if (IS_TRACKED(op))
          _PyObject_GC_UNTRACK(op);
  }

These macros are:
  #define IS_TRACKED(o) ((AS_GC(o))->gc.gc_refs != GC_UNTRACKED)
(from gcmodule.c)

and:
  #define _PyObject_GC_UNTRACK(o) do { \
      PyGC_Head *g = _Py_AS_GC(o); \
      assert(g->gc.gc_refs != _PyGC_REFS_UNTRACKED); \
      g->gc.gc_refs = _PyGC_REFS_UNTRACKED; \
      g->gc.gc_prev->gc.gc_next = g->gc.gc_next; \
      g->gc.gc_next->gc.gc_prev = g->gc.gc_prev; \
      g->gc.gc_next = NULL; \
      } while (0);
from objimpl.h

For (b): the tuple was allocated by PyObject_GC_NewVar(), a macro (in objimpl.h) to _PyObject_GC_NewVar, defined in Modules/gcmodule.c, which calls _PyObject_GC_Malloc, which sets g->gc.gc_refs = GC_UNTRACKED, a macro to _PyGC_REFS_UNTRACKED.

If this happened via (a), then this suggests to me that a tuple object got deleted (due to ob_refcnt reaching 0), but something still had a pointer to it and it got back into a garbage collector generation

If this happened via (b), then this suggest to me some kind of memory corruption: a random pointer splat bug touching the memory, perhaps?
Comment 5 Frantisek Reznicek 2012-09-06 13:08:53 UTC 
Hello David,
thanks for very good questions, see my comments inlined...

(In reply to comment #3)
> Do you have a more minimal reproducer?
> 

No, I don't have minimal reproducer as it appeared just twice and since that I was not able to trigger.

> What is the frequency with which this behavior occurs?
> 

Reproducibility is very low, happened twice of ~50 runs. 

> What shared libraries are loaded into the python process when it happens?
> (see /proc/PID/maps)

[root@dhcp-27-147 ~]# cat /proc/1675/maps ; pmap 1675
00400000-00401000 r-xp 00000000 08:01 6153                               /usr/bin/python
00600000-00601000 rw-p 00000000 08:01 6153                               /usr/bin/python
02562000-02eca000 rw-p 00000000 00:00 0                                  [heap]
7f8454000000-7f8454426000 rw-p 00000000 00:00 0
7f8454426000-7f8458000000 ---p 00000000 00:00 0
7f845a569000-7f845a56d000 r-xp 00000000 08:01 132110                     /usr/lib64/sasl2/libanonymous.so.2.0.23
7f845a56d000-7f845a76c000 ---p 00004000 08:01 132110                     /usr/lib64/sasl2/libanonymous.so.2.0.23
7f845a76c000-7f845a76d000 r--p 00003000 08:01 132110                     /usr/lib64/sasl2/libanonymous.so.2.0.23
7f845a76d000-7f845a76e000 rw-p 00004000 08:01 132110                     /usr/lib64/sasl2/libanonymous.so.2.0.23
7f845a76e000-7f845a772000 r-xp 00000000 08:01 140095                     /usr/lib64/sasl2/libcrammd5.so.2.0.23
7f845a772000-7f845a972000 ---p 00004000 08:01 140095                     /usr/lib64/sasl2/libcrammd5.so.2.0.23
7f845a972000-7f845a973000 r--p 00004000 08:01 140095                     /usr/lib64/sasl2/libcrammd5.so.2.0.23
7f845a973000-7f845a974000 rw-p 00005000 08:01 140095                     /usr/lib64/sasl2/libcrammd5.so.2.0.23
7f845a974000-7f845aae3000 r-xp 00000000 08:01 261739                     /lib64/libdb-4.7.so
7f845aae3000-7f845ace2000 ---p 0016f000 08:01 261739                     /lib64/libdb-4.7.so
7f845ace2000-7f845ace8000 rw-p 0016e000 08:01 261739                     /lib64/libdb-4.7.so
7f845ace8000-7f845aced000 r-xp 00000000 08:01 132113                     /usr/lib64/sasl2/libsasldb.so.2.0.23
7f845aced000-7f845aeec000 ---p 00005000 08:01 132113                     /usr/lib64/sasl2/libsasldb.so.2.0.23
7f845aeec000-7f845aeed000 r--p 00004000 08:01 132113                     /usr/lib64/sasl2/libsasldb.so.2.0.23
7f845aeed000-7f845aeee000 rw-p 00005000 08:01 132113                     /usr/lib64/sasl2/libsasldb.so.2.0.23
7f845aeee000-7f845aefa000 r-xp 00000000 08:01 140098                     /usr/lib64/sasl2/libdigestmd5.so.2.0.23
7f845aefa000-7f845b0f9000 ---p 0000c000 08:01 140098                     /usr/lib64/sasl2/libdigestmd5.so.2.0.23
7f845b0f9000-7f845b0fa000 r--p 0000b000 08:01 140098                     /usr/lib64/sasl2/libdigestmd5.so.2.0.23
7f845b0fa000-7f845b0fb000 rw-p 0000c000 08:01 140098                     /usr/lib64/sasl2/libdigestmd5.so.2.0.23
7f845b0fb000-7f845b0fc000 ---p 00000000 00:00 0
7f845b0fc000-7f845bafc000 rw-p 00000000 00:00 0
7f845bafc000-7f845bb01000 r-xp 00000000 08:01 261685                     /lib64/libnss_dns-2.12.so
7f845bb01000-7f845bd00000 ---p 00005000 08:01 261685                     /lib64/libnss_dns-2.12.so
7f845bd00000-7f845bd01000 r--p 00004000 08:01 261685                     /lib64/libnss_dns-2.12.so
7f845bd01000-7f845bd02000 rw-p 00005000 08:01 261685                     /lib64/libnss_dns-2.12.so
7f845bd02000-7f845bd0e000 r-xp 00000000 08:01 261687                     /lib64/libnss_files-2.12.so
7f845bd0e000-7f845bf0e000 ---p 0000c000 08:01 261687                     /lib64/libnss_files-2.12.so
7f845bf0e000-7f845bf0f000 r--p 0000c000 08:01 261687                     /lib64/libnss_files-2.12.so
7f845bf0f000-7f845bf10000 rw-p 0000d000 08:01 261687                     /lib64/libnss_files-2.12.so
7f845bf10000-7f845bf18000 r-xp 00000000 08:01 16199                      /usr/lib64/python2.6/lib-dynload/arraymodule.so
7f845bf18000-7f845c117000 ---p 00008000 08:01 16199                      /usr/lib64/python2.6/lib-dynload/arraymodule.so
7f845c117000-7f845c11a000 rw-p 00007000 08:01 16199                      /usr/lib64/python2.6/lib-dynload/arraymodule.so
7f845c11a000-7f845c11d000 r-xp 00000000 08:01 7305                       /usr/lib64/python2.6/lib-dynload/_hashlib.so
7f845c11d000-7f845c31c000 ---p 00003000 08:01 7305                       /usr/lib64/python2.6/lib-dynload/_hashlib.so
7f845c31c000-7f845c31e000 rw-p 00002000 08:01 7305                       /usr/lib64/python2.6/lib-dynload/_hashlib.so
7f845c31e000-7f845c322000 r-xp 00000000 08:01 5537                       /usr/lib64/python2.6/lib-dynload/selectmodule.so
7f845c322000-7f845c522000 ---p 00004000 08:01 5537                       /usr/lib64/python2.6/lib-dynload/selectmodule.so
7f845c522000-7f845c524000 rw-p 00004000 08:01 5537                       /usr/lib64/python2.6/lib-dynload/selectmodule.so
7f845c524000-7f845c581000 r-xp 00000000 08:01 261664                     /lib64/libfreebl3.so
7f845c581000-7f845c780000 ---p 0005d000 08:01 261664                     /lib64/libfreebl3.so
7f845c780000-7f845c781000 r--p 0005c000 08:01 261664                     /lib64/libfreebl3.so
7f845c781000-7f845c782000 rw-p 0005d000 08:01 261664                     /lib64/libfreebl3.so
7f845c782000-7f845c786000 rw-p 00000000 00:00 0
7f845c786000-7f845c78d000 r-xp 00000000 08:01 261675                     /lib64/libcrypt-2.12.so
7f845c78d000-7f845c98d000 ---p 00007000 08:01 261675                     /lib64/libcrypt-2.12.so
7f845c98d000-7f845c98e000 r--p 00007000 08:01 261675                     /lib64/libcrypt-2.12.so
7f845c98e000-7f845c98f000 rw-p 00008000 08:01 261675                     /lib64/libcrypt-2.12.so
7f845c98f000-7f845c9bd000 rw-p 00000000 00:00 0
7f845c9bd000-7f845c9d3000 r-xp 00000000 08:01 261640                     /lib64/libgcc_s-4.4.6-20120305.so.1
7f845c9d3000-7f845cbd2000 ---p 00016000 08:01 261640                     /lib64/libgcc_s-4.4.6-20120305.so.1
7f845cbd2000-7f845cbd3000 rw-p 00015000 08:01 261640                     /lib64/libgcc_s-4.4.6-20120305.so.1
7f845cbd3000-7f845ccbb000 r-xp 00000000 08:01 3548                       /usr/lib64/libstdc++.so.6.0.13
7f845ccbb000-7f845cebb000 ---p 000e8000 08:01 3548                       /usr/lib64/libstdc++.so.6.0.13
7f845cebb000-7f845cec2000 r--p 000e8000 08:01 3548                       /usr/lib64/libstdc++.so.6.0.13
7f845cec2000-7f845cec4000 rw-p 000ef000 08:01 3548                       /usr/lib64/libstdc++.so.6.0.13
7f845cec4000-7f845ced9000 rw-p 00000000 00:00 0
7f845ced9000-7f845cef2000 r-xp 00000000 08:01 3591                       /usr/lib64/libsasl2.so.2.0.23
7f845cef2000-7f845d0f1000 ---p 00019000 08:01 3591                       /usr/lib64/libsasl2.so.2.0.23
7f845d0f1000-7f845d0f2000 r--p 00018000 08:01 3591                       /usr/lib64/libsasl2.so.2.0.23
7f845d0f2000-7f845d0f3000 rw-p 00019000 08:01 3591                       /usr/lib64/libsasl2.so.2.0.23
7f845d0f3000-7f845d0f9000 r-xp 00000000 08:01 17626                      /usr/lib64/libsaslwrapper.so.1.0.0
7f845d0f9000-7f845d2f8000 ---p 00006000 08:01 17626                      /usr/lib64/libsaslwrapper.so.1.0.0
7f845d2f8000-7f845d2f9000 rw-p 00005000 08:01 17626                      /usr/lib64/libsaslwrapper.so.1.0.0
7f845d2f9000-7f845d305000 r-xp 00000000 08:01 276171                     /usr/lib64/python2.6/site-packages/_saslwrapper.so
7f845d305000-7f845d505000 ---p 0000c000 08:01 276171                     /usr/lib64/python2.6/site-packages/_saslwrapper.so
7f845d505000-7f845d507000 rw-p 0000c000 08:01 276171                     /usr/lib64/python2.6/site-packages/_saslwrapper.so
7f845d507000-7f845d50a000 r-xp 00000000 08:01 10196                      /usr/lib64/python2.6/lib-dynload/_heapq.so
7f845d50a000-7f845d70a000 ---p 00003000 08:01 10196                      /usr/lib64/python2.6/lib-dynload/_heapq.so
7f845d70a000-7f845d70c000 rw-p 00003000 08:01 10196                      /usr/lib64/python2.6/lib-dynload/_heapq.so
7f845d70c000-7f845d70e000 r-xp 00000000 08:01 5479                       /usr/lib64/python2.6/lib-dynload/_bisectmodule.so
7f845d70e000-7f845d90d000 ---p 00002000 08:01 5479                       /usr/lib64/python2.6/lib-dynload/_bisectmodule.so
7f845d90d000-7f845d90e000 rw-p 00001000 08:01 5479                       /usr/lib64/python2.6/lib-dynload/_bisectmodule.so
7f845d90e000-7f845d918000 r-xp 00000000 08:01 5526                       /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so
7f845d918000-7f845db17000 ---p 0000a000 08:01 5526                       /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so
7f845db17000-7f845db1b000 rw-p 00009000 08:01 5526                       /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so
7f845db1b000-7f845db38000 r-xp 00000000 08:01 261746                     /lib64/libselinux.so.1
7f845db38000-7f845dd37000 ---p 0001d000 08:01 261746                     /lib64/libselinux.so.1
7f845dd37000-7f845dd38000 r--p 0001c000 08:01 261746                     /lib64/libselinux.so.1
7f845dd38000-7f845dd39000 rw-p 0001d000 08:01 261746                     /lib64/libselinux.so.1
7f845dd39000-7f845dd3a000 rw-p 00000000 00:00 0
7f845dd3a000-7f845dd50000 r-xp 00000000 08:01 261697                     /lib64/libresolv-2.12.so
7f845dd50000-7f845df50000 ---p 00016000 08:01 261697                     /lib64/libresolv-2.12.so
7f845df50000-7f845df51000 r--p 00016000 08:01 261697                     /lib64/libresolv-2.12.so
7f845df51000-7f845df52000 rw-p 00017000 08:01 261697                     /lib64/libresolv-2.12.so
7f845df52000-7f845df54000 rw-p 00000000 00:00 0
7f845df54000-7f845df56000 r-xp 00000000 08:01 262093                     /lib64/libkeyutils.so.1.3
7f845df56000-7f845e155000 ---p 00002000 08:01 262093                     /lib64/libkeyutils.so.1.3
7f845e155000-7f845e156000 r--p 00001000 08:01 262093                     /lib64/libkeyutils.so.1.3
7f845e156000-7f845e157000 rw-p 00002000 08:01 262093                     /lib64/libkeyutils.so.1.3
7f845e157000-7f845e161000 r-xp 00000000 08:01 262102                     /lib64/libkrb5support.so.0.1
7f845e161000-7f845e360000 ---p 0000a000 08:01 262102                     /lib64/libkrb5support.so.0.1
7f845e360000-7f845e361000 r--p 00009000 08:01 262102                     /lib64/libkrb5support.so.0.1
7f845e361000-7f845e362000 rw-p 0000a000 08:01 262102                     /lib64/libkrb5support.so.0.1
7f845e362000-7f845e377000 r-xp 00000000 08:01 261726                     /lib64/libz.so.1.2.3
7f845e377000-7f845e576000 ---p 00015000 08:01 261726                     /lib64/libz.so.1.2.3
7f845e576000-7f845e577000 r--p 00014000 08:01 261726                     /lib64/libz.so.1.2.3
7f845e577000-7f845e578000 rw-p 00015000 08:01 261726                     /lib64/libz.so.1.2.3
7f845e578000-7f845e5a2000 r-xp 00000000 08:01 262098                     /lib64/libk5crypto.so.3.1
7f845e5a2000-7f845e7a1000 ---p 0002a000 08:01 262098                     /lib64/libk5crypto.so.3.1
7f845e7a1000-7f845e7a3000 r--p 00029000 08:01 262098                     /lib64/libk5crypto.so.3.1
7f845e7a3000-7f845e7a4000 rw-p 0002b000 08:01 262098                     /lib64/libk5crypto.so.3.1
7f845e7a4000-7f845e7a7000 r-xp 00000000 08:01 261738                     /lib64/libcom_err.so.2.1
7f845e7a7000-7f845e9a6000 ---p 00003000 08:01 261738                     /lib64/libcom_err.so.2.1
7f845e9a6000-7f845e9a7000 r--p 00002000 08:01 261738                     /lib64/libcom_err.so.2.1
7f845e9a7000-7f845e9a8000 rw-p 00003000 08:01 261738                     /lib64/libcom_err.so.2.1
7f845e9a8000-7f845ea7c000 r-xp 00000000 08:01 262100                     /lib64/libkrb5.so.3.3
7f845ea7c000-7f845ec7c000 ---p 000d4000 08:01 262100                     /lib64/libkrb5.so.3.3
7f845ec7c000-7f845ec85000 r--p 000d4000 08:01 262100                     /lib64/libkrb5.so.3.3
7f845ec85000-7f845ec87000 rw-p 000dd000 08:01 262100                     /lib64/libkrb5.so.3.3
7f845ec87000-7f845ecc6000 r-xp 00000000 08:01 262094                     /lib64/libgssapi_krb5.so.2.2
7f845ecc6000-7f845eec6000 ---p 0003f000 08:01 262094                     /lib64/libgssapi_krb5.so.2.2
7f845eec6000-7f845eec7000 r--p 0003f000 08:01 262094                     /lib64/libgssapi_krb5.so.2.2
7f845eec7000-7f845eec9000 rw-p 00040000 08:01 262094                     /lib64/libgssapi_krb5.so.2.2
7f845eec9000-7f845f03c000 r-xp 00000000 08:01 4757                       /usr/lib64/libcrypto.so.1.0.0
7f845f03c000-7f845f23c000 ---p 00173000 08:01 4757                       /usr/lib64/libcrypto.so.1.0.0
7f845f23c000-7f845f255000 r--p 00173000 08:01 4757                       /usr/lib64/libcrypto.so.1.0.0
7f845f255000-7f845f25f000 rw-p 0018c000 08:01 4757                       /usr/lib64/libcrypto.so.1.0.0
7f845f25f000-7f845f263000 rw-p 00000000 00:00 0
7f845f263000-7f845f2b6000 r-xp 00000000 08:01 4759                       /usr/lib64/libssl.so.1.0.0
7f845f2b6000-7f845f4b6000 ---p 00053000 08:01 4759                       /usr/lib64/libssl.so.1.0.0
7f845f4b6000-7f845f4b9000 r--p 00053000 08:01 4759                       /usr/lib64/libssl.so.1.0.0
7f845f4b9000-7f845f4be000 rw-p 00056000 08:01 4759                       /usr/lib64/libssl.so.1.0.0
7f845f4be000-7f845f4c5000 r-xp 00000000 08:01 16196                      /usr/lib64/python2.6/lib-dynload/_ssl.so
7f845f4c5000-7f845f6c4000 ---p 00007000 08:01 16196                      /usr/lib64/python2.6/lib-dynload/_ssl.so
7f845f6c4000-7f845f6c6000 rw-p 00006000 08:01 16196                      /usr/lib64/python2.6/lib-dynload/_ssl.so
7f845f6c6000-7f845f787000 rw-p 00000000 00:00 0
7f845f787000-7f845f792000 r-xp 00000000 08:01 16194                      /usr/lib64/python2.6/lib-dynload/_socketmodule.so
7f845f792000-7f845f992000 ---p 0000b000 08:01 16194                      /usr/lib64/python2.6/lib-dynload/_socketmodule.so
7f845f992000-7f845f996000 rw-p 0000b000 08:01 16194                      /usr/lib64/python2.6/lib-dynload/_socketmodule.so
7f845f996000-7f845f9a8000 r-xp 00000000 08:01 16202                      /usr/lib64/python2.6/lib-dynload/cPickle.so
7f845f9a8000-7f845fba7000 ---p 00012000 08:01 16202                      /usr/lib64/python2.6/lib-dynload/cPickle.so
7f845fba7000-7f845fba9000 rw-p 00011000 08:01 16202                      /usr/lib64/python2.6/lib-dynload/cPickle.so
7f845fba9000-7f845fbad000 r-xp 00000000 08:01 261865                     /lib64/libuuid.so.1.3.0
7f845fbad000-7f845fdac000 ---p 00004000 08:01 261865                     /lib64/libuuid.so.1.3.0
7f845fdac000-7f845fdad000 rw-p 00003000 08:01 261865                     /lib64/libuuid.so.1.3.0
7f845fdad000-7f845fdaf000 r-xp 00000000 08:01 5521                       /usr/lib64/python2.6/lib-dynload/fcntlmodule.so
7f845fdaf000-7f845ffaf000 ---p 00002000 08:01 5521                       /usr/lib64/python2.6/lib-dynload/fcntlmodule.so
7f845ffaf000-7f845ffb1000 rw-p 00002000 08:01 5521                       /usr/lib64/python2.6/lib-dynload/fcntlmodule.so
7f845ffb1000-7f845ffb5000 r-xp 00000000 08:01 16203                      /usr/lib64/python2.6/lib-dynload/cStringIO.so
7f845ffb5000-7f84601b4000 ---p 00004000 08:01 16203                      /usr/lib64/python2.6/lib-dynload/cStringIO.so
7f84601b4000-7f84601b6000 rw-p 00003000 08:01 16203                      /usr/lib64/python2.6/lib-dynload/cStringIO.so
7f84601b6000-7f84601b9000 r-xp 00000000 08:01 5505                       /usr/lib64/python2.6/lib-dynload/_randommodule.so
7f84601b9000-7f84603b8000 ---p 00003000 08:01 5505                       /usr/lib64/python2.6/lib-dynload/_randommodule.so
7f84603b8000-7f84603b9000 rw-p 00002000 08:01 5505                       /usr/lib64/python2.6/lib-dynload/_randommodule.so
7f84603b9000-7f84603be000 r-xp 00000000 08:01 5513                       /usr/lib64/python2.6/lib-dynload/binascii.so
7f84603be000-7f84605bd000 ---p 00005000 08:01 5513                       /usr/lib64/python2.6/lib-dynload/binascii.so
7f84605bd000-7f84605be000 rw-p 00004000 08:01 5513                       /usr/lib64/python2.6/lib-dynload/binascii.so
7f84605be000-7f84605c3000 r-xp 00000000 08:01 5528                       /usr/lib64/python2.6/lib-dynload/mathmodule.so
7f84605c3000-7f84607c3000 ---p 00005000 08:01 5528                       /usr/lib64/python2.6/lib-dynload/mathmodule.so
7f84607c3000-7f84607c5000 rw-p 00005000 08:01 5528                       /usr/lib64/python2.6/lib-dynload/mathmodule.so
7f84607c5000-7f84607cc000 r-xp 00000000 08:01 4938                       /usr/lib64/libffi.so.5.0.6
7f84607cc000-7f84609cc000 ---p 00007000 08:01 4938                       /usr/lib64/libffi.so.5.0.6
7f84609cc000-7f84609cd000 rw-p 00007000 08:01 4938                       /usr/lib64/libffi.so.5.0.6
7f84609cd000-7f84609e7000 r-xp 00000000 08:01 5491                       /usr/lib64/python2.6/lib-dynload/_ctypes.so
7f84609e7000-7f8460be7000 ---p 0001a000 08:01 5491                       /usr/lib64/python2.6/lib-dynload/_ctypes.so
7f8460be7000-7f8460beb000 rw-p 0001a000 08:01 5491                       /usr/lib64/python2.6/lib-dynload/_ctypes.so
7f8460beb000-7f8460bfb000 r-xp 00000000 08:01 5518                       /usr/lib64/python2.6/lib-dynload/datetime.so
7f8460bfb000-7f8460dfb000 ---p 00010000 08:01 5518                       /usr/lib64/python2.6/lib-dynload/datetime.so
7f8460dfb000-7f8460dff000 rw-p 00010000 08:01 5518                       /usr/lib64/python2.6/lib-dynload/datetime.so
7f8460dff000-7f8460e06000 r-xp 00000000 08:01 16197                      /usr/lib64/python2.6/lib-dynload/_struct.so
7f8460e06000-7f8461006000 ---p 00007000 08:01 16197                      /usr/lib64/python2.6/lib-dynload/_struct.so
7f8461006000-7f8461008000 rw-p 00007000 08:01 16197                      /usr/lib64/python2.6/lib-dynload/_struct.so
7f8461008000-7f8461010000 r-xp 00000000 08:01 5531                       /usr/lib64/python2.6/lib-dynload/operator.so
7f8461010000-7f8461210000 ---p 00008000 08:01 5531                       /usr/lib64/python2.6/lib-dynload/operator.so
7f8461210000-7f8461212000 rw-p 00008000 08:01 5531                       /usr/lib64/python2.6/lib-dynload/operator.so
7f8461212000-7f8461218000 r-xp 00000000 08:01 5488                       /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so
7f8461218000-7f8461417000 ---p 00006000 08:01 5488                       /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so
7f8461417000-7f8461419000 rw-p 00005000 08:01 5488                       /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so
7f8461419000-7f846141c000 r-xp 00000000 08:01 16214                      /usr/lib64/python2.6/lib-dynload/timemodule.so
7f846141c000-7f846161c000 ---p 00003000 08:01 16214                      /usr/lib64/python2.6/lib-dynload/timemodule.so
7f846161c000-7f846161e000 rw-p 00003000 08:01 16214                      /usr/lib64/python2.6/lib-dynload/timemodule.so
7f846161e000-7f8461620000 r-xp 00000000 08:01 5496                       /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so
7f8461620000-7f8461820000 ---p 00002000 08:01 5496                       /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so
7f8461820000-7f8461821000 rw-p 00002000 08:01 5496                       /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so
7f8461821000-7f8461826000 r-xp 00000000 08:01 5539                       /usr/lib64/python2.6/lib-dynload/stropmodule.so
7f8461826000-7f8461a25000 ---p 00005000 08:01 5539                       /usr/lib64/python2.6/lib-dynload/stropmodule.so
7f8461a25000-7f8461a27000 rw-p 00004000 08:01 5539                       /usr/lib64/python2.6/lib-dynload/stropmodule.so
7f8461a27000-7f84678b8000 r--p 00000000 08:01 16193                      /usr/lib/locale/locale-archive
7f84678b8000-7f8467a41000 r-xp 00000000 08:01 261671                     /lib64/libc-2.12.so
7f8467a41000-7f8467c40000 ---p 00189000 08:01 261671                     /lib64/libc-2.12.so
7f8467c40000-7f8467c44000 r--p 00188000 08:01 261671                     /lib64/libc-2.12.so
7f8467c44000-7f8467c45000 rw-p 0018c000 08:01 261671                     /lib64/libc-2.12.so
7f8467c45000-7f8467c4a000 rw-p 00000000 00:00 0
7f8467c4a000-7f8467ccd000 r-xp 00000000 08:01 261679                     /lib64/libm-2.12.so
7f8467ccd000-7f8467ecc000 ---p 00083000 08:01 261679                     /lib64/libm-2.12.so
7f8467ecc000-7f8467ecd000 r--p 00082000 08:01 261679                     /lib64/libm-2.12.so
7f8467ecd000-7f8467ece000 rw-p 00083000 08:01 261679                     /lib64/libm-2.12.so
7f8467ece000-7f8467ed0000 r-xp 00000000 08:01 261703                     /lib64/libutil-2.12.so
7f8467ed0000-7f84680cf000 ---p 00002000 08:01 261703                     /lib64/libutil-2.12.so
7f84680cf000-7f84680d0000 r--p 00001000 08:01 261703                     /lib64/libutil-2.12.so
7f84680d0000-7f84680d1000 rw-p 00002000 08:01 261703                     /lib64/libutil-2.12.so
7f84680d1000-7f84680d3000 r-xp 00000000 08:01 261677                     /lib64/libdl-2.12.so
7f84680d3000-7f84682d3000 ---p 00002000 08:01 261677                     /lib64/libdl-2.12.so
7f84682d3000-7f84682d4000 r--p 00002000 08:01 261677                     /lib64/libdl-2.12.so
7f84682d4000-7f84682d5000 rw-p 00003000 08:01 261677                     /lib64/libdl-2.12.so
7f84682d5000-7f84682ec000 r-xp 00000000 08:01 261695                     /lib64/libpthread-2.12.so
7f84682ec000-7f84684ec000 ---p 00017000 08:01 261695                     /lib64/libpthread-2.12.so
7f84684ec000-7f84684ed000 r--p 00017000 08:01 261695                     /lib64/libpthread-2.12.so
7f84684ed000-7f84684ee000 rw-p 00018000 08:01 261695                     /lib64/libpthread-2.12.so
7f84684ee000-7f84684f2000 rw-p 00000000 00:00 0
7f84684f2000-7f8468663000 r-xp 00000000 08:01 3735                       /usr/lib64/libpython2.6.so.1.0
7f8468663000-7f8468862000 ---p 00171000 08:01 3735                       /usr/lib64/libpython2.6.so.1.0
7f8468862000-7f846889e000 rw-p 00170000 08:01 3735                       /usr/lib64/libpython2.6.so.1.0
7f846889e000-7f84688ac000 rw-p 00000000 00:00 0
7f84688ac000-7f84688cc000 r-xp 00000000 08:01 265982                     /lib64/ld-2.12.so
7f8468945000-7f8468986000 rw-p 00000000 00:00 0
7f84689c7000-7f8468a08000 rw-p 00000000 00:00 0
7f8468a3a000-7f8468ac1000 rw-p 00000000 00:00 0
7f8468ac6000-7f8468ac8000 rw-p 00000000 00:00 0
7f8468ac8000-7f8468ac9000 rw-s 00000000 08:01 275594                     /tmp/ffioaMQUo (deleted)
7f8468ac9000-7f8468aca000 r-xs 00000000 08:01 275594                     /tmp/ffioaMQUo (deleted)
7f8468aca000-7f8468acb000 rw-p 00000000 00:00 0
7f8468acb000-7f8468acc000 r--p 0001f000 08:01 265982                     /lib64/ld-2.12.so
7f8468acc000-7f8468acd000 rw-p 00020000 08:01 265982                     /lib64/ld-2.12.so
7f8468acd000-7f8468ace000 rw-p 00000000 00:00 0
7fff59796000-7fff597c8000 rw-p 00000000 00:00 0                          [stack]
7fff597ff000-7fff59800000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
1675:   python ./sync3.py dhcp-27-79... dhcp-27-79... dhcp-27-69... dhcp-27-147... dhcp-27-251...
0000000000400000      4K r-x--  /usr/bin/python
0000000000600000      4K rw---  /usr/bin/python
0000000002562000   9632K rw---    [ anon ]
00007f8454000000   4248K rw---    [ anon ]
00007f8454426000  61288K -----    [ anon ]
00007f845a569000     16K r-x--  /usr/lib64/sasl2/libanonymous.so.2.0.23
00007f845a56d000   2044K -----  /usr/lib64/sasl2/libanonymous.so.2.0.23
00007f845a76c000      4K r----  /usr/lib64/sasl2/libanonymous.so.2.0.23
00007f845a76d000      4K rw---  /usr/lib64/sasl2/libanonymous.so.2.0.23
00007f845a76e000     16K r-x--  /usr/lib64/sasl2/libcrammd5.so.2.0.23
00007f845a772000   2048K -----  /usr/lib64/sasl2/libcrammd5.so.2.0.23
00007f845a972000      4K r----  /usr/lib64/sasl2/libcrammd5.so.2.0.23
00007f845a973000      4K rw---  /usr/lib64/sasl2/libcrammd5.so.2.0.23
00007f845a974000   1468K r-x--  /lib64/libdb-4.7.so
00007f845aae3000   2044K -----  /lib64/libdb-4.7.so
00007f845ace2000     24K rw---  /lib64/libdb-4.7.so
00007f845ace8000     20K r-x--  /usr/lib64/sasl2/libsasldb.so.2.0.23
00007f845aced000   2044K -----  /usr/lib64/sasl2/libsasldb.so.2.0.23
00007f845aeec000      4K r----  /usr/lib64/sasl2/libsasldb.so.2.0.23
00007f845aeed000      4K rw---  /usr/lib64/sasl2/libsasldb.so.2.0.23
00007f845aeee000     48K r-x--  /usr/lib64/sasl2/libdigestmd5.so.2.0.23
00007f845aefa000   2044K -----  /usr/lib64/sasl2/libdigestmd5.so.2.0.23
00007f845b0f9000      4K r----  /usr/lib64/sasl2/libdigestmd5.so.2.0.23
00007f845b0fa000      4K rw---  /usr/lib64/sasl2/libdigestmd5.so.2.0.23
00007f845b0fb000      4K -----    [ anon ]
00007f845b0fc000  10240K rw---    [ anon ]
00007f845bafc000     20K r-x--  /lib64/libnss_dns-2.12.so
00007f845bb01000   2044K -----  /lib64/libnss_dns-2.12.so
00007f845bd00000      4K r----  /lib64/libnss_dns-2.12.so
00007f845bd01000      4K rw---  /lib64/libnss_dns-2.12.so
00007f845bd02000     48K r-x--  /lib64/libnss_files-2.12.so
00007f845bd0e000   2048K -----  /lib64/libnss_files-2.12.so
00007f845bf0e000      4K r----  /lib64/libnss_files-2.12.so
00007f845bf0f000      4K rw---  /lib64/libnss_files-2.12.so
00007f845bf10000     32K r-x--  /usr/lib64/python2.6/lib-dynload/arraymodule.so
00007f845bf18000   2044K -----  /usr/lib64/python2.6/lib-dynload/arraymodule.so
00007f845c117000     12K rw---  /usr/lib64/python2.6/lib-dynload/arraymodule.so
00007f845c11a000     12K r-x--  /usr/lib64/python2.6/lib-dynload/_hashlib.so
00007f845c11d000   2044K -----  /usr/lib64/python2.6/lib-dynload/_hashlib.so
00007f845c31c000      8K rw---  /usr/lib64/python2.6/lib-dynload/_hashlib.so
00007f845c31e000     16K r-x--  /usr/lib64/python2.6/lib-dynload/selectmodule.so
00007f845c322000   2048K -----  /usr/lib64/python2.6/lib-dynload/selectmodule.so
00007f845c522000      8K rw---  /usr/lib64/python2.6/lib-dynload/selectmodule.so
00007f845c524000    372K r-x--  /lib64/libfreebl3.so
00007f845c581000   2044K -----  /lib64/libfreebl3.so
00007f845c780000      4K r----  /lib64/libfreebl3.so
00007f845c781000      4K rw---  /lib64/libfreebl3.so
00007f845c782000     16K rw---    [ anon ]
00007f845c786000     28K r-x--  /lib64/libcrypt-2.12.so
00007f845c78d000   2048K -----  /lib64/libcrypt-2.12.so
00007f845c98d000      4K r----  /lib64/libcrypt-2.12.so
00007f845c98e000      4K rw---  /lib64/libcrypt-2.12.so
00007f845c98f000    184K rw---    [ anon ]
00007f845c9bd000     88K r-x--  /lib64/libgcc_s-4.4.6-20120305.so.1
00007f845c9d3000   2044K -----  /lib64/libgcc_s-4.4.6-20120305.so.1
00007f845cbd2000      4K rw---  /lib64/libgcc_s-4.4.6-20120305.so.1
00007f845cbd3000    928K r-x--  /usr/lib64/libstdc++.so.6.0.13
00007f845ccbb000   2048K -----  /usr/lib64/libstdc++.so.6.0.13
00007f845cebb000     28K r----  /usr/lib64/libstdc++.so.6.0.13
00007f845cec2000      8K rw---  /usr/lib64/libstdc++.so.6.0.13
00007f845cec4000     84K rw---    [ anon ]
00007f845ced9000    100K r-x--  /usr/lib64/libsasl2.so.2.0.23
00007f845cef2000   2044K -----  /usr/lib64/libsasl2.so.2.0.23
00007f845d0f1000      4K r----  /usr/lib64/libsasl2.so.2.0.23
00007f845d0f2000      4K rw---  /usr/lib64/libsasl2.so.2.0.23
00007f845d0f3000     24K r-x--  /usr/lib64/libsaslwrapper.so.1.0.0
00007f845d0f9000   2044K -----  /usr/lib64/libsaslwrapper.so.1.0.0
00007f845d2f8000      4K rw---  /usr/lib64/libsaslwrapper.so.1.0.0
00007f845d2f9000     48K r-x--  /usr/lib64/python2.6/site-packages/_saslwrapper.so
00007f845d305000   2048K -----  /usr/lib64/python2.6/site-packages/_saslwrapper.so
00007f845d505000      8K rw---  /usr/lib64/python2.6/site-packages/_saslwrapper.so
00007f845d507000     12K r-x--  /usr/lib64/python2.6/lib-dynload/_heapq.so
00007f845d50a000   2048K -----  /usr/lib64/python2.6/lib-dynload/_heapq.so
00007f845d70a000      8K rw---  /usr/lib64/python2.6/lib-dynload/_heapq.so
00007f845d70c000      8K r-x--  /usr/lib64/python2.6/lib-dynload/_bisectmodule.so
00007f845d70e000   2044K -----  /usr/lib64/python2.6/lib-dynload/_bisectmodule.so
00007f845d90d000      4K rw---  /usr/lib64/python2.6/lib-dynload/_bisectmodule.so
00007f845d90e000     40K r-x--  /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so
00007f845d918000   2044K -----  /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so
00007f845db17000     16K rw---  /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so
00007f845db1b000    116K r-x--  /lib64/libselinux.so.1
00007f845db38000   2044K -----  /lib64/libselinux.so.1
00007f845dd37000      4K r----  /lib64/libselinux.so.1
00007f845dd38000      4K rw---  /lib64/libselinux.so.1
00007f845dd39000      4K rw---    [ anon ]
00007f845dd3a000     88K r-x--  /lib64/libresolv-2.12.so
00007f845dd50000   2048K -----  /lib64/libresolv-2.12.so
00007f845df50000      4K r----  /lib64/libresolv-2.12.so
00007f845df51000      4K rw---  /lib64/libresolv-2.12.so
00007f845df52000      8K rw---    [ anon ]
00007f845df54000      8K r-x--  /lib64/libkeyutils.so.1.3
00007f845df56000   2044K -----  /lib64/libkeyutils.so.1.3
00007f845e155000      4K r----  /lib64/libkeyutils.so.1.3
00007f845e156000      4K rw---  /lib64/libkeyutils.so.1.3
00007f845e157000     40K r-x--  /lib64/libkrb5support.so.0.1
00007f845e161000   2044K -----  /lib64/libkrb5support.so.0.1
00007f845e360000      4K r----  /lib64/libkrb5support.so.0.1
00007f845e361000      4K rw---  /lib64/libkrb5support.so.0.1
00007f845e362000     84K r-x--  /lib64/libz.so.1.2.3
00007f845e377000   2044K -----  /lib64/libz.so.1.2.3
00007f845e576000      4K r----  /lib64/libz.so.1.2.3
00007f845e577000      4K rw---  /lib64/libz.so.1.2.3
00007f845e578000    168K r-x--  /lib64/libk5crypto.so.3.1
00007f845e5a2000   2044K -----  /lib64/libk5crypto.so.3.1
00007f845e7a1000      8K r----  /lib64/libk5crypto.so.3.1
00007f845e7a3000      4K rw---  /lib64/libk5crypto.so.3.1
00007f845e7a4000     12K r-x--  /lib64/libcom_err.so.2.1
00007f845e7a7000   2044K -----  /lib64/libcom_err.so.2.1
00007f845e9a6000      4K r----  /lib64/libcom_err.so.2.1
00007f845e9a7000      4K rw---  /lib64/libcom_err.so.2.1
00007f845e9a8000    848K r-x--  /lib64/libkrb5.so.3.3
00007f845ea7c000   2048K -----  /lib64/libkrb5.so.3.3
00007f845ec7c000     36K r----  /lib64/libkrb5.so.3.3
00007f845ec85000      8K rw---  /lib64/libkrb5.so.3.3
00007f845ec87000    252K r-x--  /lib64/libgssapi_krb5.so.2.2
00007f845ecc6000   2048K -----  /lib64/libgssapi_krb5.so.2.2
00007f845eec6000      4K r----  /lib64/libgssapi_krb5.so.2.2
00007f845eec7000      8K rw---  /lib64/libgssapi_krb5.so.2.2
00007f845eec9000   1484K r-x--  /usr/lib64/libcrypto.so.1.0.0
00007f845f03c000   2048K -----  /usr/lib64/libcrypto.so.1.0.0
00007f845f23c000    100K r----  /usr/lib64/libcrypto.so.1.0.0
00007f845f255000     40K rw---  /usr/lib64/libcrypto.so.1.0.0
00007f845f25f000     16K rw---    [ anon ]
00007f845f263000    332K r-x--  /usr/lib64/libssl.so.1.0.0
00007f845f2b6000   2048K -----  /usr/lib64/libssl.so.1.0.0
00007f845f4b6000     12K r----  /usr/lib64/libssl.so.1.0.0
00007f845f4b9000     20K rw---  /usr/lib64/libssl.so.1.0.0
00007f845f4be000     28K r-x--  /usr/lib64/python2.6/lib-dynload/_ssl.so
00007f845f4c5000   2044K -----  /usr/lib64/python2.6/lib-dynload/_ssl.so
00007f845f6c4000      8K rw---  /usr/lib64/python2.6/lib-dynload/_ssl.so
00007f845f6c6000    772K rw---    [ anon ]
00007f845f787000     44K r-x--  /usr/lib64/python2.6/lib-dynload/_socketmodule.so
00007f845f792000   2048K -----  /usr/lib64/python2.6/lib-dynload/_socketmodule.so
00007f845f992000     16K rw---  /usr/lib64/python2.6/lib-dynload/_socketmodule.so
00007f845f996000     72K r-x--  /usr/lib64/python2.6/lib-dynload/cPickle.so
00007f845f9a8000   2044K -----  /usr/lib64/python2.6/lib-dynload/cPickle.so
00007f845fba7000      8K rw---  /usr/lib64/python2.6/lib-dynload/cPickle.so
00007f845fba9000     16K r-x--  /lib64/libuuid.so.1.3.0
00007f845fbad000   2044K -----  /lib64/libuuid.so.1.3.0
00007f845fdac000      4K rw---  /lib64/libuuid.so.1.3.0
00007f845fdad000      8K r-x--  /usr/lib64/python2.6/lib-dynload/fcntlmodule.so
00007f845fdaf000   2048K -----  /usr/lib64/python2.6/lib-dynload/fcntlmodule.so
00007f845ffaf000      8K rw---  /usr/lib64/python2.6/lib-dynload/fcntlmodule.so
00007f845ffb1000     16K r-x--  /usr/lib64/python2.6/lib-dynload/cStringIO.so
00007f845ffb5000   2044K -----  /usr/lib64/python2.6/lib-dynload/cStringIO.so
00007f84601b4000      8K rw---  /usr/lib64/python2.6/lib-dynload/cStringIO.so
00007f84601b6000     12K r-x--  /usr/lib64/python2.6/lib-dynload/_randommodule.so
00007f84601b9000   2044K -----  /usr/lib64/python2.6/lib-dynload/_randommodule.so
00007f84603b8000      4K rw---  /usr/lib64/python2.6/lib-dynload/_randommodule.so
00007f84603b9000     20K r-x--  /usr/lib64/python2.6/lib-dynload/binascii.so
00007f84603be000   2044K -----  /usr/lib64/python2.6/lib-dynload/binascii.so
00007f84605bd000      4K rw---  /usr/lib64/python2.6/lib-dynload/binascii.so
00007f84605be000     20K r-x--  /usr/lib64/python2.6/lib-dynload/mathmodule.so
00007f84605c3000   2048K -----  /usr/lib64/python2.6/lib-dynload/mathmodule.so
00007f84607c3000      8K rw---  /usr/lib64/python2.6/lib-dynload/mathmodule.so
00007f84607c5000     28K r-x--  /usr/lib64/libffi.so.5.0.6
00007f84607cc000   2048K -----  /usr/lib64/libffi.so.5.0.6
00007f84609cc000      4K rw---  /usr/lib64/libffi.so.5.0.6
00007f84609cd000    104K r-x--  /usr/lib64/python2.6/lib-dynload/_ctypes.so
00007f84609e7000   2048K -----  /usr/lib64/python2.6/lib-dynload/_ctypes.so
00007f8460be7000     16K rw---  /usr/lib64/python2.6/lib-dynload/_ctypes.so
00007f8460beb000     64K r-x--  /usr/lib64/python2.6/lib-dynload/datetime.so
00007f8460bfb000   2048K -----  /usr/lib64/python2.6/lib-dynload/datetime.so
00007f8460dfb000     16K rw---  /usr/lib64/python2.6/lib-dynload/datetime.so
00007f8460dff000     28K r-x--  /usr/lib64/python2.6/lib-dynload/_struct.so
00007f8460e06000   2048K -----  /usr/lib64/python2.6/lib-dynload/_struct.so
00007f8461006000      8K rw---  /usr/lib64/python2.6/lib-dynload/_struct.so
00007f8461008000     32K r-x--  /usr/lib64/python2.6/lib-dynload/operator.so
00007f8461010000   2048K -----  /usr/lib64/python2.6/lib-dynload/operator.so
00007f8461210000      8K rw---  /usr/lib64/python2.6/lib-dynload/operator.so
00007f8461212000     24K r-x--  /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so
00007f8461218000   2044K -----  /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so
00007f8461417000      8K rw---  /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so
00007f8461419000     12K r-x--  /usr/lib64/python2.6/lib-dynload/timemodule.so
00007f846141c000   2048K -----  /usr/lib64/python2.6/lib-dynload/timemodule.so
00007f846161c000      8K rw---  /usr/lib64/python2.6/lib-dynload/timemodule.so
00007f846161e000      8K r-x--  /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so
00007f8461620000   2048K -----  /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so
00007f8461820000      4K rw---  /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so
00007f8461821000     20K r-x--  /usr/lib64/python2.6/lib-dynload/stropmodule.so
00007f8461826000   2044K -----  /usr/lib64/python2.6/lib-dynload/stropmodule.so
00007f8461a25000      8K rw---  /usr/lib64/python2.6/lib-dynload/stropmodule.so
00007f8461a27000  96836K r----  /usr/lib/locale/locale-archive
00007f84678b8000   1572K r-x--  /lib64/libc-2.12.so
00007f8467a41000   2044K -----  /lib64/libc-2.12.so
00007f8467c40000     16K r----  /lib64/libc-2.12.so
00007f8467c44000      4K rw---  /lib64/libc-2.12.so
00007f8467c45000     20K rw---    [ anon ]
00007f8467c4a000    524K r-x--  /lib64/libm-2.12.so
00007f8467ccd000   2044K -----  /lib64/libm-2.12.so
00007f8467ecc000      4K r----  /lib64/libm-2.12.so
00007f8467ecd000      4K rw---  /lib64/libm-2.12.so
00007f8467ece000      8K r-x--  /lib64/libutil-2.12.so
00007f8467ed0000   2044K -----  /lib64/libutil-2.12.so
00007f84680cf000      4K r----  /lib64/libutil-2.12.so
00007f84680d0000      4K rw---  /lib64/libutil-2.12.so
00007f84680d1000      8K r-x--  /lib64/libdl-2.12.so
00007f84680d3000   2048K -----  /lib64/libdl-2.12.so
00007f84682d3000      4K r----  /lib64/libdl-2.12.so
00007f84682d4000      4K rw---  /lib64/libdl-2.12.so
00007f84682d5000     92K r-x--  /lib64/libpthread-2.12.so
00007f84682ec000   2048K -----  /lib64/libpthread-2.12.so
00007f84684ec000      4K r----  /lib64/libpthread-2.12.so
00007f84684ed000      4K rw---  /lib64/libpthread-2.12.so
00007f84684ee000     16K rw---    [ anon ]
00007f84684f2000   1476K r-x--  /usr/lib64/libpython2.6.so.1.0
00007f8468663000   2044K -----  /usr/lib64/libpython2.6.so.1.0
00007f8468862000    240K rw---  /usr/lib64/libpython2.6.so.1.0
00007f846889e000     56K rw---    [ anon ]
00007f84688ac000    128K r-x--  /lib64/ld-2.12.so
00007f8468945000    260K rw---    [ anon ]
00007f84689c7000    260K rw---    [ anon ]
00007f8468a3a000    540K rw---    [ anon ]
00007f8468ac6000      8K rw---    [ anon ]
00007f8468ac8000      4K rw-s-  /tmp/ffioaMQUo (deleted)
00007f8468ac9000      4K r-xs-  /tmp/ffioaMQUo (deleted)
00007f8468aca000      4K rw---    [ anon ]
00007f8468acb000      4K r----  /lib64/ld-2.12.so
00007f8468acc000      4K rw---  /lib64/ld-2.12.so
00007f8468acd000      4K rw---    [ anon ]
00007fff59796000    200K rw---    [ stack ]
00007fff597ff000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]
 total           309320K

> Does the behavior happen if you run the reproducer under valgrind?

No it does not:
==1711== HEAP SUMMARY:
==1711==     in use at exit: 5,676,308 bytes in 32,968 blocks
==1711==   total heap usage: 25,649,195 allocs, 25,616,227 frees, 2,212,309,853 bytes allocated
==1711==
==1711== LEAK SUMMARY:
==1711==    definitely lost: 0 bytes in 0 blocks
==1711==    indirectly lost: 0 bytes in 0 blocks
==1711==      possibly lost: 186,132 bytes in 399 blocks
==1711==    still reachable: 5,490,176 bytes in 32,569 blocks
==1711==         suppressed: 0 bytes in 0 blocks
==1711== Rerun with --leak-check=full to see details of leaked memory
==1711==
==1711== For counts of detected and suppressed errors, rerun with: -v
==1711== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 156 from 9)


I'm afraid this behavior is again related to my VM desktop F10 like kvm-74 issues, lowering severity until I reproduce with rhel5.8 / 6.3 virtualization.
Comment 6 Dave Malcolm 2012-09-07 15:51:57 UTC 
Thanks for the information.

It sounds like you're having some VM issues, so maybe that's it.

If not, looking at the /proc/PID/maps you posted, I see the only extension module loaded into the process (other than those in the standard library) is:
  /usr/lib64/python2.6/site-packages/_saslwrapper.so
I believe this is from the python-saslwrapper subpackage of saslwrapper.

I often see a big drop-off in code quality when going from the standard library to 3rd-party modules, so if it's not a virtualization issue, I would advise investigating that package next.
Comment 7 Frantisek Reznicek 2012-09-13 08:44:36 UTC 
(In reply to comment #6)
> Thanks for the information.
> 
> It sounds like you're having some VM issues, so maybe that's it.

I'm still unclear about that, whether it is my virtualization. I'll try to reproduce on RHEL VM host.

> 
> If not, looking at the /proc/PID/maps you posted, I see the only extension
> module loaded into the process (other than those in the standard library) is:
>   /usr/lib64/python2.6/site-packages/_saslwrapper.so
> I believe this is from the python-saslwrapper subpackage of saslwrapper.
> 
> I often see a big drop-off in code quality when going from the standard
> library to 3rd-party modules, so if it's not a virtualization issue, I would
> advise investigating that package next.

I'll try to re-run w/o python-saslwrapper which should be possible (no auth needed)
Comment 9 RHEL Program Management 2012-10-11 16:06:20 UTC 
Development Management has reviewed and declined this request.
You may appeal this decision by reopening this request.
Note You need to log in before you can comment on or make changes to this bug.