Bug 772976

Summary: yum / yum-complete-transaction sometimes hang / dumps '/usr/bin/python: double free or corruption' message
Product: Red Hat Enterprise Linux 6 Reporter: Frantisek Reznicek <freznice>
Component: rpmAssignee: Panu Matilainen <pmatilai>
Status: CLOSED NOTABUG QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.2CC: esammons, pknirsch
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-03-08 12:36:39 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Frantisek Reznicek 2012-01-10 15:22:43 UTC 
Description of problem:

Occasionly I saw my VM rhel6.2 x86_64 hanging / dumping glibc memory allocation check alerts:

for instance:
# yum update
  Updating   : kernel-headers-2.6.32-220.2.1.el6.x86_64                                                                               12/29
  Updating   : jasper-libs-1.900.1-15.el6_1.1.x86_64                                                                                  13/29
*** glibc detected *** /usr/bin/python: malloc(): smallbin double linked list corrupted: 0x0000000006a29ae0 ***
<hang>


in process of fixing that:
# yum-complete-transaction
...
  Erasing    : krb5-libs.x86_64                                                                                                       11/14
  Erasing    : 1:net-snmp-libs-5.5-37.el6.x86_64                                                                                      12/14
Error in POSTUN scriptlet in rpm package net-snmp-libs
  Erasing    : libicu-4.2.1-9.el6.x86_64                                                                                              13/14
error: %postun(net-snmp-libs-1:5.5-37.el6.x86_64) scriptlet failed, signal 11
  Erasing    : jasper-libs-1.900.1-15.el6.x86_64                                                                                      14/14
*** glibc detected *** /usr/bin/python: double free or corruption (fasttop): 0x000000000653f3e0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x750c6)[0x7ffaf27b70c6]
/lib64/libselinux.so.1(selinux_raw_to_trans_context+0xd0)[0x7ffaec2263c0]
/lib64/libselinux.so.1(getfilecon+0x2a)[0x7ffaec21ea8a]
/lib64/libselinux.so.1(rpm_execcon+0x75)[0x7ffaec224d15]
/usr/lib64/librpm.so.1(+0x34b56)[0x7ffaec465b56]
/usr/lib64/librpm.so.1(+0x34ffa)[0x7ffaec465ffa]
/usr/lib64/librpm.so.1(+0x358c6)[0x7ffaec4668c6]
/usr/lib64/librpm.so.1(+0x3584b)[0x7ffaec46684b]
/usr/lib64/librpm.so.1(rpmtsRun+0x8c8)[0x7ffaec47fd98]
/usr/lib64/python2.6/site-packages/rpm/_rpmmodule.so(+0xe51c)[0x7ffaec6a951c]
/usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7ffaf33ce043]
/usr/lib64/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7ffaf3462aa3]
/usr/lib64/libpython2.6.so.1.0(+0x5eaac)[0x7ffaf33e8aac]
/usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7ffaf33ce043]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x47d4)[0x7ffaf3467d84]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef)[0x7ffaf346999f]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927)[0x7ffaf346a467]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304)[0x7ffaf34688b4]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef)[0x7ffaf346999f]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 292123                             /usr/bin/python
00600000-00601000 rw-p 00000000 08:01 292123                             /usr/bin/python
01f2a000-0698d000 rw-p 00000000 00:00 0                                  [heap]
7ffae0069000-7ffae0121000 rw-s 00000000 08:01 80810                      /var/lib/rpm/__db.004
7ffae0121000-7ffae0263000 rw-s 00000000 08:01 80809                      /var/lib/rpm/__db.003
7ffae0263000-7ffae029b000 rw-s 00000000 08:01 80808                      /var/lib/rpm/__db.002
7ffae029b000-7ffae0353000 rw-s 00000000 08:01 80810                      /var/lib/rpm/__db.004
7ffae0353000-7ffae0495000 rw-s 00000000 08:01 80809                      /var/lib/rpm/__db.003
7ffae0495000-7ffae049e000 r-xp 00000000 08:01 323782                     /usr/lib64/python2.6/lib-dynload/_elementtree.so
7ffae049e000-7ffae069e000 ---p 00009000 08:01 323782                     /usr/lib64/python2.6/lib-dynload/_elementtree.so
7ffae069e000-7ffae069f000 rw-p 00009000 08:01 323782                     /usr/lib64/python2.6/lib-dynload/_elementtree.so
7ffae069f000-7ffae06a1000 r-xp 00000000 08:01 323828                     /usr/lib64/python2.6/lib-dynload/syslog.so
7ffae06a1000-7ffae08a0000 ---p 00002000 08:01 323828                     /usr/lib64/python2.6/lib-dynload/syslog.so
7ffae08a0000-7ffae08a1000 rw-p 00001000 08:01 323828                     /usr/lib64/python2.6/lib-dynload/syslog.so
7ffae08a1000-7ffae08c7000 r-xp 00000000 08:01 186031                     /lib64/libexpat.so.1.5.2
7ffae08c7000-7ffae0ac6000 ---p 00026000 08:01 186031                     /lib64/libexpat.so.1.5.2
7ffae0ac6000-7ffae0ac9000 rw-p 00025000 08:01 186031                     /lib64/libexpat.so.1.5.2
7ffae0ac9000-7ffae0ad4000 r-xp 00000000 08:01 323822                     /usr/lib64/python2.6/lib-dynload/pyexpat.so
7ffae0ad4000-7ffae0cd3000 ---p 0000b000 08:01 323822                     /usr/lib64/python2.6/lib-dynload/pyexpat.so
7ffae0cd3000-7ffae0cd5000 rw-p 0000a000 08:01 323822                     /usr/lib64/python2.6/lib-dynload/pyexpat.so
7ffae0cd5000-7ffae0d39000 r-xp 00000000 08:01 324904                     /usr/lib64/python2.6/site-packages/M2Crypto/__m2crypto.so
7ffae0d39000-7ffae0f38000 ---p 00064000 08:01 324904                     /usr/lib64/python2.6/site-packages/M2Crypto/__m2crypto.so
7ffae0f38000-7ffae0f42000 rw-p 00063000 08:01 324904                     /usr/lib64/python2.6/site-packages/M2Crypto/__m2crypto.so
7ffae0f42000-7ffae0f49000 r-xp 00000000 08:01 324741                     /usr/lib64/python2.6/site-packages/simplejson/_speedups.so
7ffae0f49000-7ffae1149000 ---p 00007000 08:01 324741                     /usr/lib64/python2.6/site-packages/simplejson/_speedups.so
7ffae1149000-7ffae114a000 rw-p 00007000 08:01 324741                     /usr/lib64/python2.6/site-packages/simplejson/_speedups.so
7ffae114a000-7ffae1167000 r-xp 00000000 08:01 185896                     /lib64/libtinfo.so.5.7
7ffae1167000-7ffae1367000 ---p 0001d000 08:01 185896                     /lib64/libtinfo.so.5.7
7ffae1367000-7ffae136b000 rw-p 0001d000 08:01 185896                     /lib64/libtinfo.so.5.7
7ffae136b000-7ffae1399000 r-xp 00000000 08:01 185894                     /lib64/libncursesw.so.5.7
7ffae1399000-7ffae1599000 ---p 0002e000 08:01 185894                     /lib64/libncursesw.so.5.7
7ffae1599000-7ffae159a000 rw-p 0002e000 08:01 185894                     /lib64/libncursesw.so.5.7
7ffae159a000-7ffae15ab000 r-xp 00000000 08:01 323780                     /usr/lib64/python2.6/lib-dynload/_curses.so
7ffae15ab000-7ffae17ab000 ---p 00011000 08:01 323780                     /usr/lib64/python2.6/lib-dynload/_curses.so
7ffae17ab000-7ffae17ad000 rw-p 00011000 08:01 323780                     /usr/lib64/python2.6/lib-dynload/_curses.so
7ffae17ad000-7ffae1870000 rw-p 00000000 00:00 0
7ffae1870000-7ffae1874000 r-xp 00000000 08:01 323829                     /usr/lib64/python2.6/lib-dynload/termios.so
7ffae1874000-7ffae1a74000 ---p 00004000 08:01 323829                     /usr/lib64/python2.6/lib-dynload/termios.so
7ffae1a74000-7ffae1a76000 rw-p 00004000 08:01 323829                     /usr/lib64/python2.6/lib-dynload/termios.so
7ffae1a76000-7ffae1ab7000 rw-p 00000000 00:00 0
7ffae1ab7000-7ffae1bfe000 r-xp 00000000 08:01 291907                     /usr/lib64/libxml2.so.2.7.6
7ffae1bfe000-7ffae1dfd000 ---p 00147000 08:01 291907                     /usr/lib64/libxml2.so.2.7.6
7ffae1dfd000-7ffae1e07000 rw-p 00146000 08:01 291907                     /usr/lib64/libxml2.so.2.7.6
7ffae1e07000-7ffae1e08000 rw-p 00000000 00:00 0
7ffae1e08000-7ffae1eec000 r-xp 00000000 08:01 185926                     /lib64/libglib-2.0.so.0.2200.5
7ffae1eec000-7ffae20ec000 ---p 000e4000 08:01 185926                     /lib64/libglib-2.0.so.0.2200.5
7ffae20ec000-7ffae20ed000 rw-p 000e4000 08:01 185926                     /lib64/libglib-2.0.so.0.2200.5
7ffae20ed000-7ffae20ee000 rw-p 00000000 00:00 0
7ffae20ee000-7ffae20f8000 r-xp 00000000 08:01 325436                     /usr/lib64/python2.6/site-packages/_sqlitecache.so
7ffae20f8000-7ffae22f8000 ---p 0000a000 08:01 325436                     /usr/lib64/python2.6/site-packages/_sqlitecache.so
7ffae22f8000-7ffae22f9000 rw-p 0000a000 08:01 325436                     /usr/lib64/python2.6/site-packages/_sqlitecache.so
7ffae22f9000-7ffae230a000 r-xp 00000000 08:01 323795                     /usr/lib64/python2.6/lib-dynload/_sqlite3.so
7ffae230a000-7ffae2509000 ---p 00011000 08:01 323795                     /usr/lib64/python2.6/lib-dynload/_sqlite3.so
7ffae2509000-7ffae250c000 rw-p 00010000 08:01 323795                     /usr/lib64/python2.6/lib-dynload/_sqlite3.so
7ffae250c000-7ffae251c000 r-xp 00000000 08:01 323806                     /usr/lib64/python2.6/lib-dynload/datetime.so
7ffae251c000-7ffae271c000 ---p 00010000 08:01 323806                     /usr/lib64/python2.6/lib-dynload/datetime.so
7ffae271c000-7ffae2720000 rw-p 00010000 08:01 323806                     /usr/lib64/python2.6/lib-dynload/datetime.so
7ffae2720000-7ffae2721000 r-xp 00000000 08:01 323798                     /usr/lib64/python2.6/lib-dynload/_weakref.so
7ffae2721000-7ffae2921000 ---p 00001000 08:01 323798                     /usr/lib64/python2.6/lib-dynload/_weakref.so
7ffae2921000-7ffae2922000 rw-p 00001000 08:01 323798                     /usr/lib64/python2.6/lib-dynload/_weakref.so
7ffae2922000-7ffae29a4000 rw-p 00000000 00:00 0
7ffae29a4000-7ffae29a6000 r-xp 00000000 08:01 323812                     /usr/lib64/python2.6/lib-dynload/grpmodule.so
7ffae29a6000-7ffae2ba5000 ---p 00002000 08:01 323812                     /usr/lib64/python2.6/lib-dynload/grpmodule.so
7ffae2ba5000-7ffae2ba6000 rw-p 00001000 08:01 323812                     /usr/lib64/python2.6/lib-dynload/grpmodule.so
7ffae2ba6000-7ffae2c28000 rw-p 00000000 00:00 0
7ffae2c28000-7ffae2c2b000 r-xp 00000000 08:01 186035                     /lib64/libgpg-error.so.0.5.0
7ffae2c2b000-7ffae2e2a000 ---p 00003000 08:01 186035                     /lib64/libgpg-error.so.0.5.0
7ffae2e2a000-7ffae2e2b000 r--p 00002000 08:01 186035                     /lib64/libgpg-error.so.0.5.0
7ffae2e2b000-7ffae2e2c000 rw-p 00003000 08:01 186035                     /lib64/libgpg-error.so.0.5.0
7ffae2e2c000-7ffae2e5a000 r-xp 00000000 08:01 292437                     /usr/lib64/libgpgme.so.11.6.6
7ffae2e5a000-7ffae305a000 ---p 0002e000 08:01 292437                     /usr/lib64/libgpgme.so.11.6.6
7ffae305a000-7ffae305c000 rw-p 0002e000 08:01 292437                     /usr/lib64/libgpgme.so.11.6.6
7ffae305c000-7ffae305d000 rw-p 00000000 00:00 0
7ffae305d000-7ffae306d000 r-xp 00000000 08:01 331754                     /usr/lib64/python2.6/site-packages/gpgme/_gpgme.so
7ffae306d000-7ffae326d000 ---p 00010000 08:01 331754                     /usr/lib64/python2.6/site-packages/gpgme/_gpgme.so
7ffae326d000-7ffae3270000 rw-p 00010000 08:01 331754                     /usr/lib64/python2.6/site-packages/gpgme/_gpgme.so
7ffae3270000-7ffae3277000 r-xp 00000000 08:01 323802                     /usr/lib64/python2.6/lib-dynload/bz2.so
7ffae3277000-7ffae3476000 ---p 00007000 08:01 323802                     /usr/lib64/python2.6/lib-dynload/bz2.so
7ffae3476000-7ffae3479000 rw-p 00006000 08:01 323802                     /usr/lib64/python2.6/lib-dynload/bz2.so
7ffae3479000-7ffae34ba000 rw-p 00000000 00:00 0
7ffae34ba000-7ffae34be000 r-xp 00000000 08:01 323789                     /usr/lib64/python2.6/lib-dynload/_localemodule.so
7ffae34be000-7ffae36be000 ---p 00004000 08:01 323789                     /usr/lib64/python2.6/lib-dynload/_localemodule.so
7ffae36be000-7ffae36bf000 rw-p 00004000 08:01 323789                     /usr/lib64/python2.6/lib-dynload/_localemodule.so
7ffae36bf000-7ffae36c4000 r-xp 00000000 08:01 323834                     /usr/lib64/python2.6/lib-dynload/zlibmodule.so
7ffae36c4000-7ffae38c3000 ---p 00005000 08:01 323834                     /usr/lib64/python2.6/lib-dynload/zlibmodule.so
7ffae38c3000-7ffae38c5000 rw-p 00004000 08:01 323834                     /usr/lib64/python2.6/lib-dynload/zlibmodule.so
7ffae38c5000-7ffae3906000 rw-p 00000000 00:00 0
7ffae3906000-7ffae390d000 r-xp 00000000 08:01 185859                     /lib64/libcrypt-2.12.so
7ffae390d000-7ffae3b0d000 ---p 00007000 08:01 185859                     /lib64/libcrypt-2.12.so
7ffae3b0d000-7ffae3b0e000 r--p 00007000 08:01 185859                     /lib64/libcrypt-2.12.so
7ffae3b0e000-7ffae3b0f000 rw-p 00008000 08:01 185859                     /lib64/libcrypt-2.12.so
7ffae3b0f000-7ffae3b3d000 rw-p 00000000 00:00 0
7ffae3b3d000-7ffae3b56000 r-xp 00000000 08:01 291888                     /usr/lib64/libsasl2.so.2.0.23
7ffae3b56000-7ffae3d55000 ---p 00019000 08:01 291888                     /usr/lib64/libsasl2.so.2.0.23
7ffae3d55000-7ffae3d56000 r--p 00018000 08:01 291888                     /usr/lib64/libsasl2.so.2.0.23
7ffae3d56000-7ffae3d57000 rw-p 00019000 08:01 291888                     /usr/lib64/libsasl2.so.2.0.23
7ffae3d57000-7ffae3d65000 r-xp 00000000 08:01 186182                     /lib64/liblber-2.4.so.2.5.6
7ffae3d65000-7ffae3f64000 ---p 0000e000 08:01 186182                     /lib64/liblber-2.4.so.2.5.6
7ffae3f64000-7ffae3f65000 r--p 0000d000 08:01 186182                     /lib64/liblber-2.4.so.2.5.6
7ffae3f65000-7ffae3f66000 rw-p 0000e000 08:01 186182                     /lib64/liblber-2.4.so.2.5.6
7ffae3f66000-7ffae3f8c000 r-xp 00000000 08:01 292389                     /usr/lib64/libssh2.so.1.0.1
7ffae3f8c000-7ffae418b000 ---p 00026000 08:01 292389                     /usr/lib64/libssh2.so.1.0.1
7ffae418b000-7ffae418c000 rw-p 00025000 08:01 292389                     /usr/lib64/libssh2.so.1.0.1
7ffae418c000-7ffae41b5000 r-xp 00000000 08:01 292936                     /usr/lib64/libsmime3.soError in POSTUN scriptlet in rpm package jasper-libs
Installed products updated.
jasper-libs-1.900.1-15.el6.x86_64 was supposed to be removed but is not!
1:net-snmp-libs-5.5-37.el6.x86_64 was supposed to be removed but is not!

Removed:
  dhclient.x86_64 12:4.1.1-25.P1.el6          dhcp-common.x86_64 12:4.1.1-25.P1.el6      kernel.x86_64 0:2.6.32-195.el6
  kernel-firmware.noarch 0:2.6.32-220.el6     kernel-headers.x86_64 0:2.6.32-220.el6     krb5-libs.x86_64 0:1.9-21.el6
  krb5-libs.x86_64 0:1.9-22.el6               libicu.x86_64 0:4.2.1-9.el6                nss.x86_64 0:3.12.10-16.el6
  nss-sysinit.x86_64 0:3.12.10-16.el6         selinux-policy.noarch 0:3.7.19-126.el6     selinux-policy-targeted.noarch 0:3.7.19-126.el6

Failed:
  jasper-libs.x86_64 0:1.900.1-15.el6                                   net-snmp-libs.x86_64 1:5.5-37.el6

Complete!
Not removing old transaction files



This was seen on RHEL 6.2 x86_64 (twice), never seen on i686.



Version-Release number of selected component (if applicable):
[root@dhcp-27-50 ~]# rpm -q kernel yum rpm
kernel-2.6.32-214.el6.x86_64
kernel-2.6.32-220.el6.x86_64
kernel-2.6.32-220.2.1.el6.x86_64
yum-3.2.29-22.el6.noarch
rpm-4.8.0-19.el6.x86_64
[root@dhcp-27-50 ~]# uname -a
Linux dhcp-27-50... #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64 x86_64 x86_64 GNU/Linux


How reproducible:
<5% (hardly)

Steps to Reproduce:
1. upgrade rhel6.2 x86_64
2. yum upgrade
3. eventually yum-complete-transaction
  
Actual results:
yum / yum-complete-transaction -> python does invalid allocation operation.

Expected results:
yum / yum-complete-transaction should not perform invalid allocation operation.
Comment 4 Phil Knirsch 2012-02-28 12:43:09 UTC 
This reminds me strongly of a kernel bug we had in RHEL-6 Beta where Transparent Hugetables were corrupting the guest memory in very rare cases (which got triggered by rpm and bdb).

Are you running the latest 6.2 updates on the host and guest systems?

Thanks & regards, Phil
Comment 5 Frantisek Reznicek 2012-02-28 16:28:45 UTC 
I'm going to retest to see if it is still happening.

At the time of report guests were (getting) updated.
Virtualization host is not RHEL (F8 with custom virtualization:)
  kvm-65-15.fc8.x86_64
  libvirt-0.6.1-1.fc8.x86_64
  libvirt-devel-0.6.1-1.fc8.x86_64
  libvirt-python-0.6.1-1.fc8.x86_64
  python-virtinst-0.400.3-3.fc9.noarch
  qemu-0.9.1-6.fc8.x86_64
  qemu-img-0.9.1-6.fc8.x86_64
  virt-manager-0.7.0-8.fc8.x86_64
  virt-manager-debuginfo-0.7.0-8.fc8.x86_64
  virt-viewer-0.0.3-1.fc8.x86_64
Comment 6 Frantisek Reznicek 2012-02-29 12:14:48 UTC 
I reproduced the issue again on updated rhel6.2 guest (x86_64).
I'm not seeing this behavior with RHEL's virtualization (kvm-*83-239.el5 libvirt-*0.8.2-22.el5) I'm going to confirm by building them on f8.
Comment 7 Frantisek Reznicek 2012-03-08 12:36:39 UTC 
The issue is indeed caused by virtualization host kvm issues.
When switched to kvm-74 issue gone away which correlates with fact that this behavior was never detected on rhel6 guests virtualized on RHEL5.x virt. hosts.

Issue has no content.

-> CLOSED / NOT-A-BUG
Comment 8 Panu Matilainen 2012-03-08 12:54:29 UTC 
Ok. Thanks for confirming it was indeed a virtualization bug, rpm/bdb seems to be quite a sucker for those.