772976 – yum / yum-complete-transaction sometimes hang / dumps '/usr/bin/python: double free or corruption' message

Bug 772976 - yum / yum-complete-transaction sometimes hang / dumps '/usr/bin/python: double free or corruption' message

Summary: yum / yum-complete-transaction sometimes hang / dumps '/usr/bin/python: doubl...

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	rpm
Sub Component:
Version:	6.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Panu Matilainen
QA Contact:	BaseOS QE Security Team
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-01-10 15:22 UTC by Frantisek Reznicek
Modified:	2015-11-16 01:13 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2012-03-08 12:36:39 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Frantisek Reznicek 2012-01-10 15:22:43 UTC

Description of problem:

Occasionly I saw my VM rhel6.2 x86_64 hanging / dumping glibc memory allocation check alerts:

for instance:
# yum update
  Updating   : kernel-headers-2.6.32-220.2.1.el6.x86_64                                                                               12/29
  Updating   : jasper-libs-1.900.1-15.el6_1.1.x86_64                                                                                  13/29
*** glibc detected *** /usr/bin/python: malloc(): smallbin double linked list corrupted: 0x0000000006a29ae0 ***
<hang>


in process of fixing that:
# yum-complete-transaction
...
  Erasing    : krb5-libs.x86_64                                                                                                       11/14
  Erasing    : 1:net-snmp-libs-5.5-37.el6.x86_64                                                                                      12/14
Error in POSTUN scriptlet in rpm package net-snmp-libs
  Erasing    : libicu-4.2.1-9.el6.x86_64                                                                                              13/14
error: %postun(net-snmp-libs-1:5.5-37.el6.x86_64) scriptlet failed, signal 11
  Erasing    : jasper-libs-1.900.1-15.el6.x86_64                                                                                      14/14
*** glibc detected *** /usr/bin/python: double free or corruption (fasttop): 0x000000000653f3e0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x750c6)[0x7ffaf27b70c6]
/lib64/libselinux.so.1(selinux_raw_to_trans_context+0xd0)[0x7ffaec2263c0]
/lib64/libselinux.so.1(getfilecon+0x2a)[0x7ffaec21ea8a]
/lib64/libselinux.so.1(rpm_execcon+0x75)[0x7ffaec224d15]
/usr/lib64/librpm.so.1(+0x34b56)[0x7ffaec465b56]
/usr/lib64/librpm.so.1(+0x34ffa)[0x7ffaec465ffa]
/usr/lib64/librpm.so.1(+0x358c6)[0x7ffaec4668c6]
/usr/lib64/librpm.so.1(+0x3584b)[0x7ffaec46684b]
/usr/lib64/librpm.so.1(rpmtsRun+0x8c8)[0x7ffaec47fd98]
/usr/lib64/python2.6/site-packages/rpm/_rpmmodule.so(+0xe51c)[0x7ffaec6a951c]
/usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7ffaf33ce043]
/usr/lib64/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7ffaf3462aa3]
/usr/lib64/libpython2.6.so.1.0(+0x5eaac)[0x7ffaf33e8aac]
/usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7ffaf33ce043]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x47d4)[0x7ffaf3467d84]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef)[0x7ffaf346999f]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927)[0x7ffaf346a467]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304)[0x7ffaf34688b4]
/usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef)[0x7ffaf346999f]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 292123                             /usr/bin/python
00600000-00601000 rw-p 00000000 08:01 292123                             /usr/bin/python
01f2a000-0698d000 rw-p 00000000 00:00 0                                  [heap]
7ffae0069000-7ffae0121000 rw-s 00000000 08:01 80810                      /var/lib/rpm/__db.004
7ffae0121000-7ffae0263000 rw-s 00000000 08:01 80809                      /var/lib/rpm/__db.003
7ffae0263000-7ffae029b000 rw-s 00000000 08:01 80808                      /var/lib/rpm/__db.002
7ffae029b000-7ffae0353000 rw-s 00000000 08:01 80810                      /var/lib/rpm/__db.004
7ffae0353000-7ffae0495000 rw-s 00000000 08:01 80809                      /var/lib/rpm/__db.003
7ffae0495000-7ffae049e000 r-xp 00000000 08:01 323782                     /usr/lib64/python2.6/lib-dynload/_elementtree.so
7ffae049e000-7ffae069e000 ---p 00009000 08:01 323782                     /usr/lib64/python2.6/lib-dynload/_elementtree.so
7ffae069e000-7ffae069f000 rw-p 00009000 08:01 323782                     /usr/lib64/python2.6/lib-dynload/_elementtree.so
7ffae069f000-7ffae06a1000 r-xp 00000000 08:01 323828                     /usr/lib64/python2.6/lib-dynload/syslog.so
7ffae06a1000-7ffae08a0000 ---p 00002000 08:01 323828                     /usr/lib64/python2.6/lib-dynload/syslog.so
7ffae08a0000-7ffae08a1000 rw-p 00001000 08:01 323828                     /usr/lib64/python2.6/lib-dynload/syslog.so
7ffae08a1000-7ffae08c7000 r-xp 00000000 08:01 186031                     /lib64/libexpat.so.1.5.2
7ffae08c7000-7ffae0ac6000 ---p 00026000 08:01 186031                     /lib64/libexpat.so.1.5.2
7ffae0ac6000-7ffae0ac9000 rw-p 00025000 08:01 186031                     /lib64/libexpat.so.1.5.2
7ffae0ac9000-7ffae0ad4000 r-xp 00000000 08:01 323822                     /usr/lib64/python2.6/lib-dynload/pyexpat.so
7ffae0ad4000-7ffae0cd3000 ---p 0000b000 08:01 323822                     /usr/lib64/python2.6/lib-dynload/pyexpat.so
7ffae0cd3000-7ffae0cd5000 rw-p 0000a000 08:01 323822                     /usr/lib64/python2.6/lib-dynload/pyexpat.so
7ffae0cd5000-7ffae0d39000 r-xp 00000000 08:01 324904                     /usr/lib64/python2.6/site-packages/M2Crypto/__m2crypto.so
7ffae0d39000-7ffae0f38000 ---p 00064000 08:01 324904                     /usr/lib64/python2.6/site-packages/M2Crypto/__m2crypto.so
7ffae0f38000-7ffae0f42000 rw-p 00063000 08:01 324904                     /usr/lib64/python2.6/site-packages/M2Crypto/__m2crypto.so
7ffae0f42000-7ffae0f49000 r-xp 00000000 08:01 324741                     /usr/lib64/python2.6/site-packages/simplejson/_speedups.so
7ffae0f49000-7ffae1149000 ---p 00007000 08:01 324741                     /usr/lib64/python2.6/site-packages/simplejson/_speedups.so
7ffae1149000-7ffae114a000 rw-p 00007000 08:01 324741                     /usr/lib64/python2.6/site-packages/simplejson/_speedups.so
7ffae114a000-7ffae1167000 r-xp 00000000 08:01 185896                     /lib64/libtinfo.so.5.7
7ffae1167000-7ffae1367000 ---p 0001d000 08:01 185896                     /lib64/libtinfo.so.5.7
7ffae1367000-7ffae136b000 rw-p 0001d000 08:01 185896                     /lib64/libtinfo.so.5.7
7ffae136b000-7ffae1399000 r-xp 00000000 08:01 185894                     /lib64/libncursesw.so.5.7
7ffae1399000-7ffae1599000 ---p 0002e000 08:01 185894                     /lib64/libncursesw.so.5.7
7ffae1599000-7ffae159a000 rw-p 0002e000 08:01 185894                     /lib64/libncursesw.so.5.7
7ffae159a000-7ffae15ab000 r-xp 00000000 08:01 323780                     /usr/lib64/python2.6/lib-dynload/_curses.so
7ffae15ab000-7ffae17ab000 ---p 00011000 08:01 323780                     /usr/lib64/python2.6/lib-dynload/_curses.so
7ffae17ab000-7ffae17ad000 rw-p 00011000 08:01 323780                     /usr/lib64/python2.6/lib-dynload/_curses.so
7ffae17ad000-7ffae1870000 rw-p 00000000 00:00 0
7ffae1870000-7ffae1874000 r-xp 00000000 08:01 323829                     /usr/lib64/python2.6/lib-dynload/termios.so
7ffae1874000-7ffae1a74000 ---p 00004000 08:01 323829                     /usr/lib64/python2.6/lib-dynload/termios.so
7ffae1a74000-7ffae1a76000 rw-p 00004000 08:01 323829                     /usr/lib64/python2.6/lib-dynload/termios.so
7ffae1a76000-7ffae1ab7000 rw-p 00000000 00:00 0
7ffae1ab7000-7ffae1bfe000 r-xp 00000000 08:01 291907                     /usr/lib64/libxml2.so.2.7.6
7ffae1bfe000-7ffae1dfd000 ---p 00147000 08:01 291907                     /usr/lib64/libxml2.so.2.7.6
7ffae1dfd000-7ffae1e07000 rw-p 00146000 08:01 291907                     /usr/lib64/libxml2.so.2.7.6
7ffae1e07000-7ffae1e08000 rw-p 00000000 00:00 0
7ffae1e08000-7ffae1eec000 r-xp 00000000 08:01 185926                     /lib64/libglib-2.0.so.0.2200.5
7ffae1eec000-7ffae20ec000 ---p 000e4000 08:01 185926                     /lib64/libglib-2.0.so.0.2200.5
7ffae20ec000-7ffae20ed000 rw-p 000e4000 08:01 185926                     /lib64/libglib-2.0.so.0.2200.5
7ffae20ed000-7ffae20ee000 rw-p 00000000 00:00 0
7ffae20ee000-7ffae20f8000 r-xp 00000000 08:01 325436                     /usr/lib64/python2.6/site-packages/_sqlitecache.so
7ffae20f8000-7ffae22f8000 ---p 0000a000 08:01 325436                     /usr/lib64/python2.6/site-packages/_sqlitecache.so
7ffae22f8000-7ffae22f9000 rw-p 0000a000 08:01 325436                     /usr/lib64/python2.6/site-packages/_sqlitecache.so
7ffae22f9000-7ffae230a000 r-xp 00000000 08:01 323795                     /usr/lib64/python2.6/lib-dynload/_sqlite3.so
7ffae230a000-7ffae2509000 ---p 00011000 08:01 323795                     /usr/lib64/python2.6/lib-dynload/_sqlite3.so
7ffae2509000-7ffae250c000 rw-p 00010000 08:01 323795                     /usr/lib64/python2.6/lib-dynload/_sqlite3.so
7ffae250c000-7ffae251c000 r-xp 00000000 08:01 323806                     /usr/lib64/python2.6/lib-dynload/datetime.so
7ffae251c000-7ffae271c000 ---p 00010000 08:01 323806                     /usr/lib64/python2.6/lib-dynload/datetime.so
7ffae271c000-7ffae2720000 rw-p 00010000 08:01 323806                     /usr/lib64/python2.6/lib-dynload/datetime.so
7ffae2720000-7ffae2721000 r-xp 00000000 08:01 323798                     /usr/lib64/python2.6/lib-dynload/_weakref.so
7ffae2721000-7ffae2921000 ---p 00001000 08:01 323798                     /usr/lib64/python2.6/lib-dynload/_weakref.so
7ffae2921000-7ffae2922000 rw-p 00001000 08:01 323798                     /usr/lib64/python2.6/lib-dynload/_weakref.so
7ffae2922000-7ffae29a4000 rw-p 00000000 00:00 0
7ffae29a4000-7ffae29a6000 r-xp 00000000 08:01 323812                     /usr/lib64/python2.6/lib-dynload/grpmodule.so
7ffae29a6000-7ffae2ba5000 ---p 00002000 08:01 323812                     /usr/lib64/python2.6/lib-dynload/grpmodule.so
7ffae2ba5000-7ffae2ba6000 rw-p 00001000 08:01 323812                     /usr/lib64/python2.6/lib-dynload/grpmodule.so
7ffae2ba6000-7ffae2c28000 rw-p 00000000 00:00 0
7ffae2c28000-7ffae2c2b000 r-xp 00000000 08:01 186035                     /lib64/libgpg-error.so.0.5.0
7ffae2c2b000-7ffae2e2a000 ---p 00003000 08:01 186035                     /lib64/libgpg-error.so.0.5.0
7ffae2e2a000-7ffae2e2b000 r--p 00002000 08:01 186035                     /lib64/libgpg-error.so.0.5.0
7ffae2e2b000-7ffae2e2c000 rw-p 00003000 08:01 186035                     /lib64/libgpg-error.so.0.5.0
7ffae2e2c000-7ffae2e5a000 r-xp 00000000 08:01 292437                     /usr/lib64/libgpgme.so.11.6.6
7ffae2e5a000-7ffae305a000 ---p 0002e000 08:01 292437                     /usr/lib64/libgpgme.so.11.6.6
7ffae305a000-7ffae305c000 rw-p 0002e000 08:01 292437                     /usr/lib64/libgpgme.so.11.6.6
7ffae305c000-7ffae305d000 rw-p 00000000 00:00 0
7ffae305d000-7ffae306d000 r-xp 00000000 08:01 331754                     /usr/lib64/python2.6/site-packages/gpgme/_gpgme.so
7ffae306d000-7ffae326d000 ---p 00010000 08:01 331754                     /usr/lib64/python2.6/site-packages/gpgme/_gpgme.so
7ffae326d000-7ffae3270000 rw-p 00010000 08:01 331754                     /usr/lib64/python2.6/site-packages/gpgme/_gpgme.so
7ffae3270000-7ffae3277000 r-xp 00000000 08:01 323802                     /usr/lib64/python2.6/lib-dynload/bz2.so
7ffae3277000-7ffae3476000 ---p 00007000 08:01 323802                     /usr/lib64/python2.6/lib-dynload/bz2.so
7ffae3476000-7ffae3479000 rw-p 00006000 08:01 323802                     /usr/lib64/python2.6/lib-dynload/bz2.so
7ffae3479000-7ffae34ba000 rw-p 00000000 00:00 0
7ffae34ba000-7ffae34be000 r-xp 00000000 08:01 323789                     /usr/lib64/python2.6/lib-dynload/_localemodule.so
7ffae34be000-7ffae36be000 ---p 00004000 08:01 323789                     /usr/lib64/python2.6/lib-dynload/_localemodule.so
7ffae36be000-7ffae36bf000 rw-p 00004000 08:01 323789                     /usr/lib64/python2.6/lib-dynload/_localemodule.so
7ffae36bf000-7ffae36c4000 r-xp 00000000 08:01 323834                     /usr/lib64/python2.6/lib-dynload/zlibmodule.so
7ffae36c4000-7ffae38c3000 ---p 00005000 08:01 323834                     /usr/lib64/python2.6/lib-dynload/zlibmodule.so
7ffae38c3000-7ffae38c5000 rw-p 00004000 08:01 323834                     /usr/lib64/python2.6/lib-dynload/zlibmodule.so
7ffae38c5000-7ffae3906000 rw-p 00000000 00:00 0
7ffae3906000-7ffae390d000 r-xp 00000000 08:01 185859                     /lib64/libcrypt-2.12.so
7ffae390d000-7ffae3b0d000 ---p 00007000 08:01 185859                     /lib64/libcrypt-2.12.so
7ffae3b0d000-7ffae3b0e000 r--p 00007000 08:01 185859                     /lib64/libcrypt-2.12.so
7ffae3b0e000-7ffae3b0f000 rw-p 00008000 08:01 185859                     /lib64/libcrypt-2.12.so
7ffae3b0f000-7ffae3b3d000 rw-p 00000000 00:00 0
7ffae3b3d000-7ffae3b56000 r-xp 00000000 08:01 291888                     /usr/lib64/libsasl2.so.2.0.23
7ffae3b56000-7ffae3d55000 ---p 00019000 08:01 291888                     /usr/lib64/libsasl2.so.2.0.23
7ffae3d55000-7ffae3d56000 r--p 00018000 08:01 291888                     /usr/lib64/libsasl2.so.2.0.23
7ffae3d56000-7ffae3d57000 rw-p 00019000 08:01 291888                     /usr/lib64/libsasl2.so.2.0.23
7ffae3d57000-7ffae3d65000 r-xp 00000000 08:01 186182                     /lib64/liblber-2.4.so.2.5.6
7ffae3d65000-7ffae3f64000 ---p 0000e000 08:01 186182                     /lib64/liblber-2.4.so.2.5.6
7ffae3f64000-7ffae3f65000 r--p 0000d000 08:01 186182                     /lib64/liblber-2.4.so.2.5.6
7ffae3f65000-7ffae3f66000 rw-p 0000e000 08:01 186182                     /lib64/liblber-2.4.so.2.5.6
7ffae3f66000-7ffae3f8c000 r-xp 00000000 08:01 292389                     /usr/lib64/libssh2.so.1.0.1
7ffae3f8c000-7ffae418b000 ---p 00026000 08:01 292389                     /usr/lib64/libssh2.so.1.0.1
7ffae418b000-7ffae418c000 rw-p 00025000 08:01 292389                     /usr/lib64/libssh2.so.1.0.1
7ffae418c000-7ffae41b5000 r-xp 00000000 08:01 292936                     /usr/lib64/libsmime3.soError in POSTUN scriptlet in rpm package jasper-libs
Installed products updated.
jasper-libs-1.900.1-15.el6.x86_64 was supposed to be removed but is not!
1:net-snmp-libs-5.5-37.el6.x86_64 was supposed to be removed but is not!

Removed:
  dhclient.x86_64 12:4.1.1-25.P1.el6          dhcp-common.x86_64 12:4.1.1-25.P1.el6      kernel.x86_64 0:2.6.32-195.el6
  kernel-firmware.noarch 0:2.6.32-220.el6     kernel-headers.x86_64 0:2.6.32-220.el6     krb5-libs.x86_64 0:1.9-21.el6
  krb5-libs.x86_64 0:1.9-22.el6               libicu.x86_64 0:4.2.1-9.el6                nss.x86_64 0:3.12.10-16.el6
  nss-sysinit.x86_64 0:3.12.10-16.el6         selinux-policy.noarch 0:3.7.19-126.el6     selinux-policy-targeted.noarch 0:3.7.19-126.el6

Failed:
  jasper-libs.x86_64 0:1.900.1-15.el6                                   net-snmp-libs.x86_64 1:5.5-37.el6

Complete!
Not removing old transaction files



This was seen on RHEL 6.2 x86_64 (twice), never seen on i686.



Version-Release number of selected component (if applicable):
[root@dhcp-27-50 ~]# rpm -q kernel yum rpm
kernel-2.6.32-214.el6.x86_64
kernel-2.6.32-220.el6.x86_64
kernel-2.6.32-220.2.1.el6.x86_64
yum-3.2.29-22.el6.noarch
rpm-4.8.0-19.el6.x86_64
[root@dhcp-27-50 ~]# uname -a
Linux dhcp-27-50... #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64 x86_64 x86_64 GNU/Linux


How reproducible:
<5% (hardly)

Steps to Reproduce:
1. upgrade rhel6.2 x86_64
2. yum upgrade
3. eventually yum-complete-transaction
  
Actual results:
yum / yum-complete-transaction -> python does invalid allocation operation.

Expected results:
yum / yum-complete-transaction should not perform invalid allocation operation.

Comment 4 Phil Knirsch 2012-02-28 12:43:09 UTC

This reminds me strongly of a kernel bug we had in RHEL-6 Beta where Transparent Hugetables were corrupting the guest memory in very rare cases (which got triggered by rpm and bdb).

Are you running the latest 6.2 updates on the host and guest systems?

Thanks & regards, Phil

Comment 5 Frantisek Reznicek 2012-02-28 16:28:45 UTC

I'm going to retest to see if it is still happening.

At the time of report guests were (getting) updated.
Virtualization host is not RHEL (F8 with custom virtualization:)
  kvm-65-15.fc8.x86_64
  libvirt-0.6.1-1.fc8.x86_64
  libvirt-devel-0.6.1-1.fc8.x86_64
  libvirt-python-0.6.1-1.fc8.x86_64
  python-virtinst-0.400.3-3.fc9.noarch
  qemu-0.9.1-6.fc8.x86_64
  qemu-img-0.9.1-6.fc8.x86_64
  virt-manager-0.7.0-8.fc8.x86_64
  virt-manager-debuginfo-0.7.0-8.fc8.x86_64
  virt-viewer-0.0.3-1.fc8.x86_64

Comment 6 Frantisek Reznicek 2012-02-29 12:14:48 UTC

I reproduced the issue again on updated rhel6.2 guest (x86_64).
I'm not seeing this behavior with RHEL's virtualization (kvm-*83-239.el5 libvirt-*0.8.2-22.el5) I'm going to confirm by building them on f8.

Comment 7 Frantisek Reznicek 2012-03-08 12:36:39 UTC

The issue is indeed caused by virtualization host kvm issues.
When switched to kvm-74 issue gone away which correlates with fact that this behavior was never detected on rhel6 guests virtualized on RHEL5.x virt. hosts.

Issue has no content.

-> CLOSED / NOT-A-BUG

Comment 8 Panu Matilainen 2012-03-08 12:54:29 UTC

Ok. Thanks for confirming it was indeed a virtualization bug, rpm/bdb seems to be quite a sucker for those.

Note You need to log in before you can comment on or make changes to this bug.