Bug 612627

doubt it's python, but david has all the magic scripts to work out whose fault it is :).

Thanks for filing this bug.

Do you have a core dump of the python process (yum)?

Do you have a list of packages that were installed?  (ideally from when the yum command began)  Otherwise, what's the output of "rpm -qa" ?

How far did "yum" get before the error?  Was there any more output beforehand?

I'm guessing this is an x86_64 machine (from the 64-bit address cited in comment #0, and from bug 612611) - is that correct?

Thanks!

(In reply to comment #2)
> doubt it's python, but david has all the magic scripts to work out whose fault
> it is :).
I have some magic scripts to help analyze gdb backtraces, but sadly they can't help us here.

I've not managed to get a coredump because ulimit was set (though I don't think the process tried to dump; it just locked up)  unfortunately I ended up killing it so I can't now force a coredump either.  

I got more output from a rerun, but the crash is different and unfortunately not reproducible any more.  

[root@localhost ~]# yum install nss-pam-ldapd
Loaded plugins: refresh-packagekit, rhnplugin
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them.
--> Running transaction check
---> Package nss-pam-ldapd.x86_64 0:0.7.5-3.el6 set to be updated
--> Processing Dependency: /lib64/security/pam_ldap.so for package: nss-pam-ldapd-0.7.5-3.el6.x86_64
--> Processing Dependency: nscd for package: nss-pam-ldapd-0.7.5-3.el6.x86_64
--> Running transaction check
---> Package nscd.x86_64 0:2.12-1.2.el6 set to be updated
---> Package pam_ldap.x86_64 0:185-5.el6 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch          Version             Repository         Size
================================================================================
Installing:
 nss-pam-ldapd        x86_64        0.7.5-3.el6         rhel6-beta        147 k
Installing for dependencies:
 nscd                 x86_64        2.12-1.2.el6        rhel6-beta        196 k
 pam_ldap             x86_64        185-5.el6           rhel6-beta         87 k

Transaction Summary
================================================================================
Install       3 Package(s)
Upgrade       0 Package(s)

Total download size: 430 k
Installed size: 771 k
Is this ok [y/N]: y
Downloading Packages:
(1/3): nscd-2.12-1.2.el6.x86_64.rpm                      | 196 kB     00:00     
(2/3): nss-pam-ldapd-0.7.5-3.el6.x86_64.rpm              | 147 kB     00:00     
(3/3): pam_ldap-185-5.el6.x86_64.rpm                     |  87 kB     00:00     
--------------------------------------------------------------------------------
Total                                            18 kB/s | 430 kB     00:24     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
*** glibc detected *** /usr/bin/python: malloc(): smallbin double linked list corrupted: 0x0000000003af2770 ***
Write failed: Broken pipe

Created attachment 430714 [details]
logfiles from machine where yum is misbehaving

here is the entire /var/log directory;  included are the anaconda.yum.log and the yum.log files which should answer your about which packages were installed.  I don't think there's much more apart from package databases etc.

Thanks for providing the log files.

Unfortunately, without a coredump it's going to be very hard to track this down.

Based on the error message, there's been some kind of corruption of the heap.

The error in comment #0: "malloc(): memory corruption" is emitted by line 4396 of malloc.c within the implementation of "malloc" whilst scanning through recently freed chunks of memory: one of the chunks has a corrupt-looking value for its size field.

The error in comment #4: "malloc(): smallbin double linked list corrupted:" is emitted by line 4341 of malloc.c, again within the implementation of "malloc": again, a bookkeeping field within a free chunk of memory has become corrupt.

So in both cases the bookkeeping fields that lurk between chunks of memory are getting corrupt.  This could indicate a dynamically-allocated block of memory being used after being freed, or a write beyond the bounds of a block of memory, or various other things going wrong.  It may be a reference counting issue: perhaps something is missing a Py_INCREF a PyObject* leading to the object being freed (however if that's the case, the object is likely to be actually allocated by the arena allocator, and that may contradict the size suggested by the smallbin message).

So there's likely a bug _somewhere_ in one of the DSOs within the python process that was running yum.  Figuring out where the bug is is likely to be very difficult.  The anaconda log you provided may give some clues as to the scope of the search.

This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release. It has
been denied for the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **

I noticed in the dmesg that this is a guest machine running on top of KVM:
e.g. this line:
  Booting paravirtualized kernel on KVM

This makes me suspect that this is another duplicate of bug 607650 (a hypervisor bug).

Do you remember what version of the hypervisor where you running on?

*** This bug has been marked as a duplicate of bug 607650 ***