Description of problem: While running an update, via yum, on a rawhide installation the following showed up on a screen: Cleanup : 2:texlive-amsmath-svn26389.2.13-14.fc19.noarch 1839/1963 *** Error in `/usr/bin/python': corrupted double-linked list: 0x0000000009b493d0 *** followed by a long backtrace (full backtrace attached) and terminated with 7faa88021000-7faa8c000000 ---p 0000Aborted (core dumped) Unfortunately a search found no produced core, even if limits allow for that, nor abrtd caught anything. Version-Release number of selected component (if applicable): python-2.7.3-14.fc19 Additional info: Clearly the whole yum transaction got screwed and even a followup with yum-complete-transaction cleaned up most of the resulting mess none of %posttran scripts were included in this.
Created attachment 693027 [details] a screen backtrace produced by python Around the time of this crash the following showed up in /var/log/messages: yum (15807) used greatest stack depth: 2192 bytes left I am afraid that this is the only additional information I was able to find anywhere.
(In reply to comment #1) > Around the time of this crash the following showed up in /var/log/messages: > > yum (15807) used greatest stack depth: 2192 bytes left Some googling told me that apparently this one is due to a kernel debugging option: CONFIG_DEBUG_STACK_USAGE and that the message is generally benign.
(In reply to comment #0) > *** Error in `/usr/bin/python': corrupted double-linked list: > 0x0000000009b493d0 *** Thanks for filing this bug. What's the output of: rpm -qa glibc Is this on "bare metal", or a guest on a hypervisor? The "corrupted double-linked list" message is from the implementation of malloc/free that's within glibc (aka the heap). The heap has somehow become corrupted. This could be due to various causes: (a) a double-free of an allocated block *somewhere* in the process (and thus hard to track down; valgrind could help) (b) a pointer in the heap's doubly-linked list of allocation areas could have been overwritten by a bug *somewhere* in the process (again, hard to track down. valgrind again could help) (d) a bug within glibc's heap implementation (d) a lower level issue e.g. with memory. I've seen issues like that with buggy hypervisors where memory in the guests occasionally got corrupted. All of the above are v fiddly to track down, alas. Reassigning to glibc in the hope of more insight.
(In reply to comment #3) > (In reply to comment #0) > > *** Error in `/usr/bin/python': corrupted double-linked list: > > 0x0000000009b493d0 *** > > Thanks for filing this bug. > > What's the output of: > rpm -qa glibc glibc-2.17-1.fc19.x86_64 glibc-2.17-1.fc19.i686 > Is this on "bare metal", or a guest on a hypervisor? This is a "bare metal". > (d) a lower level issue e.g. with memory. Although repeated runs of memtest never found anything I cannot absolutely discount that possibility. > All of the above are v fiddly to track down, alas. I realize that but unfortunately I was unable to collect any other information beyond reported. Not the issue is, so far, repeatable.
Could you please debug the problem and some backtraces to see what threads are doing at the time of the failure? Could you please confirm: Is the issue repeatable?
(In reply to comment #5) > Could you please debug the problem and some backtraces to see what threads > are doing at the time of the failure? Any suggestions how to do that in practice? I cannot trigger that at will. > Could you please confirm: Is the issue repeatable? So far this happened only ONCE (although over many years I have seen "corrupted double-linked list" on some occasions). As it can be seen from a quoted fragment this was a sizeable yum transaction and this corruption happened when processing step 1839 out of 1963. Something of that size happens when updates to various texlive packages show up.
(In reply to comment #6) > (In reply to comment #5) > > Could you please debug the problem and some backtraces to see what threads > > are doing at the time of the failure? > > Any suggestions how to do that in practice? I cannot trigger that at will. If you can't trigger it at will then this is of no use. What we really needed here was the core file which would have contained a lot of the information needed to start making informed guesses. > > Could you please confirm: Is the issue repeatable? > > So far this happened only ONCE (although over many years I have seen > "corrupted double-linked list" on some occasions). As it can be seen from a > quoted fragment > this was a sizeable yum transaction and this corruption happened when > processing step 1839 out of 1963. Something of that size happens when > updates to various texlive packages show up. If we can't reproduce this I'm going to have to close this as "not enough information." However, before we get there, what we need to do is double check that no core file was created. Can you use `find' to see if the core file was dropped anywhere you didn't expect? The core file is generally written into the programs current working directory. The core file would go a long way to helping and it's the only defence we have in cases like this. Cheers, Carlos.
(In reply to comment #7) > The core file would go a long way to helping and it's the only > defence we have in cases like this. Yes, it turns out that I actually do have that core file. It was on a file system not mounted by default and on the first search I missed it (and, yes, I know how to look for cores :-). Here is what gdb has to say with python-debuginfo and rpm-debuginfo installed: .... Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/bin/python /usr/bin/yum --exclude=gdm --exclude=grub2 update --skip-broken'. Program terminated with signal 6, Aborted. #0 0x00000031f6c35de5 in raise () from /lib64/libc.so.6 .... (gdb) bt #0 0x00000031f6c35de5 in raise () from /lib64/libc.so.6 #1 0x00000031f6c37598 in abort () from /lib64/libc.so.6 #2 0x00000031f6c766cb in __libc_message () from /lib64/libc.so.6 #3 0x00000031f6c7c846 in malloc_printerr () from /lib64/libc.so.6 #4 0x00000031f6c7f36a in _int_malloc () from /lib64/libc.so.6 #5 0x00000031f6c8063f in malloc () from /lib64/libc.so.6 #6 0x00000031f6c3834d in qsort_r () from /lib64/libc.so.6 #7 0x0000003ead415549 in headerSort (h=0xb5616f0) at header.c:221 #8 0x0000003ead41639e in headerImport (blob=<optimized out>, bsize=<optimized out>, flags=<optimized out>) at header.c:909 #9 0x0000003ead41cbae in rpmdbNextIterator (mi=0x8f356e0) at rpmdb.c:1895 #10 0x0000003ead41b8e6 in rpmdbGetHeaderAt (db=<optimized out>, offset=14070) at rpmdb.c:917 #11 0x0000003ead41d826 in rpmdbRemove (db=0x13833990, hdrNum=14070) at rpmdb.c:2386 #12 0x0000003ead42eef4 in rpmpsmStage (psm=0x3dbf, stage=15807) at psm.c:707 #13 0x000000001172ba50 in ?? () #14 0x0000000000000000 in ?? () Versions at the time of this crash: python-2.7.3-14.fc19.x86_64 rpm-4.11.0-0.beta1.3.fc19.x86_64 yum-3.4.3-57.fc19.noarch glibc-2.17-1.fc19.x86_64 (rpm got updated yesterday to rpm-4.11.0.1-1.fc19.x86_64). The original core file "weights" 375M. Even compressed with xz it comes to 32M. What do you want me to do with it? Try to attach to bugzilla?
Yes, please attach the compressed core file to the issue. Thanks.
Created attachment 693582 [details] core file, xz-compressed, dropped after "corrupted double-linked list" let's hope that this is not too big ...
(In reply to comment #8) > (In reply to comment #7) > > The core file would go a long way to helping and it's the only > > defence we have in cases like this. > > Yes, it turns out that I actually do have that core file. It was on a file > system not mounted by default and on the first search I missed it (and, yes, > I know how to look for cores :-). > > Here is what gdb has to say with python-debuginfo and rpm-debuginfo > installed: > > .... > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/bin/python /usr/bin/yum --exclude=gdm > --exclude=grub2 update --skip-broken'. > Program terminated with signal 6, Aborted. > #0 0x00000031f6c35de5 in raise () from /lib64/libc.so.6 > .... > > (gdb) bt > #0 0x00000031f6c35de5 in raise () from /lib64/libc.so.6 > #1 0x00000031f6c37598 in abort () from /lib64/libc.so.6 > #2 0x00000031f6c766cb in __libc_message () from /lib64/libc.so.6 > #3 0x00000031f6c7c846 in malloc_printerr () from /lib64/libc.so.6 > #4 0x00000031f6c7f36a in _int_malloc () from /lib64/libc.so.6 > #5 0x00000031f6c8063f in malloc () from /lib64/libc.so.6 > #6 0x00000031f6c3834d in qsort_r () from /lib64/libc.so.6 > #7 0x0000003ead415549 in headerSort (h=0xb5616f0) at header.c:221 > #8 0x0000003ead41639e in headerImport (blob=<optimized out>, > bsize=<optimized out>, flags=<optimized out>) at header.c:909 > #9 0x0000003ead41cbae in rpmdbNextIterator (mi=0x8f356e0) at rpmdb.c:1895 > #10 0x0000003ead41b8e6 in rpmdbGetHeaderAt (db=<optimized out>, offset=14070) > at rpmdb.c:917 > #11 0x0000003ead41d826 in rpmdbRemove (db=0x13833990, hdrNum=14070) > at rpmdb.c:2386 > #12 0x0000003ead42eef4 in rpmpsmStage (psm=0x3dbf, stage=15807) at psm.c:707 > #13 0x000000001172ba50 in ?? () > #14 0x0000000000000000 in ?? () > > Versions at the time of this crash: > python-2.7.3-14.fc19.x86_64 > rpm-4.11.0-0.beta1.3.fc19.x86_64 > yum-3.4.3-57.fc19.noarch > glibc-2.17-1.fc19.x86_64 > > (rpm got updated yesterday to rpm-4.11.0.1-1.fc19.x86_64). > > The original core file "weights" 375M. Even compressed with xz it comes to > 32M. What do you want me to do with it? Try to attach to bugzilla? CCing Panu for input from the rpm side: Panu, does the above sound like a possible new rpm bug?
(In reply to comment #11) > CCing Panu for input from the rpm side: Panu, does the above sound like a > possible new rpm bug? For the above traceback, seems unlikely. That path is a heavily excercised one and bugs there would be expected to cause things blowing up all over the place and is fairly regularly valgrind'ed by me. Random heap corruption from flaky seems more likely, but then you never *really* know...
I'd say that the above traceback is largely useless. The corruption could have happened anytime before this and we wouldn't even know it. The two likely causes I can see are: 1) Writing into a block after it is freed. 2) A buffer overrun in a memory block into an adjacent freed block. A double-free will have a different message and there's little chance of a double-free corrupting the linked list pointers. Based on the fact that it's so rare, I'd guess that this is a write-after-free rather than a buffer overrun. There's one way to find out - try replicating the transaction that caused the crash under valgrind and check if valgrind spews any warnings out. If valgrind does issue a useful warning, you have a clue regardless of whether yum crashes or not. If it doesn't and yum doesn't even crash, then I'd seriously consider (1) as the culprit since it depends on a race between the freeing context writing the linked list pointers and the writing context writing in data. The contexts could be two threads or within signal handlers, etc. (2) should always give a warning under valgrind.
(In reply to comment #13) > If valgrind does issue a useful > warning, you have a clue regardless of whether yum crashes or not. I have no idea if this is useful or not but I run a similar (not the same) yum operation under valgrind. It is slow! It took well over 12 hours to complete. This time did not crash. Still it collected things like that: ==12149== ERROR SUMMARY: 6 errors from 3 contexts (suppressed: 2 from 2) ==12149== ERROR SUMMARY: 6 errors from 3 contexts (suppressed: 2 from 2) ==12139== ERROR SUMMARY: 10 errors from 5 contexts (suppressed: 2 from 2) ==12139== ERROR SUMMARY: 10 errors from 5 contexts (suppressed: 2 from 2) ==12150== ERROR SUMMARY: 6 errors from 3 contexts (suppressed: 2 from 2) ==12150== ERROR SUMMARY: 6 errors from 3 contexts (suppressed: 2 from 2) ==12144== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 2 from 2) ==12144== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 2 from 2) ==12169== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 2 from 2) ==12169== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 2 from 2) ==17867== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 2 from 2) ==17867== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 2 from 2) ==17868== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 2 from 2) ==17868== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 2 from 2) ==2544== ERROR SUMMARY: 18162 errors from 4 contexts (suppressed: 2 from 2) ==2544== ERROR SUMMARY: 18162 errors from 4 contexts (suppressed: 2 from 2) and quite a number of things in this style: ==17849== definitely lost: 320 bytes in 2 blocks ==17849== indirectly lost: 200 bytes in 1 blocks ==17849== possibly lost: 31,047 bytes in 112 blocks ==17852== definitely lost: 320 bytes in 2 blocks ==17852== indirectly lost: 200 bytes in 1 blocks ==17852== possibly lost: 31,047 bytes in 112 blocks ==17867== possibly lost: 1,039,118 bytes in 2,980 blocks ==17868== possibly lost: 1,039,118 bytes in 2,980 blocks ==2544== definitely lost: 9,794 bytes in 103 blocks ==2544== indirectly lost: 600 bytes in 8 blocks ==2544== possibly lost: 16,458,507 bytes in 117,581 blocks and also a bunch of Syscall param ... points to uninitialised byte(s) with various things which stand for "..." looking, ahem, "interesting". If something helpful can be glimpsed from that then be my guest.
Created attachment 695166 [details] xz compressed output from valgrind watching yum updating texlive packages Expands to a file 60M in size. Edited out repeated valgrind reminders "TO DEBUG THIS PROCESS ..." and "TO CONTROL THIS PROCESS USING vgdb ...".
I've reduced the valgrind output to the relevant bits below by using the "ERROR SUMMARY" lines to look at only those proceses/threads that have errors. I have ignored the warnings in prelink and the one in libc with sendmsg(mmsg[0].msg_hdr) since the former is not relevant and the latter is a false alarm AFAICT. That leaves us with the warnings below, snipped to remove useless lines. ==2544== Thread 2: ==2544== Syscall param sendmsg(mmsg[0].msg_hdr) points to uninitialised byte(s) ==2544== at 0x31F6CF654B: sendmmsg (in /usr/lib64/libc-2.17.so) ==2544== by 0x31F940B4C6: __libc_res_nsend (in /usr/lib64/libresolv-2.17.so) ==2544== by 0x31F9408BEF: __libc_res_nquery (in /usr/lib64/libresolv-2.17.so) ==2544== by 0x31F94097C9: __libc_res_nsearch (in /usr/lib64/libresolv-2.17.so) ==2544== by 0x152F2AE7: _nss_dns_gethostbyname4_r (in /usr/lib64/libnss_dns-2.17.so) ==2544== by 0x31F6CDB59A: gaih_inet (in /usr/lib64/libc-2.17.so) ==2544== by 0x31F6CDF1F0: getaddrinfo (in /usr/lib64/libc-2.17.so) ==2544== by 0x38A9442853: ??? (in /usr/lib64/libcurl.so.4.3.0) ==2544== by 0x38A944BBC3: ??? (in /usr/lib64/libcurl.so.4.3.0) ==2544== by 0x38A9449772: ??? (in /usr/lib64/libcurl.so.4.3.0) ==2544== by 0x31F7807D14: start_thread (in /usr/lib64/libpthread-2.17.so) ==2544== by 0x31F6CF4C4C: clone (in /usr/lib64/libc-2.17.so) ==2544== Address 0x17ac4fc0 is on thread 2's stack ==2544== ==2544== Thread 1: ==2544== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==2544== at 0x31F6CF51DA: epoll_ctl (in /usr/lib64/libc-2.17.so) ==2544== by 0xEDC414F: ??? (in /usr/lib64/python2.7/lib-dynload/selectmodule.so) ==2544== by 0xEDC42A2: ??? (in /usr/lib64/python2.7/lib-dynload/selectmodule.so) ==2544== by 0x3213EDD200: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDCE70: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDDC3E: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDC232: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDDC3E: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDC232: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDDC3E: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDC232: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDCE70: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== Address 0x7feffe0a8 is on thread 1's stack ==2544== Updating : 2:texlive-base-2012-16.20130205_r29034.fc19.noarch 1/1938==2544== Syscall param pwrite64(buf) points to uninitialised byte(s) ==2544== at 0x31F780EA63: ??? (in /usr/lib64/libpthread-2.17.so) ==2544== by 0x31FAD5B218: __os_io (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD47086: ??? (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD47343: __memp_bhwrite (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD57661: __memp_sync_int (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FACEF236: __db_sync (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD00B6F: __db_sync_pp (in /usr/lib64/libdb-5.3.so) ==2544== by 0x35CDE133E0: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE19476: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE1DE69: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F15A: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F4FE: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== Address 0x319bee08 is 2,312 bytes inside a block of size 4,096 alloc'd ==2544== at 0x4A0887C: malloc (vg_replace_malloc.c:270) ==2544== by 0x31FAD588C4: __os_malloc (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD471B7: ??? (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD47343: __memp_bhwrite (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD57661: __memp_sync_int (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FACEF236: __db_sync (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD00B6F: __db_sync_pp (in /usr/lib64/libdb-5.3.so) ==2544== by 0x35CDE133E0: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE19476: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE1DE69: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F15A: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F4FE: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== Updating : 2:texlive-dcpic-svn19440.4.3.2-16.fc19.noarch 563/1938==2544== Conditional jump or move depends on uninitialised value(s) ==2544== at 0x31FAC4D8F4: __bam_stkrel (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAC3E41D: ??? (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FACF2856: __dbc_iput (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD022A0: __dbc_put_pp (in /usr/lib64/libdb-5.3.so) ==2544== by 0x35CDE135A4: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE19C0F: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE1DEA6: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F15A: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F4FE: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2FFD1: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE3F9A1: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE45D41: rpmtsRun (in /usr/lib64/librpm.so.3.1.0) ==2544== ==2544== Use --track-origins=yes to see where uninitialised values come from ==2544== ERROR SUMMARY: 18162 errors from 4 contexts (suppressed: 2 from 2) ==2544== ==2544== 2 errors in context 1 of 4: ==2544== Thread 2: ==2544== Syscall param ÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝÝ points to uninitialised byte(s) ==2544== at 0x31F6CF654B: sendmmsg (in /usr/lib64/libc-2.17.so) ==2544== by 0x31F940B4C6: __libc_res_nsend (in /usr/lib64/libresolv-2.17.so) ==2544== by 0x31F9408BEF: __libc_res_nquery (in /usr/lib64/libresolv-2.17.so) ==2544== by 0x31F94097C9: __libc_res_nsearch (in /usr/lib64/libresolv-2.17.so) ==2544== by 0x152F2AE7: ??? ==2544== by 0x31F6CDB59A: gaih_inet (in /usr/lib64/libc-2.17.so) ==2544== by 0x31F6CDF1F0: getaddrinfo (in /usr/lib64/libc-2.17.so) ==2544== by 0x38A9442853: ??? (in /usr/lib64/libcurl.so.4.3.0) ==2544== by 0x38A944BBC3: ??? (in /usr/lib64/libcurl.so.4.3.0) ==2544== by 0x38A9449772: ??? (in /usr/lib64/libcurl.so.4.3.0) ==2544== by 0x31F7807D14: start_thread (in /usr/lib64/libpthread-2.17.so) ==2544== by 0x31F6CF4C4C: clone (in /usr/lib64/libc-2.17.so) ==2544== Address 0x17ac4fc0 is on thread 2's stack ==2544== ==2544== ==2544== 4 errors in context 2 of 4: ==2544== Thread 1: ==2544== Conditional jump or move depends on uninitialised value(s) ==2544== at 0x31FAC4D8F4: __bam_stkrel (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAC3E41D: ??? (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FACF2856: __dbc_iput (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD022A0: __dbc_put_pp (in /usr/lib64/libdb-5.3.so) ==2544== by 0x35CDE135A4: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE19C0F: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE1DEA6: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F15A: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F4FE: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2FFD1: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE3F9A1: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE45D41: rpmtsRun (in /usr/lib64/librpm.so.3.1.0) ==2544== ==2544== ==2544== 969 errors in context 3 of 4: ==2544== Syscall param epoll_ctl(event) points to uninitialised byte(s) ==2544== at 0x31F6CF51DA: epoll_ctl (in /usr/lib64/libc-2.17.so) ==2544== by 0xEDC414F: ??? (in /usr/lib64/python2.7/lib-dynload/selectmodule.so) ==2544== by 0xEDC42A2: ??? (in /usr/lib64/python2.7/lib-dynload/selectmodule.so) ==2544== by 0x3213EDD200: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDCE70: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDDC3E: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDC232: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDDC3E: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDC232: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDDC3E: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDC232: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== by 0x3213EDCE70: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==2544== Address 0x7feffe0a8 is on thread 1's stack ==2544== ==2544== ==2544== 17187 errors in context 4 of 4: ==2544== Syscall param pwrite64(buf) points to uninitialised byte(s) ==2544== at 0x31F780EA63: ??? (in /usr/lib64/libpthread-2.17.so) ==2544== by 0x31FAD5B218: __os_io (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD47086: ??? (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD47343: __memp_bhwrite (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD57661: __memp_sync_int (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FACEF236: __db_sync (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD00B6F: __db_sync_pp (in /usr/lib64/libdb-5.3.so) ==2544== by 0x35CDE133E0: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE19476: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE1DE69: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F15A: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F4FE: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== Address 0x319bee08 is 2,312 bytes inside a block of size 4,096 alloc'd ==2544== at 0x4A0887C: malloc (vg_replace_malloc.c:270) ==2544== by 0x31FAD588C4: __os_malloc (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD471B7: ??? (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD47343: __memp_bhwrite (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD57661: __memp_sync_int (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FACEF236: __db_sync (in /usr/lib64/libdb-5.3.so) ==2544== by 0x31FAD00B6F: __db_sync_pp (in /usr/lib64/libdb-5.3.so) ==2544== by 0x35CDE133E0: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE19476: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE1DE69: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F15A: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== by 0x35CDE2F4FE: ??? (in /usr/lib64/librpm.so.3.1.0) ==2544== ==2544== ==2544== ERROR SUMMARY: 18162 errors from 4 contexts (suppressed: 2 from 2)
FWIW those "uninitialized bytes" errors originating from libdb (via rpm) are known false positives. Here's what BDB's build manual says about it: --enable-umrw Rational Software's Purify product and other run-time tools complain about uninitialized reads/writes of structure fields whose only purpose is padding, as well as when heap memory that was never initialized is written to disk. Specify the --enable-umrw argument during configuration to mask these errors. This argument should not be specified when configuring to build production binaries.
(In reply to comment #17) > FWIW those "uninitialized bytes" errors originating from libdb (via rpm) are > known false positives. OTOH these valgrind traces are not from a transaction which ended up with a "corrupted double-linked list" but only from one which was _similar_ in scope but completed after a loooong time. Provided with a faint hope that maybe somebody who knows what to look for will find there some clues.
(In reply to comment #18) > OTOH these valgrind traces are not from a transaction which ended up with a > "corrupted double-linked list" but only from one which was _similar_ in > scope but completed after a loooong time. Provided with a faint hope that > maybe somebody who knows what to look for will find there some clues. Like I had mentioned in comment 13, lack of a warning + successful completion means that it is highly likely to be the write-after-free scenario - I can't think of any other scenario that could trigger something like this. I don't think the race will trigger under valgrind, especially since it is already so rare without it. Also, this does not look like a glibc bug, so I think the component ought to be changed back unless there is a good enough reason to believe that it is.
I got again 247M of core from python after trying 'yum update ...' on rawhide with what looks like a significant number of packages. The whole thing segfaulted while resolving dependencies. This is with python-2.7.3-16.fc19.x86_64. Unfortunately right now ony python-debuginfo-2.7.3-25.fc19.x86_64 is available. With this installed I see: Program terminated with signal 11, Segmentation fault. #0 0x00007f8cc464d638 in ?? () from /lib64/libpython2.7.so.1.0 Missing separate debuginfos, use: debuginfo-install python-2.7.3-16.fc19.x86_64 (gdb) bt #0 0x00007f8cc464d638 in ?? () from /lib64/libpython2.7.so.1.0 #1 0x00007f8cc45cb70e in ?? () from /lib64/libpython2.7.so.1.0 #2 0x00007f8cc464d9e7 in ?? () from /lib64/libpython2.7.so.1.0 #3 0x00007f8cc464e5d7 in _PyObject_GC_Malloc () from /lib64/libpython2.7.so.1.0 #4 0x00007f8cc464e61d in _PyObject_GC_New () from /lib64/libpython2.7.so.1.0 #5 0x00007f8cc45b75ec in PyList_New () from /lib64/libpython2.7.so.1.0 #6 0x00007f8cc461da29 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #7 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #8 0x00007f8cc45b12e0 in ?? () from /lib64/libpython2.7.so.1.0 #9 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #10 0x00007f8cc459c085 in ?? () from /lib64/libpython2.7.so.1.0 #11 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #12 0x00007f8cc461df9c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #13 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #14 0x00007f8cc45b12e0 in ?? () from /lib64/libpython2.7.so.1.0 #15 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #16 0x00007f8cc459c085 in ?? () from /lib64/libpython2.7.so.1.0 #17 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #18 0x00007f8cc461df9c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #19 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #20 0x00007f8cc45b12e0 in ?? () from /lib64/libpython2.7.so.1.0 #21 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #22 0x00007f8cc459c085 in ?? () from /lib64/libpython2.7.so.1.0 #23 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #24 0x00007f8cc461df9c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #25 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #26 0x00007f8cc45b13bd in ?? () from /lib64/libpython2.7.so.1.0 #27 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #28 0x00007f8cc459c085 in ?? () from /lib64/libpython2.7.so.1.0 #29 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #30 0x00007f8cc461df9c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #31 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #32 0x00007f8cc45b12e0 in ?? () from /lib64/libpython2.7.so.1.0 #33 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #34 0x00007f8cc459c085 in ?? () from /lib64/libpython2.7.so.1.0 #35 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #36 0x00007f8cc45e0e47 in ?? () from /lib64/libpython2.7.so.1.0 #37 0x00007f8cc45e074f in ?? () from /lib64/libpython2.7.so.1.0 #38 0x00007f8cc458da73 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #39 0x00007f8cc461df9c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #40 0x00007f8cc46200cc in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #41 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #42 0x00007f8cc4620029 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #43 0x00007f8cc46200cc in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #44 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #45 0x00007f8cc4620029 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #46 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #47 0x00007f8cc4620029 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #48 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #49 0x00007f8cc4620029 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #50 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #51 0x00007f8cc4620029 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #52 0x00007f8cc462153d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #53 0x00007f8cc4621642 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 #54 0x00007f8cc463a3df in ?? () from /lib64/libpython2.7.so.1.0 #55 0x00007f8cc463b48e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 #56 0x00007f8cc463c5e7 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 #57 0x00007f8cc464cf87 in Py_Main () from /lib64/libpython2.7.so.1.0 #58 0x0000003780421b75 in __libc_start_main () from /lib64/libc.so.6 #59 0x0000000000400721 in _start () The second attempt of the same at least resolved dependencies succesfully. A transaction summary looks like this: Install 3 Packages (+4 Dependent packages) Upgrade 1912 Packages Remove 2 Packages Skipped (dependency problems) 87 Packages
I've switched the component back to python. I don't see any evidence that this is a glibc issue. Please feel free to keep us in the loop if you need any questions answered.
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle. Changing version to '19'. (As we did not run this process for some time, it could affect also pre-Fedora 19 development cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.) More information and reason for this action is here: https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
This message is a notice that Fedora 19 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 19. It is Fedora's policy to close all bug reports from releases that are no longer maintained. Approximately 4 (four) weeks from now this bug will be closed as EOL if it remains open with a Fedora 'version' of '19'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 19 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
It's back (or still there) on Fedora 23 with python3 and dnf: [root@localhost ~]# /usr/bin/dnf repoquery --quiet --queryformat '%{NAME}_|-%{EPOCH}:%{VERSION}_|-%{RELEASE}_|-%{ARCH}_|-%{REPOID}' --whatprovides 'unzip tar nano' *** Error in `/usr/bin/python3': corrupted double-linked list: 0x000055b0e9ef1110 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x77da5)[0x7f1a245fcda5] /lib64/libc.so.6(+0x7ee23)[0x7f1a24603e23] /lib64/libc.so.6(+0x819a6)[0x7f1a246069a6] /lib64/libc.so.6(__libc_malloc+0x50)[0x7f1a24608950] /lib64/libc.so.6(realloc+0x368)[0x7f1a24609198] /lib64/libpython3.4m.so.1.0(+0x1645e9)[0x7f1a253d05e9] /lib64/libpython3.4m.so.1.0(+0x1687ca)[0x7f1a253d47ca] /lib64/libpython3.4m.so.1.0(+0x16e423)[0x7f1a253da423] /lib64/libpython3.4m.so.1.0(+0x170127)[0x7f1a253dc127] /lib64/libpython3.4m.so.1.0(+0x170a8f)[0x7f1a253dca8f] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x82f1)[0x7f1a25386651] /lib64/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x8c6)[0x7f1a25387636] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x66cf)[0x7f1a25384a2f] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x7aeb)[0x7f1a25385e4b] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x7aeb)[0x7f1a25385e4b] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x7aeb)[0x7f1a25385e4b] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x7aeb)[0x7f1a25385e4b] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x7aeb)[0x7f1a25385e4b] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x7aeb)[0x7f1a25385e4b] /lib64/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x8c6)[0x7f1a25387636] /lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x66cf)[0x7f1a25384a2f] /lib64/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x8c6)[0x7f1a25387636] /lib64/libpython3.4m.so.1.0(PyEval_EvalCode+0x1b)[0x7f1a253876db] /lib64/libpython3.4m.so.1.0(+0x137954)[0x7f1a253a3954] /lib64/libpython3.4m.so.1.0(PyRun_FileExFlags+0x95)[0x7f1a253a5b95] /lib64/libpython3.4m.so.1.0(PyRun_SimpleFileExFlags+0x113)[0x7f1a253a6c13] /lib64/libpython3.4m.so.1.0(Py_Main+0xdc4)[0x7f1a253bd964] /usr/bin/python3(main+0x167)[0x55b0e8057be7] /lib64/libc.so.6(__libc_start_main+0xf0)[0x7f1a245a5580] /usr/bin/python3(_start+0x29)[0x55b0e8057c89]
On fedora 22: [rolf.fokkens@f22builder ~]$ sudo -i -bash-4.3# /usr/bin/dnf repoquery --quiet --queryformat '%{NAME}_|-%{EPOCH}:%{VERSION}_|-%{RELEASE}_|-%{ARCH}_|-%{REPOID}' --whatprovides 'unzip tar nano' *** Error in `/usr/bin/python': malloc(): smallbin double linked list corrupted: 0x000000000205fa90 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x77a8d)[0x7fcf558cba8d] /lib64/libc.so.6(+0x81838)[0x7fcf558d5838] /lib64/libc.so.6(__libc_calloc+0xd4)[0x7fcf558d8d04] /lib64/libsolv.so.0(solv_calloc+0x1b)[0x7fcf42ca5cbb] /lib64/libhawkey.so.2(hy_query_apply+0x1598)[0x7fcf42ef6f48] /lib64/libhawkey.so.2(hy_query_run_set+0x9)[0x7fcf42ef7a39] /usr/lib64/python2.7/site-packages/hawkey/_hawkeymodule.so(+0xdc12)[0x7fcf4310fc12] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b37)[0x7fcf56620c17] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x834)[0x7fcf566226b4] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54e6)[0x7fcf566215c6] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x834)[0x7fcf566226b4] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54e6)[0x7fcf566215c6] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x834)[0x7fcf566226b4] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54e6)[0x7fcf566215c6] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x834)[0x7fcf566226b4] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54e6)[0x7fcf566215c6] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x834)[0x7fcf566226b4] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54e6)[0x7fcf566215c6] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x834)[0x7fcf566226b4] /lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54e6)[0x7fcf566215c6] /lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x834)[0x7fcf566226b4] /lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x19)[0x7fcf566227d9] /lib64/libpython2.7.so.1.0(+0xfcbdf)[0x7fcf5663bbdf] /lib64/libpython2.7.so.1.0(PyRun_FileExFlags+0x72)[0x7fcf5663cdb2] /lib64/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xe7)[0x7fcf5663dfc7] /lib64/libpython2.7.so.1.0(Py_Main+0xcc1)[0x7fcf566501e1] /lib64/libc.so.6(__libc_start_main+0xf0)[0x7fcf55874700] /usr/bin/python(_start+0x29)[0x400729]
Could be dnf specific, because it's happening both on Python3.4 and Python2.7.
This message is a reminder that Fedora 23 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 23. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '23'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 23 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.