Hide Forgot
Description of problem: libvirtd virFDStreamEvent references struct virFDStreamData which is deallocated by virFDStreamClose called from remoteStreamEvent callback, when qemu guest shuts down. The net result is, occasional crash or hang. If run under mudflap memory checker, you always get warning messages. Version-Release number of selected component (if applicable): 0.9.2 How reproducible: Always Steps to Reproduce: 1. virsh # start domain1 --console and leave it open. 2. On qemu guest vnc console, type halt as root. 3. quit virsh after guest is down. Actual results: occasional libvirtd crash or hang Expected results: Additional info: mudflap message: ******* mudflap violation 487 (check/write): time=1308902706.288811 ptr=0x22e8030 size=44 pc=0x7fade326a7d9 location=`fdstream.c:161:5 (virFDStreamEvent)' /usr/lib/libmudflapth.so.0(__mf_check+0x59) [0x7fade326a7d9] /home/kanda/lib/libvirt.so.0 [0x7fade553dc54] /home/kanda/lib/libvirt.so.0 [0x7fade53e2885] Nearby object 1: checked region begins 192B before and ends 149B before mudflap object 0x22b2a60: name=`malloc region' bounds=[0x22e80f0,0x22e8100] size=17 area=heap check=1r/0w liveness=1 alloc time=1308902580.828627 pc=0x7fade3269e69 thread=3863275456 /usr/lib/libmudflapth.so.0(__mf_register+0x59) [0x7fade3269e69] /usr/lib/libmudflapth.so.0(__real_malloc+0xfe) [0x7fade326b68e] /lib/libc.so.6(__strdup+0x22) [0x7fade274ad92] /usr/lib/libhal.so.1(libhal_device_get_property_string+0x159) [0x7fade5182319] Nearby object 2: checked region begins 0B into and ends 43B into mudflap dead object 0x22e6ad0: name=`calloc region' bounds=[0x22e8030,0x22e809f] size=112 area=heap-init check=15r/1w liveness=16 alloc time=1308902647.420223 pc=0x7fade3269e69 thread=1123649872 /usr/lib/libmudflapth.so.0(__mf_register+0x59) [0x7fade3269e69] /usr/lib/libmudflapth.so.0(__wrap_calloc+0x11c) [0x7fade326bc6c] /home/kanda/lib/libvirt.so.0(virAlloc+0x23) [0x7fade53f8ea3] /home/kanda/lib/libvirt.so.0 [0x7fade553c976] dealloc time=1308902706.284764 pc=0x7fade326992c thread=1106864464 number of nearby objects: 2 ******* mudflap violation 488 (check/read): time=1308902706.289175 ptr=0x22e8030 size=40 pc=0x7fade326a7d9 location=`fdstream.c:162:9 (virFDStreamEvent)' /usr/lib/libmudflapth.so.0(__mf_check+0x59) [0x7fade326a7d9] /home/kanda/lib/libvirt.so.0 [0x7fade553dc36] /home/kanda/lib/libvirt.so.0 [0x7fade53e2885] Nearby object 1: checked region begins 192B before and ends 153B before mudflap object 0x22b2a60: name=`malloc region' Nearby object 2: checked region begins 0B into and ends 39B into mudflap dead object 0x22e6ad0: name=`calloc region' number of nearby objects: 2 ******* gdb session when libvirtd hangs: (gdb) bt #0 0x00007faf9ab6f594 in __lll_robust_lock_wait () from /lib/libpthread.so.0 #1 0x00007faf9ab6ac26 in _L_robust_lock_494 () from /lib/libpthread.so.0 #2 0x00007faf9ab6a74f in pthread_mutex_lock () from /lib/libpthread.so.0 #3 0x00007faf9d478a8a in virFDStreamEvent (watch=<value optimized out>, fd=<value optimized out>, events=13, opaque=0x284a480) at fdstream.c:160 #4 0x00007faf9d31d885 in virEventPollRunOnce () at util/event_poll.c:482 #5 0x00007faf9d31abb5 in virEventRunDefaultImpl () at util/event.c:191 #6 0x0000000000430d0e in qemudOneLoop () at libvirtd.c:2277 #7 0x0000000000431f9f in qemudRunLoop (opaque=0x1f1aca0) at libvirtd.c:2387 #8 0x00007faf9b1b31fa in ?? () from /usr/lib/libmudflapth.so.0 #9 0x00007faf9ab68fc7 in start_thread () from /lib/libpthread.so.0 #10 0x00007faf9a6da5ad in clone () from /lib/libc.so.6 (gdb) frame 3 #3 0x00007faf9d478a8a in virFDStreamEvent (watch=<value optimized out>, fd=<value optimized out>, events=13, opaque=0x284a480) at fdstream.c:160 160 virMutexLock(&fdst->lock); (gdb) info locals fdst = (struct virFDStreamData *) 0x1d5e960 cbopaque = (void *) 0x221fcc0 ff = (virFreeCallback) 0 fdst is a junk. (gdb) p *fdst $1 = {fd = 0, errfd = 0, cmd = 0xffffffff, offset = 140392198196841, length = 1308898643, watch = 802838, cbRemoved = 0, dispatching = 30464512, cb = 0x4, opaque = 0x427f0950, ff = 0x1, lock = {lock = {__data = { __lock = -1692776148, __count = 32687, __owner = 1308898643, __nusers = 0, __kind = 803088, __spins = 0, __list = { __prev = 0x1d86ab0, __next = 0x4}}, __size = ",I\032\233・ｯ\177\000\000S5\004N\000\000\000\000\020A\f\000\000\000\000\000・ｰj・禄001\000\000\000\000\004\000\000\000\000\000\000", __align = 140392198195500}}} (gdb) x/20gx 0x1d5e960 0x1d5e960: 0x0000000000000000 0x00000000ffffffff 0x1d5e970: 0x00007faf9b1a4e69 0x000000004e043553 0x1d5e980: 0x00000000000c4016 0x0000000001d0da00 0x1d5e990: 0x0000000000000004 0x00000000427f0950 0x1d5e9a0: 0x0000000000000001 0x00007faf9b1a492c 0x1d5e9b0: 0x000000004e043553 0x00000000000c4110 0x1d5e9c0: 0x0000000001d86ab0 0x0000000000000004 0x1d5e9d0: 0x00000000427f0950 0x0000000000000021 0x1d5e9e0: 0x0000000001ed5ab0 0x0000000001e4be10 0x1d5e9f0: 0x0000000000000020 0x00000000000000a0 another gdb session with stream->privateData assertion added: modified virFDStreamEvent source code virMutexUnlock(&fdst->lock); cb(stream, events, cbopaque); if (!stream->privateData) abort(); // BUG virMutexLock(&fdst->lock); fdst->dispatching = 0; Caught abort signal dumping internal log buffer: ====== start of log ===== 17:44:06.839: 29846: debug : remoteStreamEvent:88 : st=0x2c3b610 events=13 17:44:06.839: 29846: debug : remoteStreamHandleRead:559 : stream=0x2c5a760 17:44:06.840: 29846: debug : virStreamRecv:12631 : stream=0x2c3b610, data=0x7f70ac0ac200, nbytes=262120 17:44:06.840: 29846: debug : remoteSendStreamData:620 : client=0x7f70ac022aa0 stream=0x2c5a760 data=0x7f70ac0ac200 len=0 17:44:06.841: 29846: debug : virEventPollUpdateHandle:144 : Update handle w=7 e=3 17:44:06.842: 29846: debug : virEventPollInterruptLocked:686 : Skip interrupt, 1 1102911824 17:44:06.842: 29846: debug : virStreamEventRemoveCallback:12969 : stream=0x2c3b610 17:44:06.842: 29846: debug : virEventPollRemoveHandle:171 : Remove handle w=9 17:44:06.843: 29846: debug : virEventPollRemoveHandle:184 : mark delete 8 13 17:44:06.843: 29846: debug : virEventPollInterruptLocked:686 : Skip interrupt, 1 1102911824 17:44:06.844: 29846: debug : virStreamAbort:13053 : stream=0x2c3b610 17:44:06.844: 29846: debug : virFDStreamClose:221 : st=0x2c3b610 17:44:06.844: 29846: debug : remoteSerializeError:132 : prog=536903814 ver=1 proc=201 type=3 serial=6, msg=stream had I/O failure ... ====== end of log ===== Aborted (core dumped) (gdb) bt #0 0x00007f70b5deded5 in raise () from /lib/libc.so.6 #1 0x00007f70b5def385 in abort () from /lib/libc.so.6 #2 0x00007f70b8c2ad65 in virFDStreamEvent (watch=<value optimized out>, fd=<value optimized out>, events=13, opaque=0x2c3b610) at fdstream.c:160 #3 0x00007f70b8ace885 in virEventPollRunOnce () at util/event_poll.c:482 #4 0x00007f70b8acbbb5 in virEventRunDefaultImpl () at util/event.c:191 #5 0x0000000000430d0e in qemudOneLoop () at libvirtd.c:2277 #6 0x0000000000431f9f in qemudRunLoop (opaque=0x2259ca0) at libvirtd.c:2387 #7 0x00007f70b69641fa in ?? () from /usr/lib/libmudflapth.so.0 #8 0x00007f70b6319fc7 in start_thread () from /lib/libpthread.so.0 #9 0x00007f70b5e8b5ad in clone () from /lib/libc.so.6 #10 0x0000000000000000 in ?? () (gdb) frame 2 #2 0x00007f70b8c2ad65 in virFDStreamEvent (watch=<value optimized out>, fd=<value optimized out>, events=13, opaque=0x2c3b610) at fdstream.c:160 160 if (!stream->privateData) abort(); // BUG (gdb) info locals fdst = (struct virFDStreamData *) 0x2b7e320 cbopaque = (void *) 0x7f70ac022aa0 ff = (virFreeCallback) 0 (gdb) p stream $1 = 0 (gdb) thread apply all bt Thread 7 (process 29851): #0 0x00007f70b631dd29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f70b8b03a23 in virCondWait (c=0x2259ccc, m=0x80) at util/threads-pthread.c:117 #2 0x0000000000434b33 in qemudWorker (data=0x23af818) at libvirtd.c:1597 #3 0x00007f70b69641fa in ?? () from /usr/lib/libmudflapth.so.0 #4 0x00007f70b6319fc7 in start_thread () from /lib/libpthread.so.0 #5 0x00007f70b5e8b5ad in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 6 (process 29849): #0 0x00007f70b631dd29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f70b8b03a23 in virCondWait (c=0x2259ccc, m=0x80) at util/threads-pthread.c:117 #2 0x0000000000434b33 in qemudWorker (data=0x23af7e8) at libvirtd.c:1597 #3 0x00007f70b69641fa in ?? () from /usr/lib/libmudflapth.so.0 #4 0x00007f70b6319fc7 in start_thread () from /lib/libpthread.so.0 #5 0x00007f70b5e8b5ad in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 5 (process 29852): #0 0x00007f70b631dd29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f70b8b03a23 in virCondWait (c=0x2259ccc, m=0x80) at util/threads-pthread.c:117 #2 0x0000000000434b33 in qemudWorker (data=0x23af830) at libvirtd.c:1597 #3 0x00007f70b69641fa in ?? () from /usr/lib/libmudflapth.so.0 #4 0x00007f70b6319fc7 in start_thread () from /lib/libpthread.so.0 #5 0x00007f70b5e8b5ad in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 4 (process 29850): #0 0x00007f70b631dd29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f70b8b03a23 in virCondWait (c=0x2259ccc, m=0x80) at util/threads-pthread.c:117 #2 0x0000000000434b33 in qemudWorker (data=0x23af800) at libvirtd.c:1597 #3 0x00007f70b69641fa in ?? () from /usr/lib/libmudflapth.so.0 #4 0x00007f70b6319fc7 in start_thread () from /lib/libpthread.so.0 #5 0x00007f70b5e8b5ad in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 3 (process 29847): #0 0x00007f70b631dd29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f70b8b03a23 in virCondWait (c=0x2259ccc, m=0x80) at util/threads-pthread.c:117 #2 0x0000000000434b33 in qemudWorker (data=0x23af7d0) at libvirtd.c:1597 #3 0x00007f70b69641fa in ?? () from /usr/lib/libmudflapth.so.0 #4 0x00007f70b6319fc7 in start_thread () from /lib/libpthread.so.0 #5 0x00007f70b5e8b5ad in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 2 (process 29845): #0 0x00007f70b631a715 in pthread_join () from /lib/libpthread.so.0 #1 0x000000000043e8bd in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:3418 Thread 1 (process 29846): #0 0x00007f70b5deded5 in raise () from /lib/libc.so.6 #1 0x00007f70b5def385 in abort () from /lib/libc.so.6 #2 0x00007f70b8c2ad65 in virFDStreamEvent (watch=<value optimized out>, fd=<value optimized out>, events=13, opaque=0x2c3b610) at fdstream.c:160 #3 0x00007f70b8ace885 in virEventPollRunOnce () at util/event_poll.c:482 #4 0x00007f70b8acbbb5 in virEventRunDefaultImpl () at util/event.c:191 #5 0x0000000000430d0e in qemudOneLoop () at libvirtd.c:2277 #6 0x0000000000431f9f in qemudRunLoop (opaque=0x2259ca0) at libvirtd.c:2387 #7 0x00007f70b69641fa in ?? () from /usr/lib/libmudflapth.so.0 #8 0x00007f70b6319fc7 in start_thread () from /lib/libpthread.so.0 #9 0x00007f70b5e8b5ad in clone () from /lib/libc.so.6
Created attachment 510294 [details] Delay stream free when aborting from a callback
This fix is committed upstream as commit d97093437f59223e1bc57ad39762b0f7521e856a Author: Daniel P. Berrange <berrange> Date: Tue Jun 28 15:44:22 2011 +0100 Fix crash when aborting a stream from a I/O callback If a callback being invoked from a stream issues a virStreamAbort operation, the stream data will be free'd but the callback will then still try to use this. Delay free'ing of the stream data when a callback is dispatching * src/fdstream.c: Delay stream free when callback is active
Putting in POST per comment 2.
So that should be fixed since libvirt-0.9.3-1.el6 was built. Updating with a more recent build Daniel
I retest this bug with libvirt-0.9.2-1.el6.x86_64, after tried 16 times, when did step 3: 3. quit virsh after guest is down got some output msg(a), after then, i tried several times again and got other output msg(b). But libvirtd didn't crash or hang. So please help to check these situations. Thanks! All output pasted as following: output msg(b): virsh # start rhel6-x86_64 --console Domain rhel6-x86_64 started Connected to domain rhel6-x86_64 Escape character is ^] �Segmentation fault (core dumped) output msg(a) : virsh # start rhel6-x86_64 --console Domain rhel6-x86_64 started Connected to domain rhel6-x86_64 Escape character is ^] �*** glibc detected *** virsh: free(): invalid next size (fast): 0x0000000001fe32f0 *** ======= Backtrace: ========= /lib64/libc.so.6[0x355a275716] /usr/lib64/libvirt.so.0(virFree+0x29)[0x7f22a550ca89] /usr/lib64/libvirt.so.0(virLogMessage+0x1f1)[0x7f22a5509331] /usr/lib64/libvirt.so.0(+0x375f5)[0x7f22a55025f5] /usr/lib64/libvirt.so.0(+0x37fa6)[0x7f22a5502fa6] /usr/lib64/libvirt.so.0(virEventRunDefaultImpl+0x37)[0x7f22a5501ee7] virsh[0x408b1c] virsh[0x412463] virsh[0x41264d] virsh[0x4103f1] virsh[0x41d559] /lib64/libc.so.6(__libc_start_main+0xfd)[0x355a21ec9d] virsh[0x408819] ======= Memory map: ======== 00400000-00435000 r-xp 00000000 08:01 273166 /usr/bin/virsh 00635000-00636000 rw-p 00035000 08:01 273166 /usr/bin/virsh 01fb4000-02044000 rw-p 00000000 00:00 0 [heap] 3559a00000-3559a20000 r-xp 00000000 08:01 2359716 /lib64/ld-2.12.so 3559c1f000-3559c20000 r--p 0001f000 08:01 2359716 /lib64/ld-2.12.so 3559c20000-3559c21000 rw-p 00020000 08:01 2359716 /lib64/ld-2.12.so 3559c21000-3559c22000 rw-p 00000000 00:00 0 3559e00000-3559e3a000 r-xp 00000000 08:01 2359770 /lib64/libreadline.so.6.0 3559e3a000-355a03a000 ---p 0003a000 08:01 2359770 /lib64/libreadline.so.6.0 355a03a000-355a042000 rw-p 0003a000 08:01 2359770 /lib64/libreadline.so.6.0 355a042000-355a043000 rw-p 00000000 00:00 0 355a200000-355a387000 r-xp 00000000 08:01 2359717 /lib64/libc-2.12.so 355a387000-355a587000 ---p 00187000 08:01 2359717 /lib64/libc-2.12.so 355a587000-355a58b000 r--p 00187000 08:01 2359717 /lib64/libc-2.12.so 355a58b000-355a58c000 rw-p 0018b000 08:01 2359717 /lib64/libc-2.12.so 355a58c000-355a591000 rw-p 00000000 00:00 0 355a600000-355a602000 r-xp 00000000 08:01 2359719 /lib64/libdl-2.12.so 355a602000-355a802000 ---p 00002000 08:01 2359719 /lib64/libdl-2.12.so 355a802000-355a803000 r--p 00002000 08:01 2359719 /lib64/libdl-2.12.so 355a803000-355a804000 rw-p 00003000 08:01 2359719 /lib64/libdl-2.12.so 355aa00000-355aa17000 r-xp 00000000 08:01 2359725 /lib64/libpthread-2.12.so 355aa17000-355ac17000 ---p 00017000 08:01 2359725 /lib64/libpthread-2.12.so 355ac17000-355ac18000 r--p 00017000 08:01 2359725 /lib64/libpthread-2.12.so 355ac18000-355ac19000 rw-p 00018000 08:01 2359725 /lib64/libpthread-2.12.so 355ac19000-355ac1d000 rw-p 00000000 00:00 0 355ae00000-355ae83000 r-xp 00000000 08:01 2359740 /lib64/libm-2.12.so 355ae83000-355b082000 ---p 00083000 08:01 2359740 /lib64/libm-2.12.so 355b082000-355b083000 r--p 00082000 08:01 2359740 /lib64/libm-2.12.so 355b083000-355b084000 rw-p 00083000 08:01 2359740 /lib64/libm-2.12.so 355b200000-355b207000 r-xp 00000000 08:01 2359736 /lib64/librt-2.12.so 355b207000-355b406000 ---p 00007000 08:01 2359736 /lib64/librt-2.12.so 355b406000-355b407000 r--p 00006000 08:01 2359736 /lib64/librt-2.12.so 355b407000-355b408000 rw-p 00007000 08:01 2359736 /lib64/librt-2.12.so 355b600000-355b61d000 r-xp 00000000 08:01 2359722 /lib64/libselinux.so.1 355b61d000-355b81c000 ---p 0001d000 08:01 2359722 /lib64/libselinux.so.1 355b81c000-355b81d000 r--p 0001c000 08:01 2359722 /lib64/libselinux.so.1 355b81d000-355b81e000 rw-p 0001d000 08:01 2359722 /lib64/libselinux.so.1 355b81e000-355b81f000 rw-p 00000000 00:00 0 355ba00000-355ba15000 r-xp 00000000 08:01 2359742 /lib64/libz.so.1.2.3 355ba15000-355bc14000 ---p 00015000 08:01 2359742 /lib64/libz.so.1.2.3 355bc14000-355bc15000 rw-p 00014000 08:01 2359742 /lib64/libz.so.1.2.3 355be00000-355be16000 r-xp 00000000 08:01 2359721 /lib64/libresolv-2.12.so 355be16000-355c016000 ---p 00016000 08:01 2359721 /lib64/libresolv-2.12.so 355c016000-355c017000 r--p 00016000 08:01 2359721 /lib64/libresolv-2.12.so 355c017000-355c018000 rw-p 00017000 08:01 2359721 /lib64/libresolv-2.12.so 355c018000-355c01a000 rw-p 00000000 00:00 0 355da00000-355da08000 r-xp 00000000 08:01 277390 /usr/lib64/libnuma.so.1 355da08000-355dc07000 ---p 00008000 08:01 277390 /usr/lib64/libnuma.so.1 355dc07000-355dc08000 rw-p 00007000 08:01 277390 /usr/lib64/libnuma.so.1 355e200000-355e23c000 r-xp 00000000 08:01 2359350 /lib64/libsepol.so.1 355e23c000-355e43b000 ---p 0003c000 08:01 2359350 /lib64/libsepol.so.1 355e43b000-355e43c000 rw-p 0003b000 08:01 2359350 /lib64/libsepol.so.1 3560600000-3560747000 r-xp 00000000 08:01 283844 /usr/lib64/libxml2.so.2.7.6 3560747000-3560946000 ---p 00147000 08:01 283844 /usr/lib64/libxml2.so.2.7.6 3560946000-3560950000 rw-p 00146000 08:01 283844 /usr/lib64/libxml2.so.2.7.6 3560950000-3560951000 rw-p 00000000 00:00 0 3564600000-3564635000 r-xp 00000000 08:01 284011 /usr/lib64/libpcap.so.1.0.0 3564635000-3564834000 ---p 00035000 08:01 284011 /usr/lib64/libpcap.so.1.0.0 3564834000-3564837000 rw-p 00034000 08:01 284011 /usr/lib64/libpcap.so.1.0.0 3564a00000-3564a32000 r-xp 00000000 08:01 2359746 /lib64/libidn.so.11.6.1 3564a32000-3564c31000 ---p 00032000 08:01 2359746 /lib64/libidn.so.11.6.1 3564c31000-3564c32000 rw-p 00031000 08:01 2359746 /lib64/libidn.so.11.6.1 3564e00000-3564e25000 r-xp 00000000 08:01 2359764 /lib64/libdevmapper.so.1.02 3564e25000-3565024000 ---p 00025000 08:01 2359764 /lib64/libdevmapper.so.1.02 3565024000-3565026000 rw-p 00024000 08:01 2359764 /lib64/libdevmapper.so.1.02 3565200000-3565203000 r-xp 00000000 08:01 2359726 /lib64/libcom_err.so.2.1 3565203000-3565402000 ---p 00003000 08:01 2359726 /lib64/libcom_err.so.2.1 3565402000-3565403000 rw-p 00002000 08:01 2359726 /lib64/libcom_err.so.2.1 3565600000-356565d000 r-xp 00000000 08:01 2359729 /lib64/libfreebl3.so 356565d000-356585c000 ---p 0005d000 08:01 2359729 /lib64/libfreebl3.so 356585c000-356585e000 rw-p 0005c000 08:01 2359729 /lib64/libfreebl3.so 356585e000-3565862000 rw-p 00000000 00:00 0 3565a00000-3565a07000 r-xp 00000000 08:01 2359730 /lib64/libcrypt-2.12.so 3565a07000-3565c07000 ---p 00007000 08:01 2359730 /lib64/libcrypt-2.12.so 3565c07000-3565c08000 r--p 00007000 08:01 2359730 /lib64/libcrypt-2.12.so 3565c08000-3565c09000 rw-p 00008000 08:01 2359730 /lib64/libcrypt-2.12.so 3565c09000-3565c37000 rw-p 00000000 00:00 0 3565e00000-3565e4d000 r-xp 00000000 08:01 2359769 /lib64/libnl.so.1.1 3565e4d000-356604d000 ---p 0004d000 08:01 2359769 /lib64/libnl.so.1.1 356604d000-3566052000 rw-p 0004d000 08:01 2359769 /lib64/libnl.so.1.1 3566200000-3566204000 r-xp 00000000 08:01 2359650 /lib64/libcap-ng.so.0.0.0 3566204000-3566403000 ---p 00004000 08:01 2359650 /lib64/libcap-ng.so.0.0.0 3566403000-3566404000 r--p 00003000 08:01 2359650 /lib64/libcap-ng.so.0.0.0 3566404000-3566405000 rw-p 00004000 08:01 2359650 /lib64/libcap-ng.so.0.0.0 3566600000-3566607000 r-xp 00000000 08:01 274092 /usr/lib64/libyajl.so.1.0.7 3566607000-3566806000 ---p 00007000 08:01 274092 /usr/lib64/libyajl.so.1.0.7 3566806000-3566807000 rw-p 00006000 08:01 274092 /usr/lib64/libyajl.so.1.0.7 3566a00000-3566a0e000 r-xp 00000000 08:01 2359757 /lib64/libudev.so.0.5.1 3566a0e000-3566c0d000 ---p 0000e000 08:01 2359757 /lib64/libudev.so.0.5.1 3566c0d000-3566c0e000 r--p 0000d000 08:01 2359757 /lib64/libudev.so.0.5.1 3566c0e000-3566c0f000 rw-p 00000000 00:00 0 3566e00000-3566e2a000 r-xp 00000000 08:01 2359724 /lib64/libk5crypto.so.3.1 3566e2a000-356702a000 ---p 0002a000 08:01 2359724 /lib64/libk5crypto.so.3.1 356702a000-356702c000 rw-p 0002a000 08:01 2359724 /lib64/libk5crypto.so.3.1 3567200000-356720a000 r-xp 00000000 08:01 2359723 /lib64/libkrb5support.so.0.1 356720a000-3567409000 ---p 0000a000 08:01 2359723 /lib64/libkrb5support.so.0.1 3567409000-356740a000 rw-p 00009000 08:01 2359723 /lib64/libkrb5support.so.0.1 3567600000-3567602000 r-xp 00000000 08:01 2359720 /lib64/libkeyutils.so.1.3 3567602000-3567801000 ---p 00002000 08:01 2359720 /lib64/libkeyutils.so.1.3 3567801000-3567802000 rw-p 00001000 08:01 2359720 /lib64/libkeyutils.so.1.3 3567a00000-3567b71000 r-xp 00000000 08:01 283845 /usr/lib64/libcrypto.so.1.0.0 3567b71000-3567d70000 ---p 00171000 08:01 283845 /usr/lib64/libcrypto.so.1.0.0 3567d70000-3567d93000 rw-p 00170000 08:01 283845 /usr/lib64/libcrypto.so.1.0.0 3567d93000-3567d96000 rw-p 00000000 00:00 0 3567e00000-3567ed4000 r-xp 00000000 08:01 2359727 /lib64/libkrb5.so.3.3 3567ed4000-35680d4000 ---p 000d4000 08:01 2359727 /lib64/libkrb5.so.3.3 35680d4000-35680df000 rw-p 000d4000 08:01 2359727 /lib64/libkrb5.so.3.3 3568200000-356823f000 r-xp 00000000 08:01 2359743 /lib64/libgssapi_krb5.so.2.2 356823f000-356843e000 ---p 0003f000 08:01 2359743 /lib64/libgssapi_krb5.so.2.2 356843e000-3568441000 rw-p 0003e000 08:01 2359743 /lib64/libgssapi_krb5.so.2.2 3568600000-3568616000 r-xp 00000000 08:01 2359728 /lib64/libaudit.so.1.0.0 3568616000-3568815000 ---p 00016000 08:01 2359728 /lib64/libaudit.so.1.0.0 3568815000-3568816000 r--p 00015000 08:01 2359728 /lib64/libaudit.so.1.0.0 3568816000-3568817000 rw-p 00016000 08:01 2359728 /lib64/libaudit.so.1.0.0 3568a00000-3568a03000 r-xp 00000000 08:01 2359749 /lib64/libplds4.so 3568a03000-3568c02000 ---p 00003000 08:01 2359749 /lib64/libplds4.so 3568c02000-3568c03000 rw-p 00002000 08:01 2359749 /lib64/libplds4.so 3568e00000-3568e04000 r-xp 00000000 08:01 2359748 /lib64/libplc4.so 3568e04000-3569003000 ---p 00004000 08:01 2359748 /lib64/libplc4.so 3569003000-3569004000 rw-p 00003000 08:01 2359748 /lib64/libplc4.so 3569200000-3569238000 r-xp 00000000 08:01 2359747 /lib64/libnspr4.so 3569238000-3569438000 ---p 00038000 08:01 2359747 /lib64/libnspr4.so 3569438000-356943a000 rw-p 00038000 08:01 2359747 /lib64/libnspr4.so 356943a000-356943d000 rw-p 00000000 00:00 0 3569600000-3569733000 r-xp 00000000 08:01 283866 /usr/lib64/libnss3.so 3569733000-3569933000 ---p 00133000 08:01 283866 /usr/lib64/libnss3.so 3569933000-356993a000 rw-p 00133000 08:01 283866 /usr/lib64/libnss3.so 356993a000-356993b000 rw-p 00000000 00:00 0 3569a00000-3569a03000 r-xp 00000000 08:01 2359636 /lib64/libgpg-error.so.0.5.0 3569a03000-3569c02000 ---p 00003000 08:01 2359636 /lib64/libgpg-error.so.0.5.0 3569c02000-3569c03000 rw-p 00002000 08:01 2359636 /lib64/libgpg-error.so.0.5.0 3569e00000-3569e53000 r-xp 00000000 08:01 283846 /usr/lib64/libssl.so.1.0.0 3569e53000-356a053000 ---p 00053000 08:01 283846 /usr/lib64/libssl.so.1.0.0 356a053000-356a05b000 rw-p 00053000 08:01 283846 /usr/lib64/libssl.so.1.0.0 356a200000-356a21a000 r-xp 00000000 08:01 275186 /usr/lib64/libnssutil3.so 356a21a000-356a419000 ---p 0001a000 08:01 275186 /usr/lib64/libnssutil3.so 356a419000-356a41f000 rw-p 00019000 08:01 275186 /usr/lib64/libnssutil3.so 356aa00000-356aa28000 r-xp 00000000 08:01 283868 /usr/lib64/libsmime3.so 356aa28000-356ac28000 ---p 00028000 08:01 283868 /usr/lib64/libsmime3.so 356ac28000-356ac2c000 rw-p 00028000 08:01 283868 /usr/lib64/libsmime3.so 356ae00000-356ae32000 r-xp 00000000 08:01 283867 /usr/lib64/libssl3.so 356ae32000-356b031000 ---p 00032000 08:01 283867 /usr/lib64/libssl3.so 356b031000-356b034000 rw-p 00031000 08:01 283867 /usr/lib64/libssl3.so 356b200000-356b21d000 r-xp 00000000 08:01 2359754 /lib64/libtinfo.so.5.7 356b21d000-356b41d000 ---p 0001d000 08:01 2359754 /lib64/libtinfo.so.5.7 356b41d000-356b421000 rw-p 0001d000 08:01 2359754 /lib64/libtinfo.so.5.7 356b600000-356b613000 r-xp 00000000 08:01 280276 /usr/lib64/libexslt.so.0.8.15 356b613000-356b813000 ---p 00013000 08:01 280276 /usr/lib64/libexslt.so.0.8.15 356b813000-356b814000 rw-p 00013000 08:01 280276 /usr/lib64/libexslt.so.0.8.15 356be00000-356be72000 r-xp 00000000 08:01 2359762 /lib64/libgcrypt.so.11.5.3 356be72000-356c071000 ---p 00072000 08:01 2359762 /lib64/libgcrypt.so.11.5.3 356c071000-356c075000 rw-p 00071000 08:01 2359762 /lib64/libgcrypt.so.11.5.3 356ca00000-356ca9b000 r-xp 00000000 08:01 283912 /usr/lib64/libgnutls.so.26.14.12 356ca9b000-356cc9a000 ---p 0009b000 08:01 283912 /usr/lib64/libgnutls.so.26.14.12 356cc9a000-356cca1000 rw-p 0009a000 08:01 283912 /usr/lib64/libgnutls.so.26.14.12 356ce00000-356ce51000 r-xp 00000000 08:01 283871 /usr/lib64/libcurl.so.4.1.1 356ce51000-356d050000 ---p 00051000 08:01 283871 /usr/lib64/libcurl.so.4.1.1 356d050000-356d053000 rw-p 00050000 08:01 283871 /usr/lib64/libcurl.so.4.1.1 356d200000-356d210000 r-xp 00000000 08:01 264108 /usr/lib64/libtasn1.so.3.1.6 356d210000-356d410000 ---p 00010000 08:01 264108 /usr/lib64/libtasn1.so.3.1.6 356d410000-356d411000 rw-p 00010000 08:01 264108 /usr/lib64/libtasn1.so.3.1.6 356d600000-356d626000 r-xp 00000000 08:01 283870 /usr/lib64/libssh2.so.1.0.1 356d626000-356d826000 ---p 00026000 08:01 283870 /usr/lib64/libssh2.so.1.0.1 356d826000-356d827000 rw-p 00026000 08:01 283870 /usr/lib64/libssh2.so.1.0.1 356da00000-356da49000 r-xp 00000000 08:01 284009 /usr/lib64/libaugeas.so.0.11.0 356da49000-356dc48000 ---p 00049000 08:01 284009 /usr/lib64/libaugeas.so.0.11.0 356dc48000-356dc4a000 rw-p 00048000 08:01 284009 /usr/lib64/libaugeas.so.0.11.0 356de00000-356de3b000 r-xp 00000000 08:01 268678 /usr/lib64/libxslt.so.1.1.26 356de3b000-356e03a000 ---p 0003b000 08:01 268678 /usr/lib64/libxslt.so.1.1.26 356e03a000-356e03c000 rw-p 0003a000 08:01 268678 /usr/lib64/libxslt.so.1.1.26 356e200000-356e219000 r-xp 00000000 08:01 283869 /usr/lib64/libsasl2.so.2.0.23 356e219000-356e419000 ---p 00019000 08:01 283869 /usr/lib64/libsasl2.so.2.0.23 356e419000-356e41a000 rw-p 00019000 08:01 283869 /usr/lib64/libsasl2.so.2.0.23 356e600000-356e648000 r-xp 00000000 08:01 2359751 /lib64/libldap-2.4.so.2.5.6 356e648000-356e847000 ---p 00048000 08:01 2359751 /lib64/libldap-2.4.so.2.5.6 356e847000-356e84a000 rw-p 00047000 08:01 2359751 /lib64/libldap-2.4.so.2.5.6 356ea00000-356ea21000 r-xp 00000000 08:01 274087 /usr/lib64/libfa.so.1.3.1 356ea21000-356ec20000 ---p 00021000 08:01 274087 /usr/lib64/libfa.so.1.3.1 356ec20000-356ec21000 rw-p 00020000 08:01 274087 /usr/lib64/libfa.so.1.3.1 356ee00000-356ee0e000 r-xp 00000000 08:01 2359750 /lib64/liblber-2.4.so.2.5.6 356ee0e000-356f00d000 ---p 0000e000 08:01 2359750 /lib64/liblber-2.4.so.2.5.6 356f00d000-356f00e000 rw-p 0000d000 08:01 2359750 /lib64/liblber-2.4.so.2.5.6 3653e00000-3653e0e000 r-xp 00000000 08:01 264220 /usr/lib64/libnetcf.so.1.4.0 3653e0e000-365400d000 ---p 0000e000 08:01 264220 /usr/lib64/libnetcf.so.1.4.0 365400d000-365400e000 rw-p 0000d000 08:01 264220 /usr/lib64/libnetcf.so.1.4.0 7f229efaa000-7f229efc0000 r-xp 00000000 08:01 2359745 /lib64/libgcc_s-4.4.5-20110214.so.1 7f229efc0000-7f229f1bf000 ---p 00016000 08:01 2359745 /lib64/libgcc_s-4.4.5-20110214.so.1 7f229f1bf000-7f229f1c0000 rw-p 00015000 08:01 2359745 /lib64/libgcc_s-4.4.5-20110214.so.1 7f229f1c0000-7f229f203000 rw-p 00000000 00:00 0 7f229f203000-7f229f20f000 r-xp 00000000 08:01 2359326 /lib64/libnss_files-2.12.so 7f229f20f000-7f229f40e000 ---p 0000c000 08:01 2359326 /lib64/libnss_files-2.12.so 7f229f40e000-7f229f40f000 r--p 0000b000 08:01 2359326 /lib64/libnss_files-2.12.so 7f229f40f000-7f229f410000 rw-p 0000c000 08:01 2359326 /lib64/libnss_files-2.12.so 7f229f41f000-7f22a52b0000 r--p 00000000 08:01 263751 /usr/lib/locale/locale-archive 7f22a52b0000-7f22a52ca000 rw-p 00000000 00:00 0 7f22a52ca000-7f22a52cb000 r-xp 00000000 08:01 264265 /usr/lib64/libvirt-qemu.so.0.9.2 7f22a52cb000-7f22a54ca000 ---p 00001000 08:01 264265 /usr/lib64/libvirt-qemu.so.0.9.2 7f22a54ca000-7f22a54cb000 rw-p 00000000 08:01 264265 /usr/lib64/libvirt-qemu.so.0.9.2 7f22a54cb000-7f22a5654000 r-xp 00000000 08:01 273186 /usr/lib64/libvirt.so.0.9.2 7f22a5654000-7f22a5854000 ---p 00189000 08:01 273186 /usr/lib64/libvirt.so.0.9.2 7f22a5854000-7f22a585f000 rw-p 00189000 08:01 273186 /usr/lib64/libvirt.so.0.9.2 7f22a585f000-7f22a5861000 rw-p 00000000 00:00 0 7f22a5868000-7f22a5869000 rw-p 00000000 00:00 0 7f22a5869000-7f22a5870000 r--s 00000000 08:01 264005 /usr/lib64/gconv/gconv-modules.cache 7f22a5870000-7f22a5871000 rw-p 00000000 00:00 0 7fff01f64000-7fff01f79000 rw-p 00000000 00:00 0 [stack] 7fff01fff000-7fff02000000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted (core dumped)
Can you file a separate bug about the virsh crash/abort?
Since i got the problem(comment 7) with libvirt-0.9.2-1.el6.x86_64 when trying to reproduce this bug, i will retest it with the latest libvirt version when i'm free to confirm if it's a new bug.
Retest the bug with libvirt-0.9.4-4.el6.x86_64, and file a new bug 731673.
Test with libvirt-0.9.4-11.el6.x86_64, after shutdown guest almost 20 times, get the following error: virsh # start fff --console error: Failed to start domain fff error: internal error Process exited while reading console log output: char device redirected to /dev/pts/2 do_spice_init: starting 0.8.2 do_spice_init: statistics shm_open failed, Permission denied then retry to start the guest, and it can start normally. BTW, every time the guest shutdown, there's always 4 error output in ibvirtd.log: ------ 15:00:06.252: 32231: error : daemonStreamEvent:208 : stream had I/O failure 15:00:06.252: 32231: error : virNetSocketReadWire:910 : End of file while reading data: Input/output error 15:00:06.253: 32231: error : virFDStreamUpdateCallback:111 : internal error stream is not open 15:00:06.253: 32231: error : qemuMonitorIO:577 : internal error End of file from monitor ------
The libvirt errors you see when a guest shuts down are normal. The error when starting a guest is more interesting but is not connected to this BZ. Qemu failed to start most likely because do_spice_init failed. But I have no idea why that happened.
Following the reproduce steps, retest this bug with libvirt-0.9.4-18.el6.x86_64, and after more than 20 times, everything goes well, can't get the error mentioned above. So move to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2011-1513.html