Description of problem: When destroying the VM facing the I/O error and Failed to reconnect to the hypervisor. Version-Release number of selected component (if applicable): libvirt -1.2.0 and xen-4.2.1. How reproducible: Steps to Reproduce: 1.create the vm in virsh(create xxxx.xml) 2.Try to destroy the vm. (destroy instanceid) Actual results: Facing I/O error and failed to reconnect to the Hypervisor. daemon destroyed. Expected results: VM have to destroy without any error. Additional info: during GDB i got the following information: #gdb -p $(pgrep libvirtd) GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Attaching to process 18173 Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/lt-libvirtd...done. Reading symbols from /opt/libvirt-1.2.0/src/.libs/libvirt-lxc.so.0...done. Loaded symbols for /opt/libvirt-1.2.0/src/.libs/libvirt-lxc.so.0 Reading symbols from /opt/libvirt-1.2.0/src/.libs/libvirt-qemu.so.0...done. Loaded symbols for /opt/libvirt-1.2.0/src/.libs/libvirt-qemu.so.0 Reading symbols from /opt/libvirt-1.2.0/src/.libs/libvirt.so.0...done. Loaded symbols for /opt/libvirt-1.2.0/src/.libs/libvirt.so.0 Reading symbols from /usr/lib/x86_64-linux-gnu/libyajl.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libyajl.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libnl.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libnl.so.1 Reading symbols from /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libxml2.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libavahi-common.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libavahi-common.so.3 Reading symbols from /usr/lib/x86_64-linux-gnu/libavahi-client.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libavahi-client.so.3 Reading symbols from /lib/x86_64-linux-gnu/libdbus-1.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libdbus-1.so.3 Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls.so.26...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libgnutls.so.26 Reading symbols from /usr/lib/x86_64-linux-gnu/libssh2.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libssh2.so.1 Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.11...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libgcrypt.so.11 Reading symbols from /usr/lib/x86_64-linux-gnu/libcurl.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libcurl.so.4 Reading symbols from /lib/x86_64-linux-gnu/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libselinux.so.1 Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1 Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7f81650ac700 (LWP 18183)] [New Thread 0x7f81658ad700 (LWP 18182)] [New Thread 0x7f81660ae700 (LWP 18181)] [New Thread 0x7f81668af700 (LWP 18180)] [New Thread 0x7f81670b0700 (LWP 18179)] [New Thread 0x7f81678b1700 (LWP 18178)] [New Thread 0x7f81680b2700 (LWP 18177)] [New Thread 0x7f81688b3700 (LWP 18176)] [New Thread 0x7f81690b4700 (LWP 18175)] [New Thread 0x7f81698b5700 (LWP 18174)] Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0 Reading symbols from /lib/x86_64-linux-gnu/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libutil.so.1 Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2 Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6 Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1 Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6 Reading symbols from /lib/x86_64-linux-gnu/libudev.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libudev.so.0 Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1 Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/liblzma.so.5 Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libtasn1.so.3 Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libgpg-error.so.0 Reading symbols from /usr/lib/x86_64-linux-gnu/libidn.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libidn.so.11 Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 Reading symbols from /usr/lib/x86_64-linux-gnu/librtmp.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/librtmp.so.0 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libsasl2.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5.so.3 Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libcom_err.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libkeyutils.so.1 Reading symbols from /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_network.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_network.so Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_storage.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_storage.so Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_secret.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_secret.so Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_nwfilter.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_nwfilter.so Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_xen.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_xen.so Reading symbols from /usr/lib/libxenstore.so.3.0...done. Loaded symbols for /usr/lib/libxenstore.so.3.0 Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_libxl.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_libxl.so Reading symbols from /usr/lib/libxenlight.so.2.0...done. Loaded symbols for /usr/lib/libxenlight.so.2.0 Reading symbols from /usr/lib/libxenctrl.so.4.2...done. Loaded symbols for /usr/lib/libxenctrl.so.4.2 Reading symbols from /usr/lib/libxenguest.so.4.2...done. Loaded symbols for /usr/lib/libxenguest.so.4.2 Reading symbols from /usr/lib/libblktapctl.so.1.0...done. Loaded symbols for /usr/lib/libblktapctl.so.1.0 Reading symbols from /lib/x86_64-linux-gnu/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libuuid.so.1 Reading symbols from /lib/x86_64-linux-gnu/libbz2.so.1.0...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libbz2.so.1.0 Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_qemu.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_qemu.so Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_lxc.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_lxc.so Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_uml.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_uml.so Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_vbox.so...done. Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_vbox.so Reading symbols from /usr/lib/virtualbox/VBoxXPCOMC.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/virtualbox/VBoxXPCOMC.so Reading symbols from /usr/lib/virtualbox/VBoxRT.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/virtualbox/VBoxRT.so Reading symbols from /usr/lib/virtualbox/VBoxXPCOM.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/virtualbox/VBoxXPCOM.so Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6 Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1 Reading symbols from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1 Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2 0x00007f816d405df3 in poll () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) c Continuing. [New Thread 0x7f8170682700 (LWP 18504)] Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () ***** in Virsh console virsh # destroy 3(instanceid) error: Failed to destroy domain 3 error: End of file while reading data: Input/output error error: Failed to reconnect to the hypervisor virsh # list error: failed to connect to the hypervisor error: no valid connection error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused *****
Created attachment 838763 [details] Log file logs which logged during the process.
(In reply to Dharmalingam from comment #0) > Continuing. > [New Thread 0x7f8170682700 (LWP 18504)] > > Program received signal SIGSEGV, Segmentation fault. > 0x0000000000000000 in ?? () > Not very useful, is it? You have to issue a 'bt' command in gdb, to print the stack trace and have at least a hint of where the problem is originating. Notice that, in order to such stacktrace to be useful, you need debugging symbols, do you have them (from the above, it seems to me that you don't)? BTW, `virsh destroy' works for me, using both libvirt and xen git tip. Dario
(In reply to Dario Faggioli from comment #2) > (In reply to Dharmalingam from comment #0) > > Continuing. > > [New Thread 0x7f8170682700 (LWP 18504)] > > > > Program received signal SIGSEGV, Segmentation fault. > > 0x0000000000000000 in ?? () > > > Not very useful, is it? > > You have to issue a 'bt' command in gdb, to print the stack trace and have > at least a hint of where the problem is originating. > > Notice that, in order to such stacktrace to be useful, you need debugging > symbols, do you have them (from the above, it seems to me that you don't)? > > BTW, `virsh destroy' works for me, using both libvirt and xen git tip. > > Dario ____ I am newbie to this gdb debugging. now i did the bt in the debugging. In my side still it showing the error. Following are the stacktrace which i got. Kindly refer the following logs for full information: **** [New Thread 0x7f8503695700 (LWP 20012)] Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039 #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216 #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at rpc/virnetserver.c:1112 #7 0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb) ****** regards, cooldharma06.
Dharmalingam, if you're building from git, it should be fairly simple to get debug symbols. Just run: export CFLAGS="-O0 -ggdb3" git clean -fxd; ./autogen.sh --system && make After this, you have debug symbols in libvirt (not the underlying libraries though). After this, you should be able to reproduce, and in the 'bt' command in gdb you should not see the <optimized out> but real pointers. Moreover, you can produce a coredump (a file which will help developers to see what's going on) you can issue "generate-core-file" command in the gdb console. It will produce a file which you should be able to attach to this bug.
As per the discussion in the libvir mailing list, this behavior is due to the fact that Xen 4.2.1 (which is what is used) is lacking the following bugfixes: libxl: fix stale timeout event callback race libxl: fix stale fd event callback race Which are both present in the Xen branch stable-4.2, and hence in Xen 4.2.3, as commits: 6f0f339dd4378d062a211969f45cd23af12bf386 a87ef897295ec17788e41e9a8f4c0ada7a5a45f8 See here: http://bugs.xenproject.org/xen/mid/%3C21170.56484.749112.480737@mariner.uk.xensource.com%3E I think this can be considered enough for closing this bug.
thanks to all people. :) its working in new version of xen-4.3.1 and libvirt-1.2.0. and once again thanks to all who are guided me.. **** regards, cooldharma06 (In reply to Dario Faggioli from comment #5) > As per the discussion in the libvir mailing list, this behavior is due to > the fact that Xen 4.2.1 (which is what is used) is lacking the following > bugfixes: > libxl: fix stale timeout event callback race > libxl: fix stale fd event callback race > > Which are both present in the Xen branch stable-4.2, and hence in Xen 4.2.3, > as commits: > 6f0f339dd4378d062a211969f45cd23af12bf386 > a87ef897295ec17788e41e9a8f4c0ada7a5a45f8 > > See here: > http://bugs.xenproject.org/xen/mid/%3C21170.56484.749112.480737@mariner.uk. > xensource.com%3E > > I think this can be considered enough for closing this bug.