Bug 1044838 - Facing the segmentation fault when destroying the VM.
Summary: Facing the segmentation fault when destroying the VM.
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: x86_64
OS: Other
unspecified
high
Target Milestone: ---
Assignee: Michal Privoznik
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-19 05:17 UTC by Dharmalingam
Modified: 2013-12-20 08:51 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-12-20 08:51:12 UTC


Attachments (Terms of Use)
Log file (590.25 KB, text/plain)
2013-12-19 05:21 UTC, Dharmalingam
no flags Details

Description Dharmalingam 2013-12-19 05:17:48 UTC
Description of problem:
When destroying the VM facing the I/O error and Failed to reconnect to the hypervisor.

Version-Release number of selected component (if applicable):
libvirt -1.2.0 and xen-4.2.1.

How reproducible:


Steps to Reproduce:
1.create the vm in virsh(create xxxx.xml)
2.Try to destroy the vm. (destroy instanceid)

Actual results:
Facing I/O error and failed to reconnect to the Hypervisor.
daemon destroyed.


Expected results:
VM have to destroy without any error.

Additional info:
during GDB i got the following information:

#gdb -p $(pgrep libvirtd)
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 18173
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/lt-libvirtd...done.
Reading symbols from /opt/libvirt-1.2.0/src/.libs/libvirt-lxc.so.0...done.
Loaded symbols for /opt/libvirt-1.2.0/src/.libs/libvirt-lxc.so.0
Reading symbols from /opt/libvirt-1.2.0/src/.libs/libvirt-qemu.so.0...done.
Loaded symbols for /opt/libvirt-1.2.0/src/.libs/libvirt-qemu.so.0
Reading symbols from /opt/libvirt-1.2.0/src/.libs/libvirt.so.0...done.
Loaded symbols for /opt/libvirt-1.2.0/src/.libs/libvirt.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libyajl.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libyajl.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libnl.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libnl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libxml2.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libavahi-common.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libavahi-common.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libavahi-client.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libavahi-client.so.3
Reading symbols from /lib/x86_64-linux-gnu/libdbus-1.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdbus-1.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgnutls.so.26
Reading symbols from /usr/lib/x86_64-linux-gnu/libssh2.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libssh2.so.1
Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcrypt.so.11
Reading symbols from /usr/lib/x86_64-linux-gnu/libcurl.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcurl.so.4
Reading symbols from /lib/x86_64-linux-gnu/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libselinux.so.1
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7f81650ac700 (LWP 18183)]
[New Thread 0x7f81658ad700 (LWP 18182)]
[New Thread 0x7f81660ae700 (LWP 18181)]
[New Thread 0x7f81668af700 (LWP 18180)]
[New Thread 0x7f81670b0700 (LWP 18179)]
[New Thread 0x7f81678b1700 (LWP 18178)]
[New Thread 0x7f81680b2700 (LWP 18177)]
[New Thread 0x7f81688b3700 (LWP 18176)]
[New Thread 0x7f81690b4700 (LWP 18175)]
[New Thread 0x7f81698b5700 (LWP 18174)]
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libutil.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libudev.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libudev.so.0
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/liblzma.so.5
Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libtasn1.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgpg-error.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libidn.so.11
Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/librtmp.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/librtmp.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libsasl2.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcom_err.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libkeyutils.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_network.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_network.so
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_storage.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_storage.so
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_secret.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_secret.so
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_nwfilter.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_nwfilter.so
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_xen.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_xen.so
Reading symbols from /usr/lib/libxenstore.so.3.0...done.
Loaded symbols for /usr/lib/libxenstore.so.3.0
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_libxl.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_libxl.so
Reading symbols from /usr/lib/libxenlight.so.2.0...done.
Loaded symbols for /usr/lib/libxenlight.so.2.0
Reading symbols from /usr/lib/libxenctrl.so.4.2...done.
Loaded symbols for /usr/lib/libxenctrl.so.4.2
Reading symbols from /usr/lib/libxenguest.so.4.2...done.
Loaded symbols for /usr/lib/libxenguest.so.4.2
Reading symbols from /usr/lib/libblktapctl.so.1.0...done.
Loaded symbols for /usr/lib/libblktapctl.so.1.0
Reading symbols from /lib/x86_64-linux-gnu/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libuuid.so.1
Reading symbols from /lib/x86_64-linux-gnu/libbz2.so.1.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libbz2.so.1.0
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_qemu.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_qemu.so
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_lxc.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_lxc.so
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_uml.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_uml.so
Reading symbols from /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_vbox.so...done.
Loaded symbols for /opt/libvirt-1.2.0/daemon/.libs/../../src/.libs/libvirt_driver_vbox.so
Reading symbols from /usr/lib/virtualbox/VBoxXPCOMC.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/virtualbox/VBoxXPCOMC.so
Reading symbols from /usr/lib/virtualbox/VBoxRT.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/virtualbox/VBoxRT.so
Reading symbols from /usr/lib/virtualbox/VBoxXPCOM.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/virtualbox/VBoxXPCOM.so
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
0x00007f816d405df3 in poll () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) c
Continuing.
[New Thread 0x7f8170682700 (LWP 18504)]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()


*****
in Virsh console
virsh # destroy 3(instanceid)
error: Failed to destroy domain 3
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor

virsh # list
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
*****

Comment 1 Dharmalingam 2013-12-19 05:21:46 UTC
Created attachment 838763 [details]
Log file

logs which logged during the process.

Comment 2 Dario Faggioli 2013-12-19 09:20:12 UTC
(In reply to Dharmalingam from comment #0)
> Continuing.
> [New Thread 0x7f8170682700 (LWP 18504)]
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000000000000 in ?? ()
> 
Not very useful, is it?

You have to issue a 'bt' command in gdb, to print the stack trace and have at least a hint of where the problem is originating.

Notice that, in order to such stacktrace to be useful, you need debugging symbols, do you have them (from the above, it seems to me that you don't)?

BTW, `virsh destroy' works for me, using both libvirt and xen git tip.

Dario

Comment 3 Dharmalingam 2013-12-19 09:41:18 UTC
(In reply to Dario Faggioli from comment #2)
> (In reply to Dharmalingam from comment #0)
> > Continuing.
> > [New Thread 0x7f8170682700 (LWP 18504)]
> > 
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x0000000000000000 in ?? ()
> > 
> Not very useful, is it?
> 
> You have to issue a 'bt' command in gdb, to print the stack trace and have
> at least a hint of where the problem is originating.
> 
> Notice that, in order to such stacktrace to be useful, you need debugging
> symbols, do you have them (from the above, it seems to me that you don't)?
> 
> BTW, `virsh destroy' works for me, using both libvirt and xen git tip.
> 
> Dario
____


I am newbie to this gdb debugging. now i did the bt in the debugging. In my side still it showing the error. Following are the stacktrace which i got. Kindly refer the following logs for full information: 

****
[New Thread 0x7f8503695700 (LWP 20012)]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039
#2  0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216
#3  0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451
#4  virEventPollRunOnce () at util/vireventpoll.c:644
#5  0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274
#6  0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at rpc/virnetserver.c:1112
#7  0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513
(gdb)

******

regards,
cooldharma06.

Comment 4 Michal Privoznik 2013-12-19 12:00:15 UTC
Dharmalingam,

if you're building from git, it should be fairly simple to get debug symbols. Just run:

export CFLAGS="-O0 -ggdb3"
git clean -fxd; ./autogen.sh --system && make

After this, you have debug symbols in libvirt (not the underlying libraries though). After this, you should be able to reproduce, and in the 'bt' command in gdb you should not see the <optimized out> but real pointers. Moreover, you can produce a coredump (a file which will help developers to see what's going on) you can issue "generate-core-file" command in the gdb console. It will produce a file which you should be able to attach to this bug.

Comment 5 Dario Faggioli 2013-12-19 23:13:38 UTC
As per the discussion in the libvir mailing list, this behavior is due to the fact that Xen 4.2.1 (which is what is used) is lacking the following bugfixes:
  libxl: fix stale timeout event callback race
  libxl: fix stale fd event callback race

Which are both present in the Xen branch stable-4.2, and hence in Xen 4.2.3, as commits:
  6f0f339dd4378d062a211969f45cd23af12bf386
  a87ef897295ec17788e41e9a8f4c0ada7a5a45f8

See here:
 http://bugs.xenproject.org/xen/mid/%3C21170.56484.749112.480737@mariner.uk.xensource.com%3E

I think this can be considered enough for closing this bug.

Comment 6 Dharmalingam 2013-12-20 08:51:12 UTC
thanks to all people. :)
its working in new version of xen-4.3.1 and libvirt-1.2.0.
and once again thanks to all who are guided me..

****
regards,
cooldharma06



(In reply to Dario Faggioli from comment #5)
> As per the discussion in the libvir mailing list, this behavior is due to
> the fact that Xen 4.2.1 (which is what is used) is lacking the following
> bugfixes:
>   libxl: fix stale timeout event callback race
>   libxl: fix stale fd event callback race
> 
> Which are both present in the Xen branch stable-4.2, and hence in Xen 4.2.3,
> as commits:
>   6f0f339dd4378d062a211969f45cd23af12bf386
>   a87ef897295ec17788e41e9a8f4c0ada7a5a45f8
> 
> See here:
>  http://bugs.xenproject.org/xen/mid/%3C21170.56484.749112.480737@mariner.uk.
> xensource.com%3E
> 
> I think this can be considered enough for closing this bug.


Note You need to log in before you can comment on or make changes to this bug.