Bug 1701783

Summary:	libvirt daemon hangs when restarting on top of Xen
Product:	[Community] Virtualization Tools	Reporter:	mathieu.tarral
Component:	libvirt	Assignee:	Libvirt Maintainers <libvirt-maint>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	berrange, libvirt-maint, tburke
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-04-24 08:13:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description mathieu.tarral 2019-04-21 21:37:49 UTC

Description of problem:

Restarting libvirt with systemctl results in systemctl hanging and waiting undefintitely, unable to kill and restart libvirt.

Version-Release number of selected component (if applicable):
libvirt-4.7.0-1.fc29

How reproducible:
always

Steps to Reproduce:
1. install Xen from package 4.11.1
2. install latest libvirt
3. reboot on Xen
4. start and restart libvirt daemon

Actual results:
systemctl hangs, unable to restart libvirt daemon

Expected results:
libvirt daemon should have restarted

Additional info:

This behavior has been tested on a Vagrant box, provisioned by an Ansible playbook which installs Xen and libvirt.

Comment 1 Daniel Berrangé 2019-04-23 09:15:36 UTC

Please can you capture a stack trace of the libvirtd process using GDB.

First, make sure the libvirt-debuginfo RPM is installed then:

 $ gdb
 (gdb) attach PID-OF-LIBVIRTD
 (gdb) thread apply all backtrace

Comment 2 mathieu.tarral 2019-04-23 22:05:47 UTC

Hi,

to give you more details, my Ansible playbook hangs at:
https://github.com/Wenzel/vagrant-xen-pyvmidbg/blob/6223224440dfd98c790f47a51f6a881362f240a3/ansible/roles/libvirt/tasks/main.yml#L25

I opened another shell, installed GDB and libvirt's debug packages, but when I tried to attach,
GDB itself is hanging...


(gdb) attach 9295
Attaching to process 9295
[New LWP 9296]
[New LWP 9297]
[New LWP 9298]
[New LWP 9299]
[New LWP 9300]
[New LWP 9301]
[New LWP 9302]
[New LWP 9303]
[New LWP 9304]
[New LWP 9305]
[New LWP 9306]
[New LWP 9307]
[New LWP 9308]
[New LWP 9309]
[New LWP 9310]
[New LWP 9311]
[New LWP 9329]
[New LWP 9380]
Missing separate debuginfo for /usr/sbin/libvirtd
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/08/1dc67b73087a2c243f150be1ca62a23e940410.debug
Missing separate debuginfo for /lib64/libvirt-lxc.so.0
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/72/fd8838ef0e4ba6aee01af567246f4943390ce8.debug
Missing separate debuginfo for /lib64/libvirt-qemu.so.0
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/56/9528c6e1e1add0aed771d3ad891f12d9310069.debug
Missing separate debuginfo for /lib64/libvirt.so.0
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/9f/ca789faaee7706d90280a7e236b7d83bc1b4a1.debug
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
...
...


I tried to send some CTRL-C, but it doesn't work.

What do you suggests ?

Thanks.

Comment 3 mathieu.tarral 2019-04-23 22:09:32 UTC

Well, it looks like the xl toolstack is hanging for no reasons:

[vagrant@localhost ~]$ sudo xl list
Name                                        ID   Mem VCPUs      State   Time(s)
...
...

let me investigate this.

Comment 4 mathieu.tarral 2019-04-23 22:27:47 UTC

So, it appears that Xen alone, on a fully upgraded Fedora, in a Virtual Machine, is hanging.

Libvirt has nothing to do with it, as far as i know.

Comment 5 Daniel Berrangé 2019-04-24 08:13:26 UTC

Ok, best report that to the xen project then.