Bug 1098376

Summary: libvirtd fails to start on Xen instances (rackspace in particular)
Product: [Fedora] Fedora Reporter: Ian Wienand <iwienand>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 20CC: agedosier, aortega, berrange, clalancette, crobinso, itamar, jdenemar, jforbes, laine, libvirt-maint, veillard, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.1.3.5-2.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-05-24 23:23:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ian Wienand 2014-05-16 03:24:08 UTC
Description of problem:

libvirtd fails to start on Xen instances, particularly the "Fedora 20 (Heisenbug) (PVHVM)" instance type on Rackspace

---
[root@devstack-test libvirt-1.1.3.5]# libvirtd 
2014-05-16 03:00:31.376+0000: 23764: info : libvirt version: 1.1.3.5, package: 1.fc20 (Fedora Project, 2014-05-03-21:46:09, buildvm-16.phx2.fedoraproject.org)
2014-05-16 03:00:31.376+0000: 23764: info : virNetlinkEventServiceStart:518 : starting netlink event service with protocol 0
2014-05-16 03:00:31.376+0000: 23764: info : virNetlinkEventServiceStart:518 : starting netlink event service with protocol 15
2014-05-16 03:00:31.386+0000: 23775: info : dnsmasqCapsSetFromBuffer:682 : dnsmasq version is 2.68, --bind-dynamic is present, SO_BINDTODEVICE is in use
2014-05-16 03:00:31.392+0000: 23775: info : networkReloadFirewallRules:1553 : Reloading iptables rules
2014-05-16 03:00:31.805+0000: 23775: info : virIpTablesOnceInit:69 : firewall-cmd found but disabled for iptables
2014-05-16 03:00:32.024+0000: 23775: info : networkRefreshDaemons:1525 : Refreshing network daemons
2014-05-16 03:00:32.024+0000: 23775: info : networkRefreshDhcpDaemon:1171 : Refreshing dnsmasq for network virbr0
2014-05-16 03:00:32.333+0000: 23775: info : ebiptablesDriverInitWithFirewallD:4201 : firewalld support disabled for nwfilter
2014-05-16 03:00:32.375+0000: 23775: error : libxlDriverConfigNew:1089 : cannot initialize libxenlight context, probably not running in a Xen Dom0, disabling driver
2014-05-16 03:00:32.375+0000: 23775: error : virStateInitialize:854 : Initialization of LIBXL state driver failed: Unknown problem
2014-05-16 03:00:32.375+0000: 23775: error : daemonRunStateInit:909 : Driver state initialization failed
2014-05-16 03:00:32.376+0000: 23764: info : virNetlinkEventServiceStopAll:421 : stopping all netlink event services
---

I believe the problem there is the "Initialization of LIBXL state driver failed: Unknown problem" which actually makes libvirtd bail out.  libxlDriverShouldLoad() sees /proc/xen/capabilities looking like:

---
[root@devstack-test libvirt-1.1.3.5]# ls -l /proc/xen
total 0
-r--r--r--. 1 root root 0 May 16 00:26 capabilities
-rw-------. 1 root root 0 May 16 00:26 privcmd
-rw-------. 1 root root 0 May 16 00:26 xenbus
[root@devstack-test libvirt-1.1.3.5]# cat /proc/xen/capabilities 
[root@devstack-test libvirt-1.1.3.5]# 
---

This allows the plugin to load, which calls libxlStateInitialize()->libxlDriverConfigNew(), which returns -1 as it can't find what it's looking for and libvirtd stops.

Looking at upstream git I believe this has been addressed with [1] which checks the /proc/xen/capabilities more carefully.  I believe we should make this available in Fedora and RHEL7; running libvirtd inside instances is critical for upstream OpenStack testing on these platforms.

---
[root@devstack-test libvirt-1.1.3.5]# rpm -qa | grep libvirt
libvirt-daemon-qemu-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-interface-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-nwfilter-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-libxl-1.1.3.5-1.fc20.x86_64
libvirt-1.1.3.5-1.fc20.x86_64
libvirt-daemon-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-secret-1.1.3.5-1.fc20.x86_64
libvirt-daemon-config-network-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-uml-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-vbox-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-network-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-qemu-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-xen-1.1.3.5-1.fc20.x86_64
libvirt-debuginfo-1.1.3.5-1.fc20.x86_64
libvirt-client-1.1.3.5-1.fc20.x86_64
libvirt-daemon-config-nwfilter-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-nodedev-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-storage-1.1.3.5-1.fc20.x86_64
libvirt-daemon-driver-lxc-1.1.3.5-1.fc20.x86_64
libvirt-python-1.1.3.5-1.fc20.x86_64
libvirt-daemon-kvm-1.1.3.5-1.fc20.x86_64
---

[1] http://libvirt.org/git/?p=libvirt.git;a=commit;h=8c869ad9f9c5d57ba0ef516835a08afdba7ad828

Comment 1 Jiri Denemark 2014-05-16 08:54:25 UTC
(In reply to Ian Wienand from comment #0)
> Looking at upstream git I believe this has been addressed with [1] which
> checks the /proc/xen/capabilities more carefully.  I believe we should make
> this available in Fedora and RHEL7; running libvirtd inside instances is
> critical for upstream OpenStack testing on these platforms.

There's no libxl driver in RHEL7 and thus the patch may only be needed in Fedora.

Comment 2 Fedora Update System 2014-05-19 23:26:18 UTC
libvirt-1.1.3.5-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/libvirt-1.1.3.5-2.fc20

Comment 3 Fedora Update System 2014-05-21 23:31:35 UTC
Package libvirt-1.1.3.5-2.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.1.3.5-2.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-6586/libvirt-1.1.3.5-2.fc20
then log in and leave karma (feedback).

Comment 4 Fedora Update System 2014-05-24 23:23:58 UTC
libvirt-1.1.3.5-2.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.