Created attachment 958476 [details] Guest XML used to test Description of problem ---------------------- When defining a libvirt KVM guest XML with libvirt, $ grep kvm devstack2.xml -A1 -B1 <domain type='kvm'> <name>devstack2</name> -- <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> $ virsh define devstack2.xml error: Failed to define domain from devstack2.xml error: internal error: unexpected domain type kvm, expecting lxc With libvirt debug logs enabled: [. . .] 2014-11-18 07:57:14.785+0000: 662: error : virDomainDefParseXML:12154 : internal error: unexpected domain type kvm, expecting lxc 2014-11-18 07:58:12.446+0000: 1035: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7fc6ed25f440 sock=0x7fc6ed25f040 2014-11-18 07:58:12.459+0000: 1035: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7fc6ed25dc90 sock=0x7fc6ed25d890 2014-11-18 08:00:19.490+0000: 1035: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7fc6ed25e5d0 sock=0x7fc6ed25e1d0 2014-11-18 08:00:19.502+0000: 1035: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7fc6ed25dad0 sock=0x7fc6ed25d6d0 2014-11-18 08:05:17.833+0000: 1178: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7f454ffd16f0 sock=0x7f454ffd12f0 2014-11-18 08:07:02.488+0000: 1178: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7f454ffd0890 sock=0x7f454ffd0360 2014-11-18 08:07:02.502+0000: 1181: error : virDomainDefParseXML:12154 : internal error: unexpected domain type kvm, expecting lxc 2014-11-18 08:12:02.140+0000: 1178: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7f454ffd1b20 sock=0x7f454ffd1440 2014-11-18 08:12:02.157+0000: 1183: error : virDomainDefParseXML:12154 : internal error: unexpected domain type kvm, expecting lxc [. . .] Version ------- $ rpm -qa | grep -i libvirt libvirt-daemon-driver-storage-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-lxc-1.2.10-2.fc22.x86_64 libvirt-docs-1.2.10-2.fc22.x86_64 libvirt-daemon-qemu-1.2.10-2.fc22.x86_64 libvirt-devel-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-interface-1.2.10-2.fc22.x86_64 libvirt-daemon-config-nwfilter-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-libxl-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-uml-1.2.10-2.fc22.x86_64 libvirt-client-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-nodedev-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-nwfilter-1.2.10-2.fc22.x86_64 libvirt-daemon-kvm-1.2.10-2.fc22.x86_64 libvirt-python-1.2.10-1.fc22.x86_64 libvirt-daemon-driver-xen-1.2.10-2.fc22.x86_64 libvirt-daemon-config-network-1.2.10-2.fc22.x86_64 libvirt-daemon-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-qemu-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-vbox-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-secret-1.2.10-2.fc22.x86_64 libvirt-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-network-1.2.10-2.fc22.x86_64 libvirt-debuginfo-1.2.10-2.fc22.x86_64 $ uname -r; rpm -q qemu-system-x86 3.18.0-0.rc4.git2.1.fc22.x86_64 qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64 How reproducible: Consistently. Steps to Reproduce ------------------ 1. Install the said libvirt/QEMU versions from Rawhide $ sudo yum update libvirt\* qemu\* --enablerepo=rawhide 2. Define the guest XML (in attachment) $ virsh define devstack2.xml Actual results -------------- $ virsh define devstack2.xml error: Failed to define domain from devstack2.xml error: internal error: unexpected domain type kvm, expecting lxc Expected results ---------------- KVM guest XML should be successfully defined. Additional info --------------- (a) The same guest XML was defined successfully by libvirt on a Fedora-21 machine with the below versions: $ uname -r; rpm -q libvirt-daemon-kvm qemu-system-x86 3.17.1-302.fc21.x86_64 libvirt-daemon-kvm-1.2.9-4.fc21.x86_64 qemu-system-x86-2.1.2-6.fc21.x86_64 (b) Sanity checks: $ systemctl status libvirtd | grep -i Active Active: active (running) since Tue 2014-11-18 04:06:55 EST; 3min 30s ago $ virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking for device /dev/kvm : PASS QEMU: Checking for device /dev/vhost-net : PASS QEMU: Checking for device /dev/net/tun : PASS LXC: Checking for Linux >= 2.6.26 : PASS
Could you please post the result of "virsh uri".
(In reply to Peter Krempa from comment #1) > Could you please post the result of "virsh uri". Sure, weird - it's set to LXC: $ virsh uri lxc:///
And, setting the LIBVIRT_DEFAULT_URI explicitly doesn't help here either. $ export LIBVIRT_DEFAULT_URI=qemu:///system $ echo $LIBVIRT_DEFAULT_URI qemu:///system $ rpm -ql libvirt-daemon-driver-qemu /etc/libvirt/qemu /etc/libvirt/qemu-lockd.conf /etc/libvirt/qemu.conf /etc/logrotate.d/libvirtd.qemu /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/share/augeas/lenses/libvirtd_qemu.aug /usr/share/augeas/lenses/tests/test_libvirtd_qemu.aug /var/cache/libvirt/qemu /var/lib/libvirt/qemu /var/lib/libvirt/qemu/channel /var/lib/libvirt/qemu/channel/target /var/lib/libvirt/qemu/nvram /var/log/libvirt/qemu /var/run/libvirt/qemu And, I couldn't reproduce this case again on a fresh F21, where I upgraded the libvirt/QEMU RPMs from Rawhide: $ rpm -qa | grep -i libvirt; rpm -q qemu-system-x86; uname -r libvirt-client-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-secret-1.2.10-2.fc22.x86_64 libvirt-daemon-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-storage-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-nodedev-1.2.10-2.fc22.x86_64 libvirt-daemon-kvm-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-network-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-interface-1.2.10-2.fc22.x86_64 libvirt-python-1.2.10-1.fc22.x86_64 libvirt-daemon-driver-nwfilter-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-qemu-1.2.10-2.fc22.x86_64 qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64 3.17.2-300.fc21.x86_64 $ virsh uri qemu:///system
(In reply to Kashyap Chamarthy from comment #3) > And, setting the LIBVIRT_DEFAULT_URI explicitly doesn't help here either. > > $ export LIBVIRT_DEFAULT_URI=qemu:///system > $ echo $LIBVIRT_DEFAULT_URI > qemu:///system > > $ rpm -ql libvirt-daemon-driver-qemu > /etc/libvirt/qemu > /etc/libvirt/qemu-lockd.conf > /etc/libvirt/qemu.conf > /etc/logrotate.d/libvirtd.qemu > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > /usr/share/augeas/lenses/libvirtd_qemu.aug > /usr/share/augeas/lenses/tests/test_libvirtd_qemu.aug > /var/cache/libvirt/qemu > /var/lib/libvirt/qemu > /var/lib/libvirt/qemu/channel > /var/lib/libvirt/qemu/channel/target > /var/lib/libvirt/qemu/nvram > /var/log/libvirt/qemu > /var/run/libvirt/qemu Just thoguht I'd claridy, on the above F21 host, where setting the DEFAULT URI as above didn't make a difference, as it still couldn't find the driver to QEMU somehow, desipte having the libvirt-daemon-driver-qemu RPM installed: $ virsh uri error: failed to connect to the hypervisor error: no valid connection error: no connection driver available for qemu:///system error: Failed to reconnect to the hypervisor > > And, the below is on a completely different F21 host which was freshly installed where I couldn't reproduce the issue. > And, I couldn't reproduce this case again on a fresh F21, where I upgraded > the libvirt/QEMU RPMs from Rawhide: > > $ rpm -qa | grep -i libvirt; rpm -q qemu-system-x86; uname -r > libvirt-client-1.2.10-2.fc22.x86_64 > libvirt-daemon-driver-secret-1.2.10-2.fc22.x86_64 > libvirt-daemon-1.2.10-2.fc22.x86_64 > libvirt-daemon-driver-storage-1.2.10-2.fc22.x86_64 > libvirt-daemon-driver-nodedev-1.2.10-2.fc22.x86_64 > libvirt-daemon-kvm-1.2.10-2.fc22.x86_64 > libvirt-daemon-driver-network-1.2.10-2.fc22.x86_64 > libvirt-daemon-driver-interface-1.2.10-2.fc22.x86_64 > libvirt-python-1.2.10-1.fc22.x86_64 > libvirt-daemon-driver-nwfilter-1.2.10-2.fc22.x86_64 > libvirt-daemon-driver-qemu-1.2.10-2.fc22.x86_64 > qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64 > 3.17.2-300.fc21.x86_64 > > $ virsh uri > qemu:///system
> Just thoguht I'd claridy, on the above F21 host, where setting the DEFAULT > URI as above didn't make a difference, as it still couldn't find the driver > to QEMU somehow, desipte having the libvirt-daemon-driver-qemu RPM installed: > > $ virsh uri > error: failed to connect to the hypervisor > error: no valid connection > error: no connection driver available for qemu:///system > error: Failed to reconnect to the hypervisor So, since you have the RPM installed, the only way you'd get this error is if libvirtd was unable to load the QEMU driver module. This should be visible as an error message in the libvirtd logs, so please check the journald logs for libvirtd.service.
(In reply to Daniel Berrange from comment #5) > > Just thoguht I'd claridy, on the above F21 host, where setting the DEFAULT > > URI as above didn't make a difference, as it still couldn't find the driver > > to QEMU somehow, desipte having the libvirt-daemon-driver-qemu RPM installed: > > > > $ virsh uri > > error: failed to connect to the hypervisor > > error: no valid connection > > error: no connection driver available for qemu:///system > > error: Failed to reconnect to the hypervisor > > So, since you have the RPM installed, the only way you'd get this error is > if libvirtd was unable to load the QEMU driver module. This should be > visible as an error message in the libvirtd logs, so please check the > journald logs for libvirtd.service. Ah, thank you - seems like we found the root cause (need newer 'device-mapper'). From `journalctl`: $ journalctl -u libvirtd --since=yesterday -p err [. . .] failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference . . [. . .] failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: virStorageFileCreate On the machine, I'm using: rpm -qa | grep -i device-mapper device-mapper-libs-1.02.88-2.fc21.x86_64 device-mapper-persistent-data-0.3.2-4.fc21.x86_64 device-mapper-1.02.88-2.fc21.x86_64 device-mapper-event-1.02.88-2.fc21.x86_64 device-mapper-devel-1.02.88-2.fc21.x86_64 device-mapper-event-libs-1.02.88-2.fc21.x86_64 Updating to version resolved the issue: device-mapper-1.02.90-1.fc21.x86_64 A slightly more verbose `journalctl` error (note - I truncated the long hostname to 'foobar' from the `journalctl` output): ------------ $ journalctl -u libvirtd --since=yesterday -p err [. . .] Nov 17 04:33:47 foobar libvirtd[662]: no connection driver available for <null> Nov 17 04:33:47 foobar libvirtd[662]: End of file while reading data: Input/output error Nov 17 04:33:47 foobar libvirtd[662]: no connection driver available for <null> Nov 17 04:33:47 foobar libvirtd[662]: End of file while reading data: Input/output error Nov 17 04:33:55 foobar libvirtd[10800]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference Nov 17 04:33:55 foobar libvirtd[10800]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: virStorageFileCreate Nov 17 04:52:27 foobar libvirtd[10936]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference Nov 17 04:52:27 foobar libvirtd[10936]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: virStorageFileCreate <null> Nov 17 06:46:15 foobar libvirtd[11127]: End of file while reading data: Input/output error ------------ (And as it can be guesed: on the F21 machine where I couldn't reproduce with identical libvirt/QEMU RPMs, it had the correct 'device-mapper' RPM version: device-mapper-1.02.90-1.fc21.x86_64)
You can reproduce this issue by downgrading to device-mapper-1.02.88-2.fc21.x86_64.rpm (and its dependencies), it's a sub-package of lvm2-2.02.109-2.fc21. Builds can be found from here: http://koji.fedoraproject.org/koji/buildinfo?buildID=561650
Since it's a device-mapper-libs bug (offending version: device-mapper-1.02.88-2.fc21.x86_64), and it's fixed in version: device-mapper-libs-1.02.90-1.fc21.x86_64.rpm (provided as part of lvm2-2.02.111-1.fc21), which is in Fedora-21 stable: $ bodhi -L lvm2 | grep f21 [. . .] f21-updates-testing lvm2-2.02.111-1.fc21 f21 lvm2-2.02.111-1.fc21 f21-updates-candidate lvm2-2.02.111-1.fc21 See rationale in related bug: https://bugzilla.redhat.com/show_bug.cgi?id=1164773#c12 libvirt-1.2.10-2.fc22 fails to connect to hypervisor; error: undefined symbol: virStorageFileCreate [Closing this as NOTABUG, as it's not a libvirt bug.]