Bug 1165068 - libvirt-1.2.10-2.fc22: fails to define KVM guest XML - internal error: unexpected domain type kvm, expecting lxc
Summary: libvirt-1.2.10-2.fc22: fails to define KVM guest XML - internal error: unexpe...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-18 09:23 UTC by Kashyap Chamarthy
Modified: 2014-11-20 15:33 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-11-20 15:33:51 UTC
Embargoed:


Attachments (Terms of Use)
Guest XML used to test (3.48 KB, text/html)
2014-11-18 09:23 UTC, Kashyap Chamarthy
no flags Details

Description Kashyap Chamarthy 2014-11-18 09:23:11 UTC
Created attachment 958476 [details]
Guest XML used to test

Description of problem
----------------------

When defining a libvirt KVM guest XML with libvirt,

  $ grep kvm devstack2.xml  -A1 -B1
  
  <domain type='kvm'>
    <name>devstack2</name>
  --
    <devices>
      <emulator>/usr/bin/qemu-kvm</emulator>
      <disk type='file' device='disk'>


  $ virsh define devstack2.xml                                                                                          
  error: Failed to define domain from devstack2.xml
  error: internal error: unexpected domain type kvm, expecting lxc


With libvirt debug logs enabled:

[. . .]
2014-11-18 07:57:14.785+0000: 662: error : virDomainDefParseXML:12154 : internal error: unexpected domain type kvm, expecting lxc
2014-11-18 07:58:12.446+0000: 1035: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7fc6ed25f440 sock=0x7fc6ed25f040
2014-11-18 07:58:12.459+0000: 1035: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7fc6ed25dc90 sock=0x7fc6ed25d890
2014-11-18 08:00:19.490+0000: 1035: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7fc6ed25e5d0 sock=0x7fc6ed25e1d0
2014-11-18 08:00:19.502+0000: 1035: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7fc6ed25dad0 sock=0x7fc6ed25d6d0
2014-11-18 08:05:17.833+0000: 1178: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7f454ffd16f0 sock=0x7f454ffd12f0
2014-11-18 08:07:02.488+0000: 1178: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7f454ffd0890 sock=0x7f454ffd0360
2014-11-18 08:07:02.502+0000: 1181: error : virDomainDefParseXML:12154 : internal error: unexpected domain type kvm, expecting lxc
2014-11-18 08:12:02.140+0000: 1178: debug : virNetServerClientNewInternal:388 : RPC_SERVER_CLIENT_NEW: client=0x7f454ffd1b20 sock=0x7f454ffd1440
2014-11-18 08:12:02.157+0000: 1183: error : virDomainDefParseXML:12154 : internal error: unexpected domain type kvm, expecting lxc
[. . .]

Version
-------

  $ rpm -qa | grep -i libvirt
  libvirt-daemon-driver-storage-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-lxc-1.2.10-2.fc22.x86_64
  libvirt-docs-1.2.10-2.fc22.x86_64
  libvirt-daemon-qemu-1.2.10-2.fc22.x86_64
  libvirt-devel-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-interface-1.2.10-2.fc22.x86_64
  libvirt-daemon-config-nwfilter-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-libxl-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-uml-1.2.10-2.fc22.x86_64
  libvirt-client-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-nodedev-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-nwfilter-1.2.10-2.fc22.x86_64
  libvirt-daemon-kvm-1.2.10-2.fc22.x86_64
  libvirt-python-1.2.10-1.fc22.x86_64
  libvirt-daemon-driver-xen-1.2.10-2.fc22.x86_64
  libvirt-daemon-config-network-1.2.10-2.fc22.x86_64
  libvirt-daemon-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-qemu-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-vbox-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-secret-1.2.10-2.fc22.x86_64
  libvirt-1.2.10-2.fc22.x86_64
  libvirt-daemon-driver-network-1.2.10-2.fc22.x86_64
  libvirt-debuginfo-1.2.10-2.fc22.x86_64
  
  $ uname -r; rpm -q qemu-system-x86
  3.18.0-0.rc4.git2.1.fc22.x86_64
  qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64


How reproducible: Consistently.


Steps to Reproduce
------------------

1. Install the said libvirt/QEMU versions from Rawhide

    $ sudo yum update libvirt\* qemu\* --enablerepo=rawhide

2. Define the guest XML (in attachment)

    $ virsh define devstack2.xml


Actual results 
--------------

 $ virsh define devstack2.xml
  error: Failed to define domain from devstack2.xml
  error: internal error: unexpected domain type kvm, expecting lxc


Expected results
----------------

KVM guest XML should be successfully defined.


Additional info
---------------

(a) The same guest XML was defined successfully by libvirt on a Fedora-21
    machine with the below versions:

    $ uname -r; rpm -q libvirt-daemon-kvm qemu-system-x86
    3.17.1-302.fc21.x86_64
    libvirt-daemon-kvm-1.2.9-4.fc21.x86_64
    qemu-system-x86-2.1.2-6.fc21.x86_64

(b) Sanity checks:

    $ systemctl status libvirtd | grep -i Active
    Active: active (running) since Tue 2014-11-18 04:06:55 EST; 3min 30s ago

    $ virt-host-validate
      QEMU: Checking for hardware virtualization                                 : PASS
      QEMU: Checking for device /dev/kvm                                         : PASS
      QEMU: Checking for device /dev/vhost-net                                   : PASS
      QEMU: Checking for device /dev/net/tun                                     : PASS
       LXC: Checking for Linux >= 2.6.26                                         : PASS

Comment 1 Peter Krempa 2014-11-18 09:31:45 UTC
Could you please post the result of "virsh uri".

Comment 2 Kashyap Chamarthy 2014-11-18 09:39:43 UTC
(In reply to Peter Krempa from comment #1)
> Could you please post the result of "virsh uri".

Sure, weird - it's set to LXC:

  $ virsh uri
  lxc:///

Comment 3 Kashyap Chamarthy 2014-11-18 11:10:01 UTC
And, setting the LIBVIRT_DEFAULT_URI explicitly doesn't help here either.

$ export LIBVIRT_DEFAULT_URI=qemu:///system
$ echo $LIBVIRT_DEFAULT_URI
qemu:///system

$ rpm -ql libvirt-daemon-driver-qemu
/etc/libvirt/qemu
/etc/libvirt/qemu-lockd.conf
/etc/libvirt/qemu.conf
/etc/logrotate.d/libvirtd.qemu
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
/usr/share/augeas/lenses/libvirtd_qemu.aug
/usr/share/augeas/lenses/tests/test_libvirtd_qemu.aug
/var/cache/libvirt/qemu
/var/lib/libvirt/qemu
/var/lib/libvirt/qemu/channel
/var/lib/libvirt/qemu/channel/target
/var/lib/libvirt/qemu/nvram
/var/log/libvirt/qemu
/var/run/libvirt/qemu


And, I couldn't reproduce this case again on a fresh F21, where I upgraded the libvirt/QEMU RPMs from Rawhide:

$ rpm -qa | grep -i libvirt; rpm -q qemu-system-x86; uname -r
libvirt-client-1.2.10-2.fc22.x86_64
libvirt-daemon-driver-secret-1.2.10-2.fc22.x86_64
libvirt-daemon-1.2.10-2.fc22.x86_64
libvirt-daemon-driver-storage-1.2.10-2.fc22.x86_64
libvirt-daemon-driver-nodedev-1.2.10-2.fc22.x86_64
libvirt-daemon-kvm-1.2.10-2.fc22.x86_64
libvirt-daemon-driver-network-1.2.10-2.fc22.x86_64
libvirt-daemon-driver-interface-1.2.10-2.fc22.x86_64
libvirt-python-1.2.10-1.fc22.x86_64
libvirt-daemon-driver-nwfilter-1.2.10-2.fc22.x86_64
libvirt-daemon-driver-qemu-1.2.10-2.fc22.x86_64
qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64
3.17.2-300.fc21.x86_64

$ virsh uri
qemu:///system

Comment 4 Kashyap Chamarthy 2014-11-18 11:16:11 UTC
(In reply to Kashyap Chamarthy from comment #3)
> And, setting the LIBVIRT_DEFAULT_URI explicitly doesn't help here either.
> 
> $ export LIBVIRT_DEFAULT_URI=qemu:///system
> $ echo $LIBVIRT_DEFAULT_URI
> qemu:///system
> 
> $ rpm -ql libvirt-daemon-driver-qemu
> /etc/libvirt/qemu
> /etc/libvirt/qemu-lockd.conf
> /etc/libvirt/qemu.conf
> /etc/logrotate.d/libvirtd.qemu
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> /usr/share/augeas/lenses/libvirtd_qemu.aug
> /usr/share/augeas/lenses/tests/test_libvirtd_qemu.aug
> /var/cache/libvirt/qemu
> /var/lib/libvirt/qemu
> /var/lib/libvirt/qemu/channel
> /var/lib/libvirt/qemu/channel/target
> /var/lib/libvirt/qemu/nvram
> /var/log/libvirt/qemu
> /var/run/libvirt/qemu

Just thoguht I'd claridy, on the above F21 host, where setting the DEFAULT URI as above didn't make a difference, as it still couldn't find the driver to QEMU somehow, desipte having the libvirt-daemon-driver-qemu RPM installed:

  $ virsh uri
  error: failed to connect to the hypervisor
  error: no valid connection
  error: no connection driver available for qemu:///system
  error: Failed to reconnect to the hypervisor
> 
> 

And, the below is on a completely different F21 host which was freshly installed where I couldn't reproduce the issue.

> And, I couldn't reproduce this case again on a fresh F21, where I upgraded
> the libvirt/QEMU RPMs from Rawhide:
> 
> $ rpm -qa | grep -i libvirt; rpm -q qemu-system-x86; uname -r
> libvirt-client-1.2.10-2.fc22.x86_64
> libvirt-daemon-driver-secret-1.2.10-2.fc22.x86_64
> libvirt-daemon-1.2.10-2.fc22.x86_64
> libvirt-daemon-driver-storage-1.2.10-2.fc22.x86_64
> libvirt-daemon-driver-nodedev-1.2.10-2.fc22.x86_64
> libvirt-daemon-kvm-1.2.10-2.fc22.x86_64
> libvirt-daemon-driver-network-1.2.10-2.fc22.x86_64
> libvirt-daemon-driver-interface-1.2.10-2.fc22.x86_64
> libvirt-python-1.2.10-1.fc22.x86_64
> libvirt-daemon-driver-nwfilter-1.2.10-2.fc22.x86_64
> libvirt-daemon-driver-qemu-1.2.10-2.fc22.x86_64
> qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64
> 3.17.2-300.fc21.x86_64
> 
> $ virsh uri
> qemu:///system

Comment 5 Daniel Berrangé 2014-11-18 11:42:10 UTC
> Just thoguht I'd claridy, on the above F21 host, where setting the DEFAULT
> URI as above didn't make a difference, as it still couldn't find the driver
> to QEMU somehow, desipte having the libvirt-daemon-driver-qemu RPM installed:
> 
>   $ virsh uri
>   error: failed to connect to the hypervisor
>   error: no valid connection
>   error: no connection driver available for qemu:///system
>   error: Failed to reconnect to the hypervisor

So, since you have the RPM installed, the only way you'd get this error is if libvirtd was unable to load the QEMU driver module. This should be visible as an error message in the libvirtd logs, so please check the journald logs for libvirtd.service.

Comment 6 Kashyap Chamarthy 2014-11-18 12:06:15 UTC
(In reply to Daniel Berrange from comment #5)
> > Just thoguht I'd claridy, on the above F21 host, where setting the DEFAULT
> > URI as above didn't make a difference, as it still couldn't find the driver
> > to QEMU somehow, desipte having the libvirt-daemon-driver-qemu RPM installed:
> > 
> >   $ virsh uri
> >   error: failed to connect to the hypervisor
> >   error: no valid connection
> >   error: no connection driver available for qemu:///system
> >   error: Failed to reconnect to the hypervisor
> 
> So, since you have the RPM installed, the only way you'd get this error is
> if libvirtd was unable to load the QEMU driver module. This should be
> visible as an error message in the libvirtd logs, so please check the
> journald logs for libvirtd.service.

Ah, thank you - seems like we found the root cause (need newer 'device-mapper').

From `journalctl`:

$ journalctl -u libvirtd --since=yesterday -p err
[. . .] failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference
.
.
[. . .] failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: virStorageFileCreate


On the machine, I'm using:

rpm -qa | grep -i device-mapper
device-mapper-libs-1.02.88-2.fc21.x86_64
device-mapper-persistent-data-0.3.2-4.fc21.x86_64
device-mapper-1.02.88-2.fc21.x86_64
device-mapper-event-1.02.88-2.fc21.x86_64
device-mapper-devel-1.02.88-2.fc21.x86_64
device-mapper-event-libs-1.02.88-2.fc21.x86_64

Updating to version resolved the issue: device-mapper-1.02.90-1.fc21.x86_64

A slightly more verbose `journalctl` error (note - I truncated the long hostname to 'foobar' from the `journalctl` output):

------------
$ journalctl -u libvirtd --since=yesterday -p err
[. . .]
Nov 17 04:33:47 foobar libvirtd[662]: no connection driver available for <null>
Nov 17 04:33:47 foobar libvirtd[662]: End of file while reading data: Input/output error
Nov 17 04:33:47 foobar libvirtd[662]: no connection driver available for <null>
Nov 17 04:33:47 foobar libvirtd[662]: End of file while reading data: Input/output error
Nov 17 04:33:55 foobar libvirtd[10800]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference
Nov 17 04:33:55 foobar libvirtd[10800]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: virStorageFileCreate
Nov 17 04:52:27 foobar libvirtd[10936]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference
Nov 17 04:52:27 foobar libvirtd[10936]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: virStorageFileCreate
<null>
Nov 17 06:46:15 foobar libvirtd[11127]: End of file while reading data: Input/output error
------------

(And as it can be guesed: on the F21 machine where I couldn't reproduce with identical libvirt/QEMU RPMs, it had the correct 'device-mapper' RPM version: device-mapper-1.02.90-1.fc21.x86_64)

Comment 7 Kashyap Chamarthy 2014-11-18 13:04:04 UTC
You can reproduce this issue by downgrading to device-mapper-1.02.88-2.fc21.x86_64.rpm (and its dependencies), it's a sub-package of  lvm2-2.02.109-2.fc21.

Builds can be found from here:

http://koji.fedoraproject.org/koji/buildinfo?buildID=561650

Comment 8 Kashyap Chamarthy 2014-11-20 15:33:51 UTC
Since it's a device-mapper-libs bug (offending version: 
device-mapper-1.02.88-2.fc21.x86_64), and it's fixed in version:
device-mapper-libs-1.02.90-1.fc21.x86_64.rpm (provided as part of
lvm2-2.02.111-1.fc21), which is in Fedora-21 stable:

  $ bodhi -L lvm2  | grep f21
    [. . .]
         f21-updates-testing  lvm2-2.02.111-1.fc21
                         f21  lvm2-2.02.111-1.fc21
       f21-updates-candidate  lvm2-2.02.111-1.fc21


See rationale in related bug: 

  https://bugzilla.redhat.com/show_bug.cgi?id=1164773#c12
  libvirt-1.2.10-2.fc22 fails to connect to hypervisor; 
  error: undefined symbol: virStorageFileCreate


[Closing this as NOTABUG, as it's not a libvirt bug.]


Note You need to log in before you can comment on or make changes to this bug.