Bug 1316911

Summary: 'libvirt_driver_storage.so' fails to load due to "undefined symbol: rbd_diff_iterate2"
Product: [Community] Virtualization Tools Reporter: Kashyap Chamarthy <kchamart>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED NOTABUG QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: rbalakri
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-11 13:49:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kashyap Chamarthy 2016-03-11 13:08:12 UTC
Description of problem
----------------------

Post update to libvirt-1.3.2-2.fc25 (or libvirt-1.3.2-1.fc24), and
restarting the libvirt daemon, (a) previously existing VMs are not
enumerated; (b) explict attempt to define a guest fails with:

      $ virsh define /etc/libvirt/qemu/devstack1.xml
      error: Failed to define domain from devstack1.xml
      error: invalid argument: could not find capabilities for arch=x86_64

While we're at it, I tried to explictly get capabilities this way, sure
enough it fails:

    $ virsh cpu-models x86_64
    error: failed to get CPU model names
    error: this function is not supported by the connection driver: virConnectGetCPUModelNames

Okay, this error gives us a hint to check the `virsh uri`.  Let's do
that:

    $ virsh uri
    lxc:///

Whoops!  Upgrading packages has reset the URI to "lxc:///", while we're
expecting it to be "qemu:///system".

Okay, let's try to explicitly set the URI, and try:

    $ virsh -c qemu:///system list
    error: failed to connect to the hypervisor
    error: no connection driver available for qemu:///system

Why is it so?  Let's quickly check if we have
'libvirt-daemon-driver-qemu' package is installed.  Sure, we have it:  

    $ rpm -q libvirt-daemon-driver-qemu
    libvirt-daemon-driver-qemu-1.3.2-2.fc25.x86_64

But is 'libvirt_driver_storage.so' loaded for sure?  Let's check
`journalctl` [I manually wrapped the error message]:

   $ journalctl --since=today | grep libvirt_driver_storage.so Mar 11
   06:04:04 foohost libvirtd[32474]: failed to load module
   /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so
   /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so:
   undefined symbol: rbd_diff_iterate2
   [...]

Ah-ha!  So, indeed, it failed to load due to "undefined symbol:
rbd_diff_iterate2".

    * * *

Downgrading the packages (`dnf downgrade libvirt\* qemu\*
--allowerasing`) and restarting the libvirtd restores the correct
behavior (for both defining guests, and getting CPU model info) -- as it
sets the URI back to "qemu:///system").

Maybe these are the perils of selective testing from Rawhide (or F24),
while on F23.

Version
-------

Tested with libvirt/QEMU packages updated to Rawhide

    libvirt-1.3.2-2.fc25.x86_64
    qemu-system-x86-2.5.0-9.fc25.x86_64

This is also reproducible with F24 (branched) with:

    libvirt-1.3.2-1.fc24.x86_64
    qemu-system-x86-2.5.0-8.fc24.x86_64


How reproducible: Consistently


Steps to reproduce
------------------

(1) If testing with packages Rawhide, update libvirt & QEMU packages : 

      $ sudo dnf install fedora-repos-rawhide
      $ sudo dnf --releasever=24 update libvirt\* qemu\*

    If you're on F24:

      $ sudo dnf --releasever=rawhide update libvirt\* qemu\*

(2) Restart the libvirt daemon:

      $ sudo systemctl restart libvirtd

(3) Enumerate instances (assuming you already had a few instances
    defined before doing the package update):

      $ virsh list --all

    Nothing comes up.

    So, try to explicitly define a VM:

      $ virsh define devstack1.xml 


Actual results
--------------

Post libvirt/QEMU package update, the libvirt storage driver fails to
load, which results in the libvirt URI to changed to to "lxc:///" (while
we're expecting it to be "qemu:///system").  Which further leads to the 
two commands from step (3) in the previous section fail (as expected?)
-- no instances are listed, and explicit attempt to define a guest fails
with errors mentioned in the description.


Expected results
----------------

Libvirt storage driver should be loaded successfully.

Comment 1 Kashyap Chamarthy 2016-03-11 13:49:17 UTC
As expected, updating the librbrd1* packages resolves the issue.

Update the librbd1* packages:

    $ dnf --releasever=24 update librbrd1 librbd1-devel libsolv

[NB: 'libsolv' because, DNF itself breaks with the version in F23 
(libsolv-0.6.14-7), so update to libsolv-0.6.19-2 on F24.)

Restart libvirtd:

    $ systemctl restart libvirtd

_Now_ the URI is set correctly:

    $ virsh uri
    qemu:///system


Closing it as NOTABG, as it's a dependency issue, and not libvirt bug per se.  But, still wanted to report, in case others who hit this don't have to go through this.