Bug 1658652 - Missing librbd1 dependecy for libvirt-daemon-driver-storage-rbd is causing problem with RHEL host upgrade
Summary: Missing librbd1 dependecy for libvirt-daemon-driver-storage-rbd is causing pr...
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.5
Hardware: x86_64
OS: Linux
Target Milestone: rc
: 7.6
Assignee: Jiri Denemark
QA Contact: gaojianan
: 1673101 (view as bug list)
Depends On:
Blocks: 1670798 1679569
TreeView+ depends on / blocked
Reported: 2018-12-12 15:44 UTC by Jaroslav Spanko
Modified: 2019-08-06 13:14 UTC (History)
18 users (show)

Fixed In Version: libvirt-4.5.0-11.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1679569 (view as bug list)
Last Closed: 2019-08-06 13:14:35 UTC
Target Upstream Version:

Attachments (Terms of Use)
dependencies list + ansible ovirt-host-upgrade.yml output (111.50 KB, text/plain)
2018-12-12 16:33 UTC, Jaroslav Spanko
no flags Details
dependencies list + ansible ovirt-host-upgrade.yml output (88.21 KB, text/plain)
2018-12-12 16:34 UTC, Jaroslav Spanko
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1706907 0 high CLOSED upgrade documentation for OSP9 should link to KCS covering how to deal with Ceph EOL 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 3746691 0 None None None 2019-01-03 19:47:52 UTC
Red Hat Product Errata RHSA-2019:2294 0 None None None 2019-08-06 13:14:52 UTC

Description Jaroslav Spanko 2018-12-12 15:44:40 UTC
Description of problem:
Upgrading RHEL based host in 4.2 is causing hosts Non-responsive due to failing libvirt. This is due to missing dependency so librbd1 is not updated during the upgrade.
error : virModuleLoadFile:53 : Interner Fehler: Failed to load module '/usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so': /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so: undefined symbol: rbd_diff_iterate2

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. RHEL based host 7.4 , 7.5
2. Upgrade host via GUI 
3. The librbd1 package will stay on old version and libvirt will be updated which result in libvirt fail

Actual results:
librbd1 not upgraded during the host upgrade

Expected results:
Add dependency for librbd1

Additional info:
This was reported by a few of our customers during the last week, the fix is easy, yum update or yum update librbd1 after that is host working again but this should be covered by ovirt-host during upgrade i guess ...

List of packages before upgrade
librbd1-0.94.5-2.el7.x86_64                                 Wed Nov 29 07:03:58 2017 ======================>  
libvirt-client-3.2.0-14.el7_4.7.x86_64                      Wed Mar 14 16:19:51 2018
libvirt-daemon-3.2.0-14.el7_4.7.x86_64                      Wed Mar 14 16:20:11 2018
fence-agents-rhevm-4.0.11-66.el7_4.3.x86_64                 Wed Mar 14 16:18:47 2018
qemu-img-rhev-2.9.0-16.el7_4.13.x86_64                      Wed Mar 14 16:20:17 2018
qemu-kvm-common-rhev-2.9.0-16.el7_4.13.x86_64               Wed Mar 14 16:20:46 2018
qemu-kvm-rhev-2.9.0-16.el7_4.13.x86_64                      Wed Mar 14 16:20:47 2018
qemu-kvm-tools-rhev-2.9.0-16.el7_4.13.x86_64                Wed Mar 14 16:21:36 2018

and after upgrade
libvirt-4.5.0-10.el7_6.3.x86_64                             Tue Dec 11 15:19:06 2018
libvirt-bash-completion-4.5.0-10.el7_6.3.x86_64             Tue Dec 11 15:19:05 2018
libvirt-client-4.5.0-10.el7_6.3.x86_64                      Tue Dec 11 15:19:06 2018
librbd1-0.94.5-2.el7.x86_64                                 Wed Nov 29 07:03:58 2017 ======================> only one which were not upgraded
fence-agents-rhevm-4.0.11-66.el7_4.3.x86_64                 Wed Mar 14 16:18:47 2018
qemu-img-rhev-2.12.0-18.el7_6.1.x86_64                      Tue Dec 11 15:17:38 2018
qemu-kvm-common-rhev-2.12.0-18.el7_6.1.x86_64               Tue Dec 11 15:18:17 2018
qemu-kvm-rhev-2.12.0-18.el7_6.1.x86_64                      Tue Dec 11 15:19:08 2018
qemu-kvm-tools-rhev-2.9.0-16.el7_4.13.x86_64                Wed Mar 14 16:21:36 2018

Also yum install ovirt-host does not have librbd1 as dependency.
Hope i choose good component and ovirt team ...
Thanks !

Comment 3 Jaroslav Spanko 2018-12-12 16:33:44 UTC
Created attachment 1513717 [details]
dependencies list + ansible ovirt-host-upgrade.yml  output

Comment 4 Jaroslav Spanko 2018-12-12 16:34:04 UTC
Created attachment 1513719 [details]
dependencies list + ansible ovirt-host-upgrade.yml  output

Comment 5 Sandro Bonazzola 2018-12-13 08:52:14 UTC
Moving to RHEL libvirt, seems like libvirt-daemon-driver-storage-rbd is missing proper requirement on librbd1 correct version.

Comment 7 Sandro Bonazzola 2018-12-13 08:53:49 UTC
Proposing for 7.6 batch update

Comment 8 Han Han 2018-12-18 02:50:36 UTC
It is caused by librbd API incompatibility. Look at following code:

        r = rbd_diff_iterate2(image, snaps[i].name, 0, info.size, 0, 1,
                              virStorageBackendRBDIterateCb, (void *)&diff);
        r = rbd_diff_iterate(image, snaps[i].name, 0, info.size,
                             virStorageBackendRBDIterateCb, (void *)&diff);

Libvirt codes can be compiled with LIBRBD_VERSION_CODE > 265 or LIBRBD_VERSION_CODE <= 265. However, it was compiled with librbd version <= 265 on RHEL7.4 while compiled with librbd version > 265 on RHEL7.6. They invoke different librbd API but the update of libvirt doesn't update librbd on the same time. That is why this error happened.

I thinks updating librbd will solve this problem.

For developer and QEs, the question is , how to detect these ABI incompatibility earlier, because it is common on C/C++ compiled software. See also: https://bugzilla.redhat.com/show_bug.cgi?id=1351106

Comment 9 Han Han 2018-12-18 06:51:20 UTC
To detect these 'undefined symbol' error, we can do this upgrade testing:
1. Update softwares or the system, it depends by the scenarios of customers
2. Browser all the ELF format files (the executable binary and so files), check them with ldd:
# ldd -r ELF_FILE

If its output is like following:
undefined symbol: rbd_diff_iterate2     (/usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so)
undefined symbol: virStorageBackendRegister     (/usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so)

Obviously, we catch the undefined symbol issue here.

Comment 10 Jiri Denemark 2019-02-20 12:48:54 UTC
This is caused by librbd.so library:

# rpm -ql librbd1-0.94.5-2.el7.x86_64

# rpm -ql librbd1-10.2.5-4.el7.x86_64

Even though librdb.so version is the same in both packages, they don't provide
the same set of APIs and at the same time the symbols exported by librbd.so
are not versioned. Thus when libvirt is built, rpm just adds
librbd.so.1()(64bit) dependency, which is unfortunately satisfied by both

We have to workaround this by adding an explicit dependency on new enough
librbd1 package.

Comment 12 Daniel Berrangé 2019-02-20 14:41:56 UTC
*** Bug 1673101 has been marked as a duplicate of this bug. ***

Comment 13 Daniel Berrangé 2019-02-20 14:53:54 UTC
Note that this bug is only tracking the resolution of the missing versioned dependency from libvirt to librbd.

This assumes that the newer librbd is indeed available from the host's configured yum repository.

As such this missing dep should not a problem when doing a full yum update, which would pull in all new packages. It is only a problem if selectively updating individual packages whereby you request a libvirt update but exclude a request for librbd.

If the newer librbd is not available via yum that would point to a bad yum repository configuration.

Comment 17 F. Flinstone 2019-03-03 12:08:19 UTC
Having been bitten by this on Friday I'm quite perplex by Red Hat's stance:

> If the newer librbd is not available via yum that would point to a bad yum repository configuration.

One would think Red Hat reconsiders their descsion when breaking userspace. To wit libvirt should depend on the proper upstream version.   There is nothing wrong with our yum config.

Comment 27 Jiri Denemark 2019-04-24 15:20:16 UTC
As I tried to explain on IRC that we need to know the steps which lead to such
situation. If it is a jenkins job, we need to see what steps it is doing so
that we can try to reproduce locally without jenkins.

Comment 28 John Fulton 2019-05-06 13:29:04 UTC
If you encounter this issue when upgrading please see https://access.redhat.com/solutions/4090431

Comment 29 gaojianan 2019-05-16 06:31:29 UTC
Verified on 

[root@localhost ~]# rpm -qa |grep libvirt

[root@localhost ~]# rpm -qa librbd1

[root@localhost ~]# yum update libvirt
librados2                                    x86_64        1:10.2.5-4.el7              RHEL-Server        1.8 M
 librbd1                                      x86_64        1:10.2.5-4.el7              RHEL-Server        2.4 M
 libvirt-client                               x86_64        4.5.0-17.virtcov.el7        Libvirt-CI         537 k

[root@localhost ~]# rpm -qa libvirt librbd1

Librbd1 has been updated to version 10.2.5-4.el7.x86_64
Work as expected

Comment 32 errata-xmlrpc 2019-08-06 13:14:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.