Created attachment 1513717 [details]
dependencies list + ansible ovirt-host-upgrade.yml  output

Created attachment 1513719 [details]
dependencies list + ansible ovirt-host-upgrade.yml  output

Moving to RHEL libvirt, seems like libvirt-daemon-driver-storage-rbd is missing proper requirement on librbd1 correct version.

Proposing for 7.6 batch update

It is caused by librbd API incompatibility. Look at following code:

#if LIBRBD_VERSION_CODE > 265
        r = rbd_diff_iterate2(image, snaps[i].name, 0, info.size, 0, 1,
                              virStorageBackendRBDIterateCb, (void *)&diff);
#else
        r = rbd_diff_iterate(image, snaps[i].name, 0, info.size,
                             virStorageBackendRBDIterateCb, (void *)&diff);
#endif

Libvirt codes can be compiled with LIBRBD_VERSION_CODE > 265 or LIBRBD_VERSION_CODE <= 265. However, it was compiled with librbd version <= 265 on RHEL7.4 while compiled with librbd version > 265 on RHEL7.6. They invoke different librbd API but the update of libvirt doesn't update librbd on the same time. That is why this error happened.

I thinks updating librbd will solve this problem.

For developer and QEs, the question is , how to detect these ABI incompatibility earlier, because it is common on C/C++ compiled software. See also: https://bugzilla.redhat.com/show_bug.cgi?id=1351106

To detect these 'undefined symbol' error, we can do this upgrade testing:
1. Update softwares or the system, it depends by the scenarios of customers
2. Browser all the ELF format files (the executable binary and so files), check them with ldd:
# ldd -r ELF_FILE

If its output is like following:
undefined symbol: rbd_diff_iterate2     (/usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so)
undefined symbol: virStorageBackendRegister     (/usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so)

Obviously, we catch the undefined symbol issue here.

This is caused by librbd.so library:

# rpm -ql librbd1-0.94.5-2.el7.x86_64
/usr/lib/udev/rules.d/50-rbd.rules
/usr/lib64/librbd.so.1
/usr/lib64/librbd.so.1.0.0

# rpm -ql librbd1-10.2.5-4.el7.x86_64
/usr/lib/udev/rules.d/50-rbd.rules
/usr/lib64/librbd.so.1
/usr/lib64/librbd.so.1.0.0

Even though librdb.so version is the same in both packages, they don't provide
the same set of APIs and at the same time the symbols exported by librbd.so
are not versioned. Thus when libvirt is built, rpm just adds
librbd.so.1()(64bit) dependency, which is unfortunately satisfied by both
packages.

We have to workaround this by adding an explicit dependency on new enough
librbd1 package.

*** Bug 1673101 has been marked as a duplicate of this bug. ***

Note that this bug is only tracking the resolution of the missing versioned dependency from libvirt to librbd.

This assumes that the newer librbd is indeed available from the host's configured yum repository.

As such this missing dep should not a problem when doing a full yum update, which would pull in all new packages. It is only a problem if selectively updating individual packages whereby you request a libvirt update but exclude a request for librbd.

If the newer librbd is not available via yum that would point to a bad yum repository configuration.

Having been bitten by this on Friday I'm quite perplex by Red Hat's stance:

> If the newer librbd is not available via yum that would point to a bad yum repository configuration.


One would think Red Hat reconsiders their descsion when breaking userspace. To wit libvirt should depend on the proper upstream version.   There is nothing wrong with our yum config.

As I tried to explain on IRC that we need to know the steps which lead to such
situation. If it is a jenkins job, we need to see what steps it is doing so
that we can try to reproduce locally without jenkins.

If you encounter this issue when upgrading please see https://access.redhat.com/solutions/4090431

Verified on 
libvirt-4.5.0-17.virtcov.el7.x86_64

[root@localhost ~]# rpm -qa |grep libvirt
libvirt-daemon-3.9.0-14.virtcov.el7_5.8.x86_64
libvirt-daemon-config-nwfilter-3.9.0-14.virtcov.el7_5.8.x86_64
...
libvirt-3.9.0-14.virtcov.el7_5.8.x86_64

[root@localhost ~]# rpm -qa librbd1
librbd1-0.94.5-2.el7.x86_64

[root@localhost ~]# yum update libvirt
...
librados2                                    x86_64        1:10.2.5-4.el7              RHEL-Server        1.8 M
 librbd1                                      x86_64        1:10.2.5-4.el7              RHEL-Server        2.4 M
 libvirt-client                               x86_64        4.5.0-17.virtcov.el7        Libvirt-CI         537 k
....

[root@localhost ~]# rpm -qa libvirt librbd1
librbd1-10.2.5-4.el7.x86_64
libvirt-4.5.0-17.virtcov.el7.x86_64

Librbd1 has been updated to version 10.2.5-4.el7.x86_64
Work as expected

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2294