RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1724808 - Compute logs show periodic libvirt rbd errors
Summary: Compute logs show periodic libvirt rbd errors
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 7.6
Assignee: Peter Krempa
QA Contact: Han Han
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-27 20:58 UTC by David Hill
Modified: 2023-09-07 20:11 UTC (History)
10 users (show)

Fixed In Version: libvirt-4.5.0-25.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-31 19:58:29 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4252871 0 None None None 2019-06-27 20:58:24 UTC
Red Hat Product Errata RHBA-2020:1094 0 None None None 2020-03-31 19:59:23 UTC

Description David Hill 2019-06-27 20:58:07 UTC 
Description of problem:
Compute logs show periodic libvirt rbd errors such as:

Jun 27 15:34:24 overcloud-compute-0 journal: 2019-06-27 15:34:24.821+0000: 42494: error : virStorageFileBackendForType:142 : internal error: missing storage backend for network files using rbd protocol

Version-Release number of selected component (if applicable):
libvirt-4.5.0-10.el7_6.3.x86_64                             Sat Mar 23 12:00:17 2019
libvirt-python-4.5.0-1.el7.x86_64                           Sat Mar 23 12:00:11 2019
openstack-ceilometer-api-7.1.1-7.el7ost.noarch              Sat Mar 23 12:00:32 2019
qemu-kvm-rhev-2.12.0-18.el7_6.1.x86_64                      Sat Mar 23 12:00:14 2019


How reproducible:
some VMs does it

Steps to Reproduce:
1. No clear steps
2.
3.

Actual results:
Error messages are present in the logs.

Expected results:
No error messages should be seen in the logs.

Additional info:
This happens to 2 customers with different releases as one is using RHOSP10 and the other is using RHOSP13.
Comment 2 Han Han 2019-07-23 07:05:45 UTC 
From the error msg, I guess it was caused by missing the rbd storage backend driver file /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so . But I failed to reproduce it. 
Peter, could you have a look at it?
Comment 3 Peter Krempa 2019-07-23 07:14:06 UTC 
Yes, that is the probable cause, but I'll need to investigate whether it makes sense in the code path to actually even report the error or we can just ignore it.
Comment 5 Peter Krempa 2019-08-12 16:47:24 UTC 
The fix should be quite simple (e.g. add a check whether the backend is supported prior to calling the function which causes the log entry), but I didn't get around to do it yet.
Comment 6 Peter Krempa 2019-08-15 08:29:55 UTC 
Fix proposed upstream: https://www.redhat.com/archives/libvir-list/2019-August/msg00542.html
Comment 7 Peter Krempa 2019-08-16 12:17:47 UTC 
fixed upstream:

commit 60b862cf9d6a335db65bbf2b011499dfa729ca2e
Author: Peter Krempa <pkrempa>
Date:   Wed Aug 14 18:46:09 2019 +0200

    qemu: Don't report some ignored errors in qemuDomainGetStatsOneBlockFallback
    
    The function ignores all errors from qemuStorageLimitsRefresh by calling
    virResetLastError. This still logs them. Since qemuStorageLimitsRefresh
    allows suppressing some, do so.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit ea26e22f94cea989b566e13e1930d76e06baf2c9
Author: Peter Krempa <pkrempa>
Date:   Wed Aug 14 18:15:20 2019 +0200

    qemu: Allow suppressing errors from qemuStorageLimitsRefresh
    
    qemuStorageLimitsRefresh uses qemuDomainStorageOpenStat internally and
    there are callers which don't care about the error. Propagate the
    skipInaccessible flag so that we can log less errors.
    
    Callers currently don't care about the return value change.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 3d7ea4165b890880f43074e1e5e6e10bf0ad21d6
Author: Peter Krempa <pkrempa>
Date:   Wed Aug 14 17:57:18 2019 +0200

    qemu: driver: Improve error suppression in qemuDomainStorageUpdatePhysical
    
    None of the callers of qemuDomainStorageUpdatePhysical care about
    errors.
    
    Use the new flag for qemuDomainStorageOpenStat which suppresses some
    errors and move the reset of the rest of the uncommon errors into this
    function. Document what is happening in a comment for the function.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit b074363136ddcf1599c36f11b9f5a5fe45c5b3e5
Author: Peter Krempa <pkrempa>
Date:   Wed Aug 14 17:51:23 2019 +0200

    util: storagefile: Don't report errors from virStorageSourceUpdatePhysicalSize
    
    virStorageSourceUpdatePhysicalSize is called only from
    qemuDomainStorageUpdatePhysical and all callers of it reset the libvirt
    error if -1 is returned.
    
    Don't bother setting the error in the first place.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit ba6c12df2ceb4df9bfb7ce95deef04f96bf29462
Author: Peter Krempa <pkrempa>
Date:   Wed Aug 14 16:00:27 2019 +0200

    qemu: Allow skipping some errors in qemuDomainStorageOpenStat
    
    Some callers of this function actually don't care about errors and reset
    it. The message is still logged which might irritate users in this case.
    
    Add a boolean flag which will do few checks whether it actually makes
    sense to even try opening the storage file. For local files we check
    whether it exists and for remote files we at first see whether we even
    have a storage driver backend for it in the first place before trying to
    open it.
    
    Other problems will still report errors but these are the most common
    scenarios which can happen here.
    
    This patch changes the return value of the function so that the caller
    is able to differentiate the possibilities.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 68639829c69ba3ed369413771bf0abceb33c3668
Author: Peter Krempa <pkrempa>
Date:   Tue Aug 13 12:41:40 2019 +0200

    util: Export virStorageFileSupportsBackingChainTraversal
    
    The function will be reused in the qemu snapshot code. The argument is
    turned into const similarly to the other virStorageFileSupports*
    functions.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>


commit e776194ad2501c9dcba8b4d9f1272342cd7f7c41
Author: Peter Krempa <pkrempa>
Date:   Mon Jul 29 17:43:22 2019 +0200

    util: storage: Allow checking whether virStorageFileCreate is supported
    
    Add virStorageFileSupportsCreate which allows silent check whether
    virStorageFileCreate is implemented.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit d30e0d3abc0cb082d8d97c53dcff2b978e3eb372
Author: Peter Krempa <pkrempa>
Date:   Mon Jul 29 17:28:34 2019 +0200

    util: storage: Refactor logic for using virStorageFileGetBackendForSupportCheck
    
    Modify the return value so that callers don't have to repeat logic.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

v5.6.0-203-g60b862cf9d
Comment 15 Han Han 2020-01-16 13:17:40 UTC 
Reproduced on libvirt-4.5.0-24.el7.x86_64 qemu-kvm-rhev-2.12.0-42.el7.x86_64
Steps:
Prepare a vm with rbd backed disk.
scenarios:
1. When vm is active, call `virsh domblkinfo` to get the rbd disk info
2. When vm is inactive, call `virsh domblkinfo` to get the rbd disk info
3. When vm is inactive, call `virsh domstats` to get domain stats

Verify on libvirt-4.5.0-31.el7.x86_64:
1. Keep vm active, check domstats and domblkinfo command:
Set libvirtd log conf as following:
log_level = 3
log_filters="1:qemu 1:libvirt 4:object 4:json 4:event 1:util"



➜  ~ virsh domstats pc --block
Domain: 'pc'
  block.count=2
  block.0.name=sda
  block.0.path=/var/lib/libvirt/images/pc.qcow2
  block.0.rd.reqs=7561
  block.0.rd.bytes=193991168
  block.0.rd.times=1825193494
  block.0.wr.reqs=260
  block.0.wr.bytes=32218112
  block.0.wr.times=54587758
  block.0.fl.reqs=66
  block.0.fl.times=1063439482
  block.0.allocation=866947072
  block.0.capacity=10737418240
  block.0.physical=832176128
  block.1.name=vdb
  block.1.rd.reqs=110
  block.1.rd.bytes=3588608
  block.1.rd.times=15953568
  block.1.wr.reqs=0
  block.1.wr.bytes=0
  block.1.wr.times=0
  block.1.fl.reqs=0
  block.1.fl.times=0
  block.1.allocation=0
  block.1.capacity=209715200
  block.1.physical=209715200

➜  ~ virsh domblkinfo pc vdb  
Capacity:       209715200
Allocation:     209715200
Physical:       209715200

No error like 'internal error: missing storage backend for network files using rbd protocol' is found in libvirtd log.


2. Keep vm inactive. Check domstats and domblkinfo command:
# virsh domstats pc --block
Domain: 'pc'
  block.count=2
  block.0.name=sda
  block.0.path=/var/lib/libvirt/images/pc.qcow2
  block.0.allocation=832176128
  block.0.capacity=10737418240
  block.0.physical=866975744
  block.1.name=vdb

# virsh domblkinfo pc vdb
error: internal error: missing storage backend for network files using rbd protocol

Error only appears on inactive domblkinfo.
Comment 16 Han Han 2020-02-04 08:57:12 UTC 
Verified as comment15.
Comment 18 errata-xmlrpc 2020-03-31 19:58:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1094
Note You need to log in before you can comment on or make changes to this bug.