Bug 2125432

Summary: [ceph-Dashboard] - RBD images are not loaded up in the dashboard when we have more images across pools
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Preethi <pnataraj>
Component: Ceph-DashboardAssignee: Pere Diaz Bou <pdiazbou>
Status: CLOSED ERRATA QA Contact: Preethi <pnataraj>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.0CC: ceph-eng-bugs, cephqe-warriors, ekristov, pdiazbou
Target Milestone: ---Keywords: TestBlocker
Target Release: 6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-17.2.3-25.el9cp Doc Type: Bug Fix
Doc Text:
Cause: Missing feature commits in 6.x Consequence: Fix: cherry-pick missing commits to 6.0 patches Result:
 Story Points: ---
Clone Of: Environment:
Last Closed: 2023-03-20 18:58:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
rbd_dashboard none

Description Preethi 2022-09-09 02:27:59 UTC 
Created attachment 1910582 [details]
rbd_dashboard

Created attachment 1910582 [details]
rbd_dashboard

Description of problem:[ceph-Dashboard] - RBD images are not loaded up in the dashboard when we have more images across pools


Version-Release number of selected component (if applicable):
ceph version 17.2.3-21.el9cp (c988b360f12cbb4bc1c80ee7a9771814bc0f49d6) quincy 

How reproducible:


Steps to Reproduce:
1. Have a cluster with 5.2 and perform upgrade to 6.0 with 250+images across poos cluster with 60+ filled (OS is upgraded from 8.6 to 9.0 before performing ceph upgrade to 6.0)
2. once upgrade is successful check the dashboard RBD->images sections
3. Observe the behaviour 
We " could not load images check cluster health"

NOTE: cluster health status was in good condition

Actual results:
RBD images are not listed after upgrade to 6.0

Expected results:

We should all images listed in RBS status
Additional info:


magna021
https://10.8.128.22:8443/#/login
admin123/myPassword

Snippet of mgr logs:
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/_ping): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/_ping): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/_ping): [errno: 111] Connection refused
::ffff:10.8.128.21 - - [08/Sep/2022:07:43:05] "GET /metrics HTTP/1.1" 200 1042443 "" "Prometheus/2.33.4"
::ffff:10.8.128.21 - - [08/Sep/2022:07:43:15] "GET /metrics HTTP/1.1" 200 1042442 "" "Prometheus/2.33.4"
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/_ping): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/_ping): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/_ping): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/_ping): [errno: 111] Connection refused
::ffff:10.8.128.21 - - [08/Sep/2022:07:43:25] "GET /metrics HTTP/1.1" 200 1042442 "" "Prometheus/2.33.4"
Dashboard Exception
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 258, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper
    return func(*vpath, **params)
  File "/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/share/ceph/mgr/dashboard/controllers/rbd.py", line 105, in list
    return self._rbd_list(pool_name)
  File "/usr/share/ceph/mgr/dashboard/controllers/rbd.py", line 90, in _rbd_list
    status, value = RbdService.rbd_pool_list(pool)
  File "/usr/share/ceph/mgr/dashboard/tools.py", line 245, in wrapper
    return rvc.run(fn, args, kwargs)
  File "/usr/share/ceph/mgr/dashboard/tools.py", line 233, in run
    raise ViewCacheNoDataException()
dashboard.exceptions.ViewCacheNoDataException: ViewCache: unable to retrieve data
::ffff:10.8.128.21 - - [08/Sep/2022:07:43:35] "GET /metrics HTTP/1.1" 200 1042631 "" "Prometheus/2.33.4"
Error while calling fn=<function RbdService.rbd_pool_list at 0x7ff9cd3f7790> ex=[errno 19] error generating diff from snapshot None
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/tools.py", line 147, in run
    val = self.fn(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/rbd.py", line 421, in rbd_pool_list
    stat = cls._rbd_image_stat(
  File "/usr/share/ceph/mgr/dashboard/services/rbd.py", line 386, in _rbd_image_stat
    return cls._rbd_image(ioctx, pool_name, namespace, image_name)
  File "/usr/share/ceph/mgr/dashboard/services/rbd.py", line 360, in _rbd_image
    total_prov_bytes, snaps_prov_bytes = cls._rbd_disk_usage(
  File "/usr/share/ceph/mgr/dashboard/services/rbd.py", line 256, in _rbd_disk_usage
    image.diff_iterate(0, size, prev_snap, du_callb,
  File "rbd.pyx", line 2770, in rbd.requires_not_closed.wrapper
  File "rbd.pyx", line 3925, in rbd.Image.diff_iterate
rbd.OSError: [errno 19] error generating diff from snapshot None
::ffff:10.8.128.21 - - [08/Sep/2022:07:43:45] "GET /metrics HTTP/1.1" 200 1042626 "" "Prometheus/2.33.4"
::ffff:10.8.128.21 - - [08/Sep/2022:07:43:55] "GET /metrics HTTP/1.1" 200 1042626 "" "Prometheus/2.33.4"
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/_ping): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/_ping): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/_ping): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.1:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/sysinfo/hostname): [errno: 111] Connection refused
iscsi REST API failed GET, connection error (url=http://10.1.172.2:5000/api/_ping): [errno: 111] Connection refused
Comment 11 Preethi 2022-09-12 06:57:51 UTC 
issue is not seen with the latest build of RHCS 6.0. Attached screenshot . We are able to see the RBD image details 
  "ceph version 17.2.3-25.el9cp (135b5c820655c054d65f5cc3ef9ece6051df2b5f) quincy (stable)": 5
Comment 15 Pere Diaz Bou 2022-09-29 08:26:05 UTC 
Yes, this is a patch. I guess this isn't needed in the release notes.
Comment 23 errata-xmlrpc 2023-03-20 18:58:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 6.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:1360