Description of problem (please be detailed as possible and provide log snippests): ==================================================================== With 4.16, Feature, we are introducing rados namespaces within the default blockpool. The RBD images from each ocs-storageclients would be created within their respective rados namespace We need to add ceph commands for collecting the rbd related information from these namespaces for a provider cluster. Feature: :RHSTOR-5177 Improve allocation and distribution of storage resources in Provider Mode Version of all relevant components (if applicable): ==================================================== 4.16 and above Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? =============================================================== No Is there any workaround available to the best of your knowledge? =============================================================== Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? =============================================================== 3 Can this issue reproducible? =============================================================== Yes Can this issue reproduce from the UI? =============================================================== NA If this is a regression, please provide more details to justify this: =============================================================== No Steps to Reproduce: 1. Install a provider/client cluster 2. collect must-gather ** ceph commands were not collected due to https://bugzilla.redhat.com/show_bug.cgi?id=2274381. So tried with the 4.15 image 3. Actual results: ================ We need to add rados namespace specific commands for checking the rbd information and details Expected results: ================= We need to add rados namespace specific commands for checking the rbd information and details Additional info: ls -ltrh|grep rbd -rw-r--r-- 1 nberry staff 33K Feb 8 21:35 rados_rbd_objects -rw-r--r-- 1 nberry staff 408B Feb 8 21:35 rbd_mirror_image_status -rw-r--r-- 1 nberry staff 408B Feb 8 21:35 rbd_mirror_pool_info -rw-r--r-- 1 nberry staff 408B Feb 8 21:35 rbd_mirror_pool_status -rw-r--r-- 1 nberry staff 37K Feb 8 21:35 rbd_vol_and_snap_info -rw-r--r-- 1 nberry staff 3.8K Feb 8 21:36 ceph_rbd_task_list -rw-r--r-- 1 nberry staff 861B Feb 8 21:36 pools_rbd_cephblockpool-storageconsumer-71eea242-935a-49f5-be03-084d2843a95e-78f6850c -rw-r--r-- 1 nberry staff 361B Feb 8 21:36 pools_rbd_cephblockpool-storageconsumer-a696155d-fc47-49a1-8cee-a4462cc21e88-6fbcf432 -rw-r--r-- 1 nberry staff 270B Feb 8 21:36 pools_rbd_cephblockpool-storageconsumer-bd97ae8e-57e8-4082-9f4a-889f7f464788-3c7b4b5a -rw-r--r-- 1 nberry staff 45B Feb 8 21:36 pools_rbd_cephblockpool-storageconsumer-f0ee5b2a-e786-44e4-b299-3344d5fb7941-530a656d -rw-r--r-- 1 nberry staff 3B Feb 8 21:36 rbd_mirror_snapshot_schedule_ls -rw-r--r-- 1 nberry staff 3B Feb 8 21:36 rbd_mirror_snapshot_schedule_status -rw-r--r-- 1 nberry staff 458B Feb 8 21:36 rbd_trash_ls_cephblockpool-storageconsumer-71eea242-935a-49f5-be03-084d2843a95e-78f6850c -rw-r--r-- 1 nberry staff 3B Feb 8 21:36 rbd_trash_ls_cephblockpool-storageconsumer-a696155d-fc47-49a1-8cee-a4462cc21e88-6fbcf432 -rw-r--r-- 1 nberry staff 3B Feb 8 21:36 rbd_trash_ls_cephblockpool-storageconsumer-bd97ae8e-57e8-4082-9f4a-889f7f464788-3c7b4b5a -rw-r--r-- 1 nberry staff 3B Feb 8 21:36 rbd_trash_ls_cephblockpool-storageconsumer-f0ee5b2a-e786-44e4-b299-3344d5fb7941-530a656d nberry@Nehas-MacBook-Pro must_gather_commands %
Hi, Yati any updates?
This bug is not fixed in 4.16.0-113, by mistake (automation just picks all the MODIFIED BZs) it moved to ON_QA
within must-gather folder image Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m3s default-scheduler Successfully assigned openshift-storage/must-gather-kjfmq-helper to baremetal1-01 Normal AddedInterface 2m3s multus Add eth0 [10.129.0.121/23] from ovn-kubernetes Normal Pulled 2m3s kubelet Container image "registry.redhat.io/odf4/rook-ceph-rhel9-operator@sha256:8a924399cca885e19a1004283d5d10f068455d71f5b75e19c834075051b42c7b" already present on machine Normal Created 2m3s kubelet Created container must-gather-helper Normal Started 2m3s kubelet Started container must-gather-helper quay.io/rhceph-dev/ocs-must-gather latest-4.16 a2a088b2382e 7 hours ago 549 MB quay.io/ypadia/odf-must-gather nstest11 41230d4859b4 8 days ago 549 MB ls |grep rados rados_cephfs_objects rados_ls_--pool=ocs-storagecluster-cephblockpool rados_ls_--pool=ocs-storagecluster-cephfilesystem-metadata_--namespace=csi rados_lspools rados_rbd_objects_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219 rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-1518c2a5c832628b5140165c952bb2db rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-279a78543c6d8f28fba43c9046d0ba77 rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-348fc3351f506121ec3444c7ac7f814a rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-4075113d4b7cee60e638c64f77d35b2f rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-4a4263fa3634feb0b74f8ecefb25fa3d rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-5922ae1f367ecab41f8aadaa7c327ca9 rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-5ee26effcdd716c681549c517645d8f0 rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-d0e0ee9087ba526a589f36e9a38959fa rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219 rbd_vol_and_snap_info_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219 nberry@Nehas-MacBook-Pro must_gather_commands % 1. But these files are all empty cat rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-1518c2a5c832628b5140165c952bb2db [] 2. Why do we have rbd_vol_snap_info file for only one radosnamespace.. could it be because the others do not have snapshots? nberry@Nehas-MacBook-Pro must_gather_commands % cat rbd_vol_and_snap_info_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219 Collecting image info for: ocs-storagecluster-cephblockpool/csi-snap-0ca3acb5-bab9-4a38-850c-b38a629e42ee rbd image 'csi-snap-0ca3acb5-bab9-4a38-850c-b38a629e42ee': size 30 GiB in 7680 objects order 22 (4 MiB objects) snapshot_count: 1 id: 64e8460e4620 block_name_prefix: rbd_data.64e8460e4620 format: 2 features: layering, deep-flatten, operations op_features: clone-child flags: create_timestamp: Tue May 21 14:04:32 2024 access_timestamp: Tue May 21 14:04:32 2024 modify_timestamp: Tue May 21 14:04:32 2024 parent: ocs-storagecluster-cephblockpool/cephradosnamespace-e09b8c7b6b2f387870fe943118e26219/csi-vol-a172fc33-e494-4837-ba2e-840badb92b93@901c6a1c-97b3-485f-bad1-7047b4dab80a (trash 4120c065f1c2) overlap: 30 GiB ====================================================================== >> collecting dump cephblockpoolradosnamespaces.ceph.rook.io nberry@Nehas-MacBook-Pro ceph.rook.io % pwd /Users/nberry/Documents/work/provider-client-ibm/bm1/jun5/must-gather.local.8098294318386856004/quay-io-rhceph-dev-ocs-must-gather-sha256-a1aaa16de024b48438fced9ea28c8519df3a4a7d2339f79fd9959a46dd408519/ceph/namespaces/openshift-storage/ceph.rook.io nberry@Nehas-MacBook-Pro ceph.rook.io % ls -ltrh total 0 drwxr-xr-x 11 nberry staff 352B Jun 5 11:21 cephblockpoolradosnamespaces drwxr-xr-x 3 nberry staff 96B Jun 5 11:21 cephblockpools drwxr-xr-x 43 nberry staff 1.3K Jun 5 11:21 cephclients drwxr-xr-x 3 nberry staff 96B Jun 5 11:21 cephclusters drwxr-xr-x 3 nberry staff 96B Jun 5 11:21 cephfilesystems drwxr-xr-x 10 nberry staff 320B Jun 5 11:21 cephfilesystemsubvolumegroups drwxr-xr-x 3 nberry staff 96B Jun 5 11:21 cephobjectstores drwxr-xr-x 5 nberry staff 160B Jun 5 11:21 cephobjectstoreusers nberry@Nehas-MacBook-Pro ceph.rook.io % cd cephblockpoolradosnamespaces nberry@Nehas-MacBook-Pro cephblockpoolradosnamespaces % ls -ltrh total 72 -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-e09b8c7b6b2f387870fe943118e26219.yaml -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-d0e0ee9087ba526a589f36e9a38959fa.yaml -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-5ee26effcdd716c681549c517645d8f0.yaml -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-5922ae1f367ecab41f8aadaa7c327ca9.yaml -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-4a4263fa3634feb0b74f8ecefb25fa3d.yaml -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-4075113d4b7cee60e638c64f77d35b2f.yaml -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-348fc3351f506121ec3444c7ac7f814a.yaml -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-279a78543c6d8f28fba43c9046d0ba77.yaml -rwxr-xr-x 1 nberry staff 1.8K Jun 5 11:17 cephradosnamespace-1518c2a5c832628b5140165c952bb2db.yaml nberry@Nehas-MacBook-Pro cephblockpoolradosnamespaces %
Confirmed that one of them have the rados image names % cat rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219 [{"id":"1992c92d1f6635","name":"csi-vol-c5e87bed-d5ba-4c4f-913e-a0030721e1d2"},{"id":"1992c9c1a08185","name":"csi-vol-c2317e27-cf1a-47e9-835e-3f75163bc2d1"},{"id":"41205d5eb8de","name":"csi-vol-bf78310e-321e-402a-861b-b54feb815efb"},{"id":"412070ba53d9","name":"csi-vol-cae933b0-91d8-4c95-9845-5d83b187c623"},{"id":"412072832c06","name":"csi-vol-d21f7ced-96d2-4e9f-9a26-c0fb0040b844"},{"id":"41208b8f705","name":"csi-vol-58fbc612-db7d-4455-8d83-5e9a21587f77"},{"id":"4120a345dbe8","name":"csi-vol-3cac1d7c-9fed-4bc8-903b-f758428ced73"},{"id":"4120b35ca047","name":"csi-vol-4c0b62f6-3c47-4089-b83d-2346bbc84fb2"},{"id":"4120c065f1c2","name":"csi-vol-a172fc33-e494-4837-ba2e-840badb92b93"},{"id":"4120c8498c45","name":"csi-vol-38d29d23-9410-4af8-bb75-324c33a84572"},{"id":"4120da7d9598","name":"csi-vol-1da48a1d-984b-4130-9b4a-c33e2f9c77d4"}] nberry@Nehas-MacBook-Pro must_gather_commands % cat rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-4075113d4b7cee60e638c64f77d35b2f
Please update the RDT flag/text appropriately.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:4591