Bug 2274373

Summary:	Collect rbd image details from rados namespaces within a cephblockpool in must-gather command
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Neha Berry <nberry>
Component:	must-gather	Assignee:	yati padia <ypadia>
Status:	CLOSED ERRATA	QA Contact:	Neha Berry <nberry>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.16	CC:	jarrpa, mrajanna, muagarwa, odf-bz-bot, rohgupta, ypadia
Target Milestone:	---
Target Release:	ODF 4.16.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	isf-provider
Fixed In Version:	4.16.0-118	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-07-17 13:18:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Neha Berry 2024-04-10 16:21:15 UTC

Description of problem (please be detailed as possible and provide log
snippests):
====================================================================
With 4.16, Feature, we are introducing rados namespaces within the default blockpool. The RBD images from each ocs-storageclients would be created within their respective rados namespace

We need to add ceph commands for collecting the rbd related information from these namespaces for a provider cluster. 

Feature:  :RHSTOR-5177 Improve allocation and distribution of storage resources in Provider Mode



Version of all relevant components (if applicable):
====================================================
4.16 and above


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
===============================================================
No

Is there any workaround available to the best of your knowledge?
===============================================================


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
===============================================================
3


Can this issue reproducible?
===============================================================
Yes


Can this issue reproduce from the UI?
===============================================================
NA


If this is a regression, please provide more details to justify this:
===============================================================
No


Steps to Reproduce:
1. Install a provider/client cluster
2. collect must-gather
** ceph commands were not collected due to https://bugzilla.redhat.com/show_bug.cgi?id=2274381. So tried with the 4.15 image
3.


Actual results:
================
We need to add rados namespace specific commands for checking the rbd information and details

Expected results:
=================
We need to add rados namespace specific commands for checking the rbd information and details




Additional info:



ls -ltrh|grep rbd
-rw-r--r--  1 nberry  staff    33K Feb  8 21:35 rados_rbd_objects
-rw-r--r--  1 nberry  staff   408B Feb  8 21:35 rbd_mirror_image_status
-rw-r--r--  1 nberry  staff   408B Feb  8 21:35 rbd_mirror_pool_info
-rw-r--r--  1 nberry  staff   408B Feb  8 21:35 rbd_mirror_pool_status
-rw-r--r--  1 nberry  staff    37K Feb  8 21:35 rbd_vol_and_snap_info
-rw-r--r--  1 nberry  staff   3.8K Feb  8 21:36 ceph_rbd_task_list
-rw-r--r--  1 nberry  staff   861B Feb  8 21:36 pools_rbd_cephblockpool-storageconsumer-71eea242-935a-49f5-be03-084d2843a95e-78f6850c
-rw-r--r--  1 nberry  staff   361B Feb  8 21:36 pools_rbd_cephblockpool-storageconsumer-a696155d-fc47-49a1-8cee-a4462cc21e88-6fbcf432
-rw-r--r--  1 nberry  staff   270B Feb  8 21:36 pools_rbd_cephblockpool-storageconsumer-bd97ae8e-57e8-4082-9f4a-889f7f464788-3c7b4b5a
-rw-r--r--  1 nberry  staff    45B Feb  8 21:36 pools_rbd_cephblockpool-storageconsumer-f0ee5b2a-e786-44e4-b299-3344d5fb7941-530a656d
-rw-r--r--  1 nberry  staff     3B Feb  8 21:36 rbd_mirror_snapshot_schedule_ls
-rw-r--r--  1 nberry  staff     3B Feb  8 21:36 rbd_mirror_snapshot_schedule_status
-rw-r--r--  1 nberry  staff   458B Feb  8 21:36 rbd_trash_ls_cephblockpool-storageconsumer-71eea242-935a-49f5-be03-084d2843a95e-78f6850c
-rw-r--r--  1 nberry  staff     3B Feb  8 21:36 rbd_trash_ls_cephblockpool-storageconsumer-a696155d-fc47-49a1-8cee-a4462cc21e88-6fbcf432
-rw-r--r--  1 nberry  staff     3B Feb  8 21:36 rbd_trash_ls_cephblockpool-storageconsumer-bd97ae8e-57e8-4082-9f4a-889f7f464788-3c7b4b5a
-rw-r--r--  1 nberry  staff     3B Feb  8 21:36 rbd_trash_ls_cephblockpool-storageconsumer-f0ee5b2a-e786-44e4-b299-3344d5fb7941-530a656d
nberry@Nehas-MacBook-Pro must_gather_commands %

Comment 11 Rohan Gupta 2024-05-20 11:10:31 UTC

Hi, Yati any updates?

Comment 17 Mudit Agarwal 2024-05-30 12:47:38 UTC

This bug is not fixed in 4.16.0-113, by mistake (automation just picks all the MODIFIED BZs) it moved to ON_QA

Comment 18 Neha Berry 2024-06-05 06:10:51 UTC

within must-gather folder


image Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       2m3s  default-scheduler  Successfully assigned openshift-storage/must-gather-kjfmq-helper to baremetal1-01
  Normal  AddedInterface  2m3s  multus             Add eth0 [10.129.0.121/23] from ovn-kubernetes
  Normal  Pulled          2m3s  kubelet            Container image "registry.redhat.io/odf4/rook-ceph-rhel9-operator@sha256:8a924399cca885e19a1004283d5d10f068455d71f5b75e19c834075051b42c7b" already present on machine
  Normal  Created         2m3s  kubelet            Created container must-gather-helper
  Normal  Started         2m3s  kubelet            Started container must-gather-helper


quay.io/rhceph-dev/ocs-must-gather                                                        latest-4.16  a2a088b2382e  7 hours ago        549 MB
quay.io/ypadia/odf-must-gather                                                            nstest11     41230d4859b4  8 days ago         549 MB




ls |grep rados
rados_cephfs_objects
rados_ls_--pool=ocs-storagecluster-cephblockpool
rados_ls_--pool=ocs-storagecluster-cephfilesystem-metadata_--namespace=csi
rados_lspools
rados_rbd_objects_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-1518c2a5c832628b5140165c952bb2db
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-279a78543c6d8f28fba43c9046d0ba77
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-348fc3351f506121ec3444c7ac7f814a
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-4075113d4b7cee60e638c64f77d35b2f
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-4a4263fa3634feb0b74f8ecefb25fa3d
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-5922ae1f367ecab41f8aadaa7c327ca9
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-5ee26effcdd716c681549c517645d8f0
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-d0e0ee9087ba526a589f36e9a38959fa
rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219
rbd_vol_and_snap_info_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219
nberry@Nehas-MacBook-Pro must_gather_commands %               


1. But these files are all empty

cat rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-1518c2a5c832628b5140165c952bb2db
[]

2. Why do we have rbd_vol_snap_info file for only one radosnamespace.. could it be because the others do not have snapshots?

nberry@Nehas-MacBook-Pro must_gather_commands % cat rbd_vol_and_snap_info_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219
Collecting image info for: ocs-storagecluster-cephblockpool/csi-snap-0ca3acb5-bab9-4a38-850c-b38a629e42ee
rbd image 'csi-snap-0ca3acb5-bab9-4a38-850c-b38a629e42ee':
	size 30 GiB in 7680 objects
	order 22 (4 MiB objects)
	snapshot_count: 1
	id: 64e8460e4620
	block_name_prefix: rbd_data.64e8460e4620
	format: 2
	features: layering, deep-flatten, operations
	op_features: clone-child
	flags: 
	create_timestamp: Tue May 21 14:04:32 2024
	access_timestamp: Tue May 21 14:04:32 2024
	modify_timestamp: Tue May 21 14:04:32 2024
	parent: ocs-storagecluster-cephblockpool/cephradosnamespace-e09b8c7b6b2f387870fe943118e26219/csi-vol-a172fc33-e494-4837-ba2e-840badb92b93@901c6a1c-97b3-485f-bad1-7047b4dab80a (trash 4120c065f1c2)
	overlap: 30 GiB


======================================================================

>> collecting dump cephblockpoolradosnamespaces.ceph.rook.io

nberry@Nehas-MacBook-Pro ceph.rook.io % pwd
/Users/nberry/Documents/work/provider-client-ibm/bm1/jun5/must-gather.local.8098294318386856004/quay-io-rhceph-dev-ocs-must-gather-sha256-a1aaa16de024b48438fced9ea28c8519df3a4a7d2339f79fd9959a46dd408519/ceph/namespaces/openshift-storage/ceph.rook.io
nberry@Nehas-MacBook-Pro ceph.rook.io % ls -ltrh
total 0
drwxr-xr-x  11 nberry  staff   352B Jun  5 11:21 cephblockpoolradosnamespaces
drwxr-xr-x   3 nberry  staff    96B Jun  5 11:21 cephblockpools
drwxr-xr-x  43 nberry  staff   1.3K Jun  5 11:21 cephclients
drwxr-xr-x   3 nberry  staff    96B Jun  5 11:21 cephclusters
drwxr-xr-x   3 nberry  staff    96B Jun  5 11:21 cephfilesystems
drwxr-xr-x  10 nberry  staff   320B Jun  5 11:21 cephfilesystemsubvolumegroups
drwxr-xr-x   3 nberry  staff    96B Jun  5 11:21 cephobjectstores
drwxr-xr-x   5 nberry  staff   160B Jun  5 11:21 cephobjectstoreusers
nberry@Nehas-MacBook-Pro ceph.rook.io % cd cephblockpoolradosnamespaces 
nberry@Nehas-MacBook-Pro cephblockpoolradosnamespaces % ls -ltrh
total 72
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-e09b8c7b6b2f387870fe943118e26219.yaml
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-d0e0ee9087ba526a589f36e9a38959fa.yaml
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-5ee26effcdd716c681549c517645d8f0.yaml
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-5922ae1f367ecab41f8aadaa7c327ca9.yaml
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-4a4263fa3634feb0b74f8ecefb25fa3d.yaml
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-4075113d4b7cee60e638c64f77d35b2f.yaml
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-348fc3351f506121ec3444c7ac7f814a.yaml
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-279a78543c6d8f28fba43c9046d0ba77.yaml
-rwxr-xr-x  1 nberry  staff   1.8K Jun  5 11:17 cephradosnamespace-1518c2a5c832628b5140165c952bb2db.yaml
nberry@Nehas-MacBook-Pro cephblockpoolradosnamespaces %

Comment 19 Neha Berry 2024-06-05 06:19:25 UTC

Confirmed that one of them have the rados image names

 % cat rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-e09b8c7b6b2f387870fe943118e26219

[{"id":"1992c92d1f6635","name":"csi-vol-c5e87bed-d5ba-4c4f-913e-a0030721e1d2"},{"id":"1992c9c1a08185","name":"csi-vol-c2317e27-cf1a-47e9-835e-3f75163bc2d1"},{"id":"41205d5eb8de","name":"csi-vol-bf78310e-321e-402a-861b-b54feb815efb"},{"id":"412070ba53d9","name":"csi-vol-cae933b0-91d8-4c95-9845-5d83b187c623"},{"id":"412072832c06","name":"csi-vol-d21f7ced-96d2-4e9f-9a26-c0fb0040b844"},{"id":"41208b8f705","name":"csi-vol-58fbc612-db7d-4455-8d83-5e9a21587f77"},{"id":"4120a345dbe8","name":"csi-vol-3cac1d7c-9fed-4bc8-903b-f758428ced73"},{"id":"4120b35ca047","name":"csi-vol-4c0b62f6-3c47-4089-b83d-2346bbc84fb2"},{"id":"4120c065f1c2","name":"csi-vol-a172fc33-e494-4837-ba2e-840badb92b93"},{"id":"4120c8498c45","name":"csi-vol-38d29d23-9410-4af8-bb75-324c33a84572"},{"id":"4120da7d9598","name":"csi-vol-1da48a1d-984b-4130-9b4a-c33e2f9c77d4"}]
nberry@Nehas-MacBook-Pro must_gather_commands % cat rbd_trash_ls_ocs-storagecluster-cephblockpool_cephradosnamespace-4075113d4b7cee60e638c64f77d35b2f

Comment 22 Sunil Kumar Acharya 2024-06-06 07:34:46 UTC

Please update the RDT flag/text appropriately.

Comment 23 errata-xmlrpc 2024-07-17 13:18:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591