Bug 2101916

Summary:	must-gather is not collecting ceph logs or coredumps
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Josh Durgin <jdurgin>
Component:	must-gather	Assignee:	yati padia <ypadia>
Status:	CLOSED ERRATA	QA Contact:	Joy John Pinto <jopinto>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.10	CC:	kramdoss, kseeger, muagarwa, ocs-bugs, odf-bz-bot, ypadia
Target Milestone:	---
Target Release:	ODF 4.13.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-06-21 15:22:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Josh Durgin 2022-06-28 19:02:01 UTC

Description of problem (please be detailed as possible and provide log
snippests):

Ceph daemon logs and coredumps which are critical for debugging are no longer being collected. This was implemented via: https://bugzilla.redhat.com/show_bug.cgi?id=1869406

https://github.com/red-hat-storage/ocs-operator/blob/main/must-gather/collection-scripts/gather_ceph_resources#L381-L405

Version of all relevant components (if applicable):

ODF/OCP 4.10

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

This is a supportability issue. Without historical logs, and without coredumps, many problems are impossible to debug.

Is there any workaround available to the best of your knowledge?

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?

Yes

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. Start a ceph cluster
2. crash a ceph daemon (e.g. kill -SIGABRT) to trigger a coredump
3. Run must-gather
4. Check for logs from ceph daemons, e.g. ceph-osd.0.log, and coredumps, in the must-gather output

Actual results:
No logs or coredumps

Expected results:
Log files and coredumps are saved

Additional info:

An example is https://bugzilla.redhat.com/show_bug.cgi?id=2095331 - specifically http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-aman/9jun22/ where rbd-mirror crashed multiple times (as shown by all the crash metadata files in c2/must-gather.local.4254235368011215624/quay-io-rhceph-dev-ocs-must-gather-sha256-8d5835527008fbde909bb1f570e4be06ac6b9b9bb76369cd934d55979c6021bf/ceph/must_gather_commands) but there are no logs or coredumps available. There should be a directory for each node (e.g. named compute-0 in this case) where these are saved.

Comment 4 yati padia 2022-10-13 05:27:49 UTC

@jdurgin can I get a cluster to debug the above? I need to check if the debug nodes contain the crash logs and coredumps.
As per correct code, it does extract the crash from `/host/var/lib/rook/openshift-storage/crash/`. We may first need to collect the logs in that location to extract them.

Comment 12 Joy John Pinto 2023-05-31 04:42:31 UTC

Verified in OCP 4.13.0-0.nightly-2023-05-23-145816 and ODF 4.13.0-205.stable provided by Red Hat

1. Induce osd crash by deleting osd disk from vcenter
2. Ceph crash ls output
   sh-5.1$ ceph crash ls
ID                                                                ENTITY  NEW  
2023-05-30T12:41:00.575000Z_c155cfd3-5f6f-4207-b924-ad292d900e01  osd.0    *   
sh-5.1$ 
3. Ran must gather command and verified crash log detals under 'must-gather.local.1711890767482071931/quay-io-rhceph-dev-ocs-must-gather-sha25610071ddc29383af01d60eadfa4d6f2bd631cfd4c06fcdf7efdb655a84b13a4f1/ceph/crash_compute-0/2023-05-30T12:41:00.575000Z_c155cfd3-5f6f-4207-b924-ad292d900e01'
[jopinto@jopinto 2023-05-30T12:41:00.575000Z_c155cfd3-5f6f-4207-b924-ad292d900e01]$ pwd
/home/jopinto/Desktop/temp/serv/must-gather.local.1711890767482071931/quay-io-rhceph-dev-ocs-must-gather-sha256-10071ddc29383af01d60eadfa4d6f2bd631cfd4c06fcdf7efdb655a84b13a4f1/ceph/crash_compute-0/2023-05-30T12:41:00.575000Z_c155cfd3-5f6f-4207-b924-ad292d900e01
[jopinto@jopinto 2023-05-30T12:41:00.575000Z_c155cfd3-5f6f-4207-b924-ad292d900e01]$ ls
log  meta
[jopinto@jopinto 2023-05-30T12:41:00.575000Z_c155cfd3-5f6f-4207-b924-ad292d900e01]$

[jopinto@jopinto must_gather_commands]$ cd ..
[jopinto@jopinto ceph]$ ls
ceph_daemon_log_compute-0  coredump_compute-0  journal_compute-0  kernel_compute-0  logs                              namespaces
ceph_daemon_log_compute-1  crash_compute-0     journal_compute-1  kernel_compute-1  must_gather_commands              timestamp
ceph_daemon_log_compute-2  event-filter.html   journal_compute-2  kernel_compute-2  must_gather_commands_json_output
[jopinto@jopinto ceph]$ cd coredump_compute-0/
[jopinto@jopinto coredump_compute-0]$ ls
core.ceph-osd.167.d3d56641115f497da1ffde8db3d27bc1.20207.1685450460000000.lz4

Comment 13 errata-xmlrpc 2023-06-21 15:22:14 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.13.0 enhancement and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3742