Bug 2087527 - [RFE] Limit the Health Detail MSG log size in cluster logs
Summary: [RFE] Limit the Health Detail MSG log size in cluster logs
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RADOS
Version: 6.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 7.1
Assignee: Prashant Dhange
QA Contact: Tintu Mathew
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-17 23:18 UTC by Vikhyat Umrao
Modified: 2023-07-07 17:56 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 55693 0 None None None 2022-05-17 23:23:35 UTC
Github ceph ceph pull 46959 0 None open mon: Limit health detail message in the cluster log 2022-07-05 05:54:22 UTC
Red Hat Issue Tracker RHCEPH-4320 0 None None None 2022-05-17 23:31:24 UTC

Description Vikhyat Umrao 2022-05-17 23:18:18 UTC 
Description of problem:
[RFE] Limit the Health Detail MSG log size in cluster logs

Version-Release number of selected component (if applicable):
Upstream quincy

In a recent occurrence on one of the Ceph upstream clusters, the LRC one went into DU mode due to a bug https://tracker.ceph.com/issues/54132.

Issues caused by this problem:

- SSH errors logged massive hexdump output (cephadm bug already fixed)
- These logs got stored as part of ‘ceph health detail’ as well
- Periodically we dumped ‘ceph health detail’ to the cluster log (going through paxos, each mon db, etc.) causing the mons to lose quorum and become unresponsive
Note You need to log in before you can comment on or make changes to this bug.