Bug 2247232

Summary:	[cee/sd][ceph-mon] ceph-mon does not handle gracefully wrong syntax of "ceph health mute" command
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Tomas Petr <tpetr>
Component:	RADOS	Assignee:	Prashant Dhange <pdhange>
Status:	CLOSED ERRATA	QA Contact:	Pawan <pdhiran>
Severity:	medium	Docs Contact:	Ranjini M N <rmandyam>
Priority:	medium
Version:	5.3	CC:	bhubbard, ceph-eng-bugs, cephqe-warriors, edebeaumont, ngangadh, nojha, pdhange, rmandyam, rzarzyns, sostapov, tserlin, vumrao
Target Milestone:	---
Target Release:	5.3z6
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-16.2.10-247.el8cp	Doc Type:	Bug Fix
Doc Text:	.An exception is added to handle a Ceph Monitor crash Previously, if the `ceph health mute` command was run with the incorrect syntax, it would trigger the Ceph Monitor to crash. With this exception, an exception is added to handle the `ceph health mute` command and the monitor handles this exception and the command errors out and notifies that the command has the wrong syntax.	Story Points:	---
Clone Of:
Clones:	2247402 (view as bug list)		Environment:
Last Closed:	2024-02-08 16:56:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2247402, 2258797

Description Tomas Petr 2023-10-31 13:02:49 UTC

Description of problem:
When running incorrect syntax of "ceph health mute" command, it will crash the ceph-mon doing setting the mute operation:
---
 - incorrect syntax - result ceph-mon crash:
ceph health mute 6d CEPHADM_FAILED_DAEMON
 - correct syntax - works as expected
ceph health mute 6d CEPHADM_FAILED_DAEMON
---

Ceph mon should identify wrong syntax and report "Invalid command" instead of processing the command and crash

Version-Release number of selected component (if applicable):
RHCS 5.3z5
16.2.10-208.el8cp

How reproducible:
Always

Steps to Reproduce:
1. have RHCS 5.3 cluster
2. run incorrect syntax command "ceph health mute 6d CEPHADM_FAILED_DAEMON"
3. observe ceph-mon crash

Actual results:
ceph-mon crash

Expected results:
ceph-mon does not crash and prints "Invalid command" instead

Additional info:

Comment 2 Tomas Petr 2023-10-31 13:05:46 UTC

Attached debug logs of the crash from RH internal test cluster

Adding crash message for search reference

Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  0 mon.mons-1@0(leader) e2 handle_command mon_command({"prefix": "healt
h mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"} v 0) v1
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 20 is_capable service=mon command=health mute write addr 10.0.95.68:0/2
445627122 on cap allow *
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 20  allow so far , doing grant allow *
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: terminate called after throwing an instance of 'std::invalid_argument'
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:   what():  expected digit
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug *** Caught signal (Aborted) **
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  in thread 7f19271c9700 thread_name:ms_dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: 2023-10-31T12:48:35.035+0000 7f19271c9700 20  allow all
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 mon.mons-1@0(leader) e2 _allowed_command capable
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin
' cmd=[{"prefix": "health mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"}]: dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client _send_to_mon log to self
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client  log_queue is 1 last_log 304 sent 303 num 1 unsent 1 send
ing 1
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client  will send 2023-10-31T12:48:35.036753+0000 mon.mons-1 (mo
n.0) 304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "health mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"}]: dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  7 mon.mons-1@0(leader).health prepare_update mon_command({"prefix": "h
ealth mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"} v 0) v1 from client.? 10.0.95.68:0/2445627122
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1932a64cf0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  2: gsignal()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  3: abort()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f193207609b]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f193207c53c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f193207c597]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f193207c7f8]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  8: /usr/lib64/ceph/libceph-common.so.2(+0x27c5b9) [0x7f1934d315b9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  9: (HealthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0xb69) [0x55e1523eb9c9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  10: (HealthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0xf0) [0x55e1523ec0b0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  11: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xa6d) [0x55e152465bed]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  12: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x25bf) [0x55e15234502f]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa60) [0x55e152349cd0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  14: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  15: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x941) [0x55e15234c851]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x12bc) [0x55e15234a52c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  17: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x55e15237a40c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  19: (DispatchQueue::entry()+0x126a) [0x7f1934f778da]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f193502ae21]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  21: /lib64/libpthread.so.0(+0x81ca) [0x7f1932a5a1ca]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  22: clone()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.037+0000 7f19271c9700 -1 *** Caught signal (Aborted) **
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  in thread 7f19271c9700 thread_name:ms_dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: 
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1932a64cf0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  2: gsignal()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  3: abort()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f193207609b]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f193207c53c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f193207c597]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f193207c7f8]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  8: /usr/lib64/ceph/libceph-common.so.2(+0x27c5b9) [0x7f1934d315b9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  9: (HealthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0xb69) [0x55e1523eb9c9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  10: (HealthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0xf0) [0x55e1523ec0b0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  11: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xa6d) [0x55e152465bed]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  12: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x25bf) [0x55e15234502f]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa60) [0x55e152349cd0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  14: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  15: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x941) [0x55e15234c851]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x12bc) [0x55e15234a52c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  17: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x55e15237a40c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  19: (DispatchQueue::entry()+0x126a) [0x7f1934f778da]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f193502ae21]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  21: /lib64/libpthread.so.0(+0x81ca) [0x7f1932a5a1ca]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  22: clone()

Comment 4 Scott Ostapovicz 2024-01-08 15:46:04 UTC

Any progress?

Comment 15 errata-xmlrpc 2024-02-08 16:56:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 5.3 Security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:0745

Comment 16 Red Hat Bugzilla 2024-08-23 04:25:03 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days