2247232 – [cee/sd][ceph-mon] ceph-mon does not handle gracefully wrong syntax of "ceph health mute" command

Bug 2247232 - [cee/sd][ceph-mon] ceph-mon does not handle gracefully wrong syntax of "ceph health mute" command

Summary: [cee/sd][ceph-mon] ceph-mon does not handle gracefully wrong syntax of "ceph ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	RADOS
Sub Component:
Version:	5.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	5.3z6
Assignee:	Prashant Dhange
QA Contact:	Pawan
Docs Contact:	Ranjini M N
URL:
Whiteboard:
Depends On:
Blocks:	2247402 2258797
TreeView+	depends on / blocked

Reported:	2023-10-31 13:02 UTC by Tomas Petr
Modified:	2024-08-23 04:25 UTC (History)
CC List:	12 users (show)
Fixed In Version:	ceph-16.2.10-247.el8cp
Doc Type:	Bug Fix
Doc Text:	.An exception is added to handle a Ceph Monitor crash Previously, if the `ceph health mute` command was run with the incorrect syntax, it would trigger the Ceph Monitor to crash. With this exception, an exception is added to handle the `ceph health mute` command and the monitor handles this exception and the command errors out and notifies that the command has the wrong syntax.
Clone Of:
Clones:	2247402 (view as bug list)
Environment:
Last Closed:	2024-02-08 16:56:29 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	58003	None	None	None	2024-01-09 20:54:36 UTC
Github	ceph ceph pull 48799	None	Merged	mon: add exception handling to ceph health mute	2024-01-09 20:54:36 UTC
Github	ceph ceph pull 55118	None	open	pacific: mon: add exception handling to ceph health mute	2024-01-10 01:36:23 UTC
Red Hat Issue Tracker	RHCEPH-7834	None	None	None	2023-10-31 13:05:08 UTC
Red Hat Product Errata	RHSA-2024:0745	None	None	None	2024-02-08 16:56:40 UTC

Description Tomas Petr 2023-10-31 13:02:49 UTC

Description of problem:
When running incorrect syntax of "ceph health mute" command, it will crash the ceph-mon doing setting the mute operation:
---
 - incorrect syntax - result ceph-mon crash:
ceph health mute 6d CEPHADM_FAILED_DAEMON
 - correct syntax - works as expected
ceph health mute 6d CEPHADM_FAILED_DAEMON
---

Ceph mon should identify wrong syntax and report "Invalid command" instead of processing the command and crash

Version-Release number of selected component (if applicable):
RHCS 5.3z5
16.2.10-208.el8cp

How reproducible:
Always

Steps to Reproduce:
1. have RHCS 5.3 cluster
2. run incorrect syntax command "ceph health mute 6d CEPHADM_FAILED_DAEMON"
3. observe ceph-mon crash

Actual results:
ceph-mon crash

Expected results:
ceph-mon does not crash and prints "Invalid command" instead

Additional info:

Comment 2 Tomas Petr 2023-10-31 13:05:46 UTC

Attached debug logs of the crash from RH internal test cluster

Adding crash message for search reference

Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  0 mon.mons-1@0(leader) e2 handle_command mon_command({"prefix": "healt
h mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"} v 0) v1
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 20 is_capable service=mon command=health mute write addr 10.0.95.68:0/2
445627122 on cap allow *
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 20  allow so far , doing grant allow *
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: terminate called after throwing an instance of 'std::invalid_argument'
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:   what():  expected digit
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug *** Caught signal (Aborted) **
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  in thread 7f19271c9700 thread_name:ms_dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: 2023-10-31T12:48:35.035+0000 7f19271c9700 20  allow all
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 mon.mons-1@0(leader) e2 _allowed_command capable
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin
' cmd=[{"prefix": "health mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"}]: dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client _send_to_mon log to self
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client  log_queue is 1 last_log 304 sent 303 num 1 unsent 1 send
ing 1
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client  will send 2023-10-31T12:48:35.036753+0000 mon.mons-1 (mo
n.0) 304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "health mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"}]: dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  7 mon.mons-1@0(leader).health prepare_update mon_command({"prefix": "h
ealth mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"} v 0) v1 from client.? 10.0.95.68:0/2445627122
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1932a64cf0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  2: gsignal()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  3: abort()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f193207609b]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f193207c53c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f193207c597]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f193207c7f8]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  8: /usr/lib64/ceph/libceph-common.so.2(+0x27c5b9) [0x7f1934d315b9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  9: (HealthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0xb69) [0x55e1523eb9c9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  10: (HealthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0xf0) [0x55e1523ec0b0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  11: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xa6d) [0x55e152465bed]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  12: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x25bf) [0x55e15234502f]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa60) [0x55e152349cd0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  14: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  15: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x941) [0x55e15234c851]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x12bc) [0x55e15234a52c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  17: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x55e15237a40c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  19: (DispatchQueue::entry()+0x126a) [0x7f1934f778da]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f193502ae21]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  21: /lib64/libpthread.so.0(+0x81ca) [0x7f1932a5a1ca]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  22: clone()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.037+0000 7f19271c9700 -1 *** Caught signal (Aborted) **
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  in thread 7f19271c9700 thread_name:ms_dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: 
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1932a64cf0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  2: gsignal()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  3: abort()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f193207609b]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f193207c53c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f193207c597]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f193207c7f8]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  8: /usr/lib64/ceph/libceph-common.so.2(+0x27c5b9) [0x7f1934d315b9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  9: (HealthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0xb69) [0x55e1523eb9c9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  10: (HealthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0xf0) [0x55e1523ec0b0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  11: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xa6d) [0x55e152465bed]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  12: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x25bf) [0x55e15234502f]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa60) [0x55e152349cd0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  14: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  15: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x941) [0x55e15234c851]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x12bc) [0x55e15234a52c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  17: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x55e15237a40c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  19: (DispatchQueue::entry()+0x126a) [0x7f1934f778da]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f193502ae21]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  21: /lib64/libpthread.so.0(+0x81ca) [0x7f1932a5a1ca]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  22: clone()

Comment 4 Scott Ostapovicz 2024-01-08 15:46:04 UTC

Any progress?

Comment 15 errata-xmlrpc 2024-02-08 16:56:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 5.3 Security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:0745

Comment 16 Red Hat Bugzilla 2024-08-23 04:25:03 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days

Note You need to log in before you can comment on or make changes to this bug.