2247402 – [RHCS6][cee/sd][ceph-mon] ceph-mon does not handle gracefully wrong syntax of "ceph health mute" command

Bug 2247402 - [RHCS6][cee/sd][ceph-mon] ceph-mon does not handle gracefully wrong syntax of "ceph health mute" command [NEEDINFO]

Summary: [RHCS6][cee/sd][ceph-mon] ceph-mon does not handle gracefully wrong syntax of...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	RADOS
Sub Component:
Version:	6.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	6.1z9
Assignee:	Prashant Dhange
QA Contact:	Pawan
Docs Contact:
URL:
Whiteboard:
Depends On:	2247232
Blocks:
TreeView+	depends on / blocked

Reported:	2023-11-01 08:31 UTC by Tomas Petr
Modified:	2025-04-28 05:29 UTC (History)
CC List:	11 users (show)
Fixed In Version:	ceph-17.2.6-269.el9cp
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	2247232
Environment:
Last Closed:	2025-04-28 05:29:09 UTC
Embargoed:
Dependent Products:
Flags:	pdhange: needinfo? (tpetr) pdhiran: needinfo? (rzarzyns)

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	58003	None	None	None	2024-01-09 20:55:59 UTC
Github	ceph ceph pull 48799	None	Merged	mon: add exception handling to ceph health mute	2024-01-09 20:55:59 UTC
Red Hat Issue Tracker	RHCEPH-7839	None	None	None	2023-11-01 08:31:40 UTC
Red Hat Product Errata	RHSA-2025:4238	None	None	None	2025-04-28 05:29:13 UTC

Description Tomas Petr 2023-11-01 08:31:18 UTC

+++ This bug was initially created as a clone of Bug #2247232 +++

Description of problem:
When running incorrect syntax of "ceph health mute" command, it will crash the ceph-mon doing setting the mute operation:
---
 - incorrect syntax - result ceph-mon crash:
ceph health mute 6d CEPHADM_FAILED_DAEMON
 - correct syntax - works as expected
ceph health mute 6d CEPHADM_FAILED_DAEMON
---

Ceph mon should identify wrong syntax and report "Invalid command" instead of processing the command and crash

Version-Release number of selected component (if applicable):
RHCS 5.3z5
16.2.10-208.el8cp

How reproducible:
Always

Steps to Reproduce:
1. have RHCS 5.3 cluster
2. run incorrect syntax command "ceph health mute 6d CEPHADM_FAILED_DAEMON"
3. observe ceph-mon crash

Actual results:
ceph-mon crash

Expected results:
ceph-mon does not crash and prints "Invalid command" instead

Additional info:

--- Additional comment from Tomas Petr on 2023-10-31 13:03:35 UTC ---



--- Additional comment from Tomas Petr on 2023-10-31 13:05:46 UTC ---

Attached debug logs of the crash from RH internal test cluster

Adding crash message for search reference

Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  0 mon.mons-1@0(leader) e2 handle_command mon_command({"prefix": "healt
h mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"} v 0) v1
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 20 is_capable service=mon command=health mute write addr 10.0.95.68:0/2
445627122 on cap allow *
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 20  allow so far , doing grant allow *
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: terminate called after throwing an instance of 'std::invalid_argument'
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:   what():  expected digit
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug *** Caught signal (Aborted) **
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  in thread 7f19271c9700 thread_name:ms_dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: 2023-10-31T12:48:35.035+0000 7f19271c9700 20  allow all
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 mon.mons-1@0(leader) e2 _allowed_command capable
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin
' cmd=[{"prefix": "health mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"}]: dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client _send_to_mon log to self
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client  log_queue is 1 last_log 304 sent 303 num 1 unsent 1 send
ing 1
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client  will send 2023-10-31T12:48:35.036753+0000 mon.mons-1 (mo
n.0) 304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "health mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"}]: dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  7 mon.mons-1@0(leader).health prepare_update mon_command({"prefix": "h
ealth mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"} v 0) v1 from client.? 10.0.95.68:0/2445627122
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1932a64cf0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  2: gsignal()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  3: abort()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f193207609b]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f193207c53c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f193207c597]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f193207c7f8]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  8: /usr/lib64/ceph/libceph-common.so.2(+0x27c5b9) [0x7f1934d315b9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  9: (HealthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0xb69) [0x55e1523eb9c9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  10: (HealthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0xf0) [0x55e1523ec0b0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  11: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xa6d) [0x55e152465bed]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  12: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x25bf) [0x55e15234502f]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa60) [0x55e152349cd0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  14: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  15: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x941) [0x55e15234c851]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x12bc) [0x55e15234a52c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  17: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x55e15237a40c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  19: (DispatchQueue::entry()+0x126a) [0x7f1934f778da]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f193502ae21]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  21: /lib64/libpthread.so.0(+0x81ca) [0x7f1932a5a1ca]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  22: clone()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.037+0000 7f19271c9700 -1 *** Caught signal (Aborted) **
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  in thread 7f19271c9700 thread_name:ms_dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: 
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1932a64cf0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  2: gsignal()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  3: abort()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f193207609b]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f193207c53c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f193207c597]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f193207c7f8]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  8: /usr/lib64/ceph/libceph-common.so.2(+0x27c5b9) [0x7f1934d315b9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  9: (HealthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0xb69) [0x55e1523eb9c9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  10: (HealthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0xf0) [0x55e1523ec0b0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  11: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xa6d) [0x55e152465bed]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  12: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x25bf) [0x55e15234502f]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa60) [0x55e152349cd0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  14: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  15: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x941) [0x55e15234c851]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x12bc) [0x55e15234a52c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  17: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x55e15237a40c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  19: (DispatchQueue::entry()+0x126a) [0x7f1934f778da]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f193502ae21]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  21: /lib64/libpthread.so.0(+0x81ca) [0x7f1932a5a1ca]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  22: clone()

Comment 13 errata-xmlrpc 2025-04-28 05:29:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 6.1 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:4238

Note You need to log in before you can comment on or make changes to this bug.