Bug 2247402

Summary: [RHCS6][cee/sd][ceph-mon] ceph-mon does not handle gracefully wrong syntax of "ceph health mute" command
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tomas Petr <tpetr>
Component: RADOSAssignee: Prashant Dhange <pdhange>
Status: CLOSED ERRATA QA Contact: Pawan <pdhiran>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.1CC: bhubbard, ceph-eng-bugs, cephqe-warriors, edebeaumont, nojha, pdhange, pdhiran, rzarzyns, tserlin, vips, vumrao
Target Milestone: ---Flags: pdhange: needinfo? (tpetr)
pdhiran: needinfo? (rzarzyns)
Target Release: 6.1z9   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-17.2.6-269.el9cp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2247232 Environment:
Last Closed: 2025-04-28 05:29:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2247232    
Bug Blocks:    

Description Tomas Petr 2023-11-01 08:31:18 UTC 
+++ This bug was initially created as a clone of Bug #2247232 +++

Description of problem:
When running incorrect syntax of "ceph health mute" command, it will crash the ceph-mon doing setting the mute operation:
---
 - incorrect syntax - result ceph-mon crash:
ceph health mute 6d CEPHADM_FAILED_DAEMON
 - correct syntax - works as expected
ceph health mute 6d CEPHADM_FAILED_DAEMON
---

Ceph mon should identify wrong syntax and report "Invalid command" instead of processing the command and crash

Version-Release number of selected component (if applicable):
RHCS 5.3z5
16.2.10-208.el8cp

How reproducible:
Always

Steps to Reproduce:
1. have RHCS 5.3 cluster
2. run incorrect syntax command "ceph health mute 6d CEPHADM_FAILED_DAEMON"
3. observe ceph-mon crash

Actual results:
ceph-mon crash

Expected results:
ceph-mon does not crash and prints "Invalid command" instead

Additional info:

--- Additional comment from Tomas Petr on 2023-10-31 13:03:35 UTC ---



--- Additional comment from Tomas Petr on 2023-10-31 13:05:46 UTC ---

Attached debug logs of the crash from RH internal test cluster

Adding crash message for search reference

Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  0 mon.mons-1@0(leader) e2 handle_command mon_command({"prefix": "healt
h mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"} v 0) v1
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 20 is_capable service=mon command=health mute write addr 10.0.95.68:0/2
445627122 on cap allow *
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 20  allow so far , doing grant allow *
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: terminate called after throwing an instance of 'std::invalid_argument'
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:   what():  expected digit
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug *** Caught signal (Aborted) **
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  in thread 7f19271c9700 thread_name:ms_dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: 2023-10-31T12:48:35.035+0000 7f19271c9700 20  allow all
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 mon.mons-1@0(leader) e2 _allowed_command capable
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin
' cmd=[{"prefix": "health mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"}]: dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client _send_to_mon log to self
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client  log_queue is 1 last_log 304 sent 303 num 1 unsent 1 send
ing 1
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700 10 log_client  will send 2023-10-31T12:48:35.036753+0000 mon.mons-1 (mo
n.0) 304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "health mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"}]: dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.035+0000 7f19271c9700  7 mon.mons-1@0(leader).health prepare_update mon_command({"prefix": "h
ealth mute", "code": "6d", "ttl": "CEPHADM_FAILED_DAEMON"} v 0) v1 from client.? 10.0.95.68:0/2445627122
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1932a64cf0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  2: gsignal()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  3: abort()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f193207609b]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f193207c53c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f193207c597]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f193207c7f8]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  8: /usr/lib64/ceph/libceph-common.so.2(+0x27c5b9) [0x7f1934d315b9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  9: (HealthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0xb69) [0x55e1523eb9c9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  10: (HealthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0xf0) [0x55e1523ec0b0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  11: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xa6d) [0x55e152465bed]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  12: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x25bf) [0x55e15234502f]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa60) [0x55e152349cd0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  14: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  15: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x941) [0x55e15234c851]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x12bc) [0x55e15234a52c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  17: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x55e15237a40c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  19: (DispatchQueue::entry()+0x126a) [0x7f1934f778da]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f193502ae21]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  21: /lib64/libpthread.so.0(+0x81ca) [0x7f1932a5a1ca]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  22: clone()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: debug 2023-10-31T12:48:35.037+0000 7f19271c9700 -1 *** Caught signal (Aborted) **
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  in thread 7f19271c9700 thread_name:ms_dispatch
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]: 
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1932a64cf0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  2: gsignal()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  3: abort()
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f193207609b]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f193207c53c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f193207c597]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f193207c7f8]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  8: /usr/lib64/ceph/libceph-common.so.2(+0x27c5b9) [0x7f1934d315b9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  9: (HealthMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0xb69) [0x55e1523eb9c9]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  10: (HealthMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0xf0) [0x55e1523ec0b0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  11: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xa6d) [0x55e152465bed]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  12: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x25bf) [0x55e15234502f]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  13: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xa60) [0x55e152349cd0]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  14: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  15: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x941) [0x55e15234c851]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x12bc) [0x55e15234a52c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  17: (Monitor::_ms_dispatch(Message*)+0x670) [0x55e15234b020]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x55e15237a40c]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  19: (DispatchQueue::entry()+0x126a) [0x7f1934f778da]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f193502ae21]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  21: /lib64/libpthread.so.0(+0x81ca) [0x7f1932a5a1ca]
Oct 31 08:48:35 mons-1.foursix.lab.upshift.rdu2.redhat.com ceph-e4bd31cb-890d-430b-941e-79b2877ee966-mon-mons-1[1562]:  22: clone()
Comment 13 errata-xmlrpc 2025-04-28 05:29:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 6.1 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:4238