.The Ceph Monitor no longer crashes after reducing the number of monitors
Previously, when the user reduced the number of monitors in the quorum using the `ceph orch apply mon _NUMBER_` command, `cephadm` would remove the monitor before shutting it down.
This would trigger an assertion because Ceph would assume that the monitor is shutting down before the monitor removal.
With this fix, a sanity check is added to handle the case when the current rank of the monitor is larger or equal to the quorum rank.
The monitor no longer exists in the monitor map, therefore its peers do not ping this monitor, because the address no longer exists.
As a result, the assertion is not triggered if the monitor is removed before shutdown.
Description of problem:
Monitor crashed when number of monitors were redisced from 5 to 3
Version-Release number of selected component (if applicable):
16.1.0-1323.el8cp
How reproducible:
Tried once
Steps to Reproduce:
1. Configure a 5.x cluster using cephadm
2. Add some daemons
3. Reduce number of monitors to 3 from the default 5
Actual results:
"assert_condition": "m < ranks.size()",
"assert_file": "/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h",
"assert_func": "const entity_addrvec_t& MonMap::get_addrs(unsigned int) const",
"assert_line": 404,
"assert_msg": "/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h: In function 'const entity_addrvec_t& MonMap::get_addrs(unsigned int) const' thread 7f348ecf8700 time 2021-03-31T14:28:43.421216+0000\n/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h: 404: FAILED ceph_assert(m < ranks.size())\n",
Expected results:
Monitor shouldn't crash
Additional info:
Number of monitors were reduced using
[ceph: root@pluto002 /]# ceph orch apply mon 3
Scheduled mon update...
*** Bug 2111411 has been marked as a duplicate of this bug. ***
Comment 45Kamoltat (Junior) Sirivadhna
2022-11-01 13:20:29 UTC
Hi Eliska,
anywhere with <> is where I modified the text below:
*****
Previously, when the user reduced the number of monitors in the quorum using the `ceph orch apply mon _NUMBER_` command, `cephadm` would remove the monitor before shutting it down.
This would trigger an <assertion> because Ceph would assume that the monitor is shutting down before the monitor removal.
With this fix, a sanity check is added <to handle the case when> the current rank of the monitor is larger or equal to the quorum rank.
The monitor no longer exists in the monitor map, therefore <its peers do> not ping this monitor, because the address no longer exists.
As a result, the assertion is not triggered if the monitor is removed before shutdown.
*****
Let me know what you think thank you,
Kamoltat
Comment 50Kamoltat (Junior) Sirivadhna
2022-12-05 20:13:13 UTC
So the issue was because we are hitting this issue on a different code path that I didn't consider. Therefore,
I've filed a new PR: https://github.com/ceph/ceph/pull/49259 that will cover the code paths that I've missed.
Here is also a new upstream tracker for this: https://tracker.ceph.com/issues/58155.
Comment 52Kamoltat (Junior) Sirivadhna
2022-12-15 22:33:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Red Hat Ceph Storage 6.0 Bug Fix update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2023:1360
Description of problem: Monitor crashed when number of monitors were redisced from 5 to 3 Version-Release number of selected component (if applicable): 16.1.0-1323.el8cp How reproducible: Tried once Steps to Reproduce: 1. Configure a 5.x cluster using cephadm 2. Add some daemons 3. Reduce number of monitors to 3 from the default 5 Actual results: "assert_condition": "m < ranks.size()", "assert_file": "/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h", "assert_func": "const entity_addrvec_t& MonMap::get_addrs(unsigned int) const", "assert_line": 404, "assert_msg": "/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h: In function 'const entity_addrvec_t& MonMap::get_addrs(unsigned int) const' thread 7f348ecf8700 time 2021-03-31T14:28:43.421216+0000\n/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h: 404: FAILED ceph_assert(m < ranks.size())\n", Expected results: Monitor shouldn't crash Additional info: Number of monitors were reduced using [ceph: root@pluto002 /]# ceph orch apply mon 3 Scheduled mon update...