Description of problem: Monitor crashed when number of monitors were redisced from 5 to 3 Version-Release number of selected component (if applicable): 16.1.0-1323.el8cp How reproducible: Tried once Steps to Reproduce: 1. Configure a 5.x cluster using cephadm 2. Add some daemons 3. Reduce number of monitors to 3 from the default 5 Actual results: "assert_condition": "m < ranks.size()", "assert_file": "/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h", "assert_func": "const entity_addrvec_t& MonMap::get_addrs(unsigned int) const", "assert_line": 404, "assert_msg": "/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h: In function 'const entity_addrvec_t& MonMap::get_addrs(unsigned int) const' thread 7f348ecf8700 time 2021-03-31T14:28:43.421216+0000\n/builddir/build/BUILD/ceph-16.1.0-1323-g7e7e1f4e/src/mon/MonMap.h: 404: FAILED ceph_assert(m < ranks.size())\n", Expected results: Monitor shouldn't crash Additional info: Number of monitors were reduced using [ceph: root@pluto002 /]# ceph orch apply mon 3 Scheduled mon update...
*** Bug 1961132 has been marked as a duplicate of this bug. ***
*** Bug 2010836 has been marked as a duplicate of this bug. ***
Yeah no work here so far. I'll bump it up my priority list since it's actually getting seen.
Attached crashed logs for reference captured from MON nodes. We saw the issue while upgrading from 5.x to 5.1.
*** Bug 1945272 has been marked as a duplicate of this bug. ***
*** Bug 2111411 has been marked as a duplicate of this bug. ***
Hi Eliska, anywhere with <> is where I modified the text below: ***** Previously, when the user reduced the number of monitors in the quorum using the `ceph orch apply mon _NUMBER_` command, `cephadm` would remove the monitor before shutting it down. This would trigger an <assertion> because Ceph would assume that the monitor is shutting down before the monitor removal. With this fix, a sanity check is added <to handle the case when> the current rank of the monitor is larger or equal to the quorum rank. The monitor no longer exists in the monitor map, therefore <its peers do> not ping this monitor, because the address no longer exists. As a result, the assertion is not triggered if the monitor is removed before shutdown. ***** Let me know what you think thank you, Kamoltat
So the issue was because we are hitting this issue on a different code path that I didn't consider. Therefore, I've filed a new PR: https://github.com/ceph/ceph/pull/49259 that will cover the code paths that I've missed. Here is also a new upstream tracker for this: https://tracker.ceph.com/issues/58155.
Cherry-picked to downstream
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 6.0 Bug Fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:1360