Back to bug 1945266

Who When What Removed Added
Greg Farnum 2021-04-01 00:09:03 UTC CC gfarnum
Greg Farnum 2021-04-01 00:19:24 UTC Assignee nojha gfarnum
Neha Ojha 2021-04-01 00:25:57 UTC Status NEW ASSIGNED
Neha Ojha 2021-04-01 00:40:57 UTC Link ID Ceph Project Bug Tracker 50089
Pawan 2021-05-14 08:12:24 UTC CC pdhiran
Manasa 2021-05-17 11:34:18 UTC CC mgowri
Red Hat One Jira (issues.redhat.com) 2021-06-02 19:05:32 UTC Link ID Red Hat Issue Tracker RHCEPH-221
Vasishta 2021-06-24 02:24:00 UTC Doc Type If docs needed, set a value Known Issue
Vasishta 2021-06-24 02:24:22 UTC Blocks 1959686
Ranjini M N 2021-06-24 05:37:54 UTC CC rmandyam
Flags needinfo?(gfarnum)
Red Hat Bugzilla 2021-06-29 07:41:26 UTC CC dzafman
Ranjini M N 2021-07-01 07:07:21 UTC Flags needinfo?(gfarnum)
Greg Farnum 2021-07-06 18:54:39 UTC Flags needinfo?(gfarnum) needinfo?(gfarnum) needinfo?(rmandyam)
Ranjini M N 2021-07-07 04:52:39 UTC Flags needinfo?(rmandyam) needinfo?(gfarnum)
Ranjini M N 2021-07-13 07:17:13 UTC Blocks 1959686
Greg Farnum 2021-07-16 02:30:47 UTC Flags needinfo?(gfarnum)
Red Hat Bugzilla 2021-08-19 16:38:29 UTC CC kchai
Vikhyat Umrao 2021-10-05 17:58:25 UTC CC hyelloji
Hemanth Kumar 2021-10-05 18:44:26 UTC Flags needinfo?(vumrao) needinfo?(vumrao) needinfo?(gfarnum)
Vikhyat Umrao 2021-10-05 19:03:08 UTC Flags needinfo?(vumrao) needinfo?(vumrao)
Greg Farnum 2021-10-07 17:15:48 UTC Flags needinfo?(gfarnum)
Neha Ojha 2021-10-12 18:15:46 UTC Target Release 5.1 5.2
Vikhyat Umrao 2021-11-11 17:49:41 UTC Summary Monitor crash observed when number of monitors were reduced from 5 to 3 using ceph orchestrator Monitor crash - ceph_assert(m < ranks.size()) - observed when number of monitors were reduced from 5 to 3 using ceph orchestrator
Preethi 2021-11-15 06:33:25 UTC CC pnataraj
Neha Ojha 2022-01-27 00:29:08 UTC Assignee gfarnum ksirivad
Kamoltat (Junior) Sirivadhna 2022-01-31 15:32:45 UTC Flags needinfo?(vashastr)
Neha Ojha 2022-01-31 22:01:13 UTC Flags needinfo?(pnataraj)
Vikhyat Umrao 2022-02-03 00:12:42 UTC Flags needinfo?(ksirivad)
CC ksirivad
Vasishta 2022-02-07 12:12:22 UTC Flags needinfo?(vashastr)
Preethi 2022-02-08 05:09:12 UTC Flags needinfo?(pnataraj)
Kamoltat (Junior) Sirivadhna 2022-02-09 04:57:35 UTC Flags needinfo?(ksirivad)
Kamoltat (Junior) Sirivadhna 2022-03-18 01:26:25 UTC Blocks 1959686
Neha Ojha 2022-03-21 21:07:39 UTC Target Release 5.2 6.1
Vikhyat Umrao 2022-07-27 15:19:01 UTC CC anrao
Chaithra 2022-09-29 09:48:38 UTC CC ckulal
Neha Ojha 2022-10-19 19:57:19 UTC Target Release 6.1 6.0
Kamoltat (Junior) Sirivadhna 2022-10-21 18:59:01 UTC Status ASSIGNED POST
errata-xmlrpc 2022-10-22 08:39:51 UTC CC tserlin
Fixed In Version ceph-17.2.5-2.el9cp
Status POST MODIFIED
Status MODIFIED ON_QA
Eliska 2022-10-24 08:55:29 UTC CC ekristov
Flags needinfo?(ksirivad)
Kamoltat (Junior) Sirivadhna 2022-10-24 14:40:12 UTC Doc Type Known Issue Bug Fix
Flags needinfo?(ksirivad)
Doc Text Cause:

When we reduce the number of monitors in the quorum using ``ceph orch apply mon <num>``, ceph-adm will remove the monitor before shutting it down which causes us to go into assertion because Ceph assumes that we will be shutting down the monitor before removing the monitor.

Consequence:

we go into an assertion and the monitor crashes

Fix:

We add a sanity check in the case where the current rank of the monitor might be >= quorum rank, this means that the mon no longer exists in the monmap so we shouldn't ping this monitor since the address no longer exists.

Result:
We no longer go into assertion in the case where mon is being removed before shutdown.
Eliska 2022-10-31 07:56:32 UTC Doc Text Cause:

When we reduce the number of monitors in the quorum using ``ceph orch apply mon <num>``, ceph-adm will remove the monitor before shutting it down which causes us to go into assertion because Ceph assumes that we will be shutting down the monitor before removing the monitor.

Consequence:

we go into an assertion and the monitor crashes

Fix:

We add a sanity check in the case where the current rank of the monitor might be >= quorum rank, this means that the mon no longer exists in the monmap so we shouldn't ping this monitor since the address no longer exists.

Result:
We no longer go into assertion in the case where mon is being removed before shutdown.
.The Ceph Monitor no longer crashes after reducing the number of monitors

Previously, when the user reduced the number of monitors in the quorum using the `ceph orch apply mon _NUMBER_` command, `cephadm` would remove the monitor before shutting it down.
This would trigger an assertion, because Ceph would assume that the monitor is shutting down before the monitor removal.

With this fix, a sanity check is added in the case the current rank of the monitor is larger or equal to the quorum rank.
The monitor no longer exists in the monitor map, therefore the action does not ping this monitor, because the address no longer exists.
As a result, the assertion is not triggered if the monitor is removed before shutdown.
Docs Contact ekristov
Flags needinfo?(ksirivad)
Eliska 2022-10-31 08:02:18 UTC Blocks 2126050
Pasi Karkkainen 2022-10-31 09:47:17 UTC CC pasik
Kamoltat (Junior) Sirivadhna 2022-11-01 13:20:29 UTC Flags needinfo?(ksirivad)
Eliska 2022-11-01 14:08:43 UTC Doc Text .The Ceph Monitor no longer crashes after reducing the number of monitors

Previously, when the user reduced the number of monitors in the quorum using the `ceph orch apply mon _NUMBER_` command, `cephadm` would remove the monitor before shutting it down.
This would trigger an assertion, because Ceph would assume that the monitor is shutting down before the monitor removal.

With this fix, a sanity check is added in the case the current rank of the monitor is larger or equal to the quorum rank.
The monitor no longer exists in the monitor map, therefore the action does not ping this monitor, because the address no longer exists.
As a result, the assertion is not triggered if the monitor is removed before shutdown.
.The Ceph Monitor no longer crashes after reducing the number of monitors

Previously, when the user reduced the number of monitors in the quorum using the `ceph orch apply mon _NUMBER_` command, `cephadm` would remove the monitor before shutting it down.
This would trigger an assertion because Ceph would assume that the monitor is shutting down before the monitor removal.

With this fix, a sanity check is added to handle the case when the current rank of the monitor is larger or equal to the quorum rank.
The monitor no longer exists in the monitor map, therefore its peers do not ping this monitor, because the address no longer exists.
As a result, the assertion is not triggered if the monitor is removed before shutdown.
Kamoltat (Junior) Sirivadhna 2022-11-11 19:17:17 UTC Blocks 2142141
Manohar Murthy 2022-11-15 09:10:53 UTC QA Contact mmurthy pdhiran
Pawan 2022-12-01 03:52:52 UTC Status ON_QA ASSIGNED
Flags needinfo?(ksirivad)
Bipin Kunal 2022-12-02 06:53:17 UTC CC bkunal
Kamoltat (Junior) Sirivadhna 2022-12-05 20:13:13 UTC Flags needinfo?(ksirivad)
Kamoltat (Junior) Sirivadhna 2022-12-15 22:33:27 UTC Status ASSIGNED POST
Ken Dreyer (Red Hat) 2022-12-16 02:25:14 UTC Status POST MODIFIED
CC kdreyer
Fixed In Version ceph-17.2.5-2.el9cp ceph-17.2.5-30.el9cp
errata-xmlrpc 2022-12-16 02:55:35 UTC Status MODIFIED ON_QA
Pawan 2022-12-28 09:09:51 UTC Status ON_QA VERIFIED
Red Hat Bugzilla 2022-12-31 19:06:55 UTC CC ckulal
Red Hat Bugzilla 2022-12-31 19:15:32 UTC CC mgowri
Red Hat Bugzilla 2022-12-31 19:21:46 UTC CC anrao
Red Hat Bugzilla 2022-12-31 19:24:09 UTC CC pnataraj
Red Hat Bugzilla 2022-12-31 19:32:40 UTC QA Contact pdhiran
CC pdhiran
Red Hat Bugzilla 2022-12-31 19:50:43 UTC CC hyelloji
Red Hat Bugzilla 2022-12-31 20:00:05 UTC CC sseshasa
Red Hat Bugzilla 2022-12-31 23:43:39 UTC CC rzarzyns
Red Hat Bugzilla 2022-12-31 23:45:58 UTC CC akupczyk
Red Hat Bugzilla 2023-01-01 05:35:26 UTC Assignee ksirivad nojha
CC ksirivad
Red Hat Bugzilla 2023-01-01 05:39:55 UTC CC tserlin
Red Hat Bugzilla 2023-01-01 06:03:34 UTC CC kdreyer
Red Hat Bugzilla 2023-01-01 08:29:55 UTC CC bkunal
Red Hat Bugzilla 2023-01-01 08:38:59 UTC Assignee nojha nobody
CC nojha
Red Hat Bugzilla 2023-01-01 08:50:16 UTC CC vumrao
Alasdair Kergon 2023-01-04 04:40:45 UTC CC akupczyk
Alasdair Kergon 2023-01-04 04:43:11 UTC Assignee nobody ksirivad
Alasdair Kergon 2023-01-04 04:46:10 UTC CC anrao
Alasdair Kergon 2023-01-04 04:50:47 UTC CC ckulal
Alasdair Kergon 2023-01-04 04:56:54 UTC QA Contact pdhiran
Alasdair Kergon 2023-01-04 04:57:17 UTC CC hyelloji
Alasdair Kergon 2023-01-04 05:03:42 UTC CC kdreyer
Alasdair Kergon 2023-01-04 05:08:58 UTC CC ksirivad
Alasdair Kergon 2023-01-04 05:14:10 UTC CC mgowri
Alasdair Kergon 2023-01-04 05:21:38 UTC CC nojha
Alasdair Kergon 2023-01-04 05:30:13 UTC CC pdhiran
Alasdair Kergon 2023-01-04 05:32:18 UTC CC pnataraj
Alasdair Kergon 2023-01-04 05:37:37 UTC CC rzarzyns
Alasdair Kergon 2023-01-04 05:59:30 UTC CC vumrao
Alasdair Kergon 2023-01-04 06:09:44 UTC CC bkunal
Alasdair Kergon 2023-01-04 06:25:53 UTC CC tserlin
Alasdair Kergon 2023-01-04 06:56:31 UTC CC sseshasa
Red Hat Bugzilla 2023-01-09 08:30:25 UTC CC ceph-eng-bugs
Alasdair Kergon 2023-01-09 19:43:36 UTC CC ceph-eng-bugs
errata-xmlrpc 2023-03-20 18:37:15 UTC Status VERIFIED RELEASE_PENDING
errata-xmlrpc 2023-03-20 18:55:33 UTC Status RELEASE_PENDING CLOSED
Resolution --- ERRATA
Last Closed 2023-03-20 18:55:33 UTC
errata-xmlrpc 2023-03-20 18:56:27 UTC Link ID Red Hat Product Errata RHBA-2023:1360

Back to bug 1945266