Bug 2139876

When the tie breaker mon fails or we need to replace it in stretched mode it won't let you delete it, it needs special steps.

$ ceph orch daemon rm mon.ceph-arbiter --force
mgr_module.MonCommandFailed: mon rm failed: you cannot remove stretch mode's tiebreaker monitor retval: -22

Should they be in this doc/operations guide

Here is an overview of the steps followed to delete/replace a failed arbiter monitor:

Set the crush_location for the monitor we are going to add to the cluster:

$ ceph mon add ceph03 192.168.1.207 datacenter=DC3
Deploy a new monitor using cephadm:

$ ceph orch daemon add mon ceph03
Deployed mon.ceph03 on host 'ceph03'
Prevent cephadm to delete the new monitor as we configured cephadm to deploy only 5 monitors:

$ ceph orch apply mon --unmanaged
Scheduled mon update...
Ensure we have 5 monitors in quorum:

$ ceph -s
...
  mon: 6 daemons, quorum ceph01,ceph02,ceph04,ceph05,ceph03 (age 12s), out of quorum: ceph-arbiter
...
Set the new tiebreaker as ceph03:

$ ceph mon set_new_tiebreaker ceph03
Remove the former ceph-arbiter monitor:

$ ceph orch daemon rm mon.ceph-arbiter --force
Removed mon.ceph-arbiter from host 'ceph-arbiter'
Once the monitor has been removed from ceph-arbiter, redeploy the monitor:

$ ceph mon add ceph-arbiter 192.168.1.211 datacenter=DC3
Deploy a new monitor using cephadm:

$ ceph orch daemon add mon ceph-arbiter
Deployed mon.ceph-arbiter on host 'ceph-arbiter'
Ensure we have 6 monitors in quorum:

$ ceph -s
...
mon: 6 daemons, quorum ceph01,ceph02,ceph04,ceph05,ceph03,ceph-arbiter (age 9s)
...
Set the new tiebreaker as ceph-arbiter:

$ ceph mon set_new_tiebreaker ceph-arbiter
Remove the ceph03 monitor we added:

$ ceph orch daemon rm mon.ceph03 --force
Removed mon.ceph03 from host 'ceph03'
Ensure we have 5 monitors in quorum:

$ ceph -s
...
mon: 5 daemons, quorum ceph01,ceph04,ceph05,ceph-arbiter,ceph02 (age 2h)
...
Get the monitor quorum status and verify everything is configured properly:

$ ceph daemon /var/run/ceph/0fb6800e-4efa-11ed-af7d-525400247d4f/ceph-mon.ceph01.asok mon_status
Finally, manage the monitors back with cephadm specifying the correct placement after the monitor replacement:

$ ceph orch apply mon --placement="ceph01,ceph02,ceph04,ceph05,ceph-arbiter"



Reported by: dparkes

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/operations_guide/management-of-monitors-using-the-ceph-orchestrator#annotations:fd866251-dc20-4c0a-9bff-8ee94f7aca4d