This bug was initially created as a copy of Bug #2027826

I am copying this bug because: 

The docs must be updated if the dependent fixes are made. 
As seen in https://github.com/red-hat-storage/ocs-operator/pull/1518/files and documented in 4.10 already, we need the 4.9 docs to also reflect the change.


This bug was initially created as a copy of Bug #2026007

I am copying this bug because: 

An OCS operator update is needed to expose an option to force removal of an OSD if Ceph indicates the OSD is not safe-to-destroy.

If https://bugzilla.redhat.com/show_bug.cgi?id=2027396 is approved for 4.9.z, we will also need this for 4.9.z.


Description of problem (please be detailed as possible and provide log
snippets):
Use ceph 'osd safe-to-destroy' and 'osd ok-to-stop' feature in OSD purge job

[1] mgr: implement 'osd safe-to-destroy' and 'osd ok-to-stop' commands
     https://github.com/ceph/ceph/pull/16976 
     An osd is safe to destroy if
we have osd_stat for it
osd_stat indicates no pgs stored
all pgs are known
no pgs map to it
i.e., overall data durability will not be affected
An OSD is ok to stop if

we have the pg stats we need
no PGs will drop below min_size
i.e., availability won't be immediately compromised