Description of problem (please be detailed as possible and provide log
snippests):
[DR] rbd-mirror pod does not restart even after enabling rbd_mirror_die_after_seconds option


Version of all relevant components (if applicable):

OCP version:- 4.11.0-0.nightly-2022-05-05-015322
ODF version:- 4.11.0-63
CEPH version:- ceph version 16.2.7-109.el8cp (9f20d0292c5f7b341bc2bc580fb119416323fdc1) pacific (stable)

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
yes

Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Deploy the RDR cluster
2. Enable rbd_mirror_die_after_seconds option via ceph config set global rbd_mirror_die_after_seconds 3600
3. Check the last restart time
 

Actual results:

1:- rook-ceph-rbd-mirror-a-6c86b84c75-v5s7f                           2/2     Running     1 (4h16m ago)   5h16m


2:- Output from rbd-mirror pod
debug 2022-05-06T06:28:01.468+0000 7f4c32607540  0 rbd::mirror::Mirror: 0x55f48e2598c0 init: rbd_mirror_die_after_seconds=3600, arming death timer
debug 2022-05-06T07:28:01.468+0000 7f4c1d1b8700 -1 rbd::mirror::Mirror: 0x55f48e2598c0 operator(): stopping due to rbd_mirror_die_after_seconds

Expected results:
Rbd mirror pod should restart as per design

Additional info:
I see this issue only on secondary site

This causes the daemon state to go down 

#C1:- bash-4.4$  rbd mirror pool status ocs-storagecluster-cephblockpool
health: WARNING
daemon health: OK
image health: WARNING
images: 108 total
    108 unknown

#C2:- bash-4.4$  rbd mirror pool status ocs-storagecluster-cephblockpool
health: ERROR
daemon health: WARNING
image health: ERROR
images: 108 total
    93 error
    5 stopping_replay
    10 stopped