Bug 1932478

From the Rook Operator logs, I see that Rook scheduled the rook-ceph-mon-p-canary pod at about 1500 UTC on Feb 23. The canary pod was not created by Kubernetes until 1215 the next day, almost 21 hours later (relevant lines below). The canary does not use a liveness probe, so that would not be a possible cause of that issue. Based on this behavior, I suspect the ROKS cluster may be unhealthy. Possibly running low on resources although the 3 nodes do not report any resource pressure.

    2021-02-23T09:03:52.658955896-06:00 2021-02-23 15:03:52.658897 I | op-mon: sched-mon: created canary monitor rook-ceph-mon-p-canary pvc rook-ceph-mon-p
    # ...
    2021-02-24T06:15:16.725115391-06:00 2021-02-24 12:15:16.725055 I | op-mon: mon "p" endpoint are [v2:172.21.95.180:3300,v1:172.21.95.180:6789]


I am curious about the liveness probe issue. The mon.p pod has the appropriate liveness probe (below). The mon.p logs don't show anything suggesting that they are encountering an error which is consistent with the pod being killed by the liveness probe.

    exec [env -i sh -c ceph --admin-daemon /run/ceph/ceph-mon.p.asok mon_status] delay=10s timeout=1s period=10s #success=1 #failure=3


The message returned by the liveness probe (below) doesn't have a lot of information. Additionally, it is unclear why mon.p is failing the liveness probe whereas mon.g and mon.l are not.

    Warning  Unhealthy  30m (x224 over 4h50m)  kubelet  Liveness probe failed: no valid command found; 10 closest matches:
    0
    1
    2
    abort
    assert
    config diff
    config diff get <var>
    config get <var>
    config help {<var>}
    config set <var> <val> [<val>...]
    admin_socket: invalid command


If my hypothesis that the node on which mon.p is scheduled is low on resources (or running slowly for some other reason), it's possible that the `initialDelaySeconds: 10` is not long enough for mon.p to be sufficiently started to expose its admin socket to the liveness probe.

This hypothesis seems consistent with an upstream Rook bug (below) except that this bug is a mon while the upstream bug is an OSD. 
https://github.com/rook/rook/issues/5814

This issue for the OSD was fixed by an upstream Rook PR (below) that increased the liveness probe start delay to 45s for OSDs.
https://github.com/rook/rook/pull/5607


ACTION:
As a debugging measure, use `kubectl -n openshift-storage edit deployment rook-ceph-mon-p` and modify the `livenessProbe.initialDelaySeconds` to `45`. If mon.p is no longer the mon failing, you may need to update the command to reflect a different mon ID. If this still does not work, try increasing again up to 5 minutes. Please report back your findings.

*** This bug has been marked as a duplicate of bug 1929188 ***

After more review, I am not sure this is a duplicate. I am repoening this for now but still waiting on the requested info from @ekuric

Adding @ebenahar needinfo since I haven't gotten any response in over a month. If this bug is no longer valid, can it be closed?

(In reply to Blaine Gardner from comment #7)
> Adding @ebenahar needinfo since I haven't gotten any response in
> over a month. If this bug is no longer valid, can it be closed?

I cannot reproduce this / do not see it with OCS v4.7, I can reopen it if it is visible again. , I guess it can be closed for now

Closing with CURRENTRELEASE then.