Bug 1848939

After review the reported status of the etcd mirror pod post DR recovery was stale.

```
$ oc get pods -n openshift-etcd
{"level":"debug","ts":"2020-06-19T11:34:54.422-0400","caller":"balancer/balancer.go:60","msg":"registered balancer","policy":"picker-roundrobin-balanced","name":"etcd-picker-roundrobin-balanced"}
NAME                                                           READY   STATUS      RESTARTS   AGE
etcd-ip-10-0-139-35.us-east-2.compute.internal                 4/4     Running     2          6h56m
etcd-ip-10-0-172-33.us-east-2.compute.internal                 1/1     Running     0          162m
etcd-ip-10-0-204-230.us-east-2.compute.internal                4/4     Running     0          161m
```

Kubelet was reporting that pod etcd-ip-10-0-139-35.us-east-2.compute.internal was Ready but it was not. etcd was actually not running on this node. A secondary problem was because cluster-etcd-operator relies on Pod status to conclude[1] when scaling is required it was never scaled up. After speaking with node team we believe the issue could be related to a known upstream issue[2]. Operators etc rely on accurate status reporting of Pods, while it is understandable that time travel can cause unexpected state corner cases. We expect the API to reconcile static pod status. Currently, we are reviewing if we can document a workaround for DR.


[1] https://github.com/openshift/cluster-etcd-operator/blob/release-4.5/pkg/operator/clustermembercontroller/clustermembercontroller.go#L145
[2] https://github.com/kubernetes/kubernetes/pull/89155

Moving to node team to further triage. DR is an important part of OCP leaving this as high priority as we should not ship with broken DR.

Ryan is on leave

Is this still an issue?

disruptive tests are completely failing right now
https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/?job=*disruptive

recent runs of presubmit on etcd-operator PRs
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-etcd-operator/420/pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-disruptive/1293302526090678272
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-etcd-operator/405/pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-disruptive/1293236449713328128
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-etcd-operator/397/pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-disruptive/1293231460957818880

2 of the 3 fail with

fail [github.com/openshift/origin@/test/extended/dr/quorum_restore.go:100]: Unexpected error:
    <*errors.StatusError | 0xc0016b6460>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "deployments/scale.apps \"etcd-quorum-guard\" not found",
            Reason: "NotFound",
            Details: {
                Name: "etcd-quorum-guard",
                Group: "apps",
                Kind: "deployments/scale",
                UID: "",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 404,
        },
    }
    deployments/scale.apps "etcd-quorum-guard" not found
occurred

https://github.com/kubernetes/kubernetes/pull/89155 is the alleged fix for this.

Sam reports they have a workaround, reducing severity.

I would also be interested to is if https://github.com/kubernetes/kubernetes/pull/92442 had any impact on this situation.  It is included in 4.6 now.

Talking to Sam, Suresh and Clayton about this, this is a single example of a much larger problem of etcd going back in time and list watchers not being notified or able to handling going back in time.

In particular, once the kubelet deletes a pod, it will not attempt to redelete it (because it has no idea it exist on the apiserver because it doesn't trigger the watcher) unless it is restarted and does a full relist and reconciliation.

Not sure what the future is for this bug since it is an issue much more fundamental and systemic to kube that just this situation.

etcd team does have a workaround for this however, restarting the kubelet currently and possibly moving to manual deletion of the stale mirror pod.

Only element that I have yet to understand (or the only one I know I don't know) is why this worked in 4.4 but not 4.5.

From conversation with Clayton

===
i think hard failure is the only acceptable outcome

the question is how we patch it sanely in all componets

the DR test can verify it fixes kubelet

seth can you try a quick hack PR that os.Exits(1) if lister watcher detects a stitch in time (i think it prints an error today???)

and we can run disurptive on it with cluster bot and see

it'll be in client-go

one sec

reflector.go in client-go/tools/cache

ListAndWatch()

do the following

try to convert resourceVersion to int, if you can, compare to the previous avlue

if going back in time, klog.Fatalf
===

This likely something that will not be done in the 4.6 timeframe as it will take time to validate that every user of client-go does the Right Thing with this new behavior.  If this fix goes through, it will likely not be done by the Node team as we do not own client-go.

Sending to apiserver for comments and experimenting with the change to client-go

I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. We will revisit this in a future sprint.

(It's targeted for 4.7)

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

@Abut this is blocking DR working properly I bumped the severity to high, please reach out if you have questions around Seths questions.

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

Hi sbatsche, 
I am planning to address it this sprint. I will connect with you to get more details.

work is in progress, trying to setup an integration test to reproduce the issue.
https://github.com/openshift/kubernetes/pull/453

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

> work is in progress, trying to setup an integration test to reproduce the issue.
> https://github.com/openshift/kubernetes/pull/453

I will collaborate with Maru o this in 4.8

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Close as a dup of bug 1886160?  That was filed later, but it's what the PRs got attached to.

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

We don't have a shared client-go today, and I don't believe it's a good idea to start one.

What would be a reasonable way to have that in upstream? export KUBERNETES_EXIT_ON_TIMETRAVEL=1 ?

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Whiteboard if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Dear reporter, we greatly appreciate the bug you have reported here. Unfortunately, due to migration to a new issue-tracking system (https://issues.redhat.com/), we cannot continue triaging bugs reported in Bugzilla. Since this bug has been stale for multiple days, we, therefore, decided to close this bug.

If you think this is a mistake or this bug has a higher priority or severity as set today, please feel free to reopen this bug and tell us why. We are going to move every re-opened bug to https://issues.redhat.com. 

Thank you for your patience and understanding.