Description of problem:
In the steps of addLogicalPort, if we fail at any of the intermediary steps, usually the pod add event gets added to the oc.retryPods cache and the request is retried until it succeeds.
Currently this is not happening and only some pods are being retried while others aren't being retried.
Version-Release number of selected component (if applicable):
How reproducible: Always reproducible on OCP & KIND
Steps to Reproduce:
1. Create cluster and a bunch of pods and make them fail in one of the addLogicalPort steps.
2. Only some of the pods get retried while others are silently getting removed from the retryPods queue/cache.
Originally observed when trying to debug migration jobs on PR https://github.com/openshift/cluster-network-operator/pull/1254. Later found out that its reproducible on KIND as well.
[surya@hidden-temple Downloads]$ omg get pods -n openshift-apiserver -owide
NAME READY STATUS RESTARTS AGE IP NODE
apiserver-865c74cbcd-2c2wt 2/2 Running 1 1h5m 10.129.2.14 ip-10-0-136-71.us-west-2.compute.internal
apiserver-865c74cbcd-cb7p8 2/2 Running 3 55m 10.131.0.17 ip-10-0-137-79.us-west-2.compute.internal
apiserver-865c74cbcd-rfcts 0/2 Running 1 1h10m ip-10-0-237-148.us-west-2.compute.internal
2 API server pods came up eventually through retries while the 3rd one was never retried.
Fix is really just a one-liner if https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-network-operator/1288/pull-ci-openshift-cluster-network-operator-master-e2e-network-migration/1484980132102803456 passes.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.