For example, in this 4.2 promotion informer [1]: Dec 05 02:56:05.093 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-140-154.ec2.internal invariant violation: pod may not transition Running->Pending Ryan mentioned this as a known issue with the kubelet state machine. Monitor code is [2]. Might be some related discussion in [3]. Shows up in 48 CI jobs over the past 24h [4]. [1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-rhel7-workers-4.2/106 [2]: https://github.com/openshift/origin/blob/49aa545b8d1fb0638b820357e2b22c7b0b962173/pkg/monitor/pod.go#L96-L100 [3]: https://github.com/kubernetes/kubernetes/pull/54530 [4]: https://search.svc.ci.openshift.org/chart?search=pod%20may%20not%20transition
This print was placed into origin within this commit [1]. The state machine within the Kubelet is not documented nor intuitive to reason about, and this log was put in to track the frequency. 1. https://github.com/openshift/origin/commit/4941d3c0f60531bdb3f0589be4eea779030fb834
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.2-nightly-to-4.3-nightly/102 in past 14 days there are 1038 occurrences https://search.svc.ci.openshift.org/?search=Running-%3EPending&maxAge=336h&context=2&type=all
Upstream PR: https://github.com/kubernetes/kubernetes/pull/88440 Upstream Issue: https://github.com/kubernetes/kubernetes/issues/58711
*** This bug has been marked as a duplicate of bug 1810652 ***