Bug 1780386

Summary:	Occasional: invariant violation: pod may not transition Running->Pending
Product:	OpenShift Container Platform	Reporter:	W. Trevor King <wking>
Component:	Node	Assignee:	Ryan Phillips <rphillips>
Status:	CLOSED DUPLICATE	QA Contact:	Sunil Choudhary <schoudha>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.2.z	CC:	aos-bugs, jokerman, nagrawal, tnozicka
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-26 15:30:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description W. Trevor King 2019-12-05 20:24:10 UTC

For example, in this 4.2 promotion informer [1]:

Dec 05 02:56:05.093 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-140-154.ec2.internal invariant violation: pod may not transition Running->Pending

Ryan mentioned this as a known issue with the kubelet state machine.  Monitor code is [2].  Might be some related discussion in [3].  Shows up in 48 CI jobs over the past 24h [4].

[1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-rhel7-workers-4.2/106
[2]: https://github.com/openshift/origin/blob/49aa545b8d1fb0638b820357e2b22c7b0b962173/pkg/monitor/pod.go#L96-L100
[3]: https://github.com/kubernetes/kubernetes/pull/54530
[4]: https://search.svc.ci.openshift.org/chart?search=pod%20may%20not%20transition

Comment 1 Ryan Phillips 2019-12-05 20:53:40 UTC

This print was placed into origin within this commit [1]. The state machine within the Kubelet is not documented nor intuitive to reason about, and this log was put in to track the frequency.

1. https://github.com/openshift/origin/commit/4941d3c0f60531bdb3f0589be4eea779030fb834

Comment 3 Tomáš Nožička 2020-02-19 12:56:00 UTC

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.2-nightly-to-4.3-nightly/102

in past 14 days there are 1038 occurrences

https://search.svc.ci.openshift.org/?search=Running-%3EPending&maxAge=336h&context=2&type=all

Comment 4 Ryan Phillips 2020-02-26 01:46:39 UTC

Upstream PR: https://github.com/kubernetes/kubernetes/pull/88440
Upstream Issue: https://github.com/kubernetes/kubernetes/issues/58711

Comment 5 Ryan Phillips 2020-03-26 15:30:28 UTC


*** This bug has been marked as a duplicate of bug 1810652 ***