Bug 1887857

Summary:	[sig-apps][Feature:DeploymentConfig] deploymentconfigs when run iteratively should immediately start a new deployment
Product:	OpenShift Container Platform	Reporter:	Mike Dame <mdame>
Component:	Node	Assignee:	Peter Hunt <pehunt>
Node sub component:	Kubelet	QA Contact:	Sunil Choudhary <schoudha>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	aos-bugs, jokerman, mfojtik
Version:	4.7	Keywords:	UpcomingSprint
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:	[sig-apps][Feature:DeploymentConfig] deploymentconfigs when run iteratively should immediately start a new deployment
Last Closed:	2020-11-10 16:55:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Mike Dame 2020-10-13 13:34:30 UTC

test:
[sig-apps][Feature:DeploymentConfig] deploymentconfigs when run iteratively should immediately start a new deployment 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-apps%5C%5D%5C%5BFeature%3ADeploymentConfig%5C%5D+deploymentconfigs+when+run+iteratively+should+immediately+start+a+new+deployment


Link to recent failure: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.7/1315369643245834240

fail [github.com/openshift/origin/test/extended/deployments/deployments.go:247]: Unexpected error:
    <*errors.errorString | 0xc00034c8a0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred

Comment 1 Maciej Szulik 2020-10-14 11:02:23 UTC

Both in the event:

Oct 11 19:59:03.209: INFO: At 2020-10-11 19:57:59 +0000 UTC - event for deployment-simple-1-deploy: {kubelet ip-10-0-157-24.us-west-2.compute.internal} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF

and in kubelet logs:

E1011 19:57:59.788429    1530 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "deployment-simple-1-deploy_e2e-test-cli-deployment-th5rj(0802f8f7-6d22-4266-9b60-816deee91135)" failed: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF                                                                                                                                                                                                                 
E1011 19:57:59.788456    1530 kuberuntime_manager.go:730] createPodSandbox for pod "deployment-simple-1-deploy_e2e-test-cli-deployment-th5rj(0802f8f7-6d22-4266-9b60-816deee91135)" failed: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF                                                                                                                                                                                                                
E1011 19:57:59.788559    1530 pod_workers.go:191] Error syncing pod 0802f8f7-6d22-4266-9b60-816deee91135 ("deployment-simple-1-deploy_e2e-test-cli-deployment-th5rj(0802f8f7-6d22-4266-9b60-816deee91135)"), skipping: failed to "CreatePodSandbox" for "deployment-simple-1-deploy_e2e-test-cli-deployment-th5rj(0802f8f7-6d22-4266-9b60-816deee91135)" with CreatePodSandboxError: "CreatePodSandbox for pod \"deployment-simple-1-deploy_e2e-test-cli-deployment-th5rj(0802f8f7-6d22-4266-9b60-816deee91135)\" failed: rpc error: code = Unknown desc = error reading container (probably exited) json message: EOF"

so it's either problem with kubelet or crio that needs further investigation why 

Moving on to node team and bumping to medium, since although this was only a flake in the run, such error in kubelet or crio deserves further investigation.

Comment 2 Ryan Phillips 2020-10-23 16:44:04 UTC

We see these error messages often when a pod is requested to start, and then being torn down before it has fully started. This is not necessarily a bug. We need to suppress these types of messages.