Bug 1688719
| Summary: | Pods go into RunContainerErrors, Failed create pod sandbox | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jiří Mencák <jmencak> | ||||
| Component: | Containers | Assignee: | Mrunal Patel <mpatel> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | weiwei jiang <wjiang> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.1.0 | CC: | aos-bugs, dwalsh, jokerman, mmccomas | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-03-14 17:33:17 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
*** This bug has been marked as a duplicate of bug 1688720 *** |
Created attachment 1543973 [details] Event logs Description of problem: In a larger-scale deployment (32 nodes). when deploying 2000 pods, at around 1000 pods issues with pods going into RunContainerErros, CrashLoopBackOff started to appear in 4.x cluster. Moreover, only 1999 pods out of 2000 pods were created. Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-04-234414 True False 2d20h Cluster version is 4.0.0-0.nightly-2019-03-04-234414 How reproducible: Create ~2000 pods in 32 node cluster. Actual results: nginx-passthrough-007-qvv2s 0/1 CrashLoopBackOff 3 5m13s nginx-passthrough-008-kpjp2 1/1 Running 1 5m6s nginx-passthrough-009-bhj6r 1/1 Running 0 5m5s nginx-passthrough-010-v6656 1/1 Running 0 5m nginx-passthrough-011-htlct 1/1 Running 0 4m59s nginx-passthrough-012-ctfdq 1/1 Running 0 4m55s nginx-passthrough-013-qlhv6 1/1 Running 0 4m50s nginx-passthrough-014-dqtjg 1/1 Running 0 4m45s nginx-passthrough-015-q4mzk 1/1 Running 0 4m41s nginx-passthrough-016-nmvtl 0/1 CrashLoopBackOff 4 4m41s nginx-passthrough-017-bsc8r 1/1 Running 1 4m40s nginx-passthrough-018-5rfzw 1/1 Running 0 4m31s nginx-passthrough-019-xxz9r 1/1 Running 0 4m29s nginx-passthrough-020-8g98g 1/1 Running 1 4m23s nginx-passthrough-021-wsvmg 1/1 Running 1 4m20s nginx-passthrough-022-ht9q2 1/1 Running 3 4m12s nginx-passthrough-023-b44v2 1/1 Running 0 3m33s nginx-passthrough-024-pj6rp 1/1 Running 3 3m33s nginx-passthrough-025-8zkgw 1/1 Running 2 3m30s nginx-passthrough-026-zsjrm 0/1 RunContainerError 4 3m31s nginx-passthrough-027-4crgg 1/1 Running 1 3m33s nginx-passthrough-028-x7mc2 1/1 Running 1 3m32s nginx-passthrough-029-slbk8 1/1 Running 0 3m32s nginx-passthrough-030-7vkxl 1/1 Running 0 3m34s nginx-passthrough-031-hxkdz 1/1 Running 2 3m35s nginx-passthrough-032-wq5xg 1/1 Running 1 3m35s nginx-passthrough-033-vnsbm 1/1 Running 2 3m30s nginx-passthrough-034-7szlx 1/1 Running 1 3m35s nginx-passthrough-035-fdn5j 1/1 Running 0 3m30s nginx-passthrough-036-s95cw 1/1 Running 0 3m33s nginx-passthrough-037-zrbbs 0/1 CrashLoopBackOff 2 3m34s nginx-passthrough-038-brk6d 1/1 Running 0 3m33s nginx-passthrough-039-76mk8 1/1 Running 1 3m35s 86m Warning Failed pod/nginx-passthrough-026-zsjrm Error: container nginx-passthrough is not in created state: stopped 87m Warning Failed pod/nginx-passthrough-026-zsjrm Error: container create failed: nsenter: could not ensure we are a cloned binary: Argument list to o long container_linux.go:336: starting container process caused "process_linux.go:279: running exec setns process for init caused \"exit status 23\"" 87m Normal SandboxChanged pod/nginx-passthrough-026-zsjrm Pod sandbox changed, it will be killed and re-created. Expected results: Containers being created without the errors above. Additional info: Finally, all 1999 were running after a while going through CrashLoopBackOff and RunContainerError. The missing pod that wasn't created was supposed to be in server-tls-passthrough-006 namespaces. The errors above are from server-tls-passthrough-005 namespaces. Full event logs attached.