Bug 1826498
Summary: | hyperkube: container with ID starting with ... not found: ID does not exist | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Ryan Phillips <rphillips> | ||||||||
Component: | Node | Assignee: | Ryan Phillips <rphillips> | ||||||||
Status: | CLOSED DUPLICATE | QA Contact: | Sunil Choudhary <schoudha> | ||||||||
Severity: | urgent | Docs Contact: | |||||||||
Priority: | urgent | ||||||||||
Version: | 4.5 | CC: | aos-bugs, cblecker, jminter, jokerman, mpatel, sttts, wking, zyu | ||||||||
Target Milestone: | --- | ||||||||||
Target Release: | 4.5.0 | ||||||||||
Hardware: | Unspecified | ||||||||||
OS: | Unspecified | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | |||||||||||
: | 1827325 (view as bug list) | Environment: | |||||||||
Last Closed: | 2020-05-11 15:36:30 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Bug Depends On: | |||||||||||
Bug Blocks: | 1827325 | ||||||||||
Attachments: |
|
Description
Ryan Phillips
2020-04-21 19:31:34 UTC
I'm seeing an issue where crictl shows that a pod has exited but the kubelet hasn't realised and openshift still thinks it's running. Will attach logs. The pod in question is oauth-openshift-7fd76bfc9-vv47t. It lived around 2020/04/23 04:43:26 - 04:43:34 UTC. At 04:43:27 the kubelet shows SyncLoop (PLEG) ContainerStarted *and* ContainerDied messages. Some error in cri-o which is causing it to signal ContainerDied too early? Apr 23 04:43:27 aro-master-1 hyperkube[3632442]: I0423 04:43:27.659181 3632442 kubelet.go:1953] SyncLoop (PLEG): "oauth-openshift-7fd76bfc9-vv47t_openshift-authentication(f9b3ca89-25c9-4852-af2b-4f7abf890a04)", event: &pleg.PodLifecycleEvent{ID:"f9b3ca89-25c9-4852-af2b-4f7abf890a04", Type:"ContainerDied", Data:"cf93a4b04ab3796b028640cca6c39baaf294152387f1bf75de13995b969e8906"} Apr 23 04:43:27 aro-master-1 hyperkube[3632442]: I0423 04:43:27.659472 3632442 kubelet.go:1953] SyncLoop (PLEG): "oauth-openshift-7fd76bfc9-vv47t_openshift-authentication(f9b3ca89-25c9-4852-af2b-4f7abf890a04)", event: &pleg.PodLifecycleEvent{ID:"f9b3ca89-25c9-4852-af2b-4f7abf890a04", Type:"ContainerStarted", Data:"2d805ae593298f6ab7af88544db5fa1b35bf6aba05f3d6d415203ca9b866ef6c"} crictl inspect 2d805ae593298 | grep finishedAt "finishedAt": "2020-04-23T04:43:33.401990767Z", cri-o logs don't seem to show much. ^ the above on 4.3.10. Created attachment 1681191 [details]
output of crictl inspect
Created attachment 1681192 [details]
crio logs from journal
Created attachment 1681193 [details]
kubelet log snippet
[root@aro-master-1 ~]# rpm -qa |grep ^cri- cri-o-1.16.4-1.dev.rhaos4.3.git9238eee.el8.x86_64 cri-tools-1.14.0-2.rhaos4.2.el8.x86_64 Is this a dup of bug 1819906? Yes. Thanks Trevor! *** This bug has been marked as a duplicate of bug 1819906 *** |