Bug 1853020
| Summary: | Failed to inspect image with size for layer unknown | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mrunal Patel <mpatel> |
| Component: | Node | Assignee: | Sascha Grunert <sgrunert> |
| Node sub component: | CRI-O | QA Contact: | MinLi <minmli> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | chdeshpa, dwalsh, jokerman, nagrawal, sgrunert, simore, tsweeney, wking, zkosic |
| Version: | 4.6 | Keywords: | Reopened |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-05-30 07:06:57 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Related: https://github.com/containers/storage/issues/569 It seems that this happens when a layer is being created. Can't say more without a repro :( As https://github.com/kubernetes/kubernetes/pull/94151 is now closed, I suggest we close this one as well. I am having the same issue on a 4.6.19 cluster running on ppc64le. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning InspectFailed 9m24s (x2637 over 9h) kubelet Failed to inspect image "quay.io/openshift-psap/dcgmproftester-ppc64le:11.2.2-ubi8": rpc error: code = Unknown desc = layer not known Warning Failed 4m14s (x2661 over 9h) kubelet Error: ImageInspectError $ oc describe nodes NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME X.com Ready worker 26d v1.19.0+8d12420 xxxxxxxxxxxx <none> Red ....46.82.202102190839-0 (Ootpa) 4.18.0-193.41.1.el8_2.ppc64le cri-o://1.19.1-7.rhaos4.6.git6377f68.el8 Zvonko, can we get the CRI-O logs from nodes hitting this? Sascha, can you take a look? Zvonko, can we get the CRI-O logs from nodes hitting this? Yes, we're currently reproducing. Will get you the logs ASAP. (In reply to Zvonko Kosic from comment #9) > Yes, we're currently reproducing. Will get you the logs ASAP. Hey, may I ask you to provide an update on this one? If there is no reproducer we may close this issue and reopen if it occurs again. I'm closing this one as not reproducible for now. |
Description of problem: We saw this failure in CI "startTime": "2020-06-24T15:44:09Z", "containerStatuses": [ { "name": "controller-manager", "state": { "waiting": { "reason": "ImageInspectError", "message": "Failed to inspect image \"registry.svc.ci.openshift.org/ci-op-t9zj0l6i/stable@sha256:b645cb8528a3fc8c94462ddee29b23b839fa96a2fce480ba77f421383a4809f5\": rpc error: code = Unknown desc = size for layer \"d18dc0b8cb048a281d310fcf1a32c41dfb11586f6a8fcf9192704d449dbcd33c\" is unknown, failing getSize()" } }, "lastState": {}, "ready": false, "restartCount": 0, "image": "registry.svc.ci.openshift.org/ci-op-t9zj0l6i/stable@sha256:b645cb8528a3fc8c94462ddee29b23b839fa96a2fce480ba77f421383a4809f5", "imageID": "", "started": false } Version-Release number of selected component (if applicable): cri-o-1.19.0-30.dev.rhaos4.6.git0a84af5.el8.x86_64 podman-1.9.3-1.rhaos4.6.el8.x86_64 How reproducible: Happens in CI. We don't have a deterministic reproducer right now.