Description of problem: We saw this failure in CI "startTime": "2020-06-24T15:44:09Z", "containerStatuses": [ { "name": "controller-manager", "state": { "waiting": { "reason": "ImageInspectError", "message": "Failed to inspect image \"registry.svc.ci.openshift.org/ci-op-t9zj0l6i/stable@sha256:b645cb8528a3fc8c94462ddee29b23b839fa96a2fce480ba77f421383a4809f5\": rpc error: code = Unknown desc = size for layer \"d18dc0b8cb048a281d310fcf1a32c41dfb11586f6a8fcf9192704d449dbcd33c\" is unknown, failing getSize()" } }, "lastState": {}, "ready": false, "restartCount": 0, "image": "registry.svc.ci.openshift.org/ci-op-t9zj0l6i/stable@sha256:b645cb8528a3fc8c94462ddee29b23b839fa96a2fce480ba77f421383a4809f5", "imageID": "", "started": false } Version-Release number of selected component (if applicable): cri-o-1.19.0-30.dev.rhaos4.6.git0a84af5.el8.x86_64 podman-1.9.3-1.rhaos4.6.el8.x86_64 How reproducible: Happens in CI. We don't have a deterministic reproducer right now.
Related: https://github.com/containers/storage/issues/569 It seems that this happens when a layer is being created. Can't say more without a repro :(
As https://github.com/kubernetes/kubernetes/pull/94151 is now closed, I suggest we close this one as well.
I am having the same issue on a 4.6.19 cluster running on ppc64le. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning InspectFailed 9m24s (x2637 over 9h) kubelet Failed to inspect image "quay.io/openshift-psap/dcgmproftester-ppc64le:11.2.2-ubi8": rpc error: code = Unknown desc = layer not known Warning Failed 4m14s (x2661 over 9h) kubelet Error: ImageInspectError $ oc describe nodes NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME X.com Ready worker 26d v1.19.0+8d12420 xxxxxxxxxxxx <none> Red ....46.82.202102190839-0 (Ootpa) 4.18.0-193.41.1.el8_2.ppc64le cri-o://1.19.1-7.rhaos4.6.git6377f68.el8
Zvonko, can we get the CRI-O logs from nodes hitting this?
Sascha, can you take a look?
Yes, we're currently reproducing. Will get you the logs ASAP.
(In reply to Zvonko Kosic from comment #9) > Yes, we're currently reproducing. Will get you the logs ASAP. Hey, may I ask you to provide an update on this one? If there is no reproducer we may close this issue and reopen if it occurs again.
I'm closing this one as not reproducible for now.