Description of problem: We (Assisted Installer team) have been doing WAN emulation tests and introducing network latency as well as dropped packages simulation since we are expecting to be running AI and managing deployments in such environments. During our latest tests, we noticed some failures from cri-o downloading images and, what it looks like, a failure to retry. We noticed this issue in `podman` which was fixed and backported all the way back to `4.8` (RHEL 8.4). Original issue: https://bugzilla.redhat.com/show_bug.cgi?id=1964591 Podman BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1966872 This issue is being created as requested by @mpatel Expected results: cri-o should retry to pull the images, image index should not be corrupted.
I think we can solve the issue in c/storage. Proposing a fix in https://github.com/containers/storage/pull/943 Testing the fix in https://github.com/cri-o/cri-o/pull/5005
Patches got merged and are now part of c/storage v1.32.3 Backport is in progress https://github.com/cri-o/cri-o/pull/5015
Upstream PR got merged, updated packages should be available soon and the fix should be part of the next OpenShift 4.8 patch release.
verified on 4.8.0-0.nightly-2022-02-08-181141 sh-4.4# crictl pull httpd Image is up to date for docker.io/library/httpd@sha256:5cc947a200524a822883dc6ce6456d852d7c5629ab177dfbf7e38c1b4a647705 # crictl images IMAGE TAG IMAGE ID SIZE docker.io/library/httpd latest a8ea074f4566a 148MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> c7569d8c767fa 396MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 7d1fc75e48f73 327MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 9cbefdfc7d582 331MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 7437858b89390 342MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 74c73c5ddbc38 340MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 22985b9e44282 460MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 853b9b86aa03a 389MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 4530c04b6ac9c 479MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 2dc68e509dd25 492MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 52c2eee1a5f9d 452MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 70b1fc1c4d760 403MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 1a1825758a271 739MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> a194d5e376fea 309MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 33b37ba471bb0 449MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> c5ddfa2a56e4e 553MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> bf7ebdd528f7d 610MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 04827d930943f 484MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 8445c376f8a75 335MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> c501396873225 370MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 4d80f3474c909 363MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 879a7d4d9e0a5 335MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 0a5c71b90fb10 402MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> c910e906cbc0d 413MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 5e9f6102f4162 399MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 2a23f5e6cad28 300MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 7085d32d6e79d 383MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> 42b7926db6ba4 420MB quay.io/openshift-release-dev/ocp-v4.0-art-dev <none> a7c33ba4d99e8 297MB manually delete entry matching image httpd in /var/lib/containers/storage/overlay-layers/layers.json sh-4.4# crictl inspecti a8ea074f4566a FATA[0000] image status for "a8ea074f4566a" request: rpc error: code = Unknown desc = layer not known sh-4.4# crictl pull httpd Image is up to date for docker.io/library/httpd@sha256:5cc947a200524a822883dc6ce6456d852d7c5629ab177dfbf7e38c1b4a647705 sh-4.4# crictl inspecti a8ea074f4566a { "status": { "id": "a8ea074f4566addcd01f9745397f32be471df4a4abf200f0f10c885ed14b1d28", "repoTags": [ "docker.io/library/httpd:latest" ], "repoDigests": [ "docker.io/library/httpd@sha256:5cc947a200524a822883dc6ce6456d852d7c5629ab177dfbf7e38c1b4a647705", "docker.io/library/httpd@sha256:7280c780e026f3073f2570cbf3cbe044dacd9bd90c7094f1c6e0e687444c9cec" ], "size": "147702595", "uid": null, "username": "", "spec": null ...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.8.31 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0484