Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1972063

Summary:	cri-o pull retry and index validation on slow networks
Product:	OpenShift Container Platform	Reporter:	Flavio Percoco <fpercoco>
Component:	Node	Assignee:	Sascha Grunert <sgrunert>
Node sub component:	CRI-O	QA Contact:	MinLi <minmli>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	aos-bugs, mpatel, nagrawal, sgrunert
Version:	4.8
Target Milestone:	---
Target Release:	4.8.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-02-16 06:51:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Flavio Percoco 2021-06-15 08:01:24 UTC

Description of problem:

We (Assisted Installer team) have been doing WAN emulation tests and introducing network latency as well as dropped packages simulation since we are expecting to be running AI and managing deployments in such environments. During our latest tests, we noticed some failures from cri-o downloading images and, what it looks like, a failure to retry. 

We noticed this issue in `podman` which was fixed and backported all the way back to `4.8` (RHEL 8.4).

Original issue: https://bugzilla.redhat.com/show_bug.cgi?id=1964591
Podman BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1966872


This issue is being created as requested by @mpatel


Expected results:

cri-o should retry to pull the images, image index should not be corrupted.

Comment 3 Sascha Grunert 2021-06-16 13:32:07 UTC

I think we can solve the issue in c/storage. Proposing a fix in https://github.com/containers/storage/pull/943

Testing the fix in https://github.com/cri-o/cri-o/pull/5005

Comment 7 Sascha Grunert 2021-06-18 08:48:11 UTC

Patches got merged and are now part of c/storage v1.32.3

Backport is in progress https://github.com/cri-o/cri-o/pull/5015

Comment 8 Sascha Grunert 2021-06-21 15:10:00 UTC

Upstream PR got merged, updated packages should be available soon and the fix should be part of the next OpenShift 4.8 patch release.

Comment 13 MinLi 2022-02-10 12:04:11 UTC

verified on 4.8.0-0.nightly-2022-02-08-181141

sh-4.4# crictl pull httpd
Image is up to date for docker.io/library/httpd@sha256:5cc947a200524a822883dc6ce6456d852d7c5629ab177dfbf7e38c1b4a647705

# crictl images 
IMAGE                                            TAG                 IMAGE ID            SIZE
docker.io/library/httpd                          latest              a8ea074f4566a       148MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              c7569d8c767fa       396MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              7d1fc75e48f73       327MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              9cbefdfc7d582       331MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              7437858b89390       342MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              74c73c5ddbc38       340MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              22985b9e44282       460MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              853b9b86aa03a       389MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              4530c04b6ac9c       479MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              2dc68e509dd25       492MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              52c2eee1a5f9d       452MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              70b1fc1c4d760       403MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              1a1825758a271       739MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              a194d5e376fea       309MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              33b37ba471bb0       449MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              c5ddfa2a56e4e       553MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              bf7ebdd528f7d       610MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              04827d930943f       484MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              8445c376f8a75       335MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              c501396873225       370MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              4d80f3474c909       363MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              879a7d4d9e0a5       335MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              0a5c71b90fb10       402MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              c910e906cbc0d       413MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              5e9f6102f4162       399MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              2a23f5e6cad28       300MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              7085d32d6e79d       383MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              42b7926db6ba4       420MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>              a7c33ba4d99e8       297MB

manually delete entry matching image httpd in /var/lib/containers/storage/overlay-layers/layers.json

sh-4.4# crictl inspecti a8ea074f4566a
FATA[0000] image status for "a8ea074f4566a" request: rpc error: code = Unknown desc = layer not known 

sh-4.4# crictl pull httpd
Image is up to date for docker.io/library/httpd@sha256:5cc947a200524a822883dc6ce6456d852d7c5629ab177dfbf7e38c1b4a647705

sh-4.4# crictl inspecti a8ea074f4566a
{
  "status": {
    "id": "a8ea074f4566addcd01f9745397f32be471df4a4abf200f0f10c885ed14b1d28",
    "repoTags": [
      "docker.io/library/httpd:latest"
    ],
    "repoDigests": [
      "docker.io/library/httpd@sha256:5cc947a200524a822883dc6ce6456d852d7c5629ab177dfbf7e38c1b4a647705",
      "docker.io/library/httpd@sha256:7280c780e026f3073f2570cbf3cbe044dacd9bd90c7094f1c6e0e687444c9cec"
    ],
    "size": "147702595",
    "uid": null,
    "username": "",
    "spec": null
...

Comment 15 errata-xmlrpc 2022-02-16 06:51:40 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.31 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0484