Description of problem: This is very similar to bugzilla #1669096. When an image gets pulled through crictl it fails with the following error message: # sudo crictl pull repository.apps.company.org:5002/project/image-name:version FATA[0000] pulling image failed: rpc error: code = Unknown desc = Error: blob sha256:4c61cf98ab74562b92b3cd3f8fbf08cf770be36b35adec3dfde031708805200e is already present, but with size 75827357 instead of 74032627 sha256:4c61cf98ab74562b92b3cd3f8fbf08cf770be36b35adec3dfde031708805200e is an underlying layer of the image being pulled. The same image being pulled from a different node works perfectly. Cleansing the images on the node or using podman to pull the image solves the issue and crictl can pull the image afterwards. Comparing the manifests on the ok and nok nodes. - ids are the same: e10ae1eb4f450b747303a918e08c2fc37ad9965f5f14b364fff6b94b0c187374 - diff-digests are the same: sha256:e10ae1eb4f450b747303a918e08c2fc37ad9965f5f14b364fff6b94b0c187374 - diff-sizes are the same: 213780480 - compressed-diff-digest are different: sha256:4c61cf98ab74562b92b3cd3f8fbf08cf770be36b35adec3dfde031708805200e and sha256:c9281c141a1bfec06e291d2ad29bfdedfd10a99d583fc0f48d3c26723ebe0761 - creation times are different: 2020-05-29T03:40:22.213202061Z and 2020-07-06T09:24:05.728317106Z Although this could not be confirmed I believe that the layer was pushed multiple times into the registry. There is a single registry. If the fix for bugzilla #1669096 in crio version cri-o-1.12.5-6.rhaos4.0.git80d1487.el7.x86_64 addresses this issue as well I am requesting it to be ported back to 3.11 (I have not found any backport bugzilla for it). Version-Release number of selected component (if applicable): $ crio --version crio version 1.11.16-0.8.dev.rhaos3.11.git6d43aae.el7-dev $ podman version Version: 1.4.4 RemoteAPI Version: 1 Go Version: go1.10.3 OS/Arch: linux/amd64 How reproducible: Always for specific images (that I believe have been pushed multiple times) Steps to Reproduce: 1. crictl pulls <image-name>:<image-tag> Actual results: Error message shared above and no image pulled. Expected results: No error message and image pulled
Frank, no update as of yet, we're just getting to this now. I'll ask that Peter try to give you an update in a day or two via this BZ.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.306 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4170