Bug 1867463

Summary:	crictl pull fails due to different sizes of the image layers
Product:	OpenShift Container Platform	Reporter:	Frederic Giloux <fgiloux>
Component:	Node	Assignee:	Francesco Giudici <fgiudici>
Status:	CLOSED ERRATA	QA Contact:	Weinan Liu <weinliu>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.11.0	CC:	aos-bugs, dwalsh, fgiudici, fgrosjea, joboyer, jokerman, mitr, mpatel, mtleilia, openshift-bugs-escalate, pehunt, tmicheli, tsweeney
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	cri-o-1.11.16-0.13.rhaos3.11.git5218c73.el7	Doc Type:	Bug Fix
Doc Text:	When deleting images, the hash of the compressed layers that are cached locally were not deleted as they should have been. This may lead to a failure downloading the image again if another image sharing the same layer (but with a different compression) has been pulled. The compressed hashes are now cleared from the cache when deleting the corresponding layers which resolves the issue.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-22 11:02:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Frederic Giloux 2020-08-10 05:20:31 UTC

Description of problem:
This is very similar to bugzilla #1669096.
When an image gets pulled through crictl it fails with the following error message:
# sudo crictl pull repository.apps.company.org:5002/project/image-name:version
FATA[0000] pulling image failed: rpc error: code = Unknown desc = Error: blob sha256:4c61cf98ab74562b92b3cd3f8fbf08cf770be36b35adec3dfde031708805200e is already present, but with size 75827357 instead of 74032627

sha256:4c61cf98ab74562b92b3cd3f8fbf08cf770be36b35adec3dfde031708805200e is an underlying layer of the image being pulled.

The same image being pulled from a different node works perfectly. Cleansing the images on the node or using podman to pull the image solves the issue and crictl can pull the image afterwards.

Comparing the manifests on the ok and nok nodes.
- ids are the same: e10ae1eb4f450b747303a918e08c2fc37ad9965f5f14b364fff6b94b0c187374
- diff-digests are the same: sha256:e10ae1eb4f450b747303a918e08c2fc37ad9965f5f14b364fff6b94b0c187374
- diff-sizes are the same: 213780480
- compressed-diff-digest are different: sha256:4c61cf98ab74562b92b3cd3f8fbf08cf770be36b35adec3dfde031708805200e and sha256:c9281c141a1bfec06e291d2ad29bfdedfd10a99d583fc0f48d3c26723ebe0761
- creation times are different: 2020-05-29T03:40:22.213202061Z and 2020-07-06T09:24:05.728317106Z

Although this could not be confirmed I believe that the layer was pushed multiple times into the registry. There is a single registry.

If the fix for bugzilla #1669096  in crio version cri-o-1.12.5-6.rhaos4.0.git80d1487.el7.x86_64  addresses this issue as well I am requesting it to be ported back to 3.11 (I have not found any backport bugzilla for it).

Version-Release number of selected component (if applicable):
$ crio --version
crio version 1.11.16-0.8.dev.rhaos3.11.git6d43aae.el7-dev

$ podman version
Version:            1.4.4
RemoteAPI Version:  1
Go Version:         go1.10.3
OS/Arch:            linux/amd64

How reproducible:
Always for specific images (that I believe have been pushed multiple times)

Steps to Reproduce:
1. crictl pulls <image-name>:<image-tag>

Actual results:
Error message shared above and no image pulled.

Expected results:
No error message and image pulled

Comment 2 Tom Sweeney 2020-08-10 22:08:24 UTC

Frank, no update as of yet, we're just getting to this now.   I'll ask that Peter try to give you an update in a day or two via this BZ.

Comment 48 errata-xmlrpc 2020-10-22 11:02:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 3.11.306 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4170