Bug 1732905

Summary: Can't certify a container image on RHEL 8 using podman/buildah
Product: Red Hat Enterprise Linux 8 Reporter: Josh Manning <jmanning>
Component: podmanAssignee: Brent Baude <bbaude>
Status: CLOSED NOTABUG QA Contact: atomic-bugs <atomic-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.4CC: dwalsh, jligon, jnovy, lsm5, mheon, mitr
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-24 18:19:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1731267    

Description Josh Manning 2019-07-24 16:38:38 UTC
Description of problem:
Podman/buildah changes image layer hashes when pushing to a registry. Suspect a different compression library being used, or something else causing a hash instability when compressing layers versus docker v1.13 as shipped in RHEL 7 and Fedora.

Version-Release number of selected component (if applicable):
Confirmed in podman 1.3.2 (RHEL 7) and also 1.4.4 (Fedora / RHEL 8)

How reproducible:
Every time.

Steps to Reproduce:
1. Build a container image based on RHEL or the UBI
2. Push the image to scan.connect.redhat.com (the scanning registry)
3. Check the scan result in Connect, or use skopeo to inspect the image layers in the registry once the image is pushed; the sha256 hashes of each layer are different than when using podman inspect or even docker inspect prior to pushing the image

Actual results:
The container image fails the `has_base_rh_image` check, which relies on known sha256 hashes of RHEL and UBI images (actually of the layers comprising a known RHEL/UBI image) in order to verify.

Expected results:
This should pass as expected. The layer hashes shouldn't differ from what is seen prior to pushing the image.

Additional info:

Comment 1 Matthew Heon 2019-07-24 16:51:37 UTC
Likely to be c/image if it's seen with both Podman and Buildah

Comment 2 Miloslav Trmač 2019-07-24 18:19:59 UTC
(In reply to Josh Manning from comment #0)
> Podman/buildah changes image layer hashes when pushing to a registry.
> Suspect a different compression library being used, or something else
> causing a hash instability when compressing layers versus docker v1.13 as
> shipped in RHEL 7 and Fedora.

Yes, that can easily happen. The bitstream created by compression when pushing images was never promised to be stable, and it isn’t (IIRC it changes even in point releases of the Go standard library - and the standard library does not even promise that the bitstream is deterministic at all!). Anything relying on the compressed bitstream being stable, or deterministic, needs to be changed.

Any pull+push (including pull base image + build a derived image + push the result) can, and eventually will, change the compressed bitstream.

( (skopeo copy) of a compressed image does not, unless explicitly directed otherwise, recompress and change the bitstream, but that’s not going to help for builds of images on top of RHEL/UBI.)

(There are some caches etc. that may, in specific situations, cause the original bitstream to be preserved, but that’s beside the point. The design that relies on a fixed bitstream is making incorrect assumptions; and it should not reject a valid image just because it was built without an obscure non-default option with an undocumented side effect.)


> The container image fails the `has_base_rh_image` check, which relies on
> known sha256 hashes of RHEL and UBI images (actually of the layers
> comprising a known RHEL/UBI image) in order to verify.

That check must either decompress the layers before comparing hashes, or, if the images use the schema2 or OCI formats (which was still fairly recently problematic for Quay, FWIW), extract the DiffID values from the container config (but, then, ideally, decompress the layers and verify that the layers match the DiffID values, anyway).