Description of problem:
Podman/buildah changes image layer hashes when pushing to a registry. Suspect a different compression library being used, or something else causing a hash instability when compressing layers versus docker v1.13 as shipped in RHEL 7 and Fedora.
Version-Release number of selected component (if applicable):
Confirmed in podman 1.3.2 (RHEL 7) and also 1.4.4 (Fedora / RHEL 8)
Steps to Reproduce:
1. Build a container image based on RHEL or the UBI
2. Push the image to scan.connect.redhat.com (the scanning registry)
3. Check the scan result in Connect, or use skopeo to inspect the image layers in the registry once the image is pushed; the sha256 hashes of each layer are different than when using podman inspect or even docker inspect prior to pushing the image
The container image fails the `has_base_rh_image` check, which relies on known sha256 hashes of RHEL and UBI images (actually of the layers comprising a known RHEL/UBI image) in order to verify.
This should pass as expected. The layer hashes shouldn't differ from what is seen prior to pushing the image.
Likely to be c/image if it's seen with both Podman and Buildah
(In reply to Josh Manning from comment #0)
> Podman/buildah changes image layer hashes when pushing to a registry.
> Suspect a different compression library being used, or something else
> causing a hash instability when compressing layers versus docker v1.13 as
> shipped in RHEL 7 and Fedora.
Yes, that can easily happen. The bitstream created by compression when pushing images was never promised to be stable, and it isn’t (IIRC it changes even in point releases of the Go standard library - and the standard library does not even promise that the bitstream is deterministic at all!). Anything relying on the compressed bitstream being stable, or deterministic, needs to be changed.
Any pull+push (including pull base image + build a derived image + push the result) can, and eventually will, change the compressed bitstream.
( (skopeo copy) of a compressed image does not, unless explicitly directed otherwise, recompress and change the bitstream, but that’s not going to help for builds of images on top of RHEL/UBI.)
(There are some caches etc. that may, in specific situations, cause the original bitstream to be preserved, but that’s beside the point. The design that relies on a fixed bitstream is making incorrect assumptions; and it should not reject a valid image just because it was built without an obscure non-default option with an undocumented side effect.)
> The container image fails the `has_base_rh_image` check, which relies on
> known sha256 hashes of RHEL and UBI images (actually of the layers
> comprising a known RHEL/UBI image) in order to verify.
That check must either decompress the layers before comparing hashes, or, if the images use the schema2 or OCI formats (which was still fairly recently problematic for Quay, FWIW), extract the DiffID values from the container config (but, then, ideally, decompress the layers and verify that the layers match the DiffID values, anyway).