Bug 1732905 - Can't certify a container image on RHEL 8 using podman/buildah
Summary: Can't certify a container image on RHEL 8 using podman/buildah
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Brent Baude
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1731267
TreeView+ depends on / blocked
 
Reported: 2019-07-24 16:38 UTC by Josh Manning
Modified: 2019-07-24 18:19 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-24 18:19:59 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Josh Manning 2019-07-24 16:38:38 UTC
Description of problem:
Podman/buildah changes image layer hashes when pushing to a registry. Suspect a different compression library being used, or something else causing a hash instability when compressing layers versus docker v1.13 as shipped in RHEL 7 and Fedora.

Version-Release number of selected component (if applicable):
Confirmed in podman 1.3.2 (RHEL 7) and also 1.4.4 (Fedora / RHEL 8)

How reproducible:
Every time.

Steps to Reproduce:
1. Build a container image based on RHEL or the UBI
2. Push the image to scan.connect.redhat.com (the scanning registry)
3. Check the scan result in Connect, or use skopeo to inspect the image layers in the registry once the image is pushed; the sha256 hashes of each layer are different than when using podman inspect or even docker inspect prior to pushing the image

Actual results:
The container image fails the `has_base_rh_image` check, which relies on known sha256 hashes of RHEL and UBI images (actually of the layers comprising a known RHEL/UBI image) in order to verify.

Expected results:
This should pass as expected. The layer hashes shouldn't differ from what is seen prior to pushing the image.

Additional info:

Comment 1 Matthew Heon 2019-07-24 16:51:37 UTC
Likely to be c/image if it's seen with both Podman and Buildah

Comment 2 Miloslav Trmač 2019-07-24 18:19:59 UTC
(In reply to Josh Manning from comment #0)
> Podman/buildah changes image layer hashes when pushing to a registry.
> Suspect a different compression library being used, or something else
> causing a hash instability when compressing layers versus docker v1.13 as
> shipped in RHEL 7 and Fedora.

Yes, that can easily happen. The bitstream created by compression when pushing images was never promised to be stable, and it isn’t (IIRC it changes even in point releases of the Go standard library - and the standard library does not even promise that the bitstream is deterministic at all!). Anything relying on the compressed bitstream being stable, or deterministic, needs to be changed.

Any pull+push (including pull base image + build a derived image + push the result) can, and eventually will, change the compressed bitstream.

( (skopeo copy) of a compressed image does not, unless explicitly directed otherwise, recompress and change the bitstream, but that’s not going to help for builds of images on top of RHEL/UBI.)

(There are some caches etc. that may, in specific situations, cause the original bitstream to be preserved, but that’s beside the point. The design that relies on a fixed bitstream is making incorrect assumptions; and it should not reject a valid image just because it was built without an obscure non-default option with an undocumented side effect.)


> The container image fails the `has_base_rh_image` check, which relies on
> known sha256 hashes of RHEL and UBI images (actually of the layers
> comprising a known RHEL/UBI image) in order to verify.

That check must either decompress the layers before comparing hashes, or, if the images use the schema2 or OCI formats (which was still fairly recently problematic for Quay, FWIW), extract the DiffID values from the container config (but, then, ideally, decompress the layers and verify that the layers match the DiffID values, anyway).


Note You need to log in before you can comment on or make changes to this bug.