RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1732905 - Can't certify a container image on RHEL 8 using podman/buildah
Summary: Can't certify a container image on RHEL 8 using podman/buildah
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Brent Baude
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1731267
TreeView+ depends on / blocked
 
Reported: 2019-07-24 16:38 UTC by Josh Manning
Modified: 2019-07-24 18:19 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-24 18:19:59 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Josh Manning 2019-07-24 16:38:38 UTC 
Description of problem:
Podman/buildah changes image layer hashes when pushing to a registry. Suspect a different compression library being used, or something else causing a hash instability when compressing layers versus docker v1.13 as shipped in RHEL 7 and Fedora.

Version-Release number of selected component (if applicable):
Confirmed in podman 1.3.2 (RHEL 7) and also 1.4.4 (Fedora / RHEL 8)

How reproducible:
Every time.

Steps to Reproduce:
1. Build a container image based on RHEL or the UBI
2. Push the image to scan.connect.redhat.com (the scanning registry)
3. Check the scan result in Connect, or use skopeo to inspect the image layers in the registry once the image is pushed; the sha256 hashes of each layer are different than when using podman inspect or even docker inspect prior to pushing the image

Actual results:
The container image fails the `has_base_rh_image` check, which relies on known sha256 hashes of RHEL and UBI images (actually of the layers comprising a known RHEL/UBI image) in order to verify.

Expected results:
This should pass as expected. The layer hashes shouldn't differ from what is seen prior to pushing the image.

Additional info:
Comment 1 Matthew Heon 2019-07-24 16:51:37 UTC 
Likely to be c/image if it's seen with both Podman and Buildah
Comment 2 Miloslav Trmač 2019-07-24 18:19:59 UTC 
(In reply to Josh Manning from comment #0)
> Podman/buildah changes image layer hashes when pushing to a registry.
> Suspect a different compression library being used, or something else
> causing a hash instability when compressing layers versus docker v1.13 as
> shipped in RHEL 7 and Fedora.

Yes, that can easily happen. The bitstream created by compression when pushing images was never promised to be stable, and it isn’t (IIRC it changes even in point releases of the Go standard library - and the standard library does not even promise that the bitstream is deterministic at all!). Anything relying on the compressed bitstream being stable, or deterministic, needs to be changed.

Any pull+push (including pull base image + build a derived image + push the result) can, and eventually will, change the compressed bitstream.

( (skopeo copy) of a compressed image does not, unless explicitly directed otherwise, recompress and change the bitstream, but that’s not going to help for builds of images on top of RHEL/UBI.)

(There are some caches etc. that may, in specific situations, cause the original bitstream to be preserved, but that’s beside the point. The design that relies on a fixed bitstream is making incorrect assumptions; and it should not reject a valid image just because it was built without an obscure non-default option with an undocumented side effect.)


> The container image fails the `has_base_rh_image` check, which relies on
> known sha256 hashes of RHEL and UBI images (actually of the layers
> comprising a known RHEL/UBI image) in order to verify.

That check must either decompress the layers before comparing hashes, or, if the images use the schema2 or OCI formats (which was still fairly recently problematic for Quay, FWIW), extract the DiffID values from the container config (but, then, ideally, decompress the layers and verify that the layers match the DiffID values, anyway).
Note You need to log in before you can comment on or make changes to this bug.