Bug 1794167

Summary: buildah builds images that podman can't pull (unsupported docker v2s2 media type: "")
Product: Red Hat OpenStack Reporter: Alex Schultz <aschultz>
Component: ansible-role-tripleo-modify-imageAssignee: Alex Schultz <aschultz>
Status: CLOSED ERRATA QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: amcleod, aschultz, jhajyahy, mburns, rlandy, shrjoshi, slinaber, whayutin, wznoinsk
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Fixed In Version: ansible-role-tripleo-modify-image-1.1.1-0.20200122212319.58d7a5b.el8ost Doc Type: Known Issue
Doc Text:
There is a known issue when building container images with buildah. The default format for the image is OCI, but podman 1.6.x contains stricter restrictions about container format metadata. As a result, containers that you push to the undercloud registry can fail if they were originally in OCI format. The workaround is to use the `--format docker` option to build images in docker format instead of OCI format, and you can push the containers to the undercloud registry successfully.
Story Points: ---
Clone Of: 1793960 Environment:
Last Closed: 2020-03-05 12:01:54 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1793960    
Bug Blocks:    

Description Alex Schultz 2020-01-22 19:56:40 UTC
+++ This bug was initially created as a clone of Bug #1793960 +++

Description of problem:

running command:
tripleo container image prepare -e /home/stack/containers-prepare-parameter-copy.yaml --output-env-file /home/stack/prepare_output.yaml --log-file /var/log/tripleo-container-image-prepare.log --debug

which is running tripleo-modify-image ansible role undearneath to patch the image (using buildah bud and attached Dockerfile) and then tripleo-common (https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1703) to upload it to registry (+deletes it from local host)

(see attachment yamls and logs for details)

after that, our CI job is testing whether patched + pushed image is pull'able from that registry, with:
podman pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix

which fails with:

Trying to pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix...
  unsupported docker v2s2 media type: ""
Error: error pulling image "undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix": unable to pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix: unable to pull image: Error initializing image from source docker://undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix: unsupported docker v2s2 media type: ""

also, skopeo shows the same media type: "" issue:
[stack@undercloud-0 ~]$ skopeo inspect docker://undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix
FATA[0000] Error parsing manifest for image: unsupported docker v2s2 media type: ""

Version-Release number of selected component (if applicable):
OSP16, compose: RHOS_TRUNK-16.0-RHEL-8-20200113.n.0

[stack@undercloud-0 ~]$ yum list installed | grep -Ei "tripleo|buildah|podman"
ansible-role-tripleo-modify-image.noarch          1.1.1-0.20191030113421.66a92a4.el8ost                     @rhelosp-16.0                
ansible-tripleo-ipsec.noarch                      9.2.0-0.20191022054642.ffe104c.el8ost                     @rhelosp-16.0                
buildah.x86_64                                    1.11.6-4.module+el8.1.1+5259+bcdd613a                     @rhosp-rhel-8.1-appstream    
openstack-tripleo-common.noarch                   11.3.3-0.20200107225621.47626e1.el8ost                    @rhelosp-16.0                
openstack-tripleo-common-containers.noarch        11.3.3-0.20200107225621.47626e1.el8ost                    @rhelosp-16.0                
openstack-tripleo-heat-templates.noarch           11.3.2-0.20200109050651.8f93d27.el8ost                    @rhelosp-16.0                
openstack-tripleo-image-elements.noarch           10.6.1-0.20191022065313.7338463.el8ost                    @rhelosp-16.0                
openstack-tripleo-puppet-elements.noarch          11.2.1-0.20191108131052.2ad3189.el8ost                    @rhelosp-16.0                
openstack-tripleo-validations.noarch              11.3.1-0.20191126041901.2bba53a.el8ost                    @rhelosp-16.0                
podman.x86_64                                     1.6.4-2.module+el8.1.1+5363+bf8ff1af                      @rhosp-rhel-8.1-appstream    
podman-manpages.noarch                            1.6.4-2.module+el8.1.1+5363+bf8ff1af                      @rhosp-rhel-8.1-appstream    
puppet-tripleo.noarch                             11.4.1-0.20200106153547.5946c6f.el8ost                    @rhelosp-16.0                
python3-tripleo-common.noarch                     11.3.3-0.20200107225621.47626e1.el8ost                    @rhelosp-16.0                
python3-tripleoclient.noarch                      12.3.1-0.20191230195937.585fb28.el8ost                    @rhelosp-16.0                
python3-tripleoclient-heat-installer.noarch       12.3.1-0.20191230195937.585fb28.el8ost                    @rhelosp-16.0                
tripleo-ansible.noarch                            0.4.2-0.20200110023759.ee731ba.el8ost                     @rhelosp-16.0         

How reproducible:

Steps to Reproduce:
1. run the below command
tripleo container image prepare -e /home/stack/containers-prepare-parameter-copy.yaml --output-env-file /home/stack/prepare_output.yaml --log-file /var/log/tripleo-container-image-prepare.log --debug

(the yaml file attached)

2. observe the failure


Actual results:
pulling container image fails

Expected results:
pulling container image to succeed

Additional info:
ping me on irc (wznoinsk) for fastest reponse or if you need access to machine that has this problem at this moment

--- Additional comment from Alex Schultz on 2020-01-22 17:19:11 UTC ---

Currently this can be reproduced by running the following on an undercloud:


export CONT=$(sudo buildah from centos:7)
sudo buildah config --label maintainer="Alex Schultz <aschultz>" $CONT
sudo buildah run $CONT touch /foo
sudo buildah commit $CONT foo/foo:latest
sudo podman tag localhost/foo/foo undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo openstack tripleo container image push --local undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo podman rmi localhost/foo/foo
sudo podman rmi undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo podman pull undercloud.ctlplane.localdomain:8787/foo/foo:latest


export CONT=$(sudo buildah from centos:7)
sudo buildah config --label maintainer="Alex Schultz <aschultz>" $CONT
sudo buildah run $CONT touch /foo
sudo buildah commit -f docker $CONT foo/bar:latest
sudo podman tag localhost/foo/bar undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo openstack tripleo container image push --local undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo podman rmi localhost/foo/bar
sudo podman rmi undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo podman pull undercloud.ctlplane.localdomain:8787/foo/bar:latest

The resulting metadata in the registry looks like:


[root@undercloud foo]# cat foo/manifests/sha256\:238d95b4b77978d50ebd6abc924292a57b59b51b38934babd11da693f1120a43/index.json 
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:e5b1caa22fefda782c14e076be8ebf406474898a692c025beaea06b139e73b2c",
      "size": 1306
   "layers": [
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:e6a50b627bcb03d96996bb8e836ecb178eae7425636e3424d9e8d33a918768dd",
         "size": 75167623
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:23d0487be944221e1524d0fcf062e074632b93c341d0e21c3532d0bc1f098ee8",
         "size": 197


[root@undercloud foo]# cat bar/manifests/sha256\:770cd2a5a5e6f38813b31da95745c1aecf11d139ae05b7bae0747aba2f122790/index.json 
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 2529,
      "digest": "sha256:2352c67ba58bc3958b747f3cf6b2101a8f72b598a2f6b5e5ee2a79656b09a4c6"
   "layers": [
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 75167623,
         "digest": "sha256:e6a50b627bcb03d96996bb8e836ecb178eae7425636e3424d9e8d33a918768dd"
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 198,
         "digest": "sha256:b8c30d010e150b162e99da9818e6a1cc1024cfae3fd512193212909294475a4b"

It's likely because we don't currently support the oci format correctly. 

Additional details: 

Comment 3 Alex McLeod 2020-02-19 12:48:55 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.

Comment 5 errata-xmlrpc 2020-03-05 12:01:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.