Bug 1794167

+++ This bug was initially created as a clone of Bug #1793960 +++

Description of problem:

running command:
tripleo container image prepare -e /home/stack/containers-prepare-parameter-copy.yaml --output-env-file /home/stack/prepare_output.yaml --log-file /var/log/tripleo-container-image-prepare.log --debug


which is running tripleo-modify-image ansible role undearneath to patch the image (using buildah bud and attached Dockerfile) and then tripleo-common (https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1703) to upload it to registry (+deletes it from local host)

(see attachment yamls and logs for details)

after that, our CI job is testing whether patched + pushed image is pull'able from that registry, with:
podman pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix

which fails with:
STDERR:

Trying to pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix...
  unsupported docker v2s2 media type: ""
Error: error pulling image "undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix": unable to pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix: unable to pull image: Error initializing image from source docker://undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix: unsupported docker v2s2 media type: ""


also, skopeo shows the same media type: "" issue:
[stack@undercloud-0 ~]$ skopeo inspect docker://undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1-hotfix
FATA[0000] Error parsing manifest for image: unsupported docker v2s2 media type: ""


Version-Release number of selected component (if applicable):
OSP16, compose: RHOS_TRUNK-16.0-RHEL-8-20200113.n.0

[stack@undercloud-0 ~]$ yum list installed | grep -Ei "tripleo|buildah|podman"
ansible-role-tripleo-modify-image.noarch          1.1.1-0.20191030113421.66a92a4.el8ost                     @rhelosp-16.0                
ansible-tripleo-ipsec.noarch                      9.2.0-0.20191022054642.ffe104c.el8ost                     @rhelosp-16.0                
buildah.x86_64                                    1.11.6-4.module+el8.1.1+5259+bcdd613a                     @rhosp-rhel-8.1-appstream    
openstack-tripleo-common.noarch                   11.3.3-0.20200107225621.47626e1.el8ost                    @rhelosp-16.0                
openstack-tripleo-common-containers.noarch        11.3.3-0.20200107225621.47626e1.el8ost                    @rhelosp-16.0                
openstack-tripleo-heat-templates.noarch           11.3.2-0.20200109050651.8f93d27.el8ost                    @rhelosp-16.0                
openstack-tripleo-image-elements.noarch           10.6.1-0.20191022065313.7338463.el8ost                    @rhelosp-16.0                
openstack-tripleo-puppet-elements.noarch          11.2.1-0.20191108131052.2ad3189.el8ost                    @rhelosp-16.0                
openstack-tripleo-validations.noarch              11.3.1-0.20191126041901.2bba53a.el8ost                    @rhelosp-16.0                
podman.x86_64                                     1.6.4-2.module+el8.1.1+5363+bf8ff1af                      @rhosp-rhel-8.1-appstream    
podman-manpages.noarch                            1.6.4-2.module+el8.1.1+5363+bf8ff1af                      @rhosp-rhel-8.1-appstream    
puppet-tripleo.noarch                             11.4.1-0.20200106153547.5946c6f.el8ost                    @rhelosp-16.0                
python3-tripleo-common.noarch                     11.3.3-0.20200107225621.47626e1.el8ost                    @rhelosp-16.0                
python3-tripleoclient.noarch                      12.3.1-0.20191230195937.585fb28.el8ost                    @rhelosp-16.0                
python3-tripleoclient-heat-installer.noarch       12.3.1-0.20191230195937.585fb28.el8ost                    @rhelosp-16.0                
tripleo-ansible.noarch                            0.4.2-0.20200110023759.ee731ba.el8ost                     @rhelosp-16.0         

How reproducible:
100%


Steps to Reproduce:
1. run the below command
tripleo container image prepare -e /home/stack/containers-prepare-parameter-copy.yaml --output-env-file /home/stack/prepare_output.yaml --log-file /var/log/tripleo-container-image-prepare.log --debug

(the yaml file attached)

2. observe the failure

3.

Actual results:
pulling container image fails


Expected results:
pulling container image to succeed


Additional info:
ping me on irc (wznoinsk) for fastest reponse or if you need access to machine that has this problem at this moment

--- Additional comment from Alex Schultz on 2020-01-22 17:19:11 UTC ---

Currently this can be reproduced by running the following on an undercloud:

Fails:

export CONT=$(sudo buildah from centos:7)
sudo buildah config --label maintainer="Alex Schultz <aschultz>" $CONT
sudo buildah run $CONT touch /foo
sudo buildah commit $CONT foo/foo:latest
sudo podman tag localhost/foo/foo undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo openstack tripleo container image push --local undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo podman rmi localhost/foo/foo
sudo podman rmi undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo podman pull undercloud.ctlplane.localdomain:8787/foo/foo:latest

Works:

export CONT=$(sudo buildah from centos:7)
sudo buildah config --label maintainer="Alex Schultz <aschultz>" $CONT
sudo buildah run $CONT touch /foo
sudo buildah commit -f docker $CONT foo/bar:latest
sudo podman tag localhost/foo/bar undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo openstack tripleo container image push --local undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo podman rmi localhost/foo/bar
sudo podman rmi undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo podman pull undercloud.ctlplane.localdomain:8787/foo/bar:latest



The resulting metadata in the registry looks like:

Fails:

[root@undercloud foo]# cat foo/manifests/sha256\:238d95b4b77978d50ebd6abc924292a57b59b51b38934babd11da693f1120a43/index.json 
{
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:e5b1caa22fefda782c14e076be8ebf406474898a692c025beaea06b139e73b2c",
      "size": 1306
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:e6a50b627bcb03d96996bb8e836ecb178eae7425636e3424d9e8d33a918768dd",
         "size": 75167623
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:23d0487be944221e1524d0fcf062e074632b93c341d0e21c3532d0bc1f098ee8",
         "size": 197
      }
   ]
}


Works:

[root@undercloud foo]# cat bar/manifests/sha256\:770cd2a5a5e6f38813b31da95745c1aecf11d139ae05b7bae0747aba2f122790/index.json 
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 2529,
      "digest": "sha256:2352c67ba58bc3958b747f3cf6b2101a8f72b598a2f6b5e5ee2a79656b09a4c6"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 75167623,
         "digest": "sha256:e6a50b627bcb03d96996bb8e836ecb178eae7425636e3424d9e8d33a918768dd"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 198,
         "digest": "sha256:b8c30d010e150b162e99da9818e6a1cc1024cfae3fd512193212909294475a4b"
      }
   ]
}


It's likely because we don't currently support the oci format correctly. 

Additional details: 
https://github.com/cri-o/cri-o/issues/2905
https://issues.sonatype.org/browse/NEXUS-16947