the same issue seen on RHOS_TRUNK-16.0-RHEL-8-20200117.n.1 compose

Created attachment 1654525 [details]
containers-prepare-parameter-copy.yaml

Created attachment 1654526 [details]
tripleo-container-image-prepare.log

Created attachment 1654527 [details]
prepare_output.yaml

We see the same issue with standalone builds in donwsteam sf.
The failure started on 01/17 ... we tracked it to the change in podman versions ...
 podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64
vs
podman-1.4.2-6.module+el8.1.0+4830+f49150d7.x86_64
 is the passing tests.

https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=tripleo-ci-rhel-8-standalone-rhos-16 shows when the failure began.


https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/tripleo-ci-rhel-8-standalone-rhos-16/f3a7301/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2020-01-22_00_54_37

Perhaps this is the root cause.. https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo-image-serve/templates/image-serve.conf.j2#L24-L30
It's setting Docker-Context-Digest

Perhaps this is the root cause.. https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo-image-serve/templates/image-serve.conf.j2#L24-L30
It's setting Docker-Context-Digest

Currently this can be reproduced by running the following on an undercloud:

Fails:

export CONT=$(sudo buildah from centos:7)
sudo buildah config --label maintainer="Alex Schultz <aschultz>" $CONT
sudo buildah run $CONT touch /foo
sudo buildah commit $CONT foo/foo:latest
sudo podman tag localhost/foo/foo undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo openstack tripleo container image push --local undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo podman rmi localhost/foo/foo
sudo podman rmi undercloud.ctlplane.localdomain:8787/foo/foo:latest
sudo podman pull undercloud.ctlplane.localdomain:8787/foo/foo:latest

Works:

export CONT=$(sudo buildah from centos:7)
sudo buildah config --label maintainer="Alex Schultz <aschultz>" $CONT
sudo buildah run $CONT touch /foo
sudo buildah commit -f docker $CONT foo/bar:latest
sudo podman tag localhost/foo/bar undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo openstack tripleo container image push --local undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo podman rmi localhost/foo/bar
sudo podman rmi undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo podman pull undercloud.ctlplane.localdomain:8787/foo/bar:latest



The resulting metadata in the registry looks like:

Fails:

[root@undercloud foo]# cat foo/manifests/sha256\:238d95b4b77978d50ebd6abc924292a57b59b51b38934babd11da693f1120a43/index.json 
{
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:e5b1caa22fefda782c14e076be8ebf406474898a692c025beaea06b139e73b2c",
      "size": 1306
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:e6a50b627bcb03d96996bb8e836ecb178eae7425636e3424d9e8d33a918768dd",
         "size": 75167623
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:23d0487be944221e1524d0fcf062e074632b93c341d0e21c3532d0bc1f098ee8",
         "size": 197
      }
   ]
}


Works:

[root@undercloud foo]# cat bar/manifests/sha256\:770cd2a5a5e6f38813b31da95745c1aecf11d139ae05b7bae0747aba2f122790/index.json 
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 2529,
      "digest": "sha256:2352c67ba58bc3958b747f3cf6b2101a8f72b598a2f6b5e5ee2a79656b09a4c6"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 75167623,
         "digest": "sha256:e6a50b627bcb03d96996bb8e836ecb178eae7425636e3424d9e8d33a918768dd"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 198,
         "digest": "sha256:b8c30d010e150b162e99da9818e6a1cc1024cfae3fd512193212909294475a4b"
      }
   ]
}


It's likely because we don't currently support the oci format correctly. 

Additional details: 
https://github.com/cri-o/cri-o/issues/2905
https://issues.sonatype.org/browse/NEXUS-16947

FYI: as a (early) test I've applied  https://review.opendev.org/#/c/703846/1 on top of a downstream OSP 16 CI undercloud:

[root@undercloud-0 tripleo-modify-image]# patch -p1 < 1e10b22.diff
patching file tasks/yum_install_buildah.yml
patching file tasks/yum_update_buildah.yml
Hunk #1 succeeded at 137 (offset -5 lines).
patching file vars/main.yml

which was failing before and now it is passing the 'podman pull' succesfully

good work, thanks!

*** Bug 1794305 has been marked as a duplicate of this bug. ***

Ran job: 
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/deployment/job/DFG-df-deployment-16-virthost-1cont_1comp_3ceph-ceph-ipv4-geneve-tripleo-modify-image-custom_Dockerfile-RHELOSP-41501/10/


Also ran:
export CONT=$(sudo buildah from centos:7)
sudo buildah config --label maintainer="Alex Schultz <aschultz>" $CONT
sudo buildah run $CONT touch /foo
sudo buildah commit -f docker $CONT foo/bar:latest
sudo podman tag localhost/foo/bar undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo openstack tripleo container image push --local undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo podman rmi localhost/foo/bar
sudo podman rmi undercloud.ctlplane.localdomain:8787/foo/bar:latest
sudo podman pull undercloud.ctlplane.localdomain:8787/foo/bar:latest

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283