Bug 2036202

Thanks for reporting your issue!

In order for the CoreOS team to be able to quickly and successfully triage your issue, please fill out the following template as completely as possible.

Be ready for follow-up questions and please respond in a timely manner.

If we can't reproduce a bug, we might close your issue.

---

OCP Version at Install Time: 
4.10.0-0.nightly-2021-12-21-130047

RHCOS Version at Install Time:
~~~
[root@ip-10-0-14-69 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux CoreOS release 4.10
~~~

Platform: 
AWS

Architecture: 
x86_64


What are you trying to do? What is your use case?

The containers-auth spec mentions a way to setup multiple credentials for a single registry which can be distinguished by their path:
https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md

That made it in relatively recently with commit https://github.com/containers/image/commit/2cafc0105c89b1c569f68733f7a5eaf6e326831b . That's in starting with container/image v5.14.0.

While podman works perfectly with such a setup on my laptop, and while the oc 4.10.0-0.nightly-2021-12-21-130047 binary works as well, the installation fails because the bootstrap's podman version is too old.

Here is an example of my credentials file which refers to quay.io, and 2 private repositories of my own, hosted on quay.io as well:
~~~
$ cat ~/.docker/config.json  | sed 's/"auth": ".*"/"auth":"xyz"/g'
{
  "auths": {
    "quay.io/akaris/origin-release": {
      "auth": "(...)==",
      "email": ""
    },
    "quay.io/akaris/machine-config-operator": {
      "auth": "(...)==",
      "email": ""
    },
    "registry.ci.openshift.org": {
      "auth":"xyz"
    },
    "cloud.openshift.com": {
      "auth":"xyz",
      "email": "akaris"
    },
    "quay.io": {
      "auth":"xyz",
      "email": "akaris"
    },
    "registry.connect.redhat.com": {
      "auth":"xyz",
      "email": "akaris"
    },
    "registry.redhat.io": {
      "auth":"xyz",
      "email": "akaris"
    }
  }
}
~~~


On my fedora box, podman works perfectly fine with that, and so does the latest 4.10 oc client. Crio should work, too, as the dependencies are met starting with v1.22.0:
* in podman, it's in starting with: 3.3.0 https://github.com/containers/podman/blob/v3.3.0/go.mod#L17
* in crio, it's in starting with: 1.22 https://github.com/cri-o/cri-o/blob/v1.22.0/go.mod#L19
* in oc, this is in with 4.10 latest and 4.9 latest: https://github.com/openshift/oc/blob/release-4.10/go.mod

Unfortunately, the podman version that we are using in RHCOS is too old and it doesn't match crio's behavior. Ironically, that makes the installation fail just because of podman, because whereas 4.10 ships the correct crio version:
~~~
[root@ip-10-0-14-69 ~]# rpm -qa | grep cri
cri-tools-1.22.0-1.el8.x86_64
initscripts-10.00.15-1.el8.x86_64
criu-3.15-1.module+el8.4.0+11822+6cc1e7d7.x86_64
cri-o-1.23.0-89.rhaos4.10.git367232b.el8.x86_64
crypto-policies-scripts-20210209-1.gitbfb6bed.el8_3.noarch
subscription-manager-rhsm-certificates-1.28.13-4.el8_4.x86_64
~~~

It still comes with an outdated podman 3.2.3:
~~~
[root@ip-10-0-14-69 ~]# rpm -qa | grep podman
podman-3.2.3-0.11.module+el8.4.0+12050+ef972f71.x86_64
podman-catatonit-3.2.3-0.11.module+el8.4.0+12050+ef972f71.x86_64
~~~

The bootstrap node pulls images with podman and fails:
~~~
[root@ip-10-0-14-69 ~]#  journalctl -b -u release-image.service -u bootkube.service | tail -n 2
Dec 30 11:22:13 ip-10-0-14-69 release-image-download.sh[1581]: Error: Error initializing source docker://quay.io/akaris/origin-release:v4.10-metric-50: Error reading manifest v4.10-metric-50 in quay.io/akaris/origin-release: unauthorized: access to the requested resource is not authorized
Dec 30 11:22:13 ip-10-0-14-69 release-image-download.sh[1581]: Pull failed. Retrying quay.io/akaris/origin-release:v4.10-metric-50...
[root@ip-10-0-14-69 ~]# grep 'Pull failed' -B10 /sysroot/ostree/deploy/rhcos/var/usrlocal/bin/release-image-download.sh
echo "Pulling $RELEASE_IMAGE..."
while true
do
    record_service_stage_start "pull-release-image"
    if podman pull --quiet "$RELEASE_IMAGE"
    then
        record_service_stage_success
        break
    else
        record_service_stage_failure
        echo "Pull failed. Retrying $RELEASE_IMAGE..."
~~~

What happened? What went wrong or what did you expect?

Podman should have feature parity with 4.10 crio and 4.10 oc. It should either be bumped to 3.3.0 or above, or we should ship a custom version that uses container/image v5.14.0 or above.