Description of problem: Podman build is failing while using podman version 1.6.4. Issue seems to be fixed in upstream podman version 2.0.2. ~~~ [root@master-0 ~]# podman login registry.redhat.io --log-level=debug DEBU[0000] Credentials not found Username: rhn-support-arghosh Password: DEBU[0025] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io DEBU[0025] GET https://registry.redhat.io/v2/ DEBU[0025] Ping https://registry.redhat.io/v2/ status 401 DEBU[0025] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=rhn-support-arghosh&service=docker-registry DEBU[0027] GET https://registry.redhat.io/v2/ Login Succeeded! [root@master-0 ~]# [root@master-0 ~]# [root@master-0 ~]# podman login registry.redhat.io --log-level=debug DEBU[0000] Returning credentials from /run/user/0/containers/auth.json Authenticating with existing credentials... DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io DEBU[0000] GET https://registry.redhat.io/v2/ DEBU[0000] Ping https://registry.redhat.io/v2/ status 401 DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=rhn-support-arghosh&service=docker-registry DEBU[0001] GET https://registry.redhat.io/v2/ Existing credentials are valid. Already logged in to registry.redhat.io [root@master-0 ~]# podman build /home/quicklab/dockerfile STEP 1: FROM registry.redhat.io/rhscl/httpd-24-rhel7 AS base Error: error creating build container: Error initializing source docker://registry.redhat.io/rhscl/httpd-24-rhel7:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication [root@master-0 ~]# podman version Version: 1.6.4 RemoteAPI Version: 1 Go Version: go1.12.12 OS/Arch: linux/amd64 ~~~ Version-Release number of selected component (if applicable): Podman 1.6.4 How reproducible: Always Steps to Reproduce: 1. Login to registry using 'podman login' command 2. Try to build from a Dockerfile 3. Podman version should be 1.6.4. Could not reproduce in podman 2.0.2. Actual results: Container image build is failing with authentication error Expected results: podman version 1.6.4 should be able to build container image from Dockerfile Additional info: Workaround: ----------- 1. Use '--authfile' flag while building container image with 'podman build' command 2. Use '--creds' flag while building container image with 'podman build' command 3. Copy auth.json created after 'podman login' to '/run/containers/0' directory. #mkdir -p /run/containers/0 #sudo cp /run/user/0/containers/auth.json /run/containers/0/auth.json
We do not plan on any back ports to podman 1.6.4. Is this system happening on RHEL7?
Yes Daniel, we are experiencing the issue with podman version 1.6.4 on a RHEL 7 VM. Earlier I asked customer to upgrade to version 2.0.2 but later realized Red Hat does not ship this version yet.
Having them upgrade to RHEL8 would be my preferred solution. :^)
Why is this an openshift bug?
Yes it seems like it should be against Podman on RHEL. I'm gonig to make it so. If that's an issue, please let me know.
Maybe I've misunderstood something about this issue, but I'm not able to reproduce it on RHEL 7. Here's my test: # rpm -q podman podman-1.6.4-18.el7_8.x86_64 # ls -l /run/user/0/ total 0 # cat ~/.docker/config.json { "auths": {} } # cat bz1867495/build/Dockerfile FROM registry.redhat.io/ubi8-minimal CMD sleep infinity # podman build -t test bz1867495/build/ STEP 1: FROM registry.redhat.io/ubi8-minimal Error: error creating build container: Error initializing source docker://registry.redhat.io/ubi8-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication # podman --log-level=debug login registry.redhat.io DEBU[0000] Credentials not found Username: myuser Password: DEBU[0005] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io DEBU[0005] cert: /etc/docker/certs.d/registry.redhat.io/5792274547767415938.cert DEBU[0005] key: /etc/docker/certs.d/registry.redhat.io/5792274547767415938.key DEBU[0005] GET https://registry.redhat.io/v2/ DEBU[0005] Ping https://registry.redhat.io/v2/ status 401 DEBU[0005] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=myuser&service=docker-registry DEBU[0007] GET https://registry.redhat.io/v2/ Login Succeeded! # cat /run/user/0/containers/auth.json { "auths": { "registry.redhat.io": { "auth": "cmhuLXN1cMYtoken" } } } # podman --log-level=debug login registry.redhat.io DEBU[0000] Returning credentials from /run/user/0/containers/auth.json Authenticating with existing credentials... DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io DEBU[0000] cert: /etc/docker/certs.d/registry.redhat.io/5792274547767415938.cert DEBU[0000] key: /etc/docker/certs.d/registry.redhat.io/5792274547767415938.key DEBU[0000] GET https://registry.redhat.io/v2/ DEBU[0000] Ping https://registry.redhat.io/v2/ status 401 DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=myuser&service=docker-registry DEBU[0001] GET https://registry.redhat.io/v2/ Existing credentials are valid. Already logged in to registry.redhat.io # cat /run/user/0/containers/auth.json { "auths": { "registry.redhat.io": { "auth": "cmhuLXN1cMYtoken" } } } # podman build -t test bz1867495/build/ STEP 1: FROM registry.redhat.io/ubi8-minimal Getting image source signatures Copying blob f20f68829d13 done Copying blob 41ae95b593e0 done Copying config 86c8705965 done Writing manifest to image destination Storing signatures STEP 2: CMD sleep infinity STEP 3: COMMIT test cca8a00ca49987d8a2bafe9f72ea82a9b6384cdb43eb3a3319fa153c47e3089d Arnab, can you provide the exact package version of podman you used to reproduce this issue?
Hi Derrick, I am using below podman package. ~~~ sh-4.4# rpm -q podman podman-1.6.4-10.rhaos4.3.el8.x86_64 ~~~ I have tried to reproduce again and I could reproduce. ~~~ sh-4.4# cat ~/.docker/config.json cat: /root/.docker/config.json: No such file or directory sh-4.4# cat /run/user/0/containers/auth.json { "auths": {} } sh-4.4# cat /tmp/build/Dockerfile FROM registry.redhat.io/ubi8-minimal CMD sleep infinity sh-4.4# podman build /tmp/build/ STEP 1: FROM registry.redhat.io/ubi8-minimal Error: error creating build container: Error initializing source docker://registry.redhat.io/ubi8-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication sh-4.4# podman --log-level=debug login registry.redhat.io DEBU[0000] Credentials not found Username: rhn-support-arghosh Password: DEBU[0014] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io DEBU[0014] GET https://registry.redhat.io/v2/ DEBU[0015] Ping https://registry.redhat.io/v2/ status 401 DEBU[0015] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=rhn-support-arghosh&service=docker-registry DEBU[0016] GET https://registry.redhat.io/v2/ Login Succeeded! sh-4.4# cat /run/user/0/containers/auth.json { "auths": { "registry.redhat.io": { "auth": "cmhuLXN1cHBvcnQtYXJnaG9zaDpBZzk4NzQ3MzAyQA==" } } sh-4.4# podman build /tmp/build/ STEP 1: FROM registry.redhat.io/ubi8-minimal Error: error creating build container: Error initializing source docker://registry.redhat.io/ubi8-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication ~~~ Please let me know if you have any query. Regards Arnab
I'm not able to reproduce this with podman-1.6.4-15.module+el8.2.0+7290+954fb593 from the RHEL 8 container-tools:2.0 stream either. That should represent the most up-to-date podman 1.6.4 that RHEL 8 has. Seems like the OCP 4.3 podman 1.6.4 build needs to be updated in some way. Something to note is that podman-1.6.4-10.rhaos4.3.el8 would be the podman installed on RHCOS, and it's use is exclusively for bootstrapping the OCP cluster. I'm not currently aware of any scenario where we would support using podman directly to build images on RHCOS. RHEL 7 worker nodes can be used for that use-case. Having said that, it might be good to know what fixes are missing from the 4.3 build.
Verified on 4.3.0-0.nightly-2020-09-18-202815. Podman version is updated to 1.9.3 and I can now build without error. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.0-0.nightly-2020-09-18-202815 True False 3h5m Cluster version is 4.3.0-0.nightly-2020-09-18-202815 $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-136-61.us-east-2.compute.internal Ready worker 3h20m v1.16.2+417b9fd 10.0.136.61 <none> Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa) 4.18.0-193.23.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-137-138.us-east-2.compute.internal Ready master 3h29m v1.16.2+417b9fd 10.0.137.138 <none> Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa) 4.18.0-193.23.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-147-83.us-east-2.compute.internal Ready master 3h29m v1.16.2+417b9fd 10.0.147.83 <none> Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa) 4.18.0-193.23.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-152-110.us-east-2.compute.internal Ready worker 3h20m v1.16.2+417b9fd 10.0.152.110 <none> Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa) 4.18.0-193.23.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-165-98.us-east-2.compute.internal Ready master 3h29m v1.16.2+417b9fd 10.0.165.98 <none> Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa) 4.18.0-193.23.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 ip-10-0-166-109.us-east-2.compute.internal Ready worker 3h20m v1.16.2+417b9fd 10.0.166.109 <none> Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa) 4.18.0-193.23.1.el8_2.x86_64 cri-o://1.16.6-18.rhaos4.3.git538d861.el8 sh-4.4# podman version Version: 1.9.3 RemoteAPI Version: 1 Go Version: go1.13.4 OS/Arch: linux/amd64 sh-4.4# rpm -qa | grep -i podman podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64 sh-4.4# podman login registry.redhat.io Username: myuser Password: Login Succeeded! sh-4.4# podman build -t test /root/build/ STEP 1: FROM registry.redhat.io/ubi8-minimal Getting image source signatures Copying blob aebb8c556853 done Copying blob 0fd3b5213a9b done Copying config 28095021e5 done Writing manifest to image destination Storing signatures STEP 2: CMD sleep infinity STEP 3: COMMIT test --> 5ac3ed7ffc1 5ac3ed7ffc1a25b31b22119935228f7d734b535c4a3d1b1dc5e3f314f3bf7a9d
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.3.38 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3609