Bug 1867495 - Container image build is failing with podman version 1.6.4 due to authentication issue
Summary: Container image build is failing with podman version 1.6.4 due to authenticat...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.3.z
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: 4.3.z
Assignee: Jindrich Novy
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks: 1186913
TreeView+ depends on / blocked
 
Reported: 2020-08-10 07:40 UTC by Arnab Ghosh
Modified: 2020-10-01 09:26 UTC (History)
13 users (show)

Fixed In Version: podman-1.6.4-11.rhaos4.3.el8
Doc Type: If docs needed, set a value
Doc Text:
When trying to pull images from container image registries such as registry.redhat.io, an authentication issue would occur when using Podman v1.6.4: ``` STEP 1: FROM registry.redhat.io/rhscl/httpd-24-rhel7 AS base Error: error creating build container: Error initializing source docker://registry.redhat.io/rhscl/httpd-24-rhel7:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication ``` The IdentiyTokenKey in Oauth2 authorizations was not being handled appropriately. This has been fixed with this update.
Clone Of:
Environment:
Last Closed: 2020-09-23 13:52:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:3609 0 None None None 2020-09-23 13:53:02 UTC

Description Arnab Ghosh 2020-08-10 07:40:53 UTC
Description of problem:
Podman build is failing while using podman version 1.6.4. Issue seems to be fixed in upstream podman version 2.0.2.

~~~
[root@master-0 ~]# podman login registry.redhat.io --log-level=debug
DEBU[0000] Credentials not found                        
Username: rhn-support-arghosh
Password: 
DEBU[0025] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io 
DEBU[0025] GET https://registry.redhat.io/v2/           
DEBU[0025] Ping https://registry.redhat.io/v2/ status 401 
DEBU[0025] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=rhn-support-arghosh&service=docker-registry 
DEBU[0027] GET https://registry.redhat.io/v2/           
Login Succeeded!
[root@master-0 ~]# 
[root@master-0 ~]# 
[root@master-0 ~]# podman login registry.redhat.io --log-level=debug
DEBU[0000] Returning credentials from /run/user/0/containers/auth.json 
Authenticating with existing credentials...
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io 
DEBU[0000] GET https://registry.redhat.io/v2/           
DEBU[0000] Ping https://registry.redhat.io/v2/ status 401 
DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=rhn-support-arghosh&service=docker-registry 
DEBU[0001] GET https://registry.redhat.io/v2/           
Existing credentials are valid. Already logged in to registry.redhat.io

[root@master-0 ~]# podman build /home/quicklab/dockerfile
STEP 1: FROM registry.redhat.io/rhscl/httpd-24-rhel7 AS base
Error: error creating build container: Error initializing source docker://registry.redhat.io/rhscl/httpd-24-rhel7:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication

[root@master-0 ~]# podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64
~~~

Version-Release number of selected component (if applicable):
Podman 1.6.4

How reproducible:
Always

Steps to Reproduce:
1. Login to registry using 'podman login' command
2. Try to build from a Dockerfile
3. Podman version should be 1.6.4. Could not reproduce in podman 2.0.2.

Actual results:
Container image build is failing with authentication error

Expected results:
podman version 1.6.4 should be able to build container image from Dockerfile

Additional info:

Workaround:
-----------

1. Use '--authfile' flag while building container image with 'podman build' command
2. Use '--creds' flag while building container image with 'podman build' command
3. Copy auth.json created after 'podman login' to '/run/containers/0' directory.
  #mkdir -p /run/containers/0
  #sudo cp /run/user/0/containers/auth.json /run/containers/0/auth.json

Comment 1 Daniel Walsh 2020-08-10 13:38:47 UTC
We do not plan on any back ports to podman 1.6.4.  Is this system happening on RHEL7?

Comment 2 Arnab Ghosh 2020-08-11 01:01:46 UTC
Yes Daniel, we are experiencing the issue with podman version 1.6.4 on a RHEL 7 VM. Earlier I asked customer to upgrade to version 2.0.2 but later realized Red Hat does not ship this version yet.

Comment 3 Daniel Walsh 2020-08-11 10:39:49 UTC
Having them upgrade to RHEL8 would be my preferred solution. :^)

Comment 4 Daniel Walsh 2020-08-11 10:40:29 UTC
Why is this an openshift bug?

Comment 5 Tom Sweeney 2020-08-11 18:02:33 UTC
Yes it seems like it should be against Podman on RHEL.  I'm gonig to make it so.  If that's an issue, please let me know.

Comment 10 Derrick Ornelas 2020-08-18 22:32:33 UTC
Maybe I've misunderstood something about this issue, but I'm not able to reproduce it on RHEL 7.  Here's my test:

# rpm -q podman
podman-1.6.4-18.el7_8.x86_64


# ls -l /run/user/0/
total 0


# cat ~/.docker/config.json 
{
	"auths": {}
}


# cat bz1867495/build/Dockerfile 
FROM registry.redhat.io/ubi8-minimal
CMD sleep infinity


# podman build -t test bz1867495/build/
STEP 1: FROM registry.redhat.io/ubi8-minimal
Error: error creating build container: Error initializing source docker://registry.redhat.io/ubi8-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication


# podman --log-level=debug login registry.redhat.io
DEBU[0000] Credentials not found                        
Username: myuser
Password: 
DEBU[0005] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io 
DEBU[0005]  cert: /etc/docker/certs.d/registry.redhat.io/5792274547767415938.cert 
DEBU[0005]  key: /etc/docker/certs.d/registry.redhat.io/5792274547767415938.key 
DEBU[0005] GET https://registry.redhat.io/v2/           
DEBU[0005] Ping https://registry.redhat.io/v2/ status 401 
DEBU[0005] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=myuser&service=docker-registry 
DEBU[0007] GET https://registry.redhat.io/v2/           
Login Succeeded!


# cat /run/user/0/containers/auth.json 
{
	"auths": {
		"registry.redhat.io": {
			"auth": "cmhuLXN1cMYtoken"
		}
	}
}



# podman --log-level=debug login registry.redhat.io
DEBU[0000] Returning credentials from /run/user/0/containers/auth.json 
Authenticating with existing credentials...
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io 
DEBU[0000]  cert: /etc/docker/certs.d/registry.redhat.io/5792274547767415938.cert 
DEBU[0000]  key: /etc/docker/certs.d/registry.redhat.io/5792274547767415938.key 
DEBU[0000] GET https://registry.redhat.io/v2/           
DEBU[0000] Ping https://registry.redhat.io/v2/ status 401 
DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=myuser&service=docker-registry 
DEBU[0001] GET https://registry.redhat.io/v2/           
Existing credentials are valid. Already logged in to registry.redhat.io


# cat /run/user/0/containers/auth.json 
{
	"auths": {
		"registry.redhat.io": {
			"auth": "cmhuLXN1cMYtoken"
		}
	}
}


# podman build -t test bz1867495/build/
STEP 1: FROM registry.redhat.io/ubi8-minimal
Getting image source signatures
Copying blob f20f68829d13 done  
Copying blob 41ae95b593e0 done  
Copying config 86c8705965 done  
Writing manifest to image destination
Storing signatures
STEP 2: CMD sleep infinity
STEP 3: COMMIT test
cca8a00ca49987d8a2bafe9f72ea82a9b6384cdb43eb3a3319fa153c47e3089d



Arnab, can you provide the exact package version of podman you used to reproduce this issue?

Comment 11 Arnab Ghosh 2020-08-19 02:54:07 UTC
Hi Derrick,

I am using below podman package.

~~~
sh-4.4# rpm -q podman
podman-1.6.4-10.rhaos4.3.el8.x86_64
~~~

I have tried to reproduce again and I could reproduce.

~~~
sh-4.4# cat ~/.docker/config.json 
cat: /root/.docker/config.json: No such file or directory

sh-4.4# cat /run/user/0/containers/auth.json 
{
	"auths": {}
}

sh-4.4# cat /tmp/build/Dockerfile 
FROM registry.redhat.io/ubi8-minimal
CMD sleep infinity

sh-4.4# podman build /tmp/build/                            
STEP 1: FROM registry.redhat.io/ubi8-minimal
Error: error creating build container: Error initializing source docker://registry.redhat.io/ubi8-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication

sh-4.4# podman --log-level=debug login registry.redhat.io
DEBU[0000] Credentials not found                        
Username: rhn-support-arghosh
Password: 
DEBU[0014] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io 
DEBU[0014] GET https://registry.redhat.io/v2/           
DEBU[0015] Ping https://registry.redhat.io/v2/ status 401 
DEBU[0015] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=rhn-support-arghosh&service=docker-registry 
DEBU[0016] GET https://registry.redhat.io/v2/           
Login Succeeded!

sh-4.4# cat /run/user/0/containers/auth.json
{
	"auths": {
		"registry.redhat.io": {
			"auth": "cmhuLXN1cHBvcnQtYXJnaG9zaDpBZzk4NzQ3MzAyQA=="
		}
	}

sh-4.4# podman build /tmp/build/                            
STEP 1: FROM registry.redhat.io/ubi8-minimal
Error: error creating build container: Error initializing source docker://registry.redhat.io/ubi8-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
~~~

Please let me know if you have any query.

Regards
Arnab

Comment 12 Derrick Ornelas 2020-08-19 15:01:05 UTC
I'm not able to reproduce this with podman-1.6.4-15.module+el8.2.0+7290+954fb593 from the RHEL 8 container-tools:2.0 stream either.  That should represent the most up-to-date podman 1.6.4 that RHEL 8 has.  Seems like the OCP 4.3 podman 1.6.4 build needs to be updated in some way.  


Something to note is that podman-1.6.4-10.rhaos4.3.el8 would be the podman installed on RHCOS, and it's use is exclusively for bootstrapping the OCP cluster.  I'm not currently aware of any scenario where we would support using podman directly to build images on RHCOS.  RHEL 7 worker nodes can be used for that use-case.  Having said that, it might be good to know what fixes are missing from the 4.3 build.

Comment 22 Sunil Choudhary 2020-09-21 14:17:16 UTC
Verified on 4.3.0-0.nightly-2020-09-18-202815. Podman version is updated to 1.9.3 and I can now build without error.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2020-09-18-202815   True        False         3h5m    Cluster version is 4.3.0-0.nightly-2020-09-18-202815

$ oc get nodes -o wide
NAME                                         STATUS   ROLES    AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-136-61.us-east-2.compute.internal    Ready    worker   3h20m   v1.16.2+417b9fd   10.0.136.61    <none>        Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa)   4.18.0-193.23.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-137-138.us-east-2.compute.internal   Ready    master   3h29m   v1.16.2+417b9fd   10.0.137.138   <none>        Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa)   4.18.0-193.23.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-147-83.us-east-2.compute.internal    Ready    master   3h29m   v1.16.2+417b9fd   10.0.147.83    <none>        Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa)   4.18.0-193.23.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-152-110.us-east-2.compute.internal   Ready    worker   3h20m   v1.16.2+417b9fd   10.0.152.110   <none>        Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa)   4.18.0-193.23.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-165-98.us-east-2.compute.internal    Ready    master   3h29m   v1.16.2+417b9fd   10.0.165.98    <none>        Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa)   4.18.0-193.23.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8
ip-10-0-166-109.us-east-2.compute.internal   Ready    worker   3h20m   v1.16.2+417b9fd   10.0.166.109   <none>        Red Hat Enterprise Linux CoreOS 43.82.202009181853.0 (Ootpa)   4.18.0-193.23.1.el8_2.x86_64   cri-o://1.16.6-18.rhaos4.3.git538d861.el8


sh-4.4# podman version
Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64

sh-4.4# rpm -qa | grep -i podman
podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64

sh-4.4# podman login registry.redhat.io
Username: myuser
Password: 
Login Succeeded!

sh-4.4# podman build -t test /root/build/           
STEP 1: FROM registry.redhat.io/ubi8-minimal
Getting image source signatures
Copying blob aebb8c556853 done  
Copying blob 0fd3b5213a9b done  
Copying config 28095021e5 done  
Writing manifest to image destination
Storing signatures
STEP 2: CMD sleep infinity
STEP 3: COMMIT test
--> 5ac3ed7ffc1
5ac3ed7ffc1a25b31b22119935228f7d734b535c4a3d1b1dc5e3f314f3bf7a9d

Comment 24 errata-xmlrpc 2020-09-23 13:52:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.3.38 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3609


Note You need to log in before you can comment on or make changes to this bug.