Bug 2027576

Summary: podman fail to create container for "x509: certificate signed by unknown authority" issue
Product: Red Hat Enterprise Linux 9 Reporter: yanpliu <yanpliu>
Component: podmanAssignee: Gabriela Nečasová <gnecasov>
Status: CLOSED CURRENTRELEASE QA Contact: atomic-bugs <atomic-bugs>
Severity: medium Docs Contact: Gabriela Nečasová <gnecasov>
Priority: unspecified    
Version: 9.0CC: bbaude, dornelas, dwalsh, gnecasov, jnovy, jwboyer, lkuprova, lsm5, mheon, pthomas, qianzhan, smccarty, tsweeney, umohnani
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
.Podman no longer fails to pull a container "X509: certificate signed by unknown authority" Previously, if you had your own internal registry signed by our own CA certificate, then you had to import the certificate onto your host machine. Otherwise, an error occurs: ---- x509: certificate signed by unknown authority ---- With this update, the problem has been fixed.
Story Points: ---
Clone Of:
: 2029912 2029913 (view as bug list) Environment:
Last Closed: 2022-03-17 08:06:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2029912, 2029913    

Description yanpliu 2021-11-30 06:14:37 UTC
Description of problem:
podman fail to create container for "x509: certificate signed by unknown authority" issue

Version-Release number of selected component (if applicable):
RHEL Compose: RHEL-9.0.0-20211128.3 
Satellite 6.10.1.1

subscription-manager-rhsm-certificates-1.29.21-1.el9.x86_64
libdnf-plugin-subscription-manager-1.29.21-1.el9.x86_64
python3-subscription-manager-rhsm-1.29.21-1.el9.x86_64
subscription-manager-1.29.21-1.el9.x86_64
subscription-manager-cockpit-1.29.21-1.el9.noarch
tfm-rubygem-katello-4.1.1.39-1.el7sat.noarch
katello-4.1.1-3.el7sat.noarch
candlepin-4.0.9-1.el7sat.noarch

How reproducible:
100%

Steps to Reproduce:
1. Register RHEL9.0 against to Satellite server
[root@kvm-02-guest11 ~]# subscription-manager register --username=admin --password=admin --org=Default_Organization --environment=Library
Registering to: ent-01-vm-01.lab.eng.nay.redhat.com:443/rhsm
The system has been registered with ID: b428c0d7-9301-49b3-aa0f-00404c82f317
The registered system name is: kvm-02-guest11.rhts.eng.brq.redhat.com

2. Register and auto-attach with --force
[root@kvm-02-guest11 ~]# subscription-manager register --username=admin --password=admin --org=Default_Organization --environment=Library --force --auto-attach
Unregistering from: ent-01-vm-01.lab.eng.nay.redhat.com:443/rhsm
The system with UUID b428c0d7-9301-49b3-aa0f-00404c82f317 has been unregistered
All local data removed
Registering to: ent-01-vm-01.lab.eng.nay.redhat.com:443/rhsm
The system has been registered with ID: 11f41f2d-db31-4333-9c47-f49482d568f9
The registered system name is: kvm-02-guest11.rhts.eng.brq.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 Beta
Status:       Subscribed

3.get entitlement cert
[root@kvm-02-guest11 ~]# ls /etc/pki/entitlement/
2323425857644018530-key.pem  2323425857644018530.pem

4.get entitlement cert, and entitlement certs are now world-readable
[root@kvm-02-guest11 ~]# ls -l /etc/pki/entitlement/
total 12
-rw-r--r--. 1 root root 3243 Nov 30 07:08 2323425857644018530-key.pem
-rw-r--r--. 1 root root 4374 Nov 30 07:08 2323425857644018530.pem

5. create and start container
[root@kvm-02-guest11 ~]# podman create -t --name test_container registry-proxy.engineering.redhat.com/rh-osbs/ubi9 && podman start test_container
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Error: initializing source docker://registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest: pinging container registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": x509: certificate signed by unknown authority

Actual results:
Failed to create container for "x509: certificate signed by unknown authority"

Expected results:
It should create container successfully

Additional info:

Comment 1 Tom Sweeney 2021-11-30 23:24:52 UTC
Jindrich, I'm keeping this assigned to you for now, but I think this may be something Josh or Scott will need to change/handle in the ub9 container.  I've added a few others to the cc list to.

Comment 2 Josh Boyer 2021-12-01 18:53:05 UTC
I'm guessing this is because podman is trying to validate the certificate the internal registry uses for encryption and it doesn't trust the cert.  This doesn't seem like it has anything to do with the image.

Comment 4 Tom Sweeney 2021-12-10 21:16:32 UTC
Yanpliu,

I think your example is missing a step.  WHen I try:

# subscription-manager register --username=admin --password=password --org=Default_Organization --environment=Library --force --auto-attach
Unregistering from: subscription.rhsm.redhat.com:443/subscription
The system with UUID a3207486-aece-4d76-9353-34ae51729386 has been unregistered
All local data removed
Registering to: subscription.rhsm.redhat.com:443/subscription
Error: Server does not support environments.

Which looks like I need a hosted cert first.  Do you know how/where I would do that?  I found references in the BZ, but the certs don't appear to be available from the locations stated there.  And/or @jwboyer do you happen to know?

Comment 5 yanpliu 2021-12-13 02:46:05 UTC
Hello Tom,

My bug reproduce is on Satellite environment, it specified the org and environment, your steps are registered to custom portal.
I reproduce it on Stage candlepin

[root@kvm-02-guest24 ~]#  subscription-manager  register
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Username: rhsm_integration_sca1
Password: 
The system has been registered with ID: 896b08a7-fa73-47bb-9692-ac372f79307f
The registered system name is: kvm-02-guest24.lab.eng.rdu2.redhat.com
[root@kvm-02-guest24 ~]#  subscription-manager  register  --force --auto-attach
Unregistering from: subscription.rhsm.stage.redhat.com:443/subscription
The system with UUID 896b08a7-fa73-47bb-9692-ac372f79307f has been unregistered
All local data removed
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Username: rhsm_integration_sca1
Password: 
The system has been registered with ID: d757738b-dd9e-4df9-ba12-125749d4aea7
The registered system name is: kvm-02-guest24.lab.eng.rdu2.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 Beta
Status:       Subscribed

[root@kvm-02-guest24 ~]#  ls /etc/pki/entitlement/
1379435967326307764-key.pem  1379435967326307764.pem
[root@kvm-02-guest24 ~]#  ls -l /etc/pki/entitlement/
total 12
-rw-r--r--. 1 root root 3243 Dec 12 21:41 1379435967326307764-key.pem
-rw-r--r--. 1 root root 7689 Dec 12 21:41 1379435967326307764.pem
[root@kvm-02-guest24 ~]# podman create -t --name test_container registry-proxy.engineering.redhat.com/rh-osbs/ubi9 && podman start test_container
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Error: initializing source docker://registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest: pinging container registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": x509: certificate signed by unknown authority

Comment 6 Josh Boyer 2021-12-17 12:41:06 UTC
(In reply to Tom Sweeney from comment #4)
> Yanpliu,
> 
> I think your example is missing a step.  WHen I try:
> 
> # subscription-manager register --username=admin --password=password
> --org=Default_Organization --environment=Library --force --auto-attach
> Unregistering from: subscription.rhsm.redhat.com:443/subscription
> The system with UUID a3207486-aece-4d76-9353-34ae51729386 has been
> unregistered
> All local data removed
> Registering to: subscription.rhsm.redhat.com:443/subscription
> Error: Server does not support environments.
> 
> Which looks like I need a hosted cert first.  Do you know how/where I would
> do that?  I found references in the BZ, but the certs don't appear to be
> available from the locations stated there.  And/or @jwboyer do
> you happen to know?

I don't know.  I don't think that's the problem anyway.

(In reply to yanpliu from comment #5)
> Hello Tom,
> 
> My bug reproduce is on Satellite environment, it specified the org and
> environment, your steps are registered to custom portal.
> I reproduce it on Stage candlepin
> 
> [root@kvm-02-guest24 ~]#  subscription-manager  register
> Registering to: subscription.rhsm.stage.redhat.com:443/subscription
> Username: rhsm_integration_sca1
> Password: 
> The system has been registered with ID: 896b08a7-fa73-47bb-9692-ac372f79307f
> The registered system name is: kvm-02-guest24.lab.eng.rdu2.redhat.com
> [root@kvm-02-guest24 ~]#  subscription-manager  register  --force
> --auto-attach
> Unregistering from: subscription.rhsm.stage.redhat.com:443/subscription
> The system with UUID 896b08a7-fa73-47bb-9692-ac372f79307f has been
> unregistered
> All local data removed
> Registering to: subscription.rhsm.stage.redhat.com:443/subscription
> Username: rhsm_integration_sca1
> Password: 
> The system has been registered with ID: d757738b-dd9e-4df9-ba12-125749d4aea7
> The registered system name is: kvm-02-guest24.lab.eng.rdu2.redhat.com
> Installed Product Current Status:
> Product Name: Red Hat Enterprise Linux for x86_64 Beta
> Status:       Subscribed
> 
> [root@kvm-02-guest24 ~]#  ls /etc/pki/entitlement/
> 1379435967326307764-key.pem  1379435967326307764.pem
> [root@kvm-02-guest24 ~]#  ls -l /etc/pki/entitlement/
> total 12
> -rw-r--r--. 1 root root 3243 Dec 12 21:41 1379435967326307764-key.pem
> -rw-r--r--. 1 root root 7689 Dec 12 21:41 1379435967326307764.pem
> [root@kvm-02-guest24 ~]# podman create -t --name test_container
> registry-proxy.engineering.redhat.com/rh-osbs/ubi9 && podman start
> test_container
> Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
> Error: initializing source
> docker://registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest: pinging
> container registry registry-proxy.engineering.redhat.com: Get
> "https://registry-proxy.engineering.redhat.com/v2/": x509: certificate
> signed by unknown authority

This is still just saying it doesn't trust the cert that the registry is signed with.  That registry is an internal proxy, which is different from access.redhat.com or quay.io.  If you look at that cert, it's signed by the Red Hat CA.  If that's not in your trust chain on that machine, it's going to reject it.

Common Name (CN)	registry-proxy.engineering.redhat.com
Organization (O)	Red Hat, Inc.
Organizational Unit (OU)	PnT DevOps
Common Name (CN)	Certificate Authority
Organization (O)	Red Hat
Organizational Unit (OU)	prod

This is a host setup issue.  You need to import the Red Hat CA certs on that host.

[jwboyer@zod ~]$ podman pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Error: initializing source docker://registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest: pinging container registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": x509: certificate signed by unknown authority
[jwboyer@zod ~]$ sudo su -
[root@zod ~]# cd /etc/pki/ca-trust/source/anchors/
[root@zod anchors]# curl -O https://password.corp.redhat.com/RH-IT-Root-CA.crt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1517  100  1517    0     0   1447      0  0:00:01  0:00:01 --:--:--  1447
[root@zod anchors]# update-ca-trust 
[root@zod anchors]# exit
logout
[jwboyer@zod ~]$ podman pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Getting image source signatures
Copying blob 0eea3ccadc41 done  
Copying blob aa43eab6a004 done  
Copying config a4b4bef98e done  
Writing manifest to image destination
Storing signatures
a4b4bef98ee0514be6e1d1ce034c674f432148c37192c6624085ea2ee673d8d4
[jwboyer@zod ~]$

Comment 7 yanpliu 2022-01-24 07:47:53 UTC
Hello Josh,
Thanks for your update.
This step can pass after manually update RH-IT-Root-CA.crt.
We test this scenarios as customer, customer do not need to do this on rhel guest.
And this scenario works well on RHEL9.0 Alpha/Beta and RHEl8.6/8.5 etc, all do not need to manually update RH-IT-Root-CA.crt.

Comment 8 Josh Boyer 2022-01-24 12:35:26 UTC
(In reply to yanpliu from comment #7)
> Hello Josh,
> Thanks for your update.
> This step can pass after manually update RH-IT-Root-CA.crt.
> We test this scenarios as customer, customer do not need to do this on rhel
> guest.
> And this scenario works well on RHEL9.0 Alpha/Beta and RHEl8.6/8.5 etc, all
> do not need to manually update RH-IT-Root-CA.crt.

Customers do not have access to a registry that is internal only to Red Hat and signed with RH-IT-Root-CA.crt.  Testing against that registry isn't a valid customer scenario.  The equivalent customer scenario is if the customer has their own internal registry signed with their own CA cert, where they would have to perform similar steps to get podman to pull images.  This should probably be documented in the release notes if it isn't already.

Comment 9 Tom Sweeney 2022-01-24 14:50:17 UTC
@gnecasov Do we have anything in the release notes that covers this situation?

Comment 17 yanpliu 2022-02-15 02:15:00 UTC
Hello Lenka,
Thank you very much for telling me this information.
I can find Doc Text field in "Show advanced fields".


Thank you all!

Comment 19 yanpliu 2022-03-15 01:22:19 UTC
Retest on DISTRO=RHEL-9.1.0-20220309.3 Satellite6.10, this issue did not reproduce.
[root@kvm-04-guest16 ~]# subscription-manager register --username=admin --password=admin --org=Default_Organization --environment=Library
Registering to: hpe-nehalem-02.hpe2.lab.eng.bos.redhat.com:443/rhsm
The system has been registered with ID: 64f361c9-d187-4c2c-89f7-df7a345544f7
The registered system name is: kvm-04-guest16.lab.eng.rdu2.redhat.com
[root@kvm-04-guest16 ~]# subscription-manager register --username=admin --password=admin --org=Default_Organization --environment=Library --force --auto-attach
Unregistering from: hpe-nehalem-02.hpe2.lab.eng.bos.redhat.com:443/rhsm
The system with UUID 64f361c9-d187-4c2c-89f7-df7a345544f7 has been unregistered
All local data removed
Registering to: hpe-nehalem-02.hpe2.lab.eng.bos.redhat.com:443/rhsm
The system has been registered with ID: 37760171-9f35-48a1-9c4a-2a7459c8d9d6
The registered system name is: kvm-04-guest16.lab.eng.rdu2.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 Beta
Status:       Subscribed

[root@kvm-04-guest16 ~]# ls /etc/pki/entitlement/
5695550158646135769-key.pem  5695550158646135769.pem
[root@kvm-04-guest16 ~]# ls -l /etc/pki/entitlement/
total 12
-rw-r--r--. 1 root root 3243 Mar 14 21:19 5695550158646135769-key.pem
-rw-r--r--. 1 root root 4386 Mar 14 21:19 5695550158646135769.pem
[root@kvm-04-guest16 ~]# podman create -t --name test_container registry-proxy.engineering.redhat.com/rh-osbs/ubi9 && podman start test_container
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Getting image source signatures
Copying blob ac30c696644e done  
Copying blob 0765046881e5 done  
Copying config 6c694af905 done  
Writing manifest to image destination
Storing signatures
914654dc2913059e1ba4ed15a4ed1b224b90c1653352633960b59358e85f3dd2
test_container

Comment 20 Tom Sweeney 2022-03-15 15:24:54 UTC
@

Comment 21 Tom Sweeney 2022-03-15 15:25:42 UTC
@

Comment 22 Tom Sweeney 2022-03-15 15:29:58 UTC
Yanpliu, can you verify that this is now working as it should or if the error still exists and we still need to document.  I'm not sure which is reproducing.  (apologies for the two empty comments, Bugzilla is not my friend this morning).

Comment 23 yanpliu 2022-03-16 01:35:31 UTC
Hello Tom,
I have verified this issue on RHEL-9.1.0-20220309.3 register against Satellite6.10, create and start container pass with manually update RH-IT-Root-CA.crt 

[root@kvm-04-guest16 ~]# rpm -qa |grep podman
podman-catatonit-4.0.0-6.el9.x86_64
podman-4.0.0-6.el9.x86_64
cockpit-podman-42-1.el9.noarch

You can see the verify steps on  Comment 19.
(In reply to Tom Sweeney from comment #22)
> Yanpliu, can you verify that this is now working as it should or if the
> error still exists and we still need to document.  I'm not sure which is
> reproducing.  (apologies for the two empty comments, Bugzilla is not my
> friend this morning).

(In reply to Tom Sweeney from comment #22)
> Yanpliu, can you verify that this is now working as it should or if the
> error still exists and we still need to document.  I'm not sure which is
> reproducing.  (apologies for the two empty comments, Bugzilla is not my
> friend this morning).

Comment 24 Tom Sweeney 2022-03-16 12:17:06 UTC
@gnecasov the doc text needs some tweaking.  " the them on your host" seems off.

Comment 25 Tom Sweeney 2022-03-16 12:26:03 UTC
@yanpliu I think you are saying that the bug is NOT reproducible now?  Is that correct?  If so, should we close this as fixed in Current Release and ignore the Release note?

Comment 26 yanpliu 2022-03-17 01:42:14 UTC
(In reply to Tom Sweeney from comment #25)
> @yanpliu I think you are saying that the bug is NOT reproducible
> now?  Is that correct?  If so, should we close this as fixed in Current
> Release and ignore the Release note?

Yes, you are correct. This issue fixed in the Current Release.

Comment 29 Gabi Fialová 2022-06-02 07:14:01 UTC
The release note has been republished: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/9.0_release_notes/index