RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2027576 - podman fail to create container for "x509: certificate signed by unknown authority" issue
Summary: podman fail to create container for "x509: certificate signed by unknown auth...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: podman
Version: 9.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Gabriela Nečasová
QA Contact: atomic-bugs@redhat.com
Gabriela Nečasová
URL:
Whiteboard:
Depends On:
Blocks: 2029912 2029913
TreeView+ depends on / blocked
 
Reported: 2021-11-30 06:14 UTC by yanpliu
Modified: 2022-10-27 19:31 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
.Podman no longer fails to pull a container "X509: certificate signed by unknown authority" Previously, if you had your own internal registry signed by our own CA certificate, then you had to import the certificate onto your host machine. Otherwise, an error occurs: ---- x509: certificate signed by unknown authority ---- With this update, the problem has been fixed.
Clone Of:
: 2029912 2029913 (view as bug list)
Environment:
Last Closed: 2022-03-17 08:06:08 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-104225 0 None None None 2021-11-30 06:17:31 UTC

Description yanpliu 2021-11-30 06:14:37 UTC 
Description of problem:
podman fail to create container for "x509: certificate signed by unknown authority" issue

Version-Release number of selected component (if applicable):
RHEL Compose: RHEL-9.0.0-20211128.3 
Satellite 6.10.1.1

subscription-manager-rhsm-certificates-1.29.21-1.el9.x86_64
libdnf-plugin-subscription-manager-1.29.21-1.el9.x86_64
python3-subscription-manager-rhsm-1.29.21-1.el9.x86_64
subscription-manager-1.29.21-1.el9.x86_64
subscription-manager-cockpit-1.29.21-1.el9.noarch
tfm-rubygem-katello-4.1.1.39-1.el7sat.noarch
katello-4.1.1-3.el7sat.noarch
candlepin-4.0.9-1.el7sat.noarch

How reproducible:
100%

Steps to Reproduce:
1. Register RHEL9.0 against to Satellite server
[root@kvm-02-guest11 ~]# subscription-manager register --username=admin --password=admin --org=Default_Organization --environment=Library
Registering to: ent-01-vm-01.lab.eng.nay.redhat.com:443/rhsm
The system has been registered with ID: b428c0d7-9301-49b3-aa0f-00404c82f317
The registered system name is: kvm-02-guest11.rhts.eng.brq.redhat.com

2. Register and auto-attach with --force
[root@kvm-02-guest11 ~]# subscription-manager register --username=admin --password=admin --org=Default_Organization --environment=Library --force --auto-attach
Unregistering from: ent-01-vm-01.lab.eng.nay.redhat.com:443/rhsm
The system with UUID b428c0d7-9301-49b3-aa0f-00404c82f317 has been unregistered
All local data removed
Registering to: ent-01-vm-01.lab.eng.nay.redhat.com:443/rhsm
The system has been registered with ID: 11f41f2d-db31-4333-9c47-f49482d568f9
The registered system name is: kvm-02-guest11.rhts.eng.brq.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 Beta
Status:       Subscribed

3.get entitlement cert
[root@kvm-02-guest11 ~]# ls /etc/pki/entitlement/
2323425857644018530-key.pem  2323425857644018530.pem

4.get entitlement cert, and entitlement certs are now world-readable
[root@kvm-02-guest11 ~]# ls -l /etc/pki/entitlement/
total 12
-rw-r--r--. 1 root root 3243 Nov 30 07:08 2323425857644018530-key.pem
-rw-r--r--. 1 root root 4374 Nov 30 07:08 2323425857644018530.pem

5. create and start container
[root@kvm-02-guest11 ~]# podman create -t --name test_container registry-proxy.engineering.redhat.com/rh-osbs/ubi9 && podman start test_container
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Error: initializing source docker://registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest: pinging container registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": x509: certificate signed by unknown authority

Actual results:
Failed to create container for "x509: certificate signed by unknown authority"

Expected results:
It should create container successfully

Additional info:
Comment 1 Tom Sweeney 2021-11-30 23:24:52 UTC 
Jindrich, I'm keeping this assigned to you for now, but I think this may be something Josh or Scott will need to change/handle in the ub9 container.  I've added a few others to the cc list to.
Comment 2 Josh Boyer 2021-12-01 18:53:05 UTC 
I'm guessing this is because podman is trying to validate the certificate the internal registry uses for encryption and it doesn't trust the cert.  This doesn't seem like it has anything to do with the image.
Comment 4 Tom Sweeney 2021-12-10 21:16:32 UTC 
Yanpliu,

I think your example is missing a step.  WHen I try:

# subscription-manager register --username=admin --password=password --org=Default_Organization --environment=Library --force --auto-attach
Unregistering from: subscription.rhsm.redhat.com:443/subscription
The system with UUID a3207486-aece-4d76-9353-34ae51729386 has been unregistered
All local data removed
Registering to: subscription.rhsm.redhat.com:443/subscription
Error: Server does not support environments.

Which looks like I need a hosted cert first.  Do you know how/where I would do that?  I found references in the BZ, but the certs don't appear to be available from the locations stated there.  And/or @jwboyer do you happen to know?
Comment 5 yanpliu 2021-12-13 02:46:05 UTC 
Hello Tom,

My bug reproduce is on Satellite environment, it specified the org and environment, your steps are registered to custom portal.
I reproduce it on Stage candlepin

[root@kvm-02-guest24 ~]#  subscription-manager  register
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Username: rhsm_integration_sca1
Password: 
The system has been registered with ID: 896b08a7-fa73-47bb-9692-ac372f79307f
The registered system name is: kvm-02-guest24.lab.eng.rdu2.redhat.com
[root@kvm-02-guest24 ~]#  subscription-manager  register  --force --auto-attach
Unregistering from: subscription.rhsm.stage.redhat.com:443/subscription
The system with UUID 896b08a7-fa73-47bb-9692-ac372f79307f has been unregistered
All local data removed
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Username: rhsm_integration_sca1
Password: 
The system has been registered with ID: d757738b-dd9e-4df9-ba12-125749d4aea7
The registered system name is: kvm-02-guest24.lab.eng.rdu2.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 Beta
Status:       Subscribed

[root@kvm-02-guest24 ~]#  ls /etc/pki/entitlement/
1379435967326307764-key.pem  1379435967326307764.pem
[root@kvm-02-guest24 ~]#  ls -l /etc/pki/entitlement/
total 12
-rw-r--r--. 1 root root 3243 Dec 12 21:41 1379435967326307764-key.pem
-rw-r--r--. 1 root root 7689 Dec 12 21:41 1379435967326307764.pem
[root@kvm-02-guest24 ~]# podman create -t --name test_container registry-proxy.engineering.redhat.com/rh-osbs/ubi9 && podman start test_container
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Error: initializing source docker://registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest: pinging container registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": x509: certificate signed by unknown authority
Comment 6 Josh Boyer 2021-12-17 12:41:06 UTC 
(In reply to Tom Sweeney from comment #4)
> Yanpliu,
> 
> I think your example is missing a step.  WHen I try:
> 
> # subscription-manager register --username=admin --password=password
> --org=Default_Organization --environment=Library --force --auto-attach
> Unregistering from: subscription.rhsm.redhat.com:443/subscription
> The system with UUID a3207486-aece-4d76-9353-34ae51729386 has been
> unregistered
> All local data removed
> Registering to: subscription.rhsm.redhat.com:443/subscription
> Error: Server does not support environments.
> 
> Which looks like I need a hosted cert first.  Do you know how/where I would
> do that?  I found references in the BZ, but the certs don't appear to be
> available from the locations stated there.  And/or @jwboyer do
> you happen to know?

I don't know.  I don't think that's the problem anyway.

(In reply to yanpliu from comment #5)
> Hello Tom,
> 
> My bug reproduce is on Satellite environment, it specified the org and
> environment, your steps are registered to custom portal.
> I reproduce it on Stage candlepin
> 
> [root@kvm-02-guest24 ~]#  subscription-manager  register
> Registering to: subscription.rhsm.stage.redhat.com:443/subscription
> Username: rhsm_integration_sca1
> Password: 
> The system has been registered with ID: 896b08a7-fa73-47bb-9692-ac372f79307f
> The registered system name is: kvm-02-guest24.lab.eng.rdu2.redhat.com
> [root@kvm-02-guest24 ~]#  subscription-manager  register  --force
> --auto-attach
> Unregistering from: subscription.rhsm.stage.redhat.com:443/subscription
> The system with UUID 896b08a7-fa73-47bb-9692-ac372f79307f has been
> unregistered
> All local data removed
> Registering to: subscription.rhsm.stage.redhat.com:443/subscription
> Username: rhsm_integration_sca1
> Password: 
> The system has been registered with ID: d757738b-dd9e-4df9-ba12-125749d4aea7
> The registered system name is: kvm-02-guest24.lab.eng.rdu2.redhat.com
> Installed Product Current Status:
> Product Name: Red Hat Enterprise Linux for x86_64 Beta
> Status:       Subscribed
> 
> [root@kvm-02-guest24 ~]#  ls /etc/pki/entitlement/
> 1379435967326307764-key.pem  1379435967326307764.pem
> [root@kvm-02-guest24 ~]#  ls -l /etc/pki/entitlement/
> total 12
> -rw-r--r--. 1 root root 3243 Dec 12 21:41 1379435967326307764-key.pem
> -rw-r--r--. 1 root root 7689 Dec 12 21:41 1379435967326307764.pem
> [root@kvm-02-guest24 ~]# podman create -t --name test_container
> registry-proxy.engineering.redhat.com/rh-osbs/ubi9 && podman start
> test_container
> Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
> Error: initializing source
> docker://registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest: pinging
> container registry registry-proxy.engineering.redhat.com: Get
> "https://registry-proxy.engineering.redhat.com/v2/": x509: certificate
> signed by unknown authority

This is still just saying it doesn't trust the cert that the registry is signed with.  That registry is an internal proxy, which is different from access.redhat.com or quay.io.  If you look at that cert, it's signed by the Red Hat CA.  If that's not in your trust chain on that machine, it's going to reject it.

Common Name (CN)	registry-proxy.engineering.redhat.com
Organization (O)	Red Hat, Inc.
Organizational Unit (OU)	PnT DevOps
Common Name (CN)	Certificate Authority
Organization (O)	Red Hat
Organizational Unit (OU)	prod

This is a host setup issue.  You need to import the Red Hat CA certs on that host.

[jwboyer@zod ~]$ podman pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Error: initializing source docker://registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest: pinging container registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": x509: certificate signed by unknown authority
[jwboyer@zod ~]$ sudo su -
[root@zod ~]# cd /etc/pki/ca-trust/source/anchors/
[root@zod anchors]# curl -O https://password.corp.redhat.com/RH-IT-Root-CA.crt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1517  100  1517    0     0   1447      0  0:00:01  0:00:01 --:--:--  1447
[root@zod anchors]# update-ca-trust 
[root@zod anchors]# exit
logout
[jwboyer@zod ~]$ podman pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Getting image source signatures
Copying blob 0eea3ccadc41 done  
Copying blob aa43eab6a004 done  
Copying config a4b4bef98e done  
Writing manifest to image destination
Storing signatures
a4b4bef98ee0514be6e1d1ce034c674f432148c37192c6624085ea2ee673d8d4
[jwboyer@zod ~]$
Comment 7 yanpliu 2022-01-24 07:47:53 UTC 
Hello Josh,
Thanks for your update.
This step can pass after manually update RH-IT-Root-CA.crt.
We test this scenarios as customer, customer do not need to do this on rhel guest.
And this scenario works well on RHEL9.0 Alpha/Beta and RHEl8.6/8.5 etc, all do not need to manually update RH-IT-Root-CA.crt.
Comment 8 Josh Boyer 2022-01-24 12:35:26 UTC 
(In reply to yanpliu from comment #7)
> Hello Josh,
> Thanks for your update.
> This step can pass after manually update RH-IT-Root-CA.crt.
> We test this scenarios as customer, customer do not need to do this on rhel
> guest.
> And this scenario works well on RHEL9.0 Alpha/Beta and RHEl8.6/8.5 etc, all
> do not need to manually update RH-IT-Root-CA.crt.

Customers do not have access to a registry that is internal only to Red Hat and signed with RH-IT-Root-CA.crt.  Testing against that registry isn't a valid customer scenario.  The equivalent customer scenario is if the customer has their own internal registry signed with their own CA cert, where they would have to perform similar steps to get podman to pull images.  This should probably be documented in the release notes if it isn't already.
Comment 9 Tom Sweeney 2022-01-24 14:50:17 UTC 
@gnecasov Do we have anything in the release notes that covers this situation?
Comment 17 yanpliu 2022-02-15 02:15:00 UTC 
Hello Lenka,
Thank you very much for telling me this information.
I can find Doc Text field in "Show advanced fields".


Thank you all!
Comment 19 yanpliu 2022-03-15 01:22:19 UTC 
Retest on DISTRO=RHEL-9.1.0-20220309.3 Satellite6.10, this issue did not reproduce.
[root@kvm-04-guest16 ~]# subscription-manager register --username=admin --password=admin --org=Default_Organization --environment=Library
Registering to: hpe-nehalem-02.hpe2.lab.eng.bos.redhat.com:443/rhsm
The system has been registered with ID: 64f361c9-d187-4c2c-89f7-df7a345544f7
The registered system name is: kvm-04-guest16.lab.eng.rdu2.redhat.com
[root@kvm-04-guest16 ~]# subscription-manager register --username=admin --password=admin --org=Default_Organization --environment=Library --force --auto-attach
Unregistering from: hpe-nehalem-02.hpe2.lab.eng.bos.redhat.com:443/rhsm
The system with UUID 64f361c9-d187-4c2c-89f7-df7a345544f7 has been unregistered
All local data removed
Registering to: hpe-nehalem-02.hpe2.lab.eng.bos.redhat.com:443/rhsm
The system has been registered with ID: 37760171-9f35-48a1-9c4a-2a7459c8d9d6
The registered system name is: kvm-04-guest16.lab.eng.rdu2.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 Beta
Status:       Subscribed

[root@kvm-04-guest16 ~]# ls /etc/pki/entitlement/
5695550158646135769-key.pem  5695550158646135769.pem
[root@kvm-04-guest16 ~]# ls -l /etc/pki/entitlement/
total 12
-rw-r--r--. 1 root root 3243 Mar 14 21:19 5695550158646135769-key.pem
-rw-r--r--. 1 root root 4386 Mar 14 21:19 5695550158646135769.pem
[root@kvm-04-guest16 ~]# podman create -t --name test_container registry-proxy.engineering.redhat.com/rh-osbs/ubi9 && podman start test_container
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/ubi9:latest...
Getting image source signatures
Copying blob ac30c696644e done  
Copying blob 0765046881e5 done  
Copying config 6c694af905 done  
Writing manifest to image destination
Storing signatures
914654dc2913059e1ba4ed15a4ed1b224b90c1653352633960b59358e85f3dd2
test_container
Comment 20 Tom Sweeney 2022-03-15 15:24:54 UTC 
@
Comment 21 Tom Sweeney 2022-03-15 15:25:42 UTC 
@
Comment 22 Tom Sweeney 2022-03-15 15:29:58 UTC 
Yanpliu, can you verify that this is now working as it should or if the error still exists and we still need to document.  I'm not sure which is reproducing.  (apologies for the two empty comments, Bugzilla is not my friend this morning).
Comment 23 yanpliu 2022-03-16 01:35:31 UTC 
Hello Tom,
I have verified this issue on RHEL-9.1.0-20220309.3 register against Satellite6.10, create and start container pass with manually update RH-IT-Root-CA.crt 

[root@kvm-04-guest16 ~]# rpm -qa |grep podman
podman-catatonit-4.0.0-6.el9.x86_64
podman-4.0.0-6.el9.x86_64
cockpit-podman-42-1.el9.noarch

You can see the verify steps on  Comment 19.
(In reply to Tom Sweeney from comment #22)
> Yanpliu, can you verify that this is now working as it should or if the
> error still exists and we still need to document.  I'm not sure which is
> reproducing.  (apologies for the two empty comments, Bugzilla is not my
> friend this morning).

(In reply to Tom Sweeney from comment #22)
> Yanpliu, can you verify that this is now working as it should or if the
> error still exists and we still need to document.  I'm not sure which is
> reproducing.  (apologies for the two empty comments, Bugzilla is not my
> friend this morning).
Comment 24 Tom Sweeney 2022-03-16 12:17:06 UTC 
@gnecasov the doc text needs some tweaking.  " the them on your host" seems off.
Comment 25 Tom Sweeney 2022-03-16 12:26:03 UTC 
@yanpliu I think you are saying that the bug is NOT reproducible now?  Is that correct?  If so, should we close this as fixed in Current Release and ignore the Release note?
Comment 26 yanpliu 2022-03-17 01:42:14 UTC 
(In reply to Tom Sweeney from comment #25)
> @yanpliu I think you are saying that the bug is NOT reproducible
> now?  Is that correct?  If so, should we close this as fixed in Current
> Release and ignore the Release note?

Yes, you are correct. This issue fixed in the Current Release.
Comment 29 Gabi Fialová 2022-06-02 07:14:01 UTC 
The release note has been republished: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/9.0_release_notes/index
Note You need to log in before you can comment on or make changes to this bug.