1710923 – Can't pull UBI images rootless, no access to /etc/docker/certs.d/registry.access.redhat.com/

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1710923 - Can't pull UBI images rootless, no access to /etc/docker/certs.d/registry.access.redhat.com/

Summary: Can't pull UBI images rootless, no access to /etc/docker/certs.d/registry.acc...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	subscription-manager
Sub Component:
Version:	8.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Jiri Hnidek
QA Contact:	Red Hat subscription-manager QE Team
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1186913 1718915 1739707
TreeView+	depends on / blocked

Reported:	2019-05-16 14:57 UTC by Scott McCarty
Modified:	2023-03-24 14:48 UTC (History)
CC List:	17 users (show)
Fixed In Version:	subscription-manager-1.26.5-1
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1718362 1739707 (view as bug list)
Environment:
Last Closed:	2020-04-28 16:54:49 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	candlepin subscription-manager pull 2084	'None'	closed	1710564: Make entitlement certs and keys world-readable	2021-02-08 15:02:49 UTC
Red Hat Issue Tracker	RHELPLAN-29807	None	None	None	2023-03-07 12:34:15 UTC
Red Hat Product Errata	RHBA-2020:1849	None	None	None	2020-04-28 16:55:08 UTC

Description Scott McCarty 2019-05-16 14:57:42 UTC

Description of problem:

Can't pull UBI (or other images) from registry.access.redhat.com in rootless mode because it appears non-root users don't have access to: /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key

Version-Release number of selected component (if applicable):

RHEL 8.0, Podman 1.0.2-dev

How reproducible:

100%

Steps to Reproduce:
1. podman pull registry.access.redhat.com/ubi8/ubi
2.
3.

Actual results:

Trying to pull registry.access.redhat.com/ubi8/ubi...Failed
error pulling image "registry.access.redhat.com/ubi8/ubi": unable to pull registry.access.redhat.com/ubi8/ubi: unable to pull image: Error initializing source docker://registry.access.redhat.com/ubi8/ubi:latest: open /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key: permission denied


Expected results:

Trying to pull registry.access.redhat.com/ubi8/ubi...Getting image source signatures
Skipping blob 787f47dbeaac (already present): 67.66 MiB / 67.66 MiB [=======] 0s
Skipping blob 6a5240d60dc4 (already present): 1.45 KiB / 1.45 KiB [=========] 0s
Copying config 4a0518848c7a: 4.37 KiB / 4.37 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
4a0518848c7a1332f3c39bf548e4a77bcce0481e2fea088404026122dedc3379


Additional info:

Appears to be a permissions problem. The following command as root will make things work for non-root users:

chmod 644 /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key

Comment 1 Daniel Walsh 2019-05-16 17:55:45 UTC

This is a subscription-manager issue.

Can you also move the certs out of /etc/docker to /etc/containers. Perhaps add a symbolic link from /etc/docker-> /etc/containers, so if someone installed docker-ce it will work.

Comment 2 Christian Labisch 2019-05-16 18:06:07 UTC

Confirmation of the issue on a freshly installed RHEL 8.0 system :

$ podman login registry.redhat.io --username XXXXXXXX --password XXXXXXXXXX
error authenticating creds for "registry.redhat.io": error creating new docker client: open /etc/docker/certs.d/registry.redhat.io/4275960890465482954.key: permission denied

$ podman search rhel8
ERRO[0000] error searching registry "registry.redhat.io": error creating new docker client: open /etc/docker/certs.d/registry.redhat.io/4275960890465482954.key: permission denied

Comment 3 Scott McCarty 2019-05-16 18:48:06 UTC

Here is the state at fresh install:

[root@localhost ~]# find /etc/docker/
/etc/docker/
/etc/docker/certs.d
/etc/docker/certs.d/cdn.redhat.com
/etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt
/etc/docker/certs.d/registry.access.redhat.com
/etc/docker/certs.d/access.redhat.com
/etc/docker/certs.d/registry.redhat.io

Here is the state after an RHSM register:

[root@rhel8 ~]# find /etc/docker/
/etc/docker/
/etc/docker/certs.d
/etc/docker/certs.d/cdn.redhat.com
/etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt
/etc/docker/certs.d/cdn.redhat.com/3648082865114768684.cert
/etc/docker/certs.d/cdn.redhat.com/3648082865114768684.key
/etc/docker/certs.d/registry.access.redhat.com
/etc/docker/certs.d/registry.access.redhat.com/3648082865114768684.cert
/etc/docker/certs.d/registry.access.redhat.com/3648082865114768684.key
/etc/docker/certs.d/access.redhat.com
/etc/docker/certs.d/access.redhat.com/3648082865114768684.cert
/etc/docker/certs.d/access.redhat.com/3648082865114768684.key
/etc/docker/certs.d/registry.redhat.io
/etc/docker/certs.d/registry.redhat.io/3648082865114768684.cert
/etc/docker/certs.d/registry.redhat.io/3648082865114768684.key

A further unregister, makes it look like this (and rootless podman works again):

[root@rhel8 ~]# find /etc/docker/
/etc/docker/
/etc/docker/certs.d
/etc/docker/certs.d/cdn.redhat.com
/etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt
/etc/docker/certs.d/registry.access.redhat.com
/etc/docker/certs.d/access.redhat.com
/etc/docker/certs.d/registry.redhat.io

Comment 5 Christian Labisch 2019-05-17 08:36:54 UTC

Workaround (successfully tested) : sudo rm -r /etc/docker solves the problem.

$ podman login registry.redhat.io --username XXXXXXXX --password XXXXXXXXXX
Login Succeeded!

Question is, what creates the /etc/docker directory even though docker not being installed ?
We need a general solution, which avoids the initial creation of the /etc/docker/ directory.
This directory should only be created after the installation of docker (what we don't want).

Comment 6 Daniel Walsh 2019-05-17 17:12:26 UTC

Christian, this is an artifact of some older versions of subscription-manager that were looking at controlling access to the registry via Certs.  
This is not needed anylonger, and we hope to have it removed.

Comment 7 Chris Snyder 2019-05-20 18:30:46 UTC

Daniel,

Just to verify, is the request here for us to remove the subscription-manager-container-plugin (the bit that is responsible for copy the entitlement certificates to those locations)?

Comment 8 Scott McCarty 2019-05-20 18:56:16 UTC

@csnyder I don't believe so, no. This is something different. This is managing SSL certs for the registries which are connected to when "subscription-manager register" occurs. This problem is happening before a container is ever run. There is some logic in RHSM somewhere that is pulling certs/keys from somewhere, and populating them in /etc/docker. Then, podman must search there, but doesn't have access to them (in rootless mode). 

Basically, we just need to stop populating /etc/docker with SSL certs/keys for registry server and podman rootless will work right. 

@dwalsh - one side question, why does podman even search /etc/docker?

Comment 9 Daniel Walsh 2019-05-20 19:16:25 UTC

We search /etc/containers/certs.d and /etc/docker/certs.d before connecting to regististries, just incase the registry requires a cert.

Comment 12 Jiri Hnidek 2019-06-21 13:46:53 UTC

Hi,
since the entitlement certificates are world readable (introduced in this PR: https://github.com/candlepin/subscription-manager/pull/2084), then certificates and keys copied by subscription-manager-plugin-container to /etc/docker/certs.d/*/ are also world readable too.

Here are steps of verification:

1. Make sure subscription-manager-plugin-container is installed:

[root@centos7 ~]# rpm -qa | grep subscription-manager
dnf-plugin-subscription-manager-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-rhsm-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-rhsm-certificates-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-gui-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-plugin-container-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-initial-setup-addon-1.25.11-1.git.0.51dc894.el7.x86_64


2. register to local candlepin server:

[root@centos7 ~]# subscription-manager register --username admin --password admin


3. Get list of available subscription with content type equal to "containerimage"

[root@centos7 ~]# subscription-manager list --available --matches "*docker*"
+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+
Subscription Name:   Awesome OS Docker
Provides:            Awesome OS Docker Bits
SKU:                 awesomeos-docker
Contract:            1
Pool ID:             ff8080816b6f0cf1016b6f0f97d30243
Provides Management: No
Available:           10
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              19.6.2019
Ends:                18.6.2020
Entitlement Type:    Physical


4. Attach this subscription

[root@centos7 ~]# subscription-manager attach --pool ff8080816b6f0cf1016b6f0f97d30243


5. Entitlement certificates in /etc/docker/certs.d/ are world readable:

[root@centos7 ~]# ls -l /etc/docker/certs.d/*/
/etc/docker/certs.d/access.redhat.com/:
celkem 8
-rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert
-rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key

/etc/docker/certs.d/cdn.redhat.com/:
celkem 12
-rw-r--r--. 1 root root 2305 19. čen 09.45 redhat-entitlement-authority.crt
-rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert
-rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key

/etc/docker/certs.d/redhat.com/:
celkem 0
lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem

/etc/docker/certs.d/redhat.io/:
celkem 0
lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem

/etc/docker/certs.d/registry.access.redhat.com/:
celkem 8
lrwxrwxrwx. 1 root root   27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem
-rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert
-rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key

/etc/docker/certs.d/registry.redhat.io/:
celkem 8
-rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert
-rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key

Comment 13 Christian Labisch 2019-06-21 14:10:56 UTC

Hi Jiri,

Thanks for the information ! Not sure what you want to tell us though, these cert and key files prevent podman to login to registry.redhat.io.
Dan Walsh and Scott McCarty agreed upon that it is safe to remove /etc/docker and so the task would be to avoid the initial folder population.
This is what this bug report is about. Or did I misunderstand something ?

Regards,
Christian

Comment 14 Chris Snyder 2019-06-21 18:11:35 UTC

Christian,

Do the certs and keys present an issue if they are world-readable?

We are considering removing the subscription-manager-plugin-container package entirely as it may no longer be necessary.
This is being worked on in https://bugzilla.redhat.com/show_bug.cgi?id=1718362 (if removed for that bug the package would be removed in all pertinent versions of RHEL).

Comment 16 Christian Labisch 2019-06-22 09:05:01 UTC

Hi Chris,

Unfortunately I cannot tell you if those certs and keys present an issue if they are world-readable, because I have removed /etc/docker after the discussion with Dan and Scott.
Since then the problem is solved, podman can login to registry.redhat.io. Do you suggest to remove the subscription-manager-plugin-container package in existing installations ?

Regards,
Christian

Comment 18 Scott McCarty 2019-11-18 17:02:50 UTC

Can we close this? I think it's fixed in RHEL 8.1 :-)

Comment 19 Christian Labisch 2019-11-18 17:11:27 UTC

Hi Scott, I think we can - and these ones as well :
https://bugzilla.redhat.com/show_bug.cgi?id=1718362
https://bugzilla.redhat.com/show_bug.cgi?id=1739707

Comment 20 Petr Nehez 2019-12-02 10:22:52 UTC

Guys, I am on RHEL 8.1 but I had to remove /etc/docker again.

Not sure when it appeared again in my system. 
It might be before I did upgrade to 8.1...

Files in /etc/docker are from 8 Nov 2019.

Comment 21 Derrick Ornelas 2019-12-02 23:15:21 UTC

(In reply to Petr Nehez from comment #20)
> Guys, I am on RHEL 8.1 but I had to remove /etc/docker again.
> 
> Not sure when it appeared again in my system. 
> It might be before I did upgrade to 8.1...
> 
> Files in /etc/docker are from 8 Nov 2019.

We mitigated this issue by making Red Hat's entitlement certificates under /etc/docker/certs.d/ world-readable.  If you want to avoid these unneeded certificate files/directories from being created in /etc/docker/certs.d/ in the first place then you can remove the subscription-manager-plugin-container package from your system.

Comment 23 Petr Nehez 2019-12-04 21:35:48 UTC

I guess you still have an issue somewhere.
I uninstalled > installed > uninstalled subscription-manager-plugin-container package from my system but *.key files which are located in subfolders of /etc/docker/certs.d/ still have limited permission - 0400.

Comment 24 Petr Nehez 2019-12-04 21:39:02 UTC

When I executed "sudo chmod 0644 /etc/docker/certs.d/XXX/*" for all subfolders then it started to work again.

And I've written wrong limited permission in the comment #23 - it was 0600 and not 0400.

Comment 25 Shwetha Kallesh 2020-02-05 07:12:03 UTC

Verification:

[root@kvm-04-guest02 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 2.9.21-1
subscription management rules: 5.37
subscription-manager: 1.26.10-1.el8


[root@kvm-04-guest02 ~]# subscription-manager register --serverurl subscription.rhsm.stage.redhat.com
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Username: stage_auto_syspurpose002
Password: 
The system has been registered with ID: c00d572e-72a2-41ab-88cd-97dfc9b69ff6
The registered system name is: kvm-04-guest02.hv2.lab.eng.bos.redhat.com
[root@kvm-04-guest02 ~]# subscription-manager attach --auto
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 High Touch Beta
Status:       Subscribed


Add a non-root user:

[root@kvm-04-guest02 ~]# useradd shwetha

[root@kvm-04-guest02 ~]# exit
logout
Connection to kvm-04-guest02.hv2.lab.eng.bos.redhat.com closed.

login as shwetha:

[shwetha@localhost ~]$ ssh shwetha.lab.eng.bos.redhat.com
shwetha.lab.eng.bos.redhat.com's password: 


[shwetha@kvm-04-guest02 ~]$ whoami
shwetha

[shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p **** registry.redhat.io; podman run -it --rm registry.redhat.io/ubi8 dnf repolist all
Login Succeeded!
Trying to pull registry.redhat.io/ubi8...
Getting image source signatures
Copying blob ff6f434a470a skipped: already exists
Copying blob eae5d284042d skipped: already exists
Copying config fd73e6738a done
Writing manifest to image destination
Storing signatures
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Red Hat Universal Base Image 8 (RPMs) - BaseOS                                                                                                                                                                230 kB/s | 760 kB     00:03    
Red Hat Universal Base Image 8 (RPMs) - AppStream                                                                                                                                                             1.0 MB/s | 3.1 MB     00:03    
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                                                                                                                                                     4.8 kB/s | 9.1 kB     00:01    
repo id                                                                                         repo name                                                                                                                         status
ubi-8-appstream                                                                                 Red Hat Universal Base Image 8 (RPMs) - AppStream                                                                                 enabled: 806
ubi-8-appstream-debug                                                                           Red Hat Universal Base Image 8 (Debug RPMs) - AppStream                                                                           disabled
ubi-8-appstream-source                                                                          Red Hat Universal Base Image 8 (Source RPMs) - AppStream                                                                          disabled
ubi-8-baseos                                                                                    Red Hat Universal Base Image 8 (RPMs) - BaseOS                                                                                    enabled: 664
ubi-8-baseos-debug                                                                              Red Hat Universal Base Image 8 (Debug RPMs) - BaseOS                                                                              disabled
ubi-8-baseos-source                                                                             Red Hat Universal Base Image 8 (Source RPMs) - BaseOS                                                                             disabled
ubi-8-codeready-builder                                                                         Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                                                                         enabled:  12
ubi-8-codeready-builder-debug                                                                   Red Hat Universal Base Image 8 (Debug RPMs) - CodeReady Builder                                                                   disabled
ubi-8-codeready-builder-source                                                                  Red Hat Universal Base Image 8 (Source RPMs) - CodeReady Builder                                                                  disabled
[shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p ***** registry.redhat.io; podman pull registry.access.redhat.com/ubi8/ubi
Login Succeeded!
Trying to pull registry.access.redhat.com/ubi8/ubi...
Getting image source signatures
Copying blob ff6f434a470a skipped: already exists
Copying blob eae5d284042d skipped: already exists
Copying config fd73e6738a done
Writing manifest to image destination
Storing signatures
fd73e6738a956712154a0118767d182f546ebca12a06ba3491601974ca6d198c

[shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p ****** registry.redhat.io; podman images
Login Succeeded!
REPOSITORY                            TAG      IMAGE ID       CREATED      SIZE
registry.access.redhat.com/ubi8/ubi   latest   fd73e6738a95   6 days ago   239 MB
registry.redhat.io/ubi8               latest   fd73e6738a95   6 days ago   239 MB

Comment 26 Rehana 2020-04-15 09:47:19 UTC

clearing need info based onthe above verification comments

Comment 28 errata-xmlrpc 2020-04-28 16:54:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1849

Note You need to log in before you can comment on or make changes to this bug.