Bug 1710923 - Can't pull UBI images rootless, no access to /etc/docker/certs.d/registry.access.redhat.com/ [NEEDINFO]
Summary: Can't pull UBI images rootless, no access to /etc/docker/certs.d/registry.acc...
Keywords:
Status: ON_QA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: subscription-manager
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Jiri Hnidek
QA Contact: Red Hat subscription-manager QE Team
URL:
Whiteboard:
Depends On:
Blocks: 1718915 1739707
TreeView+ depends on / blocked
 
Reported: 2019-05-16 14:57 UTC by Scott McCarty
Modified: 2020-01-13 12:09 UTC (History)
16 users (show)

Fixed In Version: subscription-manager-1.26.5-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1718362 1739707 (view as bug list)
Environment:
Last Closed:
Type: Bug
Target Upstream Version:
dornelas: needinfo? (csnyder)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github candlepin subscription-manager pull 2084 'None' closed 1710564: Make entitlement certs and keys world-readable 2020-01-24 12:06:40 UTC

Description Scott McCarty 2019-05-16 14:57:42 UTC
Description of problem:

Can't pull UBI (or other images) from registry.access.redhat.com in rootless mode because it appears non-root users don't have access to: /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key

Version-Release number of selected component (if applicable):

RHEL 8.0, Podman 1.0.2-dev

How reproducible:

100%

Steps to Reproduce:
1. podman pull registry.access.redhat.com/ubi8/ubi
2.
3.

Actual results:

Trying to pull registry.access.redhat.com/ubi8/ubi...Failed
error pulling image "registry.access.redhat.com/ubi8/ubi": unable to pull registry.access.redhat.com/ubi8/ubi: unable to pull image: Error initializing source docker://registry.access.redhat.com/ubi8/ubi:latest: open /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key: permission denied


Expected results:

Trying to pull registry.access.redhat.com/ubi8/ubi...Getting image source signatures
Skipping blob 787f47dbeaac (already present): 67.66 MiB / 67.66 MiB [=======] 0s
Skipping blob 6a5240d60dc4 (already present): 1.45 KiB / 1.45 KiB [=========] 0s
Copying config 4a0518848c7a: 4.37 KiB / 4.37 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
4a0518848c7a1332f3c39bf548e4a77bcce0481e2fea088404026122dedc3379


Additional info:

Appears to be a permissions problem. The following command as root will make things work for non-root users:

chmod 644 /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key

Comment 1 Daniel Walsh 2019-05-16 17:55:45 UTC
This is a subscription-manager issue.

Can you also move the certs out of /etc/docker to /etc/containers. Perhaps add a symbolic link from /etc/docker-> /etc/containers, so if someone installed docker-ce it will work.

Comment 2 Christian Labisch 2019-05-16 18:06:07 UTC
Confirmation of the issue on a freshly installed RHEL 8.0 system :

$ podman login registry.redhat.io --username XXXXXXXX --password XXXXXXXXXX
error authenticating creds for "registry.redhat.io": error creating new docker client: open /etc/docker/certs.d/registry.redhat.io/4275960890465482954.key: permission denied

$ podman search rhel8
ERRO[0000] error searching registry "registry.redhat.io": error creating new docker client: open /etc/docker/certs.d/registry.redhat.io/4275960890465482954.key: permission denied

Comment 3 Scott McCarty 2019-05-16 18:48:06 UTC
Here is the state at fresh install:

[root@localhost ~]# find /etc/docker/
/etc/docker/
/etc/docker/certs.d
/etc/docker/certs.d/cdn.redhat.com
/etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt
/etc/docker/certs.d/registry.access.redhat.com
/etc/docker/certs.d/access.redhat.com
/etc/docker/certs.d/registry.redhat.io

Here is the state after an RHSM register:

[root@rhel8 ~]# find /etc/docker/
/etc/docker/
/etc/docker/certs.d
/etc/docker/certs.d/cdn.redhat.com
/etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt
/etc/docker/certs.d/cdn.redhat.com/3648082865114768684.cert
/etc/docker/certs.d/cdn.redhat.com/3648082865114768684.key
/etc/docker/certs.d/registry.access.redhat.com
/etc/docker/certs.d/registry.access.redhat.com/3648082865114768684.cert
/etc/docker/certs.d/registry.access.redhat.com/3648082865114768684.key
/etc/docker/certs.d/access.redhat.com
/etc/docker/certs.d/access.redhat.com/3648082865114768684.cert
/etc/docker/certs.d/access.redhat.com/3648082865114768684.key
/etc/docker/certs.d/registry.redhat.io
/etc/docker/certs.d/registry.redhat.io/3648082865114768684.cert
/etc/docker/certs.d/registry.redhat.io/3648082865114768684.key

A further unregister, makes it look like this (and rootless podman works again):

[root@rhel8 ~]# find /etc/docker/
/etc/docker/
/etc/docker/certs.d
/etc/docker/certs.d/cdn.redhat.com
/etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt
/etc/docker/certs.d/registry.access.redhat.com
/etc/docker/certs.d/access.redhat.com
/etc/docker/certs.d/registry.redhat.io

Comment 5 Christian Labisch 2019-05-17 08:36:54 UTC
Workaround (successfully tested) : sudo rm -r /etc/docker solves the problem.

$ podman login registry.redhat.io --username XXXXXXXX --password XXXXXXXXXX
Login Succeeded!

Question is, what creates the /etc/docker directory even though docker not being installed ?
We need a general solution, which avoids the initial creation of the /etc/docker/ directory.
This directory should only be created after the installation of docker (what we don't want).

Comment 6 Daniel Walsh 2019-05-17 17:12:26 UTC
Christian, this is an artifact of some older versions of subscription-manager that were looking at controlling access to the registry via Certs.  
This is not needed anylonger, and we hope to have it removed.

Comment 7 Chris Snyder 2019-05-20 18:30:46 UTC
Daniel,

Just to verify, is the request here for us to remove the subscription-manager-container-plugin (the bit that is responsible for copy the entitlement certificates to those locations)?

Comment 8 Scott McCarty 2019-05-20 18:56:16 UTC
@csnyder@redhat.com I don't believe so, no. This is something different. This is managing SSL certs for the registries which are connected to when "subscription-manager register" occurs. This problem is happening before a container is ever run. There is some logic in RHSM somewhere that is pulling certs/keys from somewhere, and populating them in /etc/docker. Then, podman must search there, but doesn't have access to them (in rootless mode). 

Basically, we just need to stop populating /etc/docker with SSL certs/keys for registry server and podman rootless will work right. 

@dwalsh - one side question, why does podman even search /etc/docker?

Comment 9 Daniel Walsh 2019-05-20 19:16:25 UTC
We search /etc/containers/certs.d and /etc/docker/certs.d before connecting to regististries, just incase the registry requires a cert.

Comment 12 Jiri Hnidek 2019-06-21 13:46:53 UTC
Hi,
since the entitlement certificates are world readable (introduced in this PR: https://github.com/candlepin/subscription-manager/pull/2084), then certificates and keys copied by subscription-manager-plugin-container to /etc/docker/certs.d/*/ are also world readable too.

Here are steps of verification:

1. Make sure subscription-manager-plugin-container is installed:

[root@centos7 ~]# rpm -qa | grep subscription-manager
dnf-plugin-subscription-manager-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-rhsm-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-rhsm-certificates-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-gui-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-plugin-container-1.25.11-1.git.0.51dc894.el7.x86_64
subscription-manager-initial-setup-addon-1.25.11-1.git.0.51dc894.el7.x86_64


2. register to local candlepin server:

[root@centos7 ~]# subscription-manager register --username admin --password admin


3. Get list of available subscription with content type equal to "containerimage"

[root@centos7 ~]# subscription-manager list --available --matches "*docker*"
+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+
Subscription Name:   Awesome OS Docker
Provides:            Awesome OS Docker Bits
SKU:                 awesomeos-docker
Contract:            1
Pool ID:             ff8080816b6f0cf1016b6f0f97d30243
Provides Management: No
Available:           10
Suggested:           1
Service Type:        
Roles:               
Service Level:       
Usage:               
Add-ons:             
Subscription Type:   Standard
Starts:              19.6.2019
Ends:                18.6.2020
Entitlement Type:    Physical


4. Attach this subscription

[root@centos7 ~]# subscription-manager attach --pool ff8080816b6f0cf1016b6f0f97d30243


5. Entitlement certificates in /etc/docker/certs.d/ are world readable:

[root@centos7 ~]# ls -l /etc/docker/certs.d/*/
/etc/docker/certs.d/access.redhat.com/:
celkem 8
-rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert
-rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key

/etc/docker/certs.d/cdn.redhat.com/:
celkem 12
-rw-r--r--. 1 root root 2305 19. čen 09.45 redhat-entitlement-authority.crt
-rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert
-rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key

/etc/docker/certs.d/redhat.com/:
celkem 0
lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem

/etc/docker/certs.d/redhat.io/:
celkem 0
lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem

/etc/docker/certs.d/registry.access.redhat.com/:
celkem 8
lrwxrwxrwx. 1 root root   27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem
-rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert
-rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key

/etc/docker/certs.d/registry.redhat.io/:
celkem 8
-rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert
-rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key

Comment 13 Christian Labisch 2019-06-21 14:10:56 UTC
Hi Jiri,

Thanks for the information ! Not sure what you want to tell us though, these cert and key files prevent podman to login to registry.redhat.io.
Dan Walsh and Scott McCarty agreed upon that it is safe to remove /etc/docker and so the task would be to avoid the initial folder population.
This is what this bug report is about. Or did I misunderstand something ?

Regards,
Christian

Comment 14 Chris Snyder 2019-06-21 18:11:35 UTC
Christian,

Do the certs and keys present an issue if they are world-readable?

We are considering removing the subscription-manager-plugin-container package entirely as it may no longer be necessary.
This is being worked on in https://bugzilla.redhat.com/show_bug.cgi?id=1718362 (if removed for that bug the package would be removed in all pertinent versions of RHEL).

Comment 16 Christian Labisch 2019-06-22 09:05:01 UTC
Hi Chris,

Unfortunately I cannot tell you if those certs and keys present an issue if they are world-readable, because I have removed /etc/docker after the discussion with Dan and Scott.
Since then the problem is solved, podman can login to registry.redhat.io. Do you suggest to remove the subscription-manager-plugin-container package in existing installations ?

Regards,
Christian

Comment 18 Scott McCarty 2019-11-18 17:02:50 UTC
Can we close this? I think it's fixed in RHEL 8.1 :-)

Comment 19 Christian Labisch 2019-11-18 17:11:27 UTC
Hi Scott, I think we can - and these ones as well :
https://bugzilla.redhat.com/show_bug.cgi?id=1718362
https://bugzilla.redhat.com/show_bug.cgi?id=1739707

Comment 20 Petr Nehez 2019-12-02 10:22:52 UTC
Guys, I am on RHEL 8.1 but I had to remove /etc/docker again.

Not sure when it appeared again in my system. 
It might be before I did upgrade to 8.1...

Files in /etc/docker are from 8 Nov 2019.

Comment 21 Derrick Ornelas 2019-12-02 23:15:21 UTC
(In reply to Petr Nehez from comment #20)
> Guys, I am on RHEL 8.1 but I had to remove /etc/docker again.
> 
> Not sure when it appeared again in my system. 
> It might be before I did upgrade to 8.1...
> 
> Files in /etc/docker are from 8 Nov 2019.

We mitigated this issue by making Red Hat's entitlement certificates under /etc/docker/certs.d/ world-readable.  If you want to avoid these unneeded certificate files/directories from being created in /etc/docker/certs.d/ in the first place then you can remove the subscription-manager-plugin-container package from your system.

Comment 23 Petr Nehez 2019-12-04 21:35:48 UTC
I guess you still have an issue somewhere.
I uninstalled > installed > uninstalled subscription-manager-plugin-container package from my system but *.key files which are located in subfolders of /etc/docker/certs.d/ still have limited permission - 0400.

Comment 24 Petr Nehez 2019-12-04 21:39:02 UTC
When I executed "sudo chmod 0644 /etc/docker/certs.d/XXX/*" for all subfolders then it started to work again.

And I've written wrong limited permission in the comment #23 - it was 0600 and not 0400.


Note You need to log in before you can comment on or make changes to this bug.