Hide Forgot
Description of problem: Can't pull UBI (or other images) from registry.access.redhat.com in rootless mode because it appears non-root users don't have access to: /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key Version-Release number of selected component (if applicable): RHEL 8.0, Podman 1.0.2-dev How reproducible: 100% Steps to Reproduce: 1. podman pull registry.access.redhat.com/ubi8/ubi 2. 3. Actual results: Trying to pull registry.access.redhat.com/ubi8/ubi...Failed error pulling image "registry.access.redhat.com/ubi8/ubi": unable to pull registry.access.redhat.com/ubi8/ubi: unable to pull image: Error initializing source docker://registry.access.redhat.com/ubi8/ubi:latest: open /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key: permission denied Expected results: Trying to pull registry.access.redhat.com/ubi8/ubi...Getting image source signatures Skipping blob 787f47dbeaac (already present): 67.66 MiB / 67.66 MiB [=======] 0s Skipping blob 6a5240d60dc4 (already present): 1.45 KiB / 1.45 KiB [=========] 0s Copying config 4a0518848c7a: 4.37 KiB / 4.37 KiB [==========================] 0s Writing manifest to image destination Storing signatures 4a0518848c7a1332f3c39bf548e4a77bcce0481e2fea088404026122dedc3379 Additional info: Appears to be a permissions problem. The following command as root will make things work for non-root users: chmod 644 /etc/docker/certs.d/registry.access.redhat.com/1352580929635528173.key
This is a subscription-manager issue. Can you also move the certs out of /etc/docker to /etc/containers. Perhaps add a symbolic link from /etc/docker-> /etc/containers, so if someone installed docker-ce it will work.
Confirmation of the issue on a freshly installed RHEL 8.0 system : $ podman login registry.redhat.io --username XXXXXXXX --password XXXXXXXXXX error authenticating creds for "registry.redhat.io": error creating new docker client: open /etc/docker/certs.d/registry.redhat.io/4275960890465482954.key: permission denied $ podman search rhel8 ERRO[0000] error searching registry "registry.redhat.io": error creating new docker client: open /etc/docker/certs.d/registry.redhat.io/4275960890465482954.key: permission denied
Here is the state at fresh install: [root@localhost ~]# find /etc/docker/ /etc/docker/ /etc/docker/certs.d /etc/docker/certs.d/cdn.redhat.com /etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt /etc/docker/certs.d/registry.access.redhat.com /etc/docker/certs.d/access.redhat.com /etc/docker/certs.d/registry.redhat.io Here is the state after an RHSM register: [root@rhel8 ~]# find /etc/docker/ /etc/docker/ /etc/docker/certs.d /etc/docker/certs.d/cdn.redhat.com /etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt /etc/docker/certs.d/cdn.redhat.com/3648082865114768684.cert /etc/docker/certs.d/cdn.redhat.com/3648082865114768684.key /etc/docker/certs.d/registry.access.redhat.com /etc/docker/certs.d/registry.access.redhat.com/3648082865114768684.cert /etc/docker/certs.d/registry.access.redhat.com/3648082865114768684.key /etc/docker/certs.d/access.redhat.com /etc/docker/certs.d/access.redhat.com/3648082865114768684.cert /etc/docker/certs.d/access.redhat.com/3648082865114768684.key /etc/docker/certs.d/registry.redhat.io /etc/docker/certs.d/registry.redhat.io/3648082865114768684.cert /etc/docker/certs.d/registry.redhat.io/3648082865114768684.key A further unregister, makes it look like this (and rootless podman works again): [root@rhel8 ~]# find /etc/docker/ /etc/docker/ /etc/docker/certs.d /etc/docker/certs.d/cdn.redhat.com /etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt /etc/docker/certs.d/registry.access.redhat.com /etc/docker/certs.d/access.redhat.com /etc/docker/certs.d/registry.redhat.io
Workaround (successfully tested) : sudo rm -r /etc/docker solves the problem. $ podman login registry.redhat.io --username XXXXXXXX --password XXXXXXXXXX Login Succeeded! Question is, what creates the /etc/docker directory even though docker not being installed ? We need a general solution, which avoids the initial creation of the /etc/docker/ directory. This directory should only be created after the installation of docker (what we don't want).
Christian, this is an artifact of some older versions of subscription-manager that were looking at controlling access to the registry via Certs. This is not needed anylonger, and we hope to have it removed.
Daniel, Just to verify, is the request here for us to remove the subscription-manager-container-plugin (the bit that is responsible for copy the entitlement certificates to those locations)?
@csnyder@redhat.com I don't believe so, no. This is something different. This is managing SSL certs for the registries which are connected to when "subscription-manager register" occurs. This problem is happening before a container is ever run. There is some logic in RHSM somewhere that is pulling certs/keys from somewhere, and populating them in /etc/docker. Then, podman must search there, but doesn't have access to them (in rootless mode). Basically, we just need to stop populating /etc/docker with SSL certs/keys for registry server and podman rootless will work right. @dwalsh - one side question, why does podman even search /etc/docker?
We search /etc/containers/certs.d and /etc/docker/certs.d before connecting to regististries, just incase the registry requires a cert.
Hi, since the entitlement certificates are world readable (introduced in this PR: https://github.com/candlepin/subscription-manager/pull/2084), then certificates and keys copied by subscription-manager-plugin-container to /etc/docker/certs.d/*/ are also world readable too. Here are steps of verification: 1. Make sure subscription-manager-plugin-container is installed: [root@centos7 ~]# rpm -qa | grep subscription-manager dnf-plugin-subscription-manager-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-rhsm-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-rhsm-certificates-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-gui-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-plugin-container-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-initial-setup-addon-1.25.11-1.git.0.51dc894.el7.x86_64 2. register to local candlepin server: [root@centos7 ~]# subscription-manager register --username admin --password admin 3. Get list of available subscription with content type equal to "containerimage" [root@centos7 ~]# subscription-manager list --available --matches "*docker*" +-------------------------------------------+ Available Subscriptions +-------------------------------------------+ Subscription Name: Awesome OS Docker Provides: Awesome OS Docker Bits SKU: awesomeos-docker Contract: 1 Pool ID: ff8080816b6f0cf1016b6f0f97d30243 Provides Management: No Available: 10 Suggested: 1 Service Type: Roles: Service Level: Usage: Add-ons: Subscription Type: Standard Starts: 19.6.2019 Ends: 18.6.2020 Entitlement Type: Physical 4. Attach this subscription [root@centos7 ~]# subscription-manager attach --pool ff8080816b6f0cf1016b6f0f97d30243 5. Entitlement certificates in /etc/docker/certs.d/ are world readable: [root@centos7 ~]# ls -l /etc/docker/certs.d/*/ /etc/docker/certs.d/access.redhat.com/: celkem 8 -rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert -rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key /etc/docker/certs.d/cdn.redhat.com/: celkem 12 -rw-r--r--. 1 root root 2305 19. čen 09.45 redhat-entitlement-authority.crt -rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert -rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key /etc/docker/certs.d/redhat.com/: celkem 0 lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem /etc/docker/certs.d/redhat.io/: celkem 0 lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem /etc/docker/certs.d/registry.access.redhat.com/: celkem 8 lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem -rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert -rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key /etc/docker/certs.d/registry.redhat.io/: celkem 8 -rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert -rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key
Hi Jiri, Thanks for the information ! Not sure what you want to tell us though, these cert and key files prevent podman to login to registry.redhat.io. Dan Walsh and Scott McCarty agreed upon that it is safe to remove /etc/docker and so the task would be to avoid the initial folder population. This is what this bug report is about. Or did I misunderstand something ? Regards, Christian
Christian, Do the certs and keys present an issue if they are world-readable? We are considering removing the subscription-manager-plugin-container package entirely as it may no longer be necessary. This is being worked on in https://bugzilla.redhat.com/show_bug.cgi?id=1718362 (if removed for that bug the package would be removed in all pertinent versions of RHEL).
Hi Chris, Unfortunately I cannot tell you if those certs and keys present an issue if they are world-readable, because I have removed /etc/docker after the discussion with Dan and Scott. Since then the problem is solved, podman can login to registry.redhat.io. Do you suggest to remove the subscription-manager-plugin-container package in existing installations ? Regards, Christian
Can we close this? I think it's fixed in RHEL 8.1 :-)
Hi Scott, I think we can - and these ones as well : https://bugzilla.redhat.com/show_bug.cgi?id=1718362 https://bugzilla.redhat.com/show_bug.cgi?id=1739707
Guys, I am on RHEL 8.1 but I had to remove /etc/docker again. Not sure when it appeared again in my system. It might be before I did upgrade to 8.1... Files in /etc/docker are from 8 Nov 2019.
(In reply to Petr Nehez from comment #20) > Guys, I am on RHEL 8.1 but I had to remove /etc/docker again. > > Not sure when it appeared again in my system. > It might be before I did upgrade to 8.1... > > Files in /etc/docker are from 8 Nov 2019. We mitigated this issue by making Red Hat's entitlement certificates under /etc/docker/certs.d/ world-readable. If you want to avoid these unneeded certificate files/directories from being created in /etc/docker/certs.d/ in the first place then you can remove the subscription-manager-plugin-container package from your system.
I guess you still have an issue somewhere. I uninstalled > installed > uninstalled subscription-manager-plugin-container package from my system but *.key files which are located in subfolders of /etc/docker/certs.d/ still have limited permission - 0400.
When I executed "sudo chmod 0644 /etc/docker/certs.d/XXX/*" for all subfolders then it started to work again. And I've written wrong limited permission in the comment #23 - it was 0600 and not 0400.
Verification: [root@kvm-04-guest02 ~]# subscription-manager version server type: Red Hat Subscription Management subscription management server: 2.9.21-1 subscription management rules: 5.37 subscription-manager: 1.26.10-1.el8 [root@kvm-04-guest02 ~]# subscription-manager register --serverurl subscription.rhsm.stage.redhat.com Registering to: subscription.rhsm.stage.redhat.com:443/subscription Username: stage_auto_syspurpose002 Password: The system has been registered with ID: c00d572e-72a2-41ab-88cd-97dfc9b69ff6 The registered system name is: kvm-04-guest02.hv2.lab.eng.bos.redhat.com [root@kvm-04-guest02 ~]# subscription-manager attach --auto Installed Product Current Status: Product Name: Red Hat Enterprise Linux for x86_64 High Touch Beta Status: Subscribed Add a non-root user: [root@kvm-04-guest02 ~]# useradd shwetha [root@kvm-04-guest02 ~]# exit logout Connection to kvm-04-guest02.hv2.lab.eng.bos.redhat.com closed. login as shwetha: [shwetha@localhost ~]$ ssh shwetha@kvm-04-guest02.hv2.lab.eng.bos.redhat.com shwetha@kvm-04-guest02.hv2.lab.eng.bos.redhat.com's password: [shwetha@kvm-04-guest02 ~]$ whoami shwetha [shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p **** registry.redhat.io; podman run -it --rm registry.redhat.io/ubi8 dnf repolist all Login Succeeded! Trying to pull registry.redhat.io/ubi8... Getting image source signatures Copying blob ff6f434a470a skipped: already exists Copying blob eae5d284042d skipped: already exists Copying config fd73e6738a done Writing manifest to image destination Storing signatures Updating Subscription Management repositories. Unable to read consumer identity This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Red Hat Universal Base Image 8 (RPMs) - BaseOS 230 kB/s | 760 kB 00:03 Red Hat Universal Base Image 8 (RPMs) - AppStream 1.0 MB/s | 3.1 MB 00:03 Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 4.8 kB/s | 9.1 kB 00:01 repo id repo name status ubi-8-appstream Red Hat Universal Base Image 8 (RPMs) - AppStream enabled: 806 ubi-8-appstream-debug Red Hat Universal Base Image 8 (Debug RPMs) - AppStream disabled ubi-8-appstream-source Red Hat Universal Base Image 8 (Source RPMs) - AppStream disabled ubi-8-baseos Red Hat Universal Base Image 8 (RPMs) - BaseOS enabled: 664 ubi-8-baseos-debug Red Hat Universal Base Image 8 (Debug RPMs) - BaseOS disabled ubi-8-baseos-source Red Hat Universal Base Image 8 (Source RPMs) - BaseOS disabled ubi-8-codeready-builder Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder enabled: 12 ubi-8-codeready-builder-debug Red Hat Universal Base Image 8 (Debug RPMs) - CodeReady Builder disabled ubi-8-codeready-builder-source Red Hat Universal Base Image 8 (Source RPMs) - CodeReady Builder disabled [shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p ***** registry.redhat.io; podman pull registry.access.redhat.com/ubi8/ubi Login Succeeded! Trying to pull registry.access.redhat.com/ubi8/ubi... Getting image source signatures Copying blob ff6f434a470a skipped: already exists Copying blob eae5d284042d skipped: already exists Copying config fd73e6738a done Writing manifest to image destination Storing signatures fd73e6738a956712154a0118767d182f546ebca12a06ba3491601974ca6d198c [shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p ****** registry.redhat.io; podman images Login Succeeded! REPOSITORY TAG IMAGE ID CREATED SIZE registry.access.redhat.com/ubi8/ubi latest fd73e6738a95 6 days ago 239 MB registry.redhat.io/ubi8 latest fd73e6738a95 6 days ago 239 MB
clearing need info based onthe above verification comments
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1849