| Summary: | Can't pull UBI images rootless, no access to /etc/docker/certs.d/registry.access.redhat.com/ | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Scott McCarty <smccarty> | |
| Component: | subscription-manager | Assignee: | Jiri Hnidek <jhnidek> | |
| Status: | CLOSED ERRATA | QA Contact: | Red Hat subscription-manager QE Team <rhsm-qe> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 8.0 | CC: | bbreard, bcourt, clnetbox, csnyder, ddarrah, dornelas, dwalsh, fedoraproject, jhnidek, jligon, jnovy, lsm5, mheon, petr, pthomas, redakkan, skallesh | |
| Target Milestone: | rc | Keywords: | Triaged | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | subscription-manager-1.26.5-1 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1718362 1739707 (view as bug list) | Environment: | ||
| Last Closed: | 2020-04-28 16:54:49 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Bug Depends On: | ||||
| Bug Blocks: | 1186913, 1718915, 1739707 | |||
|
Description
Scott McCarty
2019-05-16 14:57:42 UTC
This is a subscription-manager issue. Can you also move the certs out of /etc/docker to /etc/containers. Perhaps add a symbolic link from /etc/docker-> /etc/containers, so if someone installed docker-ce it will work. Confirmation of the issue on a freshly installed RHEL 8.0 system : $ podman login registry.redhat.io --username XXXXXXXX --password XXXXXXXXXX error authenticating creds for "registry.redhat.io": error creating new docker client: open /etc/docker/certs.d/registry.redhat.io/4275960890465482954.key: permission denied $ podman search rhel8 ERRO[0000] error searching registry "registry.redhat.io": error creating new docker client: open /etc/docker/certs.d/registry.redhat.io/4275960890465482954.key: permission denied Here is the state at fresh install: [root@localhost ~]# find /etc/docker/ /etc/docker/ /etc/docker/certs.d /etc/docker/certs.d/cdn.redhat.com /etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt /etc/docker/certs.d/registry.access.redhat.com /etc/docker/certs.d/access.redhat.com /etc/docker/certs.d/registry.redhat.io Here is the state after an RHSM register: [root@rhel8 ~]# find /etc/docker/ /etc/docker/ /etc/docker/certs.d /etc/docker/certs.d/cdn.redhat.com /etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt /etc/docker/certs.d/cdn.redhat.com/3648082865114768684.cert /etc/docker/certs.d/cdn.redhat.com/3648082865114768684.key /etc/docker/certs.d/registry.access.redhat.com /etc/docker/certs.d/registry.access.redhat.com/3648082865114768684.cert /etc/docker/certs.d/registry.access.redhat.com/3648082865114768684.key /etc/docker/certs.d/access.redhat.com /etc/docker/certs.d/access.redhat.com/3648082865114768684.cert /etc/docker/certs.d/access.redhat.com/3648082865114768684.key /etc/docker/certs.d/registry.redhat.io /etc/docker/certs.d/registry.redhat.io/3648082865114768684.cert /etc/docker/certs.d/registry.redhat.io/3648082865114768684.key A further unregister, makes it look like this (and rootless podman works again): [root@rhel8 ~]# find /etc/docker/ /etc/docker/ /etc/docker/certs.d /etc/docker/certs.d/cdn.redhat.com /etc/docker/certs.d/cdn.redhat.com/redhat-entitlement-authority.crt /etc/docker/certs.d/registry.access.redhat.com /etc/docker/certs.d/access.redhat.com /etc/docker/certs.d/registry.redhat.io Workaround (successfully tested) : sudo rm -r /etc/docker solves the problem. $ podman login registry.redhat.io --username XXXXXXXX --password XXXXXXXXXX Login Succeeded! Question is, what creates the /etc/docker directory even though docker not being installed ? We need a general solution, which avoids the initial creation of the /etc/docker/ directory. This directory should only be created after the installation of docker (what we don't want). Christian, this is an artifact of some older versions of subscription-manager that were looking at controlling access to the registry via Certs. This is not needed anylonger, and we hope to have it removed. Daniel, Just to verify, is the request here for us to remove the subscription-manager-container-plugin (the bit that is responsible for copy the entitlement certificates to those locations)? @csnyder I don't believe so, no. This is something different. This is managing SSL certs for the registries which are connected to when "subscription-manager register" occurs. This problem is happening before a container is ever run. There is some logic in RHSM somewhere that is pulling certs/keys from somewhere, and populating them in /etc/docker. Then, podman must search there, but doesn't have access to them (in rootless mode). Basically, we just need to stop populating /etc/docker with SSL certs/keys for registry server and podman rootless will work right. @dwalsh - one side question, why does podman even search /etc/docker? We search /etc/containers/certs.d and /etc/docker/certs.d before connecting to regististries, just incase the registry requires a cert. Hi, since the entitlement certificates are world readable (introduced in this PR: https://github.com/candlepin/subscription-manager/pull/2084), then certificates and keys copied by subscription-manager-plugin-container to /etc/docker/certs.d/*/ are also world readable too. Here are steps of verification: 1. Make sure subscription-manager-plugin-container is installed: [root@centos7 ~]# rpm -qa | grep subscription-manager dnf-plugin-subscription-manager-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-rhsm-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-rhsm-certificates-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-gui-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-plugin-container-1.25.11-1.git.0.51dc894.el7.x86_64 subscription-manager-initial-setup-addon-1.25.11-1.git.0.51dc894.el7.x86_64 2. register to local candlepin server: [root@centos7 ~]# subscription-manager register --username admin --password admin 3. Get list of available subscription with content type equal to "containerimage" [root@centos7 ~]# subscription-manager list --available --matches "*docker*" +-------------------------------------------+ Available Subscriptions +-------------------------------------------+ Subscription Name: Awesome OS Docker Provides: Awesome OS Docker Bits SKU: awesomeos-docker Contract: 1 Pool ID: ff8080816b6f0cf1016b6f0f97d30243 Provides Management: No Available: 10 Suggested: 1 Service Type: Roles: Service Level: Usage: Add-ons: Subscription Type: Standard Starts: 19.6.2019 Ends: 18.6.2020 Entitlement Type: Physical 4. Attach this subscription [root@centos7 ~]# subscription-manager attach --pool ff8080816b6f0cf1016b6f0f97d30243 5. Entitlement certificates in /etc/docker/certs.d/ are world readable: [root@centos7 ~]# ls -l /etc/docker/certs.d/*/ /etc/docker/certs.d/access.redhat.com/: celkem 8 -rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert -rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key /etc/docker/certs.d/cdn.redhat.com/: celkem 12 -rw-r--r--. 1 root root 2305 19. čen 09.45 redhat-entitlement-authority.crt -rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert -rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key /etc/docker/certs.d/redhat.com/: celkem 0 lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem /etc/docker/certs.d/redhat.io/: celkem 0 lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem /etc/docker/certs.d/registry.access.redhat.com/: celkem 8 lrwxrwxrwx. 1 root root 27 21. čen 12.32 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem -rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert -rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key /etc/docker/certs.d/registry.redhat.io/: celkem 8 -rw-r--r--. 1 root root 2826 21. čen 12.43 8879416730622238074.cert -rw-r--r--. 1 root root 3243 21. čen 12.43 8879416730622238074.key Hi Jiri, Thanks for the information ! Not sure what you want to tell us though, these cert and key files prevent podman to login to registry.redhat.io. Dan Walsh and Scott McCarty agreed upon that it is safe to remove /etc/docker and so the task would be to avoid the initial folder population. This is what this bug report is about. Or did I misunderstand something ? Regards, Christian Christian, Do the certs and keys present an issue if they are world-readable? We are considering removing the subscription-manager-plugin-container package entirely as it may no longer be necessary. This is being worked on in https://bugzilla.redhat.com/show_bug.cgi?id=1718362 (if removed for that bug the package would be removed in all pertinent versions of RHEL). Hi Chris, Unfortunately I cannot tell you if those certs and keys present an issue if they are world-readable, because I have removed /etc/docker after the discussion with Dan and Scott. Since then the problem is solved, podman can login to registry.redhat.io. Do you suggest to remove the subscription-manager-plugin-container package in existing installations ? Regards, Christian Can we close this? I think it's fixed in RHEL 8.1 :-) Hi Scott, I think we can - and these ones as well : https://bugzilla.redhat.com/show_bug.cgi?id=1718362 https://bugzilla.redhat.com/show_bug.cgi?id=1739707 Guys, I am on RHEL 8.1 but I had to remove /etc/docker again. Not sure when it appeared again in my system. It might be before I did upgrade to 8.1... Files in /etc/docker are from 8 Nov 2019. (In reply to Petr Nehez from comment #20) > Guys, I am on RHEL 8.1 but I had to remove /etc/docker again. > > Not sure when it appeared again in my system. > It might be before I did upgrade to 8.1... > > Files in /etc/docker are from 8 Nov 2019. We mitigated this issue by making Red Hat's entitlement certificates under /etc/docker/certs.d/ world-readable. If you want to avoid these unneeded certificate files/directories from being created in /etc/docker/certs.d/ in the first place then you can remove the subscription-manager-plugin-container package from your system. I guess you still have an issue somewhere. I uninstalled > installed > uninstalled subscription-manager-plugin-container package from my system but *.key files which are located in subfolders of /etc/docker/certs.d/ still have limited permission - 0400. When I executed "sudo chmod 0644 /etc/docker/certs.d/XXX/*" for all subfolders then it started to work again. And I've written wrong limited permission in the comment #23 - it was 0600 and not 0400. Verification: [root@kvm-04-guest02 ~]# subscription-manager version server type: Red Hat Subscription Management subscription management server: 2.9.21-1 subscription management rules: 5.37 subscription-manager: 1.26.10-1.el8 [root@kvm-04-guest02 ~]# subscription-manager register --serverurl subscription.rhsm.stage.redhat.com Registering to: subscription.rhsm.stage.redhat.com:443/subscription Username: stage_auto_syspurpose002 Password: The system has been registered with ID: c00d572e-72a2-41ab-88cd-97dfc9b69ff6 The registered system name is: kvm-04-guest02.hv2.lab.eng.bos.redhat.com [root@kvm-04-guest02 ~]# subscription-manager attach --auto Installed Product Current Status: Product Name: Red Hat Enterprise Linux for x86_64 High Touch Beta Status: Subscribed Add a non-root user: [root@kvm-04-guest02 ~]# useradd shwetha [root@kvm-04-guest02 ~]# exit logout Connection to kvm-04-guest02.hv2.lab.eng.bos.redhat.com closed. login as shwetha: [shwetha@localhost ~]$ ssh shwetha.lab.eng.bos.redhat.com shwetha.lab.eng.bos.redhat.com's password: [shwetha@kvm-04-guest02 ~]$ whoami shwetha [shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p **** registry.redhat.io; podman run -it --rm registry.redhat.io/ubi8 dnf repolist all Login Succeeded! Trying to pull registry.redhat.io/ubi8... Getting image source signatures Copying blob ff6f434a470a skipped: already exists Copying blob eae5d284042d skipped: already exists Copying config fd73e6738a done Writing manifest to image destination Storing signatures Updating Subscription Management repositories. Unable to read consumer identity This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Red Hat Universal Base Image 8 (RPMs) - BaseOS 230 kB/s | 760 kB 00:03 Red Hat Universal Base Image 8 (RPMs) - AppStream 1.0 MB/s | 3.1 MB 00:03 Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 4.8 kB/s | 9.1 kB 00:01 repo id repo name status ubi-8-appstream Red Hat Universal Base Image 8 (RPMs) - AppStream enabled: 806 ubi-8-appstream-debug Red Hat Universal Base Image 8 (Debug RPMs) - AppStream disabled ubi-8-appstream-source Red Hat Universal Base Image 8 (Source RPMs) - AppStream disabled ubi-8-baseos Red Hat Universal Base Image 8 (RPMs) - BaseOS enabled: 664 ubi-8-baseos-debug Red Hat Universal Base Image 8 (Debug RPMs) - BaseOS disabled ubi-8-baseos-source Red Hat Universal Base Image 8 (Source RPMs) - BaseOS disabled ubi-8-codeready-builder Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder enabled: 12 ubi-8-codeready-builder-debug Red Hat Universal Base Image 8 (Debug RPMs) - CodeReady Builder disabled ubi-8-codeready-builder-source Red Hat Universal Base Image 8 (Source RPMs) - CodeReady Builder disabled [shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p ***** registry.redhat.io; podman pull registry.access.redhat.com/ubi8/ubi Login Succeeded! Trying to pull registry.access.redhat.com/ubi8/ubi... Getting image source signatures Copying blob ff6f434a470a skipped: already exists Copying blob eae5d284042d skipped: already exists Copying config fd73e6738a done Writing manifest to image destination Storing signatures fd73e6738a956712154a0118767d182f546ebca12a06ba3491601974ca6d198c [shwetha@kvm-04-guest02 ~]$ podman login -u rhelentqe -p ****** registry.redhat.io; podman images Login Succeeded! REPOSITORY TAG IMAGE ID CREATED SIZE registry.access.redhat.com/ubi8/ubi latest fd73e6738a95 6 days ago 239 MB registry.redhat.io/ubi8 latest fd73e6738a95 6 days ago 239 MB clearing need info based onthe above verification comments Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1849 |