Bug 1773806
| Summary: | ImageContentSourcePolicy is not working for a mirror requiring authentication | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Austin Dewey <adewey> |
| Component: | Node | Assignee: | Peter Hunt <pehunt> |
| Node sub component: | Kubelet | QA Contact: | Sunil Choudhary <schoudha> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | urgent | ||
| Priority: | urgent | CC: | adam.kaplan, aos-bugs, gblomqui, jforrest, jmatthew, jmontleo, jokerman, mitr, mpatel, nagrawal, obulatov, ppitonak, qitang, rphillips, scuppett, shurley, travi, wking, wzheng |
| Version: | 4.2.z | Keywords: | Reopened, TestBlocker |
| Target Milestone: | --- | Flags: | scuppett:
needinfo-
scuppett: needinfo- scuppett: needinfo- |
| Target Release: | 4.3.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-12-18 19:12:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
We did some debugging with podman as described here https://docs.openshift.com/container-platform/4.2/openshift_images/image-configuration.html#images-configuration-registry-mirror_image-configuration. Some interesting observations: - The /etc/containers/registries.conf file was configured properly on each node with the corresponding mirror. - Podman was able to pull from the mirror directly by passing in "--creds". - Podman was not able to pull from the source (with the mirror configured) by passing in "--creds". A further look at debug log-level output shows that podman did in fact redirect and try to pull from the mirror, but the pull failed and fell back to pulling from the source. **Podman output pulling directly from mirror** sh-4.4# podman pull --creds $USER:$PASS --log-level debug my-mirror.example.com/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846 ... Trying to pull my-mirror.example.com/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846...time="2019-11-19T16:37:44Z" level=debug msg="reference rewritten from 'my-mirror.example.com/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846' to 'my-mirror.example.com/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846'" ... time="2019-11-19T16:37:44Z" level=debug msg="GET https://my-mirror.example.com/v2/" time="2019-11-19T16:37:44Z" level=debug msg="Ping https://my-mirror.example.com/v2/ status 401" time="2019-11-19T16:37:44Z" level=debug msg="GET https://my-mirror.example.com/artifactory/api/docker/my-mirror/v2/token?account=my-username&scope=repository%3Aopenshift4%2Fose-metering-ansible-operator%3Apull&service=my-mirror.example.com" ... Then podman was able to pull the image. Compare that to this... **Podman output pulling from source configured with mirror** sh-4.4# podman pull --creds $USER:PASS --log-level debug registry.redhat.io/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846 ... Trying to pull registry.redhat.io/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846...time="2019-11-19T16:41:59Z" level=debug msg="reference rewritten from 'registry.redhat.io/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846' to 'my-mirror.example.com/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846'" time="2019-11-19T16:41:59Z" level=debug msg="reference rewritten from 'registry.redhat.io/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846' to 'registry.redhat.io/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846'" time="2019-11-19T16:41:59Z" level=debug msg="Trying to pull "my-mirror.example.com/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846"" ... time="2019-11-19T16:41:59Z" level=debug msg="GET https://my-mirror.example.com/v2/" time="2019-11-19T16:41:59Z" level=debug msg="Ping https://my-mirror.example.com/v2/ status 401" time="2019-11-19T16:41:59Z" level=debug msg="GET https://my-mirror.example.com/artifactory/api/docker/my-mirror/v2/token?scope=repository%3Aopenshift4%2Fose-metering-ansible-operator%3Apull&service=my-mirror.example.com" time="2019-11-19T16:41:59Z" level=debug msg="Trying to pull "registry.redhat.io/openshift4/ose-metering-ansible-operator@sha256:1b950471100eac532d59e60c483b1177fd9b6ab393e493d391124e6387d8d846"" ... The output shows that podman attempted to pull from the mirror and then fell back on the upstream registry. One thing to point out is that in the first scenario (pulling directly from mirror), the username appears to be passed to the mirror as a URL parameter called "account". In the second scenario where the mirror is failing, the "account" query parameter is does not get passed in. You say you are using a specific service account "metering-operator" which you attached the pull secret to. Was the OperatorGroup where you installed the metering operator set up to use that service account? https://github.com/operator-framework/olm-book/blob/master/docs/operator-scoping.md#defining-a-serviceaccount-for-an-operatorgroup When you look at the Deployment for the operator that OLM is creating, does it have its ServiceAccount set to "metering-operator" shurley and jesusr believe they are running into this same issue. Confirmed with them that the Deployment was correctly specifying the service account that the pull secret was attached to. So it seems we have a bug when: 1) ICSP from registry1 -> registry2 2) Pull secret for registry1 is defined globally 3) Pull secret for registry2 is defined in the namespace and attached to a ServiceAccount 4) Deployment is specified to use the ServiceAccount, and the image references are to registry1 The pod fails with an event saying "image could not be rewritten" I filed https://bugzilla.redhat.com/show_bug.cgi?id=1782852 thinking I was seeing something different, but I'm starting to suspect it's the same problem. After turning on debug and looking through the logs I see 401 Unauthorized errors when trying to pull from the internal registry using it as a mirror. It works fine if I use it as the original source. *** Bug 1780471 has been marked as a duplicate of this bug. *** Bringing this back to 4.3.0 as QE is blocked verifying 3rd party operators on Azure, AWS, vSphere. Is this a release blocker or are there workarounds we can give QE? *** Bug 1782852 has been marked as a duplicate of this bug. *** Is BZ1783829 related to this or a dupe? (In reply to Jessica Forrester from comment #2) > You say you are using a specific service account "metering-operator" which > you attached the pull secret to. Was the OperatorGroup where you installed > the metering operator set up to use that service account? > > https://github.com/operator-framework/olm-book/blob/master/docs/operator- > scoping.md#defining-a-serviceaccount-for-an-operatorgroup > > When you look at the Deployment for the operator that OLM is creating, does > it have its ServiceAccount set to "metering-operator" The OperatorGroup does not have a service account associated with it. The deployment does specify the "metering-operator" service account, however, and that service account does have a pull secret linked that gives it the ability to pull from the company's internal registry. @Stephen no that seems to be a different bug. I am not willing to say they are the same at this time. We are debugging that one in a different thread @Stephen I'm pretty sure https://bugzilla.redhat.com/show_bug.cgi?id=1783829 is a bug with ose-ansible-operator. I saw this last night after rebuilding our operator, and after pinning the build to an older tag it worked again: http://pkgs.devel.redhat.com/cgit/containers/openshift-migration-operator/commit/?h=rhcam-1.0-rhel-7&id=2c0fc4e0631cc8b0fc5bd4188c466324898ce943 (In reply to Shawn Hurley from comment #14) > @Stephen no that seems to be a different bug. I am not willing to say they > are the same at this time. We are debugging that one in a different thread Okay, great. I smelled it out right then! Thanks! After discussing with Miloslav on Slack, it turns out this behavior is a design decision. CRI-O can only authenticate to a mirror with the global pull secret usable by any pod. This decision was made because modifying the CRI would take too much time, and the kubelet doesn't know about CRI-O's mirroring. The kubelet only asks for a pull spec, and doesn't pass to CRI-O the right credentials (as it doesn't know that's what CRI-O would need). As such, a mirrored registry can't use a namespaced pull secret. |
Description of problem: OpenShift is not pulling from the specified mirrors in an ImageContentSourcePolicy. We are trying to mirror the ose-metering-ansible-operator image from an internal registry, but OpenShift continues to fall back on pulling from registry.redhat.io. The internal mirror requires authentication, but we have verified that the credentials are correct and that OpenShift can pull from the mirror when the registry URL is hardcoded in the deployment. Version-Release number of selected component (if applicable): $ oc version Client Version: v4.2.4 Server Version: 4.2.2 Kubernetes Version: v1.14.6+868bc38 How reproducible: Always Steps to Reproduce: 1. Create an ImageContentSourcePolicy like below. To reproduce our use case, the mirror must require authentication. apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: name: metering-mirror spec: repositoryDigestMirrors: - mirrors: - my-mirror.example.com/openshift4/ose-metering-ansible-operator source: registry.redhat.io/openshift4/ose-metering-ansible-operator 2. Pull the ose-metering-ansible-operator image into the mirror registry 3. Create a metering subscription: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: metering-ocp namespace: openshift-metering spec: channel: "4.2" installPlanApproval: Automatic name: metering-ocp source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: metering-operator.4.2.4-201911050122 4. In this case, the mirror pull secret was not included in the global pull secret, so a docker-registry secret was created and linked to the metering-operator service account 5. Restart the metering pod Actual results: The metering operator pulls from registry.redhat.io instead of the mirror. Expected results: The metering operator should pull from the mirror instead of upstream. Additional info: The image is referencing a digest, not a tag.