Description of problem: When manually modifying "image.config.openshift.io/cluster" resource (documented here[1]) and include "blockedRegistries" section, the PULL operations got blocked as expected but not the build PUSH. I know that for builds there is another resource called "build.config.openshift.io/cluster" but this one has no option to blacklist registries. Also, documentation about "image.config.openshift.io/cluster" clearly specifies the following: ~~~ - registrySources: Contains configuration that determines how the container runtime should treat individual registries when accessing images for builds and pods. For instance, whether or not to allow insecure access. It does not contain configuration for the internal cluster registry. - blockedRegistries: Blacklisted for image pull and push actions. All other registries are allowed. ~~~ Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.4 True False 6d2h Cluster version is 4.1.4 $ oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.1.4 True False False 6d2h cloud-credential 4.1.4 True False False 6d3h cluster-autoscaler 4.1.4 True False False 6d3h console 4.1.4 True False False 6d3h dns 4.1.4 True False False 6d3h image-registry 4.1.4 True False False 22h ingress 4.1.4 True False False 6d3h kube-apiserver 4.1.4 True False False 6d3h kube-controller-manager 4.1.4 True False False 6d3h kube-scheduler 4.1.4 True False False 6d3h machine-api 4.1.4 True False False 6d3h machine-config 4.1.4 True False False 6d3h marketplace 4.1.4 True False False 22h monitoring 4.1.4 True False False 7h2m network 4.1.4 True False False 6d3h node-tuning 4.1.4 True False False 6d3h openshift-apiserver 4.1.4 True False False 6d3h openshift-controller-manager 4.1.4 True False False 6d3h openshift-samples 4.1.4 True False False 6d3h operator-lifecycle-manager 4.1.4 True False False 6d3h operator-lifecycle-manager-catalog 4.1.4 True False False 6d3h service-ca 4.1.4 True False False 6d3h service-catalog-apiserver 4.1.4 True False False 22h service-catalog-controller-manager 4.1.4 True False False 6d2h storage 4.1.4 True False False 6d3h How reproducible: Always Steps to Reproduce: 1. OCP 4.1.4 cluster (IPI AWS installation) 2. Modify "image.config.openshift.io/cluster" resource on the following manner: ~~~ apiVersion: config.openshift.io/v1 kind: Image metadata: annotations: release.openshift.io/create-only: "true" creationTimestamp: "2019-07-11T08:45:45Z" generation: 2 name: cluster resourceVersion: "2051577" selfLink: /apis/config.openshift.io/v1/images/cluster uid: 4629cdcb-a3b8-11e9-b5da-06bdc8339008 spec: allowedRegistriesForImport: - domainName: registry.redhat.io insecure: false registrySources: blockedRegistries: - docker.io - registry.fedoraproject.org - registry.centos.org status: internalRegistryHostname: image-registry.openshift-image-registry.svc:5000 ~~~ 3. Create a test project like "ext-image-push" 4. Confirm the PULL blockade: ~~~ $ oc import-image docker.io/openshift/hello-openshift --confirm The ImageStream "hello-openshift" is invalid: * spec.tags[latest].from.name: Forbidden: registry "docker.io" not allowed by whitelist: "image-registry.openshift-image-registry.svc:5000", "registry.redhat.io:443" * status.tags[latest].items[0].dockerImageReference: Forbidden: registry "docker.io" not allowed by whitelist: "image-registry.openshift-image-registry.svc:5000", "registry.redhat.io:443" ~~~ OR (if the image stream was already present) ~~~ $ oc new-app hello-openshift --> Found image 7af3297 (15 months old) in image stream "test/hello-openshift" under tag "latest" for "hello-openshift" * This image will be deployed in deployment config "hello-openshift" * Ports 8080/tcp, 8888/tcp will be load balanced by service "hello-openshift" * Other containers can access this service through the hostname "hello-openshift" --> Creating resources ... deploymentconfig.apps.openshift.io "hello-openshift" created service "hello-openshift" created --> Success Application is not exposed. You can expose services to the outside world by executing one or more of the commands below: 'oc expose svc/hello-openshift' Run 'oc status' to view your app. $ oc get pods NAME READY STATUS RESTARTS AGE hello-openshift-1-deploy 1/1 Running 0 36s hello-openshift-1-gs9c6 0/1 ImageInspectError 0 29s $ oc get events | grep Failed 8s Warning InspectFailed pod/hello-openshift-1-gs9c6 Failed to inspect image "docker.io/openshift/hello-openshift@sha256:aaea76ff622d2f8bcb32e538e7b3cd0ef6d291953f3e7c9f556c1ba5baf47e2e": rpc error: code = Unknown desc = cannot use "docker.io/openshift/hello-openshift@sha256:aaea76ff622d2f8bcb32e538e7b3cd0ef6d291953f3e7c9f556c1ba5baf47e2e" because it's blocked 8s Warning Failed pod/hello-openshift-1-gs9c6 Error: ImageInspectError ~~~ 5. Create new-build to confirm PUSH blockade (this is the step failing): a. create the proper docker.io secret and link it with the builder serviceaccount: ~~~ $ oc create secret docker-registry dockerhub --docker-username=USERID --docker-password='PASS' --docker-email=MAIL --docker-server=docker.io $ oc secrets link builder dockerhub ~~~ b. launch a new build (in this case I'm using a modified example from [3]): ~~~ $ oc new-build http://github.com/pamoedom/time --context-dir=httpd --> Found image 0cecc39 (2 weeks old) in image stream "openshift/httpd" under tag "2.4" for "httpd" Apache httpd 2.4 ---------------- Apache httpd 2.4 available as container, is a powerful, efficient, and extensible web server. Apache supports a variety of features, many implemented as compiled modules which extend the core functionality. These can range from server-side programming language support to authentication schemes. Virtual hosting allows one Apache installation to serve many different Web sites. Tags: builder, httpd, httpd24 * A Docker build using source code from http://github.com/pamoedom/time will be created * The resulting image will be pushed to image stream tag "time:latest" * Use 'oc start-build' to trigger a new build --> Creating resources with label build=time ... imagestream.image.openshift.io "time" created buildconfig.build.openshift.io "time" created --> Success ~~~ c. modify buildconfig and change the output section as follows (change the username properly): ~~~ $ oc edit bc time ... spec: output: to: kind: DockerImage name: docker.io/pamoedo/mytime:latest pushSecret: name: dockerhub ~~~ d. start a new build and follow the process log: ~~~ $ oc start-build time $ $ oc logs -f time-3-build Caching blobs under "/var/cache/blobs". Pulling image image-registry.openshift-image-registry.svc:5000/openshift/httpd@sha256:109a5ab8afbbdff4a65167349d3af4548f8e4ad93e56bb9350670764a49418a9 ... [...] Writing manifest to image destination Storing signatures --> 9de14a3ca37adbd3d7152fd2ca2b35eeb8d3732bab0dd90394be69b126de1c1b Pushing image docker.io/pamoedo/mytime:latest ... Getting image source signatures Copying blob sha256:f81824bd20dd5943c11c89c51297f2634fbf8b90cc37ddade1ee74aeea941c0c Copying blob sha256:c0ebc657d8c5c1c122b275ffd37b0bdd48726a13671a51a6741d9ed113bf0493 Copying blob sha256:506b188c0abe2ae3c1561bbff7ce50284de2a241eff5da6f528a3a024871c606 Copying blob sha256:a4d9907173f48ee257a0d6c451d530a2ec4088c38908b2ec48e3bc8dc66c6d21 Copying blob sha256:1d72f48b3d14f64a70c8eb92dff736969b510ddcf2da89fac058c5c03ba41c58 Copying config sha256:9de14a3ca37adbd3d7152fd2ca2b35eeb8d3732bab0dd90394be69b126de1c1b Writing manifest to image destination Storing signatures Push successful ~~~ Actual results: PULL operation blocked as expected but PUSH still allowed to blacklisted registries. Expected results: Both PULL & PUSH operations should be blocked based on "blockedRegistries" configuration within "image.config.openshift.io/cluster" resource. Additional info: [1] - https://docs.openshift.com/container-platform/4.1/openshift_images/image-configuration.html#images-configuration-file_image-configuration [2] - https://docs.openshift.com/container-platform/4.1/builds/build-configuration.html#builds-configuration-parameters_build-configuration [3] - https://blog.openshift.com/pushing-application-images-to-an-external-registry/
Since latest payload is not include the pr, so tested it in ci payload, and verified it. version: 4.2.0-0.ci-2019-08-30-032806 Pushing image docker.io/pamoedo/mytime:latest ... Successfully pushed docker.io/pamoedo/mytime:latest Warning: Push failed, retrying in 5s ... Successfully pushed docker.io/pamoedo/mytime:latest Warning: Push failed, retrying in 5s ... Successfully pushed docker.io/pamoedo/mytime:latest Warning: Push failed, retrying in 5s ... Registry server Address: Registry server User Name: wewang58 Registry server Email: Registry server Password: <<non-empty>> error: build error: Failed to push image: push access to registry for "docker://pamoedo/mytime:latest" is blocked by configuration
@Pedro fix was applied in this PR: https://github.com/openshift/openshift-controller-manager/pull/21
(In reply to Adam Kaplan from comment #10) > @Pedro fix was applied in this PR: > https://github.com/openshift/openshift-controller-manager/pull/21 Thank you!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922