Bug 1730722 - image.config.openshift.io/cluster "blockedRegistries" spec is not properly blacklisting for build push operations
Summary: image.config.openshift.io/cluster "blockedRegistries" spec is not properly bl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.2.0
Assignee: Adam Kaplan
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-17 12:59 UTC by Pedro Amoedo
Modified: 2019-10-16 06:32 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Blocked registries were not set in `registries.conf` used by buildah Consequence: Buildah could push an image to a registry blocked by the cluster image policy Fix: The `registries.conf` file generated for builds includes blocked registries Result: Builds respect the blocked registries setting for image pull and push
Clone Of:
Environment:
Last Closed: 2019-10-16 06:29:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-controller-manager pull 21 0 'None' closed Bug 1730722: Add blocked registries to registries.conf 2020-10-15 20:05:08 UTC
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:32:03 UTC

Description Pedro Amoedo 2019-07-17 12:59:05 UTC
Description of problem:

When manually modifying "image.config.openshift.io/cluster" resource (documented here[1]) and include "blockedRegistries" section, the PULL operations got blocked as expected but not the build PUSH.

I know that for builds there is another resource called "build.config.openshift.io/cluster" but this one has no option to blacklist registries.

Also, documentation about "image.config.openshift.io/cluster" clearly specifies the following:

~~~
- registrySources: Contains configuration that determines how the container runtime should treat individual registries when accessing images for builds and pods. For instance, whether or not to allow insecure access. It does not contain configuration for the internal cluster registry.
- blockedRegistries: Blacklisted for image pull and push actions. All other registries are allowed.
~~~

Version-Release number of selected component (if applicable):

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.4     True        False         6d2h    Cluster version is 4.1.4

$ oc get clusteroperators
NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                       4.1.4     True        False         False      6d2h
cloud-credential                     4.1.4     True        False         False      6d3h
cluster-autoscaler                   4.1.4     True        False         False      6d3h
console                              4.1.4     True        False         False      6d3h
dns                                  4.1.4     True        False         False      6d3h
image-registry                       4.1.4     True        False         False      22h
ingress                              4.1.4     True        False         False      6d3h
kube-apiserver                       4.1.4     True        False         False      6d3h
kube-controller-manager              4.1.4     True        False         False      6d3h
kube-scheduler                       4.1.4     True        False         False      6d3h
machine-api                          4.1.4     True        False         False      6d3h
machine-config                       4.1.4     True        False         False      6d3h
marketplace                          4.1.4     True        False         False      22h
monitoring                           4.1.4     True        False         False      7h2m
network                              4.1.4     True        False         False      6d3h
node-tuning                          4.1.4     True        False         False      6d3h
openshift-apiserver                  4.1.4     True        False         False      6d3h
openshift-controller-manager         4.1.4     True        False         False      6d3h
openshift-samples                    4.1.4     True        False         False      6d3h
operator-lifecycle-manager           4.1.4     True        False         False      6d3h
operator-lifecycle-manager-catalog   4.1.4     True        False         False      6d3h
service-ca                           4.1.4     True        False         False      6d3h
service-catalog-apiserver            4.1.4     True        False         False      22h
service-catalog-controller-manager   4.1.4     True        False         False      6d2h
storage                              4.1.4     True        False         False      6d3h


How reproducible:

Always

Steps to Reproduce:
1. OCP 4.1.4 cluster (IPI AWS installation)

2. Modify "image.config.openshift.io/cluster" resource on the following manner:

~~~
apiVersion: config.openshift.io/v1
kind: Image
metadata:
  annotations:
    release.openshift.io/create-only: "true"
  creationTimestamp: "2019-07-11T08:45:45Z"
  generation: 2
  name: cluster
  resourceVersion: "2051577"
  selfLink: /apis/config.openshift.io/v1/images/cluster
  uid: 4629cdcb-a3b8-11e9-b5da-06bdc8339008
spec:
  allowedRegistriesForImport:
  - domainName: registry.redhat.io
    insecure: false
  registrySources:
    blockedRegistries:
    - docker.io
    - registry.fedoraproject.org
    - registry.centos.org
status:
  internalRegistryHostname: image-registry.openshift-image-registry.svc:5000
~~~

3. Create a test project like "ext-image-push"

4. Confirm the PULL blockade:

~~~
$ oc import-image docker.io/openshift/hello-openshift --confirm
The ImageStream "hello-openshift" is invalid: 
* spec.tags[latest].from.name: Forbidden: registry "docker.io" not allowed by whitelist: "image-registry.openshift-image-registry.svc:5000", "registry.redhat.io:443"
* status.tags[latest].items[0].dockerImageReference: Forbidden: registry "docker.io" not allowed by whitelist: "image-registry.openshift-image-registry.svc:5000", "registry.redhat.io:443"
~~~

OR (if the image stream was already present)

~~~
$ oc new-app hello-openshift
--> Found image 7af3297 (15 months old) in image stream "test/hello-openshift" under tag "latest" for "hello-openshift"

    * This image will be deployed in deployment config "hello-openshift"
    * Ports 8080/tcp, 8888/tcp will be load balanced by service "hello-openshift"
      * Other containers can access this service through the hostname "hello-openshift"

--> Creating resources ...
    deploymentconfig.apps.openshift.io "hello-openshift" created
    service "hello-openshift" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose svc/hello-openshift' 
    Run 'oc status' to view your app.

$ oc get pods
NAME                       READY   STATUS              RESTARTS   AGE
hello-openshift-1-deploy   1/1     Running             0          36s
hello-openshift-1-gs9c6    0/1     ImageInspectError   0          29s

$ oc get events | grep Failed
8s          Warning   InspectFailed       pod/hello-openshift-1-gs9c6               Failed to inspect image "docker.io/openshift/hello-openshift@sha256:aaea76ff622d2f8bcb32e538e7b3cd0ef6d291953f3e7c9f556c1ba5baf47e2e": rpc error: code = Unknown desc = cannot use "docker.io/openshift/hello-openshift@sha256:aaea76ff622d2f8bcb32e538e7b3cd0ef6d291953f3e7c9f556c1ba5baf47e2e" because it's blocked
8s          Warning   Failed              pod/hello-openshift-1-gs9c6               Error: ImageInspectError
~~~

5. Create new-build to confirm PUSH blockade (this is the step failing):

 a. create the proper docker.io secret and link it with the builder serviceaccount:
 
 ~~~
 $ oc create secret docker-registry dockerhub --docker-username=USERID --docker-password='PASS' --docker-email=MAIL --docker-server=docker.io
 
 $ oc secrets link builder dockerhub
 ~~~

 b. launch a new build (in this case I'm using a modified example from [3]):

  ~~~
  $ oc new-build http://github.com/pamoedom/time --context-dir=httpd
  --> Found image 0cecc39 (2 weeks old) in image stream "openshift/httpd" under tag "2.4" for "httpd"

      Apache httpd 2.4 
      ---------------- 
      Apache httpd 2.4 available as container, is a powerful, efficient, and extensible web server. Apache supports a variety of features, many implemented as compiled modules which extend the core functionality. These can range from server-side programming language support to authentication schemes. Virtual hosting allows one Apache installation to serve many different Web sites.

      Tags: builder, httpd, httpd24

      * A Docker build using source code from http://github.com/pamoedom/time will be created
        * The resulting image will be pushed to image stream tag "time:latest"
        * Use 'oc start-build' to trigger a new build

  --> Creating resources with label build=time ...
      imagestream.image.openshift.io "time" created
      buildconfig.build.openshift.io "time" created
  --> Success
  ~~~

  c. modify buildconfig and change the output section as follows (change the username properly):

  ~~~
  $ oc edit bc time
  ...
  spec:
    output:
      to:
        kind: DockerImage   
        name: docker.io/pamoedo/mytime:latest
      pushSecret:
        name: dockerhub
  ~~~

  d. start a new build and follow the process log:
  
  ~~~
  $ oc start-build time

  $ $ oc logs -f time-3-build
  Caching blobs under "/var/cache/blobs".

  Pulling image image-registry.openshift-image-registry.svc:5000/openshift/httpd@sha256:109a5ab8afbbdff4a65167349d3af4548f8e4ad93e56bb9350670764a49418a9 ...
  [...]
  Writing manifest to image destination
  Storing signatures
  --> 9de14a3ca37adbd3d7152fd2ca2b35eeb8d3732bab0dd90394be69b126de1c1b

  Pushing image docker.io/pamoedo/mytime:latest ...
  Getting image source signatures
  Copying blob sha256:f81824bd20dd5943c11c89c51297f2634fbf8b90cc37ddade1ee74aeea941c0c
  Copying blob sha256:c0ebc657d8c5c1c122b275ffd37b0bdd48726a13671a51a6741d9ed113bf0493
  Copying blob sha256:506b188c0abe2ae3c1561bbff7ce50284de2a241eff5da6f528a3a024871c606
  Copying blob sha256:a4d9907173f48ee257a0d6c451d530a2ec4088c38908b2ec48e3bc8dc66c6d21
  Copying blob sha256:1d72f48b3d14f64a70c8eb92dff736969b510ddcf2da89fac058c5c03ba41c58
  Copying config sha256:9de14a3ca37adbd3d7152fd2ca2b35eeb8d3732bab0dd90394be69b126de1c1b
  Writing manifest to image destination
  Storing signatures
  Push successful
  ~~~


Actual results:

PULL operation blocked as expected but PUSH still allowed to blacklisted registries.

Expected results:

Both PULL & PUSH operations should be blocked based on "blockedRegistries" configuration within "image.config.openshift.io/cluster" resource.

Additional info:

[1] - https://docs.openshift.com/container-platform/4.1/openshift_images/image-configuration.html#images-configuration-file_image-configuration
[2] - https://docs.openshift.com/container-platform/4.1/builds/build-configuration.html#builds-configuration-parameters_build-configuration
[3] - https://blog.openshift.com/pushing-application-images-to-an-external-registry/

Comment 8 wewang 2019-08-30 08:10:52 UTC
Since latest payload is not include the pr, so tested it in ci payload, and verified it.
version:
4.2.0-0.ci-2019-08-30-032806 

Pushing image docker.io/pamoedo/mytime:latest ...
Successfully pushed docker.io/pamoedo/mytime:latest
Warning: Push failed, retrying in 5s ...
Successfully pushed docker.io/pamoedo/mytime:latest
Warning: Push failed, retrying in 5s ...
Successfully pushed docker.io/pamoedo/mytime:latest
Warning: Push failed, retrying in 5s ...
Registry server Address: 
Registry server User Name: wewang58
Registry server Email: 
Registry server Password: <<non-empty>>
error: build error: Failed to push image: push access to registry for "docker://pamoedo/mytime:latest" is blocked by configuration

Comment 10 Adam Kaplan 2019-09-04 13:49:14 UTC
@Pedro fix was applied in this PR: https://github.com/openshift/openshift-controller-manager/pull/21

Comment 11 Pedro Amoedo 2019-09-04 14:03:09 UTC
(In reply to Adam Kaplan from comment #10)
> @Pedro fix was applied in this PR:
> https://github.com/openshift/openshift-controller-manager/pull/21

Thank you!

Comment 12 errata-xmlrpc 2019-10-16 06:29:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.