Bug 1741391 - imagestream import should use imagecontentsourcepolicy for SHA imports
Summary: imagestream import should use imagecontentsourcepolicy for SHA imports
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.0
Assignee: Oleg Bulatov
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On: 1770101
Blocks: 1766662
TreeView+ depends on / blocked
 
Reported: 2019-08-15 02:41 UTC by Ben Parees
Modified: 2020-06-19 15:53 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1766662 (view as bug list)
Environment:
Last Closed: 2020-01-23 11:05:05 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift openshift-apiserver pull 23 'None' closed Bug 1741391: support ImageContentSourcePolicy by ImageStreamImport 2020-06-23 12:24:52 UTC
Red Hat Product Errata RHBA-2020:0062 None None None 2020-01-23 11:05:32 UTC

Description Ben Parees 2019-08-15 02:41:18 UTC
Description of problem:
In disconnected environments, CRIO uses the imagecontentsourcepolicy to "fallback" to alternate registry sources when pulling sha-based image specs.

The imagestream import controller should do the same, again only when importing a SHA based imagestreamtag.  If the external references is a SHA, and the import against the original registry fails, it should use the imagecontentsourcepolicy to try alternate registry sources to import the SHA.

If it succeeds, then the "status" for the tag should reflect the alternate registry.

This is particularly needed to help resolve imagestreamtags that point to SHA content that's part of the release payload.

Comment 1 Ben Parees 2019-08-15 02:43:29 UTC
I'm not sure if this needs to be a blocker for 4.2 or not.  As it currently stands, some imagestreamtags (notably "cli", "tests", and "must-gather") will fail to import.  It's not clear if any components on the cluster actually attempt to use those imagestreamtags for anything, since most components could just get the pullspec for those images directly via an image-references definition, such that they'd have the pullspec available in their manifest if they needed it.  (And then any pods they created using that pullspec would go through CRIO which would handle the registry fallback logic if needed).

As such i'm not setting a severity right now, but if we can fix this for 4.2 we should, and it's possible it will become more critical if we identify something that's not working because this is missing.

Comment 2 Ben Parees 2019-08-15 02:45:03 UTC
Note: imagecontentsourcepolicy is a cluster resource, so the imagestream import controller should have access to it (the controller manager operator should watch it/set it in the config so the controller gets restarted if it changes.  Or the imagestream controller itself should watch that config resource directly, which is probably easier and avoids restarts).

Comment 4 XiuJuan Wang 2019-08-20 12:01:45 UTC
The jenkins related images failed to import too, maybe should reset the bug priority.

Comment 5 Ben Parees 2019-08-20 13:34:40 UTC
As I recall, the jenkins related images failed import because the samples operator registry hostname substitution logic didn't include them (because the registry hostname substitution logic ignored "registry.svc.ci.openshift.org", and Gabe created a PR to address that (in a different BZ i believe).  Is it still broken?  Has that PR merged?

Comment 7 Miloslav Trmač 2019-08-20 15:35:15 UTC
(In reply to Ben Parees from comment #0)
> Description of problem:
> In disconnected environments, CRIO uses the imagecontentsourcepolicy to
> "fallback" to alternate registry sources when pulling sha-based image specs.
> 
> The imagestream import controller should do the same, again only when
> importing a SHA based imagestreamtag.  If the external references is a SHA,
> and the import against the original registry fails, it should use the
> imagecontentsourcepolicy to try alternate registry sources to import the SHA.

I think it would be ideal to use the c/image client to access registries; that handles the MCO-managed registries.conf mirror configuration automatically, exactly consistently with CRI-O.

(I have no idea whether the port to c/image would be more or less work than to handle ImageContentSourcePolicy explicitly in all the ImageStream callers.)

Comment 8 Gabe Montero 2019-08-20 16:54:00 UTC
In response's to Ben's #comment 5 and XiuJuan's #comment 4 the samples operator change to include `registry.svc.ci.openshift.org` in the list of registries the samples operator will replace with his samplesRegistry config setting did land with commit https://github.com/openshift/cluster-samples-operator/commit/b0470e90ab6e9914f1c0914a3f995a29984d6f66

If XiuJuan thinks that commit was in the version being used, I would need to see the output from 

oc get is jenkins -n openshift -o yaml

and 

oc get configs.samples -o yaml

and open a separate bugzilla.

I'm in the midst of some customer/3.11 testing right now and can't get a 4.2 cluster up, but if I can before XiuJuan updates.

Comment 9 XiuJuan Wang 2019-08-21 08:39:00 UTC
Gabe, 
The latest disconnect env(4.2.0-0.nightly-2019-08-20-213632) has included the commit https://github.com/openshift/cluster-samples-operator/pull/173.
So I will report a new bug to track the jenkins issue.
Thanks

Comment 17 Ben Parees 2019-10-29 15:30:19 UTC
We are going to backport this to 4.2.z right?

Comment 18 Oleg Bulatov 2019-10-29 15:34:58 UTC
Right, but we haven't merged it yet.

Comment 20 Wenjing Zheng 2019-11-11 08:48:09 UTC
4.3 disconnected install failed due to 1770101, QE will verify this bug when 1770101 is fixed.

Comment 21 XiuJuan Wang 2019-11-18 06:01:49 UTC
In disconnect env,  Could import image with SHA imports

$ oc get is  cli  -n openshift   -o yaml 
 apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  creationTimestamp: "2019-11-18T02:02:34Z"
  generation: 5
  name: cli
  namespace: openshift
  resourceVersion: "35371"
  selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/cli
  uid: 3c6f08f0-7918-468c-ae8d-8b06377ddaf6
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f572099c3c93261cab23376857c06757d1aa680414f25fa08709efdef38a7f1e
    generation: 4
    importPolicy:
      scheduled: true
    name: latest
    referencePolicy:
      type: Source
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/cli
  publicDockerImageRepository: default-route-openshift-image-registry.apps.jialiu-663.qe.gcp.devcluster.openshift.com/openshift/cli
  tags:
  - items:
    - created: "2019-11-18T02:54:31Z"
      dockerImageReference: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f572099c3c93261cab23376857c06757d1aa680414f25fa08709efdef38a7f1e
      generation: 4
      image: sha256:f572099c3c93261cab23376857c06757d1aa680414f25fa08709efdef38a7f1e
    tag: latest


 
Before create imagecontentsourcepolicy for target image, import image with digest will fail, after creating imagecontentsourcepolicy, the importing process succeed.

1. Mirror image to mirror registry
$oc image mirror centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae jialiu-663.mirror-registry.qe.gcp.devcluster.openshift.com:5000/centos/ruby-22-centos7
2. Import image with digest before create imagecontentsourcepolicy
$oc import-image rubycentos:v1  --from=docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae -n openshift     --confirm 
error: tag  failed: Internal error occurred: docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
imagestream.image.openshift.io/rubycentos imported with errors

Name:			rubycentos
Namespace:		openshift
Created:		1 second ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2019-11-18T03:13:45Z
Image Repository:	image-registry.openshift-image-registry.svc:5000/openshift/rubycentos
Image Lookup:		local=false
Unique Images:		0
Tags:			1

v1
  tagged from docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae

  ! error: Import failed (InternalError): Internal error occurred: docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
      1 second ago
3.Create imagecontentsourcepolicy

$ oc get imagecontentsourcepolicy image-policy-centos -o yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  creationTimestamp: "2019-11-18T03:37:35Z"
  generation: 2
  name: image-policy-centos
  resourceVersion: "90520"
  selfLink: /apis/operator.openshift.io/v1alpha1/imagecontentsourcepolicies/image-policy-centos
  uid: ce4f362f-6d15-4e41-b874-b36f943c4767
spec:
  repositoryDigestMirrors:
  - mirrors:
    - jialiu-663.mirror-registry.qe.gcp.devcluster.openshift.com:5000/centos/ruby-22-centos7
    source: docker.io/centos/ruby-22-centos7
4.Import image with digest again
$ oc describe  is rubycentos -n openshift 
Name:			rubycentos
Namespace:		openshift
Created:		3 hours ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2019-11-18T05:38:22Z
Image Repository:	default-route-openshift-image-registry.apps.jialiu-663.qe.gcp.devcluster.openshift.com/openshift/rubycentos
Image Lookup:		local=false
Unique Images:		1
Tags:			1

v1
  tagged from docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae

  * docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae
      22 minutes ago

Verified with 4.3.0-0.nightly-2019-11-15-213610, and PASS

Comment 23 errata-xmlrpc 2020-01-23 11:05:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.