Bug 1829786

Summary: Imagestreams which are not managed by samples operator keep import failed
Product: OpenShift Container Platform Reporter: XiuJuan Wang <xiuwang>
Component: ImageStreamsAssignee: Ricardo Maraschini <rmarasch>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.5CC: adam.kaplan, aos-bugs, jokerman, wzheng
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Lack of retries on scheduled image stream import when the first attempt had failed. Consequence: Scheduled image stream imports were not accomplised if the initial import attempt fails. Fix: Retrying to import the image on the next scheduling cycle. Result: If the initial import fails we now retry the import on the next cycle.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:32:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
controller-manager logs
none
content of configmap none

Description XiuJuan Wang 2020-04-30 11:19:01 UTC
Created attachment 1683273 [details]
controller-manager logs

Description of problem:
On disconnect cluster, Imagestreams which are not managed by samples operator keep import failed due to 'not find any tags or repository needing import'

Version-Release number of selected component (if applicable):

4.5.0-0.nightly-2020-04-29-231711

How reproducible:
always

Steps to Reproduce:
1.Check the imagestreams which are not managed by samples operator in disconnect cluster
2.
3.

Actual results:
$oc describe is tests -n openshift -o yaml  
Error: unknown shorthand flag: 'o' in -o
See 'oc describe --help' for usage.
[wxj@console kubeconfig]$ oc get is tests -n openshift -o yaml  
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  creationTimestamp: "2020-04-30T04:23:01Z"
  generation: 2
  managedFields:
  - apiVersion: image.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:tags:
          .: {}
          k:{"name":"latest"}:
            .: {}
            f:annotations: {}
            f:from:
              .: {}
              f:kind: {}
              f:name: {}
            f:generation: {}
            f:importPolicy:
              .: {}
              f:scheduled: {}
            f:name: {}
            f:referencePolicy:
              .: {}
              f:type: {}
      f:status:
        f:publicDockerImageRepository: {}
    manager: cluster-version-operator
    operation: Update
    time: "2020-04-30T07:24:06Z"
  name: tests
  namespace: openshift
  resourceVersion: "88332"
  selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/tests
  uid: 6cc44927-157e-4456-9303-509622408735
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a97dbe61859fe84b6606de365c809f800e1048f7deed62249118f91e5d6a9ff
    generation: 2
    importPolicy:
      scheduled: true
    name: latest
    referencePolicy:
      type: Source
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/tests
  publicDockerImageRepository: default-route-openshift-image-registry.apps.qe-hashadebug1.qe.gcp.devcluster.openshift.com/openshift/tests
  tags:
  - conditions:
    - generation: 2
      lastTransitionTime: "2020-04-30T04:23:46Z"
      message: 'Internal error occurred: [qe-hashadebug1.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:5a97dbe61859fe84b6606de365c809f800e1048f7deed62249118f91e5d6a9ff:
        Get https://qe-hashadebug1.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/:
        x509: certificate signed by unknown authority, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a97dbe61859fe84b6606de365c809f800e1048f7deed62249118f91e5d6a9ff:
        Get https://quay.io/v2/: net/http: request canceled while waiting for connection
        (Client.Timeout exceeded while awaiting headers)]'
      reason: InternalError
      status: "False"
      type: ImportSuccess
    items: null
    tag: latest

Expected results:
Should import succeed automaticlly.

Additional info:
Logs of controller manager:

 853 I0430 10:49:38.368359       1 image_trigger_controller.go:347] Started syncing image stream "openshift/tests"
 854 I0430 10:49:38.368378       1 image_trigger_controller.go:349] Finished syncing image stream "openshift/tests" (36.511µs)
 855 I0430 10:49:38.368389       1 imagestream_controller.go:161] Queued import of stream openshift/installer-artifacts...
 856 I0430 10:49:38.368399       1 imagestream_controller.go:161] Queued import of stream openshift/tests...
 857 I0430 10:49:38.368400       1 imagestream_controller.go:266] Importing stream openshift/installer-artifacts partial=true...
 858 I0430 10:49:38.368407       1 imagestream_controller.go:266] Importing stream openshift/tests partial=true...
 859 I0430 10:49:38.368421       1 imagestream_controller.go:300] Did not find any tags or repository needing import
 860 I0430 10:49:38.368424       1 imagestream_controller.go:300] Did not find any tags or repository needing import

Comment 1 Oleg Bulatov 2020-04-30 11:30:31 UTC
Have you added certificates for qe-hashadebug1.mirror-registry.qe.gcp.devcluster.openshift.com:5000 to image.config's additionalTrutedCA?

Comment 2 XiuJuan Wang 2020-04-30 11:50:10 UTC
@Oleg 
Yes, have added, see details in https://bugzilla.redhat.com/show_bug.cgi?id=1818476#c14 .
I could import them successfully by manual at same time.

Comment 3 Ricardo Maraschini 2020-05-07 12:34:58 UTC
Please attach the content of image.config.openshift.io/cluster as well as the configmap where the additional CA bundle has been placed.

Comment 5 Ricardo Maraschini 2020-05-08 11:23:59 UTC
Could you attach the information to this BZ? I have tried to reach your link but it asked for a password that I don't have.

Comment 6 XiuJuan Wang 2020-05-08 12:35:14 UTC
Created attachment 1686466 [details]
content of configmap

@Ricardo
Sorry, my mistake, I used a private pastbin tool.

Comment 9 XiuJuan Wang 2020-05-28 09:13:30 UTC
Installed a disconnect cluster 4.5.0-0.nightly-2020-05-27-202943, check the imagestreams which are not managed by samples operator.

$oc describe is must-gather -n openshift
Name:			must-gather
Namespace:		openshift
Created:		2 hours ago
Labels:			<none>
Annotations:		<none>
Image Repository:	<none>
Image Lookup:		local=false
Unique Images:		0
Tags:			1

latest
  updates automatically from registry quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182c1636631667895687640c739dd1b6a266ca0eb381a75cf8ea39a1cbed9b84

  ! error: Import failed (InternalError): Internal error occurred: [vsphere.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release@sha256:182c1636631667895687640c739dd1b6a266ca0eb381a75cf8ea39a1cbed9b84: Get https://vsphere.mirror-registry.qe.devcluster.openshift.com:5000/v2/: x509: certificate signed by unknown authority, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182c1636631667895687640c739dd1b6a266ca0eb381a75cf8ea39a1cbed9b84: Get https://quay.io/v2/: dial tcp: lookup quay.io on 172.30.0.10:53: read udp 10.129.0.12:43612->172.30.0.10:53: i/o timeout]
      2 hours ago


$oc describe is must-gather -n openshift
Name:			must-gather
Namespace:		openshift
Created:		3 hours ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2020-05-28T06:05:29Z
Image Repository:	<none>
Image Lookup:		local=false
Unique Images:		1
Tags:			1

latest
  updates automatically from registry quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182c1636631667895687640c739dd1b6a266ca0eb381a75cf8ea39a1cbed9b84

  * quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182c1636631667895687640c739dd1b6a266ca0eb381a75cf8ea39a1cbed9b84
      3 minutes ago

$oc get is must-gather -n openshift -o yaml 
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/image.dockerRepositoryCheck: "2020-05-28T06:05:29Z"
  creationTimestamp: "2020-05-28T03:25:20Z"
  generation: 3
  managedFields:
  - apiVersion: image.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:tags:
          .: {}
          k:{"name":"latest"}:
            .: {}
            f:annotations: {}
            f:from:
              .: {}
              f:kind: {}
              f:name: {}
            f:generation: {}
            f:importPolicy:
              .: {}
              f:scheduled: {}
            f:name: {}
            f:referencePolicy:
              .: {}
              f:type: {}
    manager: cluster-version-operator
    operation: Update
    time: "2020-05-28T05:49:28Z"
  name: must-gather
  namespace: openshift
  resourceVersion: "85214"
  selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/must-gather
  uid: f1a08f0d-6df1-4ea7-a2f3-747fea8dfb6a
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182c1636631667895687640c739dd1b6a266ca0eb381a75cf8ea39a1cbed9b84
    generation: 3
    importPolicy:
      scheduled: true
    name: latest
    referencePolicy:
      type: Source
status:
  dockerImageRepository: ""
  tags:
  - items:
    - created: "2020-05-28T06:05:29Z"
      dockerImageReference: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:182c1636631667895687640c739dd1b6a266ca0eb381a75cf8ea39a1cbed9b84
      generation: 3
      image: sha256:182c1636631667895687640c739dd1b6a266ca0eb381a75cf8ea39a1cbed9b84
    tag: latest

Comment 10 errata-xmlrpc 2020-07-13 17:32:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409