Description of problem: Imagestreams with scheduled enabled doesn‘t update automaticly every 15 mins when it imports failed. It updated automaticlly 4 hours later. Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2020-11-11-220947 How reproducible: Always Steps to Reproduce: 1.Install a disconnect cluster 2.Update trust bundle for mirror registry $oc create configmap registry-config --from-file=${MR_Hostname}..5000="${HOME}/qe-additional-ca.crt" -n openshift-config $oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' 3.Check must-gather imagestream later Actual results: $ oc describe is must-gather -n openshift Name: must-gather Namespace: openshift Created: 4 hours ago Labels: <none> Annotations: include.release.openshift.io/self-managed-high-availability=true Image Repository: image-registry.openshift-image-registry.svc:5000/openshift/must-gather Image Lookup: local=false Unique Images: 0 Tags: 1 latest updates automatically from registry quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:919050a1cfc9b6b53a3f9a41103f5cecb20e66af18ab445bfbb3ff3e0a991475 ! error: Import failed (InternalError): Internal error occurred: [xiuwang-dis12.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:919050a1cfc9b6b53a3f9a41103f5cecb20e66af18ab445bfbb3ff3e0a991475: Get "https://xiuwang-dis12.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/": x509: certificate signed by unknown authority, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:919050a1cfc9b6b53a3f9a41103f5cecb20e66af18ab445bfbb3ff3e0a991475: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)] 4 hours ago ================================================ oc get is must-gather -o yaml -n openshift apiVersion: image.openshift.io/v1 kind: ImageStream metadata: annotations: include.release.openshift.io/self-managed-high-availability: "true" creationTimestamp: "2020-11-12T02:23:36Z" generation: 2 managedFields: - apiVersion: image.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:include.release.openshift.io/self-managed-high-availability: {} f:spec: f:tags: .: {} k:{"name":"latest"}: .: {} f:annotations: {} f:from: .: {} f:kind: {} f:name: {} f:generation: {} f:importPolicy: .: {} f:scheduled: {} f:name: {} f:referencePolicy: .: {} f:type: {} f:status: f:dockerImageRepository: {} manager: cluster-version-operator operation: Update time: "2020-11-12T06:41:25Z" name: must-gather namespace: openshift resourceVersion: "109412" selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/must-gather uid: d94aeda3-f952-406c-bf3e-1bc94cd1dadf spec: lookupPolicy: local: false tags: - annotations: null from: kind: DockerImage name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:919050a1cfc9b6b53a3f9a41103f5cecb20e66af18ab445bfbb3ff3e0a991475 generation: 2 importPolicy: scheduled: true name: latest referencePolicy: type: Source status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/must-gather tags: - conditions: - generation: 2 lastTransitionTime: "2020-11-12T02:24:05Z" message: 'Internal error occurred: [xiuwang-dis12.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:919050a1cfc9b6b53a3f9a41103f5cecb20e66af18ab445bfbb3ff3e0a991475: Get "https://xiuwang-dis12.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/": x509: certificate signed by unknown authority, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:919050a1cfc9b6b53a3f9a41103f5cecb20e66af18ab445bfbb3ff3e0a991475: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]' reason: InternalError status: "False" type: ImportSuccess items: null tag: latest Expected results: Imagestreams with scheduled enabled should update automaticly every 15 mins if import failed Additional info:
Created attachment 1728668 [details] must-gather log
I could not reproduce this issue (the scheduled image stream got imported correctly after a failure). By looking at the must-gather I could not find the logs for the openshift-controller-manager pods (we need these to understand what is going on). Could you please enable the debug logging [1] and provide me with a new must-gather (this time with openshift-controller-manager pod logs) ? [1] https://access.redhat.com/solutions/5349121
I still could reproduce this issue on 4.6.0-0.nightly-2021-05-08-154328 version openshift-controller-manager pod logs with loglevel 8 http://virt-openshift-05.lab.eng.nay.redhat.com/xiuwang/1897075/ocm.log $ oc get is -n openshift -o yaml | grep "Internal error occurred" -B 12 | grep "scheduled: true" -A 14 scheduled: true name: latest referencePolicy: type: Source status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/cli-artifacts publicDockerImageRepository: default-route-openshift-image-registry.apps.jimagcp0512.qe.gcp.devcluster.openshift.com/openshift/cli-artifacts tags: - conditions: - generation: 2 lastTransitionTime: "2021-05-12T01:25:09Z" message: 'Internal error occurred: [jimagcp0512.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:94e4d605520db185733820d8f58043a42bac9d77ae6c9691c0e843b53e0c8864: Get "https://jimagcp0512.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/": x509: certificate signed by unknown authority, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:94e4d605520db185733820d8f58043a42bac9d77ae6c9691c0e843b53e0c8864: Get "https://quay.io/v2/": dial tcp 34.224.196.162:443: i/o timeout]' -- importPolicy: scheduled: true name: latest referencePolicy: type: Source status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/installer publicDockerImageRepository: default-route-openshift-image-registry.apps.jimagcp0512.qe.gcp.devcluster.openshift.com/openshift/installer tags: - conditions: - generation: 2 lastTransitionTime: "2021-05-12T01:25:09Z" message: 'Internal error occurred: [jimagcp0512.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:fe775071ad814139d1dff7c1749a6ef5a0173edaa2f1350762cb29cfdab1b116: Get "https://jimagcp0512.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/": x509: certificate signed by unknown authority, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe775071ad814139d1dff7c1749a6ef5a0173edaa2f1350762cb29cfdab1b116: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]' -- importPolicy: scheduled: true name: latest referencePolicy: type: Source status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/must-gather publicDockerImageRepository: default-route-openshift-image-registry.apps.jimagcp0512.qe.gcp.devcluster.openshift.com/openshift/must-gather tags: - conditions: - generation: 3 lastTransitionTime: "2021-05-12T01:59:18Z" message: 'Internal error occurred: [jimagcp0512.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:17172a93363605e98dfe381eaa50969dad62b1be4e4d633d91111e159e3c70ff: Get "https://jimagcp0512.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/": x509: certificate signed by unknown authority, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17172a93363605e98dfe381eaa50969dad62b1be4e4d633d91111e159e3c70ff: Get "https://quay.io/v2/": dial tcp 3.233.133.41:443: i/o timeout]' -- importPolicy: scheduled: true name: v4.4 referencePolicy: type: Source status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/oauth-proxy publicDockerImageRepository: default-route-openshift-image-registry.apps.jimagcp0512.qe.gcp.devcluster.openshift.com/openshift/oauth-proxy tags: - conditions: - generation: 2 lastTransitionTime: "2021-05-12T01:25:24Z" message: 'Internal error occurred: [jimagcp0512.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:440effdfec3d11aff76ee73ddd593bbcc5c08b7e97baae5bad709c16b8915cde: Get "https://jimagcp0512.mirror-registry.qe.gcp.devcluster.openshift.com:5000/v2/": x509: certificate signed by unknown authority, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:440effdfec3d11aff76ee73ddd593bbcc5c08b7e97baae5bad709c16b8915cde: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]'
Could you please attach the content of the configmap called "config" in openshift-controller-manager namespace? I could not locate it in the provided must-gather.
This is being caused by https://bugzilla.redhat.com/show_bug.cgi?id=1881514 Every time the CVO unnecessarily updates the image streams their import got rescheduled, postponing the import.
It seems the CVO bug is fixed. Ricardo, is there anything left to do?
Assign it back, since this fix is not in 4.6.z
For some reason we've lost the target release. We fix bugs in a release version (4.x.0), and then backport it to z-streams. It was fixed in 4.8.0.
Can't reproduce this issue on 4.8 cluster.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.8.10 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3299
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days