Bug 1784317

Summary: ImportImageErrorsExist keeps true while no import error in openshift-samples clusteroperator.
Product: OpenShift Container Platform Reporter: XiuJuan Wang <xiuwang>
Component: SamplesAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.0CC: gmontero
Target Milestone: ---   
Target Release: 4.3.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: if samples operator was provided an invalid samplesRegistry override, leading to failed imagestream imports, the samples operator would take a very very long time to ignore errors from the invalid setting and reach a clean state even after the samplesRegistry override was corrected by the user Consequence: the samples operator would incorrectly report degraded for long stretches even after an invalid samplesRegistry setting was fixed Fix: the samples operator now properly clears out pending state on imagestreamimports when the samplesRegistry override is changed Result: the samples operator will recover in a timely fashion when and invalid samplesRegistry override is corrected.
 Story Points: ---
Clone Of: 1784315 Environment:
Last Closed: 2020-03-10 23:52:55 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1784315    
Bug Blocks:    

Description XiuJuan Wang 2019-12-17 08:49:11 UTC 
+++ This bug was initially created as a clone of Bug #1784315 +++

Description of problem:
Sepecify samplesRegistry to an external registry without adding secret.
ImportImageErrorsExist will change to true since all imagestreams import failed without secret and cert.
After revert samplesRegistry, all imagestreams import succeed, but ImportImageErrorsExist still keep true while no import error in openshift-samples clusteroperator.

Version-Release number of selected component (if applicable):
4.4.0-0.ci-2019-12-14-210519

How reproducible:
always

Steps to Reproduce:
1.Patch samplesRegistry to abc.noregistry.com
oc patch config.samples cluster -p '{"spec":{"samplesRegistry": "abc.noregistry.com"}}' --type=merge
2.Check ImportImageErrorsExist 
oc describe config.samples
3.Remove samplesRegistry parameter
4.Check ImportImageErrorsExist and openshift-samples co after a while(about two hours)

Actual results:
 oc get co  openshift-samples -o yaml  
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-12-17T00:41:38Z"
  generation: 1
  name: openshift-samples
  resourceVersion: "140693"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/openshift-samples
  uid: 5baf2fec-2cb4-4ac3-a2a0-7d97eca0d6aa
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-12-17T00:41:52Z"
    message: Samples installation successful at 4.4.0-0.ci-2019-12-14-210519
    status: "True"
    type: Available
  - lastTransitionTime: "2019-12-17T08:20:21Z"
    message: Samples installation successful at 4.4.0-0.ci-2019-12-14-210519
    status: "False"
    type: Progressing
  - lastTransitionTime: "2019-12-17T00:41:46Z"
    status: "False"
    type: Degraded
  extension: null
  relatedObjects:
  - group: samples.operator.openshift.io
    name: cluster
    resource: configs
  - group: ""
    name: openshift-cluster-samples-operator
    resource: namespaces
  - group: ""
    name: openshift
    resource: namespaces
  versions:
  - name: operator
    version: 4.4.0-0.ci-2019-12-14-210519


$ oc get  config.samples  -o yaml 
apiVersion: v1
items:
- apiVersion: samples.operator.openshift.io/v1
  kind: Config
  metadata:
    creationTimestamp: "2019-12-17T00:41:38Z"
    finalizers:
    - samples.operator.openshift.io/finalizer
    generation: 3
    name: cluster
    resourceVersion: "136358"
    selfLink: /apis/samples.operator.openshift.io/v1/configs/cluster
    uid: 53be8fae-500f-45d3-9ffa-a3eb0d9d258a
  spec:
    architectures:
    - x86_64
    managementState: Managed
  status:
    architectures:
    - x86_64
    conditions:
    - lastTransitionTime: "2019-12-17T00:41:38Z"
      lastUpdateTime: "2019-12-17T00:41:38Z"
      status: "True"
      type: ImportCredentialsExist
    - lastTransitionTime: "2019-12-17T00:41:43Z"
      lastUpdateTime: "2019-12-17T00:41:43Z"
      status: "True"
      type: ConfigurationValid
    - lastTransitionTime: "2019-12-17T07:58:52Z"
      lastUpdateTime: "2019-12-17T08:04:26Z"
      message: '<imagestream/jboss-datagrid72-openshift>Internal error occurred: abc.noregistry.com/jboss-datagrid-7/datagrid72-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datagrid72-openshift><imagestream/rhdm74-kieserver-openshift>Internal
        error occurred: abc.noregistry.com/rhdm-7/rhdm74-kieserver-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhdm74-kieserver-openshift><imagestream/fuse-apicurito-generator>Internal
        error occurred: abc.noregistry.com/fuse7/fuse-apicurito-generator:1.2: Get
        https://abc.noregistry.com/v2/: net/http: request canceled while waiting for
        connection (Client.Timeout exceeded while awaiting headers)<imagestream/fuse-apicurito-generator><imagestream/dotnet>Internal
        error occurred: abc.noregistry.com/dotnet/dotnet-21-rhel7:2.1: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/dotnet><imagestream/openjdk-11-rhel7>Internal
        error occurred: abc.noregistry.com/openjdk/openjdk-11-rhel7:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/openjdk-11-rhel7><imagestream/rhdm74-optaweb-employee-rostering-openshift>Internal
        error occurred: abc.noregistry.com/rhdm-7-tech-preview/rhdm74-optaweb-employee-rostering-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhdm74-optaweb-employee-rostering-openshift><imagestream/rhpam74-smartrouter-openshift>Internal
        error occurred: abc.noregistry.com/rhpam-7/rhpam74-smartrouter-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhpam74-smartrouter-openshift><imagestream/jboss-webserver31-tomcat7-openshift>Internal
        error occurred: abc.noregistry.com/jboss-webserver-3/webserver31-tomcat7-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver31-tomcat7-openshift><imagestream/apicurito-ui>Internal
        error occurred: abc.noregistry.com/fuse7/fuse-apicurito:1.2: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/apicurito-ui><imagestream/jenkins>Internal
        error occurred: abc.noregistry.com/ocp/4.4-2019-12-14-210519@sha256:931905ae20463879828c8dea761ef56506c182f7dd4f9a17572cb1739e2281b8:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jenkins><imagestream/jboss-datagrid71-client-openshift>Internal
        error occurred: abc.noregistry.com/jboss-datagrid-7/datagrid71-client-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datagrid71-client-openshift><imagestream/jboss-datagrid71-openshift>Internal
        error occurred: abc.noregistry.com/jboss-datagrid-7/datagrid71-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datagrid71-openshift><imagestream/jboss-datagrid65-client-openshift>Internal
        error occurred: abc.noregistry.com/jboss-datagrid-6/datagrid65-client-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datagrid65-client-openshift><imagestream/jboss-webserver30-tomcat8-openshift>Internal
        error occurred: abc.noregistry.com/jboss-webserver-3/webserver30-tomcat8-openshift:1.1:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver30-tomcat8-openshift><imagestream/redhat-sso73-openshift>Internal
        error occurred: abc.noregistry.com/redhat-sso-7/sso73-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/redhat-sso73-openshift><imagestream/mariadb>Internal
        error occurred: abc.noregistry.com/rhscl/mariadb-102-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/mariadb><imagestream/python>Internal error
        occurred: abc.noregistry.com/rhscl/python-27-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/python><imagestream/jboss-fuse70-eap-openshift>Internal
        error occurred: abc.noregistry.com/fuse7/fuse-eap-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/jboss-fuse70-eap-openshift><imagestream/java>Internal
        error occurred: abc.noregistry.com/openjdk/openjdk-11-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/java><imagestream/jenkins-agent-maven>Internal
        error occurred: abc.noregistry.com/ocp/4.4-2019-12-14-210519@sha256:04708c39ad59ca4f42cb89ba39802323a638476a05eb58ce4530ab5d788430e6:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jenkins-agent-maven><imagestream/rhpam74-kieserver-openshift>Internal
        error occurred: abc.noregistry.com/rhpam-7/rhpam74-kieserver-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhpam74-kieserver-openshift><imagestream/php>Internal
        error occurred: abc.noregistry.com/rhscl/php-70-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/php><imagestream/perl>Internal error occurred:
        abc.noregistry.com/rhscl/perl-524-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/perl><imagestream/jboss-fuse70-karaf-openshift>Internal
        error occurred: abc.noregistry.com/fuse7/fuse-karaf-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/jboss-fuse70-karaf-openshift><imagestream/rhpam74-businesscentral-monitoring-openshift>Internal
        error occurred: abc.noregistry.com/rhpam-7/rhpam74-businesscentral-monitoring-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhpam74-businesscentral-monitoring-openshift><imagestream/jboss-eap72-openshift>Internal
        error occurred: abc.noregistry.com/jboss-eap-7/eap72-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/jboss-eap72-openshift><imagestream/postgresql>Internal
        error occurred: abc.noregistry.com/rhscl/postgresql-10-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/postgresql><imagestream/fuse7-java-openshift>Internal
        error occurred: abc.noregistry.com/fuse7/fuse-java-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/fuse7-java-openshift><imagestream/redhat-sso71-openshift>Internal
        error occurred: abc.noregistry.com/redhat-sso-7/sso71-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/redhat-sso71-openshift><imagestream/jboss-amq-63>Internal
        error occurred: abc.noregistry.com/jboss-amq-6/amq63-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/jboss-amq-63><imagestream/redhat-openjdk18-openshift>Internal
        error occurred: abc.noregistry.com/redhat-openjdk-18/openjdk18-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/redhat-openjdk18-openshift><imagestream/mongodb>Internal
        error occurred: abc.noregistry.com/rhscl/mongodb-32-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/mongodb><imagestream/jboss-fuse70-java-openshift>Internal
        error occurred: abc.noregistry.com/fuse7/fuse-java-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/jboss-fuse70-java-openshift><imagestream/jboss-processserver64-openshift>Internal
        error occurred: abc.noregistry.com/jboss-processserver-6/processserver64-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-processserver64-openshift><imagestream/jboss-webserver31-tomcat8-openshift>Internal
        error occurred: abc.noregistry.com/jboss-webserver-3/webserver31-tomcat8-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver31-tomcat8-openshift><imagestream/jboss-eap70-openshift>Internal
        error occurred: abc.noregistry.com/jboss-eap-7/eap70-openshift:1.3: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/jboss-eap70-openshift><imagestream/ruby>Internal
        error occurred: abc.noregistry.com/rhscl/ruby-23-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/ruby><imagestream/jboss-eap71-openshift>Internal
        error occurred: abc.noregistry.com/jboss-eap-7/eap71-openshift:1.1: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/jboss-eap71-openshift><imagestream/jboss-eap64-openshift>Internal
        error occurred: abc.noregistry.com/jboss-eap-6/eap64-openshift:1.1: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/jboss-eap64-openshift><imagestream/dotnet-runtime>Internal
        error occurred: abc.noregistry.com/dotnet/dotnet-21-runtime-rhel7:2.1: Get
        https://abc.noregistry.com/v2/: net/http: request canceled while waiting for
        connection (Client.Timeout exceeded while awaiting headers)<imagestream/dotnet-runtime><imagestream/redhat-sso70-openshift>Internal
        error occurred: abc.noregistry.com/redhat-sso-7/sso70-openshift:1.3: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/redhat-sso70-openshift><imagestream/fuse7-karaf-openshift>Internal
        error occurred: abc.noregistry.com/fuse7/fuse-karaf-openshift:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/fuse7-karaf-openshift><imagestream/jboss-datavirt64-driver-openshift>Internal
        error occurred: abc.noregistry.com/jboss-datavirt-6/datavirt64-driver-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datavirt64-driver-openshift><imagestream/nginx>Internal
        error occurred: abc.noregistry.com/rhscl/nginx-110-rhel7:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/nginx><imagestream/jboss-webserver30-tomcat7-openshift>Internal
        error occurred: abc.noregistry.com/jboss-webserver-3/webserver30-tomcat7-openshift:1.1:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver30-tomcat7-openshift><imagestream/rhdm74-decisioncentral-openshift>Internal
        error occurred: abc.noregistry.com/rhdm-7/rhdm74-decisioncentral-openshift:1.0:
        Get https://abc.noregistry.com/v2/: net/http: request canceled while waiting
        for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhdm74-decisioncentral-openshift><imagestream/fuse7-console>Internal
        error occurred: abc.noregistry.com/fuse7/fuse-console:1.0: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/fuse7-console><imagestream/nodejs>Internal
        error occurred: abc.noregistry.com/rhoar-nodejs/nodejs-10:latest: Get https://abc.noregistry.com/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)<imagestream/nodejs>'
      reason: 'jboss-datagrid72-openshift rhdm74-kieserver-openshift fuse-apicurito-generator
        dotnet openjdk-11-rhel7 rhdm74-optaweb-employee-rostering-openshift rhpam74-smartrouter-openshift
        jboss-webserver31-tomcat7-openshift apicurito-ui jenkins jboss-datagrid71-client-openshift
        jboss-datagrid71-openshift jboss-datagrid65-client-openshift jboss-webserver30-tomcat8-openshift
        redhat-sso73-openshift mariadb python jboss-fuse70-eap-openshift java jenkins-agent-maven
        rhpam74-kieserver-openshift php perl jboss-fuse70-karaf-openshift rhpam74-businesscentral-monitoring-openshift
        jboss-eap72-openshift postgresql fuse7-java-openshift redhat-sso71-openshift
        jboss-amq-63 redhat-openjdk18-openshift mongodb jboss-fuse70-java-openshift
        jboss-processserver64-openshift jboss-webserver31-tomcat8-openshift jboss-eap70-openshift
        ruby jboss-eap71-openshift jboss-eap64-openshift dotnet-runtime redhat-sso70-openshift
        fuse7-karaf-openshift jboss-datavirt64-driver-openshift nginx jboss-webserver30-tomcat7-openshift
        rhdm74-decisioncentral-openshift fuse7-console nodejs '
      status: "True"
      type: ImportImageErrorsExist
    - lastTransitionTime: "2019-12-17T08:04:22Z"
      lastUpdateTime: "2019-12-17T08:04:22Z"
      status: "False"
      type: ImageChangesInProgress
    - lastTransitionTime: "2019-12-17T00:41:46Z"
      lastUpdateTime: "2019-12-17T00:41:46Z"
      status: "True"
      type: SamplesExist
    - lastTransitionTime: "2019-12-17T00:41:43Z"
      lastUpdateTime: "2019-12-17T00:41:43Z"
      status: "False"
      type: RemovePending
    - lastTransitionTime: "2019-12-17T00:41:43Z"
      lastUpdateTime: "2019-12-17T00:41:43Z"
      status: "False"
      type: MigrationInProgress
    managementState: Managed
    version: 4.4.0-0.ci-2019-12-14-210519
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Expected results:


Additional info:
Take jenkins imagestream as example, see details
http://pastebin.test.redhat.com/822375

This issue could reproduce in 4.3.0-0.nightly-2019-12-13-180405
Comment 1 Gabe Montero 2020-01-15 15:50:05 UTC 
waiting on 4.3.z to open up so https://github.com/openshift/cluster-samples-operator/pull/219 can be associated with it
Comment 4 XiuJuan Wang 2020-02-21 03:32:15 UTC 
Test with 4.3.0-0.nightly-2020-02-20-190050
ImportImageErrorsExist could been clean after the the import error disappeared a while.

Mark this bug as verified.
Comment 6 errata-xmlrpc 2020-03-10 23:52:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0676