Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1771321

Summary: [Disconnected] openshift-samples operator is reporting bad status to make the whole installation get failed.
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: SamplesAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.3.0CC: bparees, wzheng
Target Milestone: ---Keywords: Regression
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1772178 (view as bug list) Environment:
Last Closed: 2020-01-23 11:12:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1772178    
Attachments:
Description Flags
cluster-samples-operator.log none

Description Johnny Liu 2019-11-12 07:34:48 UTC 
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:
4.3.0-0.nightly-2019-11-12-000306

Steps to Reproduce:
1. Trigger a disconnected install on barmetal
2.
3.

Actual results:
installation failed. 

$ openshift-install wait-for install-complete --dir '/home/installer2/workspace/Launch Environment Flexy/workdir/install-dir'
level=info msg="Waiting up to 30m0s for the cluster at https://api.jialiu43-dis2.qe.devcluster.openshift.com:6443 to initialize..."

E1111 23:40:23.358556    2370 reflector.go:280] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: the server is currently unable to handle the request (get clusterversions.config.openshift.io)

level=info msg="Cluster operator insights Disabled is True with Disabled: Health reporting is disabled"
level=info msg="Cluster operator openshift-samples Progressing is True with : Samples processing to 4.3.0-0.nightly-2019-11-12-000306"
level=fatal msg="failed to initialize the cluster: Cluster operator openshift-samples is still updating"

# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          172m    Unable to apply 4.3.0-0.nightly-2019-11-12-000306: the cluster operator openshift-samples has not yet successfully rolled out

# oc get co openshift-samples
NAME                VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
openshift-samples             True        True          False      154m

openshift-samples was reporting a bad status to CVO, that would lead to `wait-for install-complete` failure.

# oc get co openshift-samples -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-11-12T04:37:11Z"
  generation: 1
  name: openshift-samples
  resourceVersion: "14353"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/openshift-samples
  uid: ef102837-e4a8-4f5a-83d0-592da6145680
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-11-12T04:37:33Z"
    message: Samples installation successful at 4.3.0-0.nightly-2019-11-12-000306
    status: "True"
    type: Available
  - lastTransitionTime: "2019-11-12T04:37:30Z"
    message: Samples processing to 4.3.0-0.nightly-2019-11-12-000306
    status: "True"
    type: Progressing
  - lastTransitionTime: "2019-11-12T04:37:30Z"
    status: "False"
    type: Degraded
  extension: null
  relatedObjects:
  - group: samples.operator.openshift.io
    name: cluster
    resource: configs
  - group: ""
    name: openshift-cluster-samples-operator
    resource: namespaces
  - group: ""
    name: openshift
    resource: namespaces


Expected results:
In disconnected env, openshift-samples should not fail the whole of installation process. Maybe the idea state should be 'VERSION' is available, but only showing PROGRESSING=True to so that notify user the operator is still in progress. User can do mirror images operations as day 2 operation to get opoenshift-samples get ready.


Additional info:
1. This issue does not happen on 4.2, such as: 4.2.0-0.nightly-2019-11-11-110741
2. This is blocking QE's ci job.
Comment 1 Gabe Montero 2019-11-12 15:42:33 UTC 
given that I see that 4.3.0-0.nightly-2019-11-12-000306 was Accepted by CI, this would appear to be an intermittent thing, or you checked too soon thing

I am not seeing it locally when I try that level.

Based on what is provided, with samples getting created at 

creationTimestamp: "2019-11-12T04:37:11Z"

and the progressing last transition time only a few seconds later

lastTransitionTime: "2019-11-12T04:37:30Z"

either the updates are not happening, or you checked too soon.

I'm going to need 

a) the results from oc get configs.samples -o yaml

b) the results from oc logs -f `oc get pods -o name` -c cluster-samples-operator -n openshift-cluster-samples-operator 

to clarify which it is.
Comment 2 Johnny Liu 2019-11-13 10:26:05 UTC 
Created attachment 1635738 [details]
cluster-samples-operator.log
Comment 3 Johnny Liu 2019-11-13 10:28:23 UTC 
Maybe my comment 0 is not clear enough. 
This issue only happened in restricted network install.

Yesterday's cluster is terminated, I launched a new one, reproduced with the same behaviour.


a). # oc get configs.samples -o yaml
apiVersion: v1
items:
- apiVersion: samples.operator.openshift.io/v1
  kind: Config
  metadata:
    creationTimestamp: "2019-11-13T08:51:22Z"
    finalizers:
    - samples.operator.openshift.io/finalizer
    generation: 1
    name: cluster
    resourceVersion: "9481"
    selfLink: /apis/samples.operator.openshift.io/v1/configs/cluster
    uid: eaad671d-6948-49ef-b435-3da5628741db
  spec:
    architectures:
    - x86_64
    managementState: Managed
  status:
    architectures:
    - x86_64
    conditions:
    - lastTransitionTime: "2019-11-13T08:51:23Z"
      lastUpdateTime: "2019-11-13T08:51:23Z"
      status: "True"
      type: ImportCredentialsExist
    - lastTransitionTime: "2019-11-13T08:51:33Z"
      lastUpdateTime: "2019-11-13T08:51:33Z"
      status: "True"
      type: ConfigurationValid
    - lastTransitionTime: "2019-11-13T08:51:24Z"
      lastUpdateTime: "2019-11-13T08:51:24Z"
      status: "False"
      type: ImportImageErrorsExist
    - lastTransitionTime: "2019-11-13T08:51:33Z"
      lastUpdateTime: "2019-11-13T08:51:33Z"
      reason: 'fuse-apicurito-generator jboss-fuse70-eap-openshift jboss-processserver64-openshift
        rhpam74-businesscentral-monitoring-openshift jboss-datagrid65-client-openshift
        jboss-datagrid65-openshift jboss-eap71-openshift redhat-sso72-openshift jboss-webserver30-tomcat7-openshift
        jboss-datavirt64-openshift php ruby jboss-decisionserver64-openshift dotnet
        mariadb rhdm74-decisioncentral-openshift jboss-datagrid73-openshift dotnet-runtime
        openjdk-11-rhel7 jboss-eap64-openshift redhat-sso70-openshift fis-karaf-openshift
        httpd mongodb apicast-gateway jboss-datagrid72-openshift jboss-eap70-openshift
        rhpam74-businesscentral-openshift jboss-webserver31-tomcat7-openshift jboss-amq-63
        java mysql redis jboss-datagrid71-client-openshift apicurito-ui nginx fuse7-java-openshift
        redhat-openjdk18-openshift eap-cd-openshift jboss-webserver31-tomcat8-openshift
        jboss-fuse70-karaf-openshift jenkins-agent-maven nodejs jboss-amq-62 jboss-datagrid71-openshift
        jboss-fuse70-console rhpam74-kieserver-openshift redhat-sso73-openshift fuse7-karaf-openshift
        jenkins postgresql perl jboss-eap72-openshift fuse7-eap-openshift jboss-fuse70-java-openshift
        golang jenkins-agent-nodejs python rhdm74-kieserver-openshift redhat-sso71-openshift
        jboss-datavirt64-driver-openshift fis-java-openshift fuse7-console jboss-webserver30-tomcat8-openshift
        jboss-webserver50-tomcat9-openshift modern-webapp rhdm74-optaweb-employee-rostering-openshift
        rhpam74-smartrouter-openshift '
      status: "True"
      type: ImageChangesInProgress
    - lastTransitionTime: "2019-11-13T08:51:36Z"
      lastUpdateTime: "2019-11-13T08:51:36Z"
      status: "True"
      type: SamplesExist
    - lastTransitionTime: "2019-11-13T08:51:33Z"
      lastUpdateTime: "2019-11-13T08:51:33Z"
      status: "False"
      type: RemovePending
    - lastTransitionTime: "2019-11-13T08:51:33Z"
      lastUpdateTime: "2019-11-13T08:51:33Z"
      status: "False"
      type: MigrationInProgress
    managementState: Managed
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

b). log is attached.
Comment 4 Gabe Montero 2019-11-13 15:45:15 UTC 
OK the data helped.

Basically Johnny:

- in 4.2, we were only retrying failed imagestream imports once ... so we gave up in your disconnected env quickly, and samples progressing went to false ... so basically you got lucky

- in 4.3, based on customer telemeter data, it was decided that we need retry on a more continual basis (10 minute gap between retries), but with so many imagestreams, we end up in a case where we are always in progress with at least one imagestream in your disconnected env, where the attempts to access registry.redhat.io will always fail

So what do we tell customers to do in a disconnected env for samples?  If you see https://docs.openshift.com/container-platform/4.2/installing/installing_restricted_networks/installing-restricted-networks-preparations.html#installation-restricted-network-samples_installing-restricted-networks-preparations we say that if you are not going to mirror in registry.redhat.io content, you should set the samples operator to Removed 

That said, Ben Parees and I have been discussing the situation this morning ... do we allow samples to stay progressing==true and cause the `wait-for install-complete` failure you noted in the description.

As such, he and I have agreed to adjust samples operator's interpretation of progressing wrt imagestream import.  We'll essentially no longer image import attempt in flight to progressing.
Comment 6 Johnny Liu 2019-11-14 07:14:58 UTC 
> So what do we tell customers to do in a disconnected env for samples?  If
> you see
> https://docs.openshift.com/container-platform/4.2/installing/
> installing_restricted_networks/installing-restricted-networks-preparations.
> html#installation-restricted-network-samples_installing-restricted-networks-
> preparations we say that if you are not going to mirror in
> registry.redhat.io content, you should set the samples operator to Removed 
I tried that, but seem like take no effect.
# oc get configs.samples.operator.openshift.io -n openshift-cluster-samples-operator cluster -o yaml
apiVersion: samples.operator.openshift.io/v1
kind: Config
metadata:
  creationTimestamp: "2019-11-14T05:15:06Z"
  finalizers:
  - samples.operator.openshift.io/finalizer
  generation: 2
  name: cluster
  resourceVersion: "41999"
  selfLink: /apis/samples.operator.openshift.io/v1/configs/cluster
  uid: 8b395671-6112-4caa-86fd-b1ac87a69e91
spec:
  architectures:
  - x86_64
  managementState: Removed
status:
  architectures:
  - x86_64
  conditions:
  - lastTransitionTime: "2019-11-14T05:15:06Z"
    lastUpdateTime: "2019-11-14T05:15:06Z"
    status: "True"
    type: ImportCredentialsExist
  - lastTransitionTime: "2019-11-14T05:15:11Z"
    lastUpdateTime: "2019-11-14T05:15:11Z"
    status: "True"
    type: ConfigurationValid
  - lastTransitionTime: "2019-11-14T05:15:06Z"
    lastUpdateTime: "2019-11-14T05:15:06Z"
    status: "False"
    type: ImportImageErrorsExist
  - lastTransitionTime: "2019-11-14T05:15:11Z"
    lastUpdateTime: "2019-11-14T05:15:11Z"
    reason: 'jboss-webserver50-tomcat9-openshift fis-karaf-openshift jenkins rhpam74-businesscentral-openshift
      redhat-sso70-openshift redhat-sso72-openshift fuse-apicurito-generator java
      modern-webapp nginx redis jboss-datagrid65-client-openshift jboss-datagrid71-openshift
      dotnet apicurito-ui golang jenkins-agent-nodejs mariadb perl jboss-decisionserver64-openshift
      jboss-eap64-openshift eap-cd-openshift jboss-processserver64-openshift redhat-sso73-openshift
      apicast-gateway rhpam74-kieserver-openshift mysql python ruby jboss-datagrid72-openshift
      fuse7-console openjdk-11-rhel7 jboss-fuse70-eap-openshift nodejs jboss-eap70-openshift
      fis-java-openshift jboss-fuse70-console redhat-openjdk18-openshift rhdm74-optaweb-employee-rostering-openshift
      jboss-webserver30-tomcat8-openshift jboss-amq-62 jboss-eap71-openshift jboss-eap72-openshift
      jboss-datagrid71-client-openshift jboss-datavirt64-driver-openshift jboss-fuse70-java-openshift
      jboss-webserver30-tomcat7-openshift jboss-amq-63 httpd rhpam74-smartrouter-openshift
      jenkins-agent-maven jboss-datagrid65-openshift dotnet-runtime jboss-fuse70-karaf-openshift
      rhdm74-kieserver-openshift rhpam74-businesscentral-monitoring-openshift jboss-webserver31-tomcat7-openshift
      jboss-datagrid73-openshift jboss-datavirt64-openshift fuse7-eap-openshift postgresql
      fuse7-java-openshift mongodb php fuse7-karaf-openshift redhat-sso71-openshift
      rhdm74-decisioncentral-openshift jboss-webserver31-tomcat8-openshift '
    status: "True"
    type: ImageChangesInProgress
  - lastTransitionTime: "2019-11-14T05:15:14Z"
    lastUpdateTime: "2019-11-14T05:15:14Z"
    status: "True"
    type: SamplesExist
  - lastTransitionTime: "2019-11-14T05:15:11Z"
    lastUpdateTime: "2019-11-14T05:15:11Z"
    status: "False"
    type: RemovePending
  - lastTransitionTime: "2019-11-14T05:15:11Z"
    lastUpdateTime: "2019-11-14T05:15:11Z"
    status: "False"
    type: MigrationInProgress
  managementState: Managed

Wait for 5 mins, and check again.
# oc get co openshift-samples
NAME                VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
openshift-samples             True        True          False      116m

Then I go back official doc, there is a NOTE:
"""
Any unmirrored imagestreams that are not skipped, or if the Samples Operator is not changed to Removed, will result in the Samples Operator reporting a Degraded status two hours after the imagestream imports start failing.
"""

Do that means I have to wait two hours, so that not fail the whole installation?

> As such, he and I have agreed to adjust samples operator's interpretation of
> progressing wrt imagestream import.  We'll essentially no longer image
> import attempt in flight to progressing.
Great! Thanks for your quick fix.
Comment 7 Gabe Montero 2019-11-14 15:02:13 UTC 
@Johnny - I have a suspicion as to what is going on with the Removed thing, but if you could get me the operator pod logs like you did before and open a separate bug for that,
I'll see if we can change the behavior there.

thanks
Comment 8 Gabe Montero 2019-11-14 18:09:46 UTC 
Actually upon review @Johnny my fix for this problem should also address the issue with Removed not getting processed when we are stuck with image in progress == true

so don't open a new bug, and simply retry setting to Removed as part of verifying this fix.
Comment 9 Johnny Liu 2019-11-15 01:50:32 UTC 
Presently the latest available nightly build is 4.3.0-0.nightly-2019-11-13-233341, which does not include your fix. Waiting for new nightly build.
Comment 10 Johnny Liu 2019-11-18 02:19:38 UTC 
Verified this bug with 4.3.0-0.nightly-2019-11-17-224250, and PASS.

Did not mirror any image stream, trigger a disconnected env, succeed.

Also test "Removed" things:
1. # oc patch config.samples.operator.openshift.io/cluster --patch '{"spec":{"managementState": "Removed"}}' --type=merge
2. # oc get config.samples.operator.openshift.io/cluster -o yaml
apiVersion: samples.operator.openshift.io/v1
kind: Config
metadata:
  creationTimestamp: "2019-11-18T01:53:28Z"
  finalizers:
  - samples.operator.openshift.io/finalizer
  generation: 3
  name: cluster
  resourceVersion: "21029"
  selfLink: /apis/samples.operator.openshift.io/v1/configs/cluster
  uid: 1d03c519-eefc-4231-a619-10f100e0287e
spec:
  architectures:
  - x86_64
  managementState: Removed
  samplesRegistry: jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000
status:
  conditions:
  - lastTransitionTime: "2019-11-18T01:53:28Z"
    lastUpdateTime: "2019-11-18T01:53:28Z"
    status: "True"
    type: ImportCredentialsExist
  - lastTransitionTime: "2019-11-18T01:53:34Z"
    lastUpdateTime: "2019-11-18T01:53:34Z"
    status: "True"
    type: ConfigurationValid
  - lastTransitionTime: "2019-11-18T01:54:12Z"
    lastUpdateTime: "2019-11-18T02:13:32Z"
    message: '<imagestream/dotnet>Internal error occurred: registry.redhat.io/dotnet/dotnet-21-rhel7:2.1:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/dotnet><imagestream/apicurito-ui>Internal
      error occurred: registry.redhat.io/fuse7/fuse-apicurito:1.2: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/apicurito-ui><imagestream/dotnet-runtime>Internal
      error occurred: registry.redhat.io/dotnet/dotnet-21-runtime-rhel7:2.1: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/dotnet-runtime><imagestream/fuse-apicurito-generator>Internal
      error occurred: registry.redhat.io/fuse7/fuse-apicurito-generator:1.2: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/fuse-apicurito-generator><imagestream/fuse7-console>Internal
      error occurred: registry.redhat.io/fuse7/fuse-console:1.0: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/fuse7-console><imagestream/apicast-gateway>Internal
      error occurred: registry.redhat.io/3scale-amp21/apicast-gateway:1.4-2: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/apicast-gateway><imagestream/eap-cd-openshift>Internal
      error occurred: registry.redhat.io/jboss-eap-7-tech-preview/eap-cd-openshift:12.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/eap-cd-openshift><imagestream/fuse7-eap-openshift>Internal
      error occurred: registry.redhat.io/fuse7/fuse-eap-openshift:1.0: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/fuse7-eap-openshift><imagestream/fuse7-java-openshift>Internal
      error occurred: registry.redhat.io/fuse7/fuse-java-openshift:1.0: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/fuse7-java-openshift><imagestream/fuse7-karaf-openshift>Internal
      error occurred: registry.redhat.io/fuse7/fuse-karaf-openshift:1.0: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/fuse7-karaf-openshift><imagestream/java>Internal
      error occurred: registry.redhat.io/openjdk/openjdk-11-rhel7:latest: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/java><imagestream/jboss-datagrid71-client-openshift>Internal
      error occurred: registry.redhat.io/jboss-datagrid-7/datagrid71-client-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datagrid71-client-openshift><imagestream/jboss-amq-62>Internal
      error occurred: registry.redhat.io/jboss-amq-6/amq62-openshift:1.1: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/jboss-amq-62><imagestream/jboss-datagrid65-client-openshift>Internal
      error occurred: registry.redhat.io/jboss-datagrid-6/datagrid65-client-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datagrid65-client-openshift><imagestream/redhat-openjdk18-openshift>Internal
      error occurred: registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/redhat-openjdk18-openshift><imagestream/redis>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/redis-32-rhel7:latest"
      not found<imagestream/redis><imagestream/jboss-datagrid65-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datagrid-6/datagrid65-openshift:1.2"
      not found<imagestream/jboss-datagrid65-openshift><imagestream/jboss-amq-63>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-amq-6/amq63-openshift:1.0"
      not found<imagestream/jboss-amq-63><imagestream/jboss-webserver31-tomcat8-openshift>Internal
      error occurred: registry.redhat.io/jboss-webserver-3/webserver31-tomcat8-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver31-tomcat8-openshift><imagestream/postgresql>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/postgresql-10-rhel7:latest"
      not found<imagestream/postgresql><imagestream/jboss-webserver30-tomcat8-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-webserver-3/webserver30-tomcat8-openshift:1.1"
      not found<imagestream/jboss-webserver30-tomcat8-openshift><imagestream/jboss-webserver50-tomcat9-openshift>Internal
      error occurred: registry.redhat.io/jboss-webserver-5/webserver50-tomcat9-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver50-tomcat9-openshift><imagestream/jboss-eap71-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-eap-7/eap71-openshift:1.1"
      not found<imagestream/jboss-eap71-openshift><imagestream/ruby>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/ruby-23-rhel7:latest"
      not found<imagestream/ruby><imagestream/jboss-processserver64-openshift>Internal
      error occurred: registry.redhat.io/jboss-processserver-6/processserver64-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-processserver64-openshift><imagestream/mongodb>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/mongodb-32-rhel7:latest"
      not found<imagestream/mongodb><imagestream/jboss-fuse70-karaf-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/fuse7/fuse-karaf-openshift:1.0"
      not found<imagestream/jboss-fuse70-karaf-openshift><imagestream/python>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/python-27-rhel7:latest"
      not found<imagestream/python><imagestream/openjdk-11-rhel7>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/openjdk/openjdk-11-rhel7:1.0"
      not found<imagestream/openjdk-11-rhel7><imagestream/jboss-datavirt64-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datavirt-6/datavirt64-openshift:1.0"
      not found<imagestream/jboss-datavirt64-openshift><imagestream/rhpam74-smartrouter-openshift>Internal
      error occurred: registry.redhat.io/rhpam-7/rhpam74-smartrouter-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhpam74-smartrouter-openshift><imagestream/mysql>Internal
      error occurred: registry.redhat.io/rhscl/mysql-57-rhel7:latest: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/mysql><imagestream/jboss-datavirt64-driver-openshift>Internal
      error occurred: registry.redhat.io/jboss-datavirt-6/datavirt64-driver-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datavirt64-driver-openshift><imagestream/modern-webapp>Internal
      error occurred: registry.redhat.io/rhoar-nodejs-tech-preview/rhoar-nodejs-10-webapp:latest:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/modern-webapp><imagestream/jboss-webserver31-tomcat7-openshift>Internal
      error occurred: registry.redhat.io/jboss-webserver-3/webserver31-tomcat7-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver31-tomcat7-openshift><imagestream/jboss-decisionserver64-openshift>Internal
      error occurred: registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-decisionserver64-openshift><imagestream/rhdm74-kieserver-openshift>Internal
      error occurred: registry.redhat.io/rhdm-7/rhdm74-kieserver-openshift:1.0: Get
      https://registry.redhat.io/v2/: net/http: request canceled while waiting for
      connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhdm74-kieserver-openshift><imagestream/jboss-datagrid73-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datagrid-7/datagrid73-openshift:1.0"
      not found<imagestream/jboss-datagrid73-openshift><imagestream/jboss-webserver30-tomcat7-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-webserver-3/webserver30-tomcat7-openshift:1.1"
      not found<imagestream/jboss-webserver30-tomcat7-openshift><imagestream/perl>Internal
      error occurred: registry.redhat.io/rhscl/perl-524-rhel7:latest: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/perl><imagestream/rhpam74-businesscentral-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhpam-7/rhpam74-businesscentral-openshift:1.0"
      not found<imagestream/rhpam74-businesscentral-openshift><imagestream/jboss-eap70-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-eap-7/eap70-openshift:1.3"
      not found<imagestream/jboss-eap70-openshift><imagestream/rhdm74-optaweb-employee-rostering-openshift>Internal
      error occurred: registry.redhat.io/rhdm-7-tech-preview/rhdm74-optaweb-employee-rostering-openshift:1.0:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhdm74-optaweb-employee-rostering-openshift><imagestream/redhat-sso71-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/redhat-sso-7/sso71-openshift:1.0"
      not found<imagestream/redhat-sso71-openshift><imagestream/jboss-fuse70-java-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/fuse7/fuse-java-openshift:1.0"
      not found<imagestream/jboss-fuse70-java-openshift><imagestream/jboss-eap72-openshift>Internal
      error occurred: registry.redhat.io/jboss-eap-7/eap72-openshift:1.0: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/jboss-eap72-openshift><imagestream/rhpam74-kieserver-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhpam-7/rhpam74-kieserver-openshift:1.0"
      not found<imagestream/rhpam74-kieserver-openshift><imagestream/redhat-sso72-openshift>Internal
      error occurred: registry.redhat.io/redhat-sso-7/sso72-openshift:1.0: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/redhat-sso72-openshift><imagestream/jboss-datagrid72-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datagrid-7/datagrid72-openshift:1.0"
      not found<imagestream/jboss-datagrid72-openshift><imagestream/jboss-fuse70-console>Internal
      error occurred: registry.redhat.io/fuse7/fuse-console:1.0: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/jboss-fuse70-console><imagestream/jboss-fuse70-eap-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/fuse7/fuse-eap-openshift:1.0"
      not found<imagestream/jboss-fuse70-eap-openshift><imagestream/nodejs>Internal
      error occurred: registry.redhat.io/rhoar-nodejs/nodejs-10:latest: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/nodejs><imagestream/redhat-sso70-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/redhat-sso-7/sso70-openshift:1.3"
      not found<imagestream/redhat-sso70-openshift><imagestream/redhat-sso73-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/redhat-sso-7/sso73-openshift:1.0"
      not found<imagestream/redhat-sso73-openshift><imagestream/httpd>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/httpd-24-rhel7:latest"
      not found<imagestream/httpd><imagestream/mariadb>Internal error occurred: registry.redhat.io/rhscl/mariadb-102-rhel7:latest:
      Get https://registry.redhat.io/v2/: net/http: request canceled while waiting
      for connection (Client.Timeout exceeded while awaiting headers)<imagestream/mariadb><imagestream/fis-java-openshift>Internal
      error occurred: registry.redhat.io/jboss-fuse-6/fis-java-openshift:1.0: Get
      https://registry.redhat.io/v2/: net/http: request canceled while waiting for
      connection (Client.Timeout exceeded while awaiting headers)<imagestream/fis-java-openshift><imagestream/php>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/php-70-rhel7:latest"
      not found<imagestream/php><imagestream/jboss-eap64-openshift>Internal error
      occurred: registry.redhat.io/jboss-eap-6/eap64-openshift:1.1: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/jboss-eap64-openshift><imagestream/nginx>Internal
      error occurred: registry.redhat.io/rhscl/nginx-110-rhel7:latest: Get https://registry.redhat.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)<imagestream/nginx><imagestream/rhdm74-decisioncentral-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhdm-7/rhdm74-decisioncentral-openshift:1.0"
      not found<imagestream/rhdm74-decisioncentral-openshift><imagestream/golang>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/devtools/go-toolset-rhel7:1.11.5"
      not found<imagestream/golang><imagestream/jboss-datagrid71-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datagrid-7/datagrid71-openshift:1.0"
      not found<imagestream/jboss-datagrid71-openshift><imagestream/rhpam74-businesscentral-monitoring-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhpam-7/rhpam74-businesscentral-monitoring-openshift:1.0"
      not found<imagestream/rhpam74-businesscentral-monitoring-openshift><imagestream/fis-karaf-openshift>dockerimage.image.openshift.io
      "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-fuse-6/fis-karaf-openshift:1.0"
      not found<imagestream/fis-karaf-openshift>'
    reason: 'dotnet apicurito-ui dotnet-runtime fuse-apicurito-generator fuse7-console
      apicast-gateway eap-cd-openshift fuse7-eap-openshift fuse7-java-openshift fuse7-karaf-openshift
      java jboss-datagrid71-client-openshift jboss-amq-62 jboss-datagrid65-client-openshift
      redhat-openjdk18-openshift redis jboss-datagrid65-openshift jboss-amq-63 jboss-webserver31-tomcat8-openshift
      postgresql jboss-webserver30-tomcat8-openshift jboss-webserver50-tomcat9-openshift
      jboss-eap71-openshift ruby jboss-processserver64-openshift mongodb jboss-fuse70-karaf-openshift
      python openjdk-11-rhel7 jboss-datavirt64-openshift rhpam74-smartrouter-openshift
      mysql jboss-datavirt64-driver-openshift modern-webapp jboss-webserver31-tomcat7-openshift
      jboss-decisionserver64-openshift rhdm74-kieserver-openshift jboss-datagrid73-openshift
      jboss-webserver30-tomcat7-openshift perl rhpam74-businesscentral-openshift jboss-eap70-openshift
      rhdm74-optaweb-employee-rostering-openshift redhat-sso71-openshift jboss-fuse70-java-openshift
      jboss-eap72-openshift rhpam74-kieserver-openshift redhat-sso72-openshift jboss-datagrid72-openshift
      jboss-fuse70-console jboss-fuse70-eap-openshift nodejs redhat-sso70-openshift
      redhat-sso73-openshift httpd mariadb fis-java-openshift php jboss-eap64-openshift
      nginx rhdm74-decisioncentral-openshift golang jboss-datagrid71-openshift rhpam74-businesscentral-monitoring-openshift
      fis-karaf-openshift '
    status: "True"
    type: ImportImageErrorsExist
  - lastTransitionTime: "2019-11-18T02:10:15Z"
    lastUpdateTime: "2019-11-18T02:10:15Z"
    status: "False"
    type: ImageChangesInProgress
  - lastTransitionTime: "2019-11-18T02:16:02Z"
    lastUpdateTime: "2019-11-18T02:16:02Z"
    status: "False"
    type: SamplesExist
  - lastTransitionTime: "2019-11-18T02:16:05Z"
    lastUpdateTime: "2019-11-18T02:16:05Z"
    status: "False"
    type: RemovePending
  - lastTransitionTime: "2019-11-18T01:53:34Z"
    lastUpdateTime: "2019-11-18T01:53:34Z"
    status: "False"
    type: MigrationInProgress
  managementState: Removed
  samplesRegistry: jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000
  version: 4.3.0-0.nightly-2019-11-17-224250
# oc get co openshift-samples
NAME                VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
openshift-samples   4.3.0-0.nightly-2019-11-17-224250   True        False         False      2m49s

3. # oc get is ruby -n openshift
Error from server (NotFound): imagestreams.image.openshift.io "ruby" not found
Comment 12 errata-xmlrpc 2020-01-23 11:12:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062