Description of problem: Version-Release number of selected component (if applicable): How reproducible: 4.3.0-0.nightly-2019-11-12-000306 Steps to Reproduce: 1. Trigger a disconnected install on barmetal 2. 3. Actual results: installation failed. $ openshift-install wait-for install-complete --dir '/home/installer2/workspace/Launch Environment Flexy/workdir/install-dir' level=info msg="Waiting up to 30m0s for the cluster at https://api.jialiu43-dis2.qe.devcluster.openshift.com:6443 to initialize..." E1111 23:40:23.358556 2370 reflector.go:280] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *v1.ClusterVersion: the server is currently unable to handle the request (get clusterversions.config.openshift.io) level=info msg="Cluster operator insights Disabled is True with Disabled: Health reporting is disabled" level=info msg="Cluster operator openshift-samples Progressing is True with : Samples processing to 4.3.0-0.nightly-2019-11-12-000306" level=fatal msg="failed to initialize the cluster: Cluster operator openshift-samples is still updating" # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 172m Unable to apply 4.3.0-0.nightly-2019-11-12-000306: the cluster operator openshift-samples has not yet successfully rolled out # oc get co openshift-samples NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE openshift-samples True True False 154m openshift-samples was reporting a bad status to CVO, that would lead to `wait-for install-complete` failure. # oc get co openshift-samples -o yaml apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: creationTimestamp: "2019-11-12T04:37:11Z" generation: 1 name: openshift-samples resourceVersion: "14353" selfLink: /apis/config.openshift.io/v1/clusteroperators/openshift-samples uid: ef102837-e4a8-4f5a-83d0-592da6145680 spec: {} status: conditions: - lastTransitionTime: "2019-11-12T04:37:33Z" message: Samples installation successful at 4.3.0-0.nightly-2019-11-12-000306 status: "True" type: Available - lastTransitionTime: "2019-11-12T04:37:30Z" message: Samples processing to 4.3.0-0.nightly-2019-11-12-000306 status: "True" type: Progressing - lastTransitionTime: "2019-11-12T04:37:30Z" status: "False" type: Degraded extension: null relatedObjects: - group: samples.operator.openshift.io name: cluster resource: configs - group: "" name: openshift-cluster-samples-operator resource: namespaces - group: "" name: openshift resource: namespaces Expected results: In disconnected env, openshift-samples should not fail the whole of installation process. Maybe the idea state should be 'VERSION' is available, but only showing PROGRESSING=True to so that notify user the operator is still in progress. User can do mirror images operations as day 2 operation to get opoenshift-samples get ready. Additional info: 1. This issue does not happen on 4.2, such as: 4.2.0-0.nightly-2019-11-11-110741 2. This is blocking QE's ci job.
given that I see that 4.3.0-0.nightly-2019-11-12-000306 was Accepted by CI, this would appear to be an intermittent thing, or you checked too soon thing I am not seeing it locally when I try that level. Based on what is provided, with samples getting created at creationTimestamp: "2019-11-12T04:37:11Z" and the progressing last transition time only a few seconds later lastTransitionTime: "2019-11-12T04:37:30Z" either the updates are not happening, or you checked too soon. I'm going to need a) the results from oc get configs.samples -o yaml b) the results from oc logs -f `oc get pods -o name` -c cluster-samples-operator -n openshift-cluster-samples-operator to clarify which it is.
Created attachment 1635738 [details] cluster-samples-operator.log
Maybe my comment 0 is not clear enough. This issue only happened in restricted network install. Yesterday's cluster is terminated, I launched a new one, reproduced with the same behaviour. a). # oc get configs.samples -o yaml apiVersion: v1 items: - apiVersion: samples.operator.openshift.io/v1 kind: Config metadata: creationTimestamp: "2019-11-13T08:51:22Z" finalizers: - samples.operator.openshift.io/finalizer generation: 1 name: cluster resourceVersion: "9481" selfLink: /apis/samples.operator.openshift.io/v1/configs/cluster uid: eaad671d-6948-49ef-b435-3da5628741db spec: architectures: - x86_64 managementState: Managed status: architectures: - x86_64 conditions: - lastTransitionTime: "2019-11-13T08:51:23Z" lastUpdateTime: "2019-11-13T08:51:23Z" status: "True" type: ImportCredentialsExist - lastTransitionTime: "2019-11-13T08:51:33Z" lastUpdateTime: "2019-11-13T08:51:33Z" status: "True" type: ConfigurationValid - lastTransitionTime: "2019-11-13T08:51:24Z" lastUpdateTime: "2019-11-13T08:51:24Z" status: "False" type: ImportImageErrorsExist - lastTransitionTime: "2019-11-13T08:51:33Z" lastUpdateTime: "2019-11-13T08:51:33Z" reason: 'fuse-apicurito-generator jboss-fuse70-eap-openshift jboss-processserver64-openshift rhpam74-businesscentral-monitoring-openshift jboss-datagrid65-client-openshift jboss-datagrid65-openshift jboss-eap71-openshift redhat-sso72-openshift jboss-webserver30-tomcat7-openshift jboss-datavirt64-openshift php ruby jboss-decisionserver64-openshift dotnet mariadb rhdm74-decisioncentral-openshift jboss-datagrid73-openshift dotnet-runtime openjdk-11-rhel7 jboss-eap64-openshift redhat-sso70-openshift fis-karaf-openshift httpd mongodb apicast-gateway jboss-datagrid72-openshift jboss-eap70-openshift rhpam74-businesscentral-openshift jboss-webserver31-tomcat7-openshift jboss-amq-63 java mysql redis jboss-datagrid71-client-openshift apicurito-ui nginx fuse7-java-openshift redhat-openjdk18-openshift eap-cd-openshift jboss-webserver31-tomcat8-openshift jboss-fuse70-karaf-openshift jenkins-agent-maven nodejs jboss-amq-62 jboss-datagrid71-openshift jboss-fuse70-console rhpam74-kieserver-openshift redhat-sso73-openshift fuse7-karaf-openshift jenkins postgresql perl jboss-eap72-openshift fuse7-eap-openshift jboss-fuse70-java-openshift golang jenkins-agent-nodejs python rhdm74-kieserver-openshift redhat-sso71-openshift jboss-datavirt64-driver-openshift fis-java-openshift fuse7-console jboss-webserver30-tomcat8-openshift jboss-webserver50-tomcat9-openshift modern-webapp rhdm74-optaweb-employee-rostering-openshift rhpam74-smartrouter-openshift ' status: "True" type: ImageChangesInProgress - lastTransitionTime: "2019-11-13T08:51:36Z" lastUpdateTime: "2019-11-13T08:51:36Z" status: "True" type: SamplesExist - lastTransitionTime: "2019-11-13T08:51:33Z" lastUpdateTime: "2019-11-13T08:51:33Z" status: "False" type: RemovePending - lastTransitionTime: "2019-11-13T08:51:33Z" lastUpdateTime: "2019-11-13T08:51:33Z" status: "False" type: MigrationInProgress managementState: Managed kind: List metadata: resourceVersion: "" selfLink: "" b). log is attached.
OK the data helped. Basically Johnny: - in 4.2, we were only retrying failed imagestream imports once ... so we gave up in your disconnected env quickly, and samples progressing went to false ... so basically you got lucky - in 4.3, based on customer telemeter data, it was decided that we need retry on a more continual basis (10 minute gap between retries), but with so many imagestreams, we end up in a case where we are always in progress with at least one imagestream in your disconnected env, where the attempts to access registry.redhat.io will always fail So what do we tell customers to do in a disconnected env for samples? If you see https://docs.openshift.com/container-platform/4.2/installing/installing_restricted_networks/installing-restricted-networks-preparations.html#installation-restricted-network-samples_installing-restricted-networks-preparations we say that if you are not going to mirror in registry.redhat.io content, you should set the samples operator to Removed That said, Ben Parees and I have been discussing the situation this morning ... do we allow samples to stay progressing==true and cause the `wait-for install-complete` failure you noted in the description. As such, he and I have agreed to adjust samples operator's interpretation of progressing wrt imagestream import. We'll essentially no longer image import attempt in flight to progressing.
> So what do we tell customers to do in a disconnected env for samples? If > you see > https://docs.openshift.com/container-platform/4.2/installing/ > installing_restricted_networks/installing-restricted-networks-preparations. > html#installation-restricted-network-samples_installing-restricted-networks- > preparations we say that if you are not going to mirror in > registry.redhat.io content, you should set the samples operator to Removed I tried that, but seem like take no effect. # oc get configs.samples.operator.openshift.io -n openshift-cluster-samples-operator cluster -o yaml apiVersion: samples.operator.openshift.io/v1 kind: Config metadata: creationTimestamp: "2019-11-14T05:15:06Z" finalizers: - samples.operator.openshift.io/finalizer generation: 2 name: cluster resourceVersion: "41999" selfLink: /apis/samples.operator.openshift.io/v1/configs/cluster uid: 8b395671-6112-4caa-86fd-b1ac87a69e91 spec: architectures: - x86_64 managementState: Removed status: architectures: - x86_64 conditions: - lastTransitionTime: "2019-11-14T05:15:06Z" lastUpdateTime: "2019-11-14T05:15:06Z" status: "True" type: ImportCredentialsExist - lastTransitionTime: "2019-11-14T05:15:11Z" lastUpdateTime: "2019-11-14T05:15:11Z" status: "True" type: ConfigurationValid - lastTransitionTime: "2019-11-14T05:15:06Z" lastUpdateTime: "2019-11-14T05:15:06Z" status: "False" type: ImportImageErrorsExist - lastTransitionTime: "2019-11-14T05:15:11Z" lastUpdateTime: "2019-11-14T05:15:11Z" reason: 'jboss-webserver50-tomcat9-openshift fis-karaf-openshift jenkins rhpam74-businesscentral-openshift redhat-sso70-openshift redhat-sso72-openshift fuse-apicurito-generator java modern-webapp nginx redis jboss-datagrid65-client-openshift jboss-datagrid71-openshift dotnet apicurito-ui golang jenkins-agent-nodejs mariadb perl jboss-decisionserver64-openshift jboss-eap64-openshift eap-cd-openshift jboss-processserver64-openshift redhat-sso73-openshift apicast-gateway rhpam74-kieserver-openshift mysql python ruby jboss-datagrid72-openshift fuse7-console openjdk-11-rhel7 jboss-fuse70-eap-openshift nodejs jboss-eap70-openshift fis-java-openshift jboss-fuse70-console redhat-openjdk18-openshift rhdm74-optaweb-employee-rostering-openshift jboss-webserver30-tomcat8-openshift jboss-amq-62 jboss-eap71-openshift jboss-eap72-openshift jboss-datagrid71-client-openshift jboss-datavirt64-driver-openshift jboss-fuse70-java-openshift jboss-webserver30-tomcat7-openshift jboss-amq-63 httpd rhpam74-smartrouter-openshift jenkins-agent-maven jboss-datagrid65-openshift dotnet-runtime jboss-fuse70-karaf-openshift rhdm74-kieserver-openshift rhpam74-businesscentral-monitoring-openshift jboss-webserver31-tomcat7-openshift jboss-datagrid73-openshift jboss-datavirt64-openshift fuse7-eap-openshift postgresql fuse7-java-openshift mongodb php fuse7-karaf-openshift redhat-sso71-openshift rhdm74-decisioncentral-openshift jboss-webserver31-tomcat8-openshift ' status: "True" type: ImageChangesInProgress - lastTransitionTime: "2019-11-14T05:15:14Z" lastUpdateTime: "2019-11-14T05:15:14Z" status: "True" type: SamplesExist - lastTransitionTime: "2019-11-14T05:15:11Z" lastUpdateTime: "2019-11-14T05:15:11Z" status: "False" type: RemovePending - lastTransitionTime: "2019-11-14T05:15:11Z" lastUpdateTime: "2019-11-14T05:15:11Z" status: "False" type: MigrationInProgress managementState: Managed Wait for 5 mins, and check again. # oc get co openshift-samples NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE openshift-samples True True False 116m Then I go back official doc, there is a NOTE: """ Any unmirrored imagestreams that are not skipped, or if the Samples Operator is not changed to Removed, will result in the Samples Operator reporting a Degraded status two hours after the imagestream imports start failing. """ Do that means I have to wait two hours, so that not fail the whole installation? > As such, he and I have agreed to adjust samples operator's interpretation of > progressing wrt imagestream import. We'll essentially no longer image > import attempt in flight to progressing. Great! Thanks for your quick fix.
@Johnny - I have a suspicion as to what is going on with the Removed thing, but if you could get me the operator pod logs like you did before and open a separate bug for that, I'll see if we can change the behavior there. thanks
Actually upon review @Johnny my fix for this problem should also address the issue with Removed not getting processed when we are stuck with image in progress == true so don't open a new bug, and simply retry setting to Removed as part of verifying this fix.
Presently the latest available nightly build is 4.3.0-0.nightly-2019-11-13-233341, which does not include your fix. Waiting for new nightly build.
Verified this bug with 4.3.0-0.nightly-2019-11-17-224250, and PASS. Did not mirror any image stream, trigger a disconnected env, succeed. Also test "Removed" things: 1. # oc patch config.samples.operator.openshift.io/cluster --patch '{"spec":{"managementState": "Removed"}}' --type=merge 2. # oc get config.samples.operator.openshift.io/cluster -o yaml apiVersion: samples.operator.openshift.io/v1 kind: Config metadata: creationTimestamp: "2019-11-18T01:53:28Z" finalizers: - samples.operator.openshift.io/finalizer generation: 3 name: cluster resourceVersion: "21029" selfLink: /apis/samples.operator.openshift.io/v1/configs/cluster uid: 1d03c519-eefc-4231-a619-10f100e0287e spec: architectures: - x86_64 managementState: Removed samplesRegistry: jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000 status: conditions: - lastTransitionTime: "2019-11-18T01:53:28Z" lastUpdateTime: "2019-11-18T01:53:28Z" status: "True" type: ImportCredentialsExist - lastTransitionTime: "2019-11-18T01:53:34Z" lastUpdateTime: "2019-11-18T01:53:34Z" status: "True" type: ConfigurationValid - lastTransitionTime: "2019-11-18T01:54:12Z" lastUpdateTime: "2019-11-18T02:13:32Z" message: '<imagestream/dotnet>Internal error occurred: registry.redhat.io/dotnet/dotnet-21-rhel7:2.1: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/dotnet><imagestream/apicurito-ui>Internal error occurred: registry.redhat.io/fuse7/fuse-apicurito:1.2: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/apicurito-ui><imagestream/dotnet-runtime>Internal error occurred: registry.redhat.io/dotnet/dotnet-21-runtime-rhel7:2.1: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/dotnet-runtime><imagestream/fuse-apicurito-generator>Internal error occurred: registry.redhat.io/fuse7/fuse-apicurito-generator:1.2: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/fuse-apicurito-generator><imagestream/fuse7-console>Internal error occurred: registry.redhat.io/fuse7/fuse-console:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/fuse7-console><imagestream/apicast-gateway>Internal error occurred: registry.redhat.io/3scale-amp21/apicast-gateway:1.4-2: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/apicast-gateway><imagestream/eap-cd-openshift>Internal error occurred: registry.redhat.io/jboss-eap-7-tech-preview/eap-cd-openshift:12.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/eap-cd-openshift><imagestream/fuse7-eap-openshift>Internal error occurred: registry.redhat.io/fuse7/fuse-eap-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/fuse7-eap-openshift><imagestream/fuse7-java-openshift>Internal error occurred: registry.redhat.io/fuse7/fuse-java-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/fuse7-java-openshift><imagestream/fuse7-karaf-openshift>Internal error occurred: registry.redhat.io/fuse7/fuse-karaf-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/fuse7-karaf-openshift><imagestream/java>Internal error occurred: registry.redhat.io/openjdk/openjdk-11-rhel7:latest: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/java><imagestream/jboss-datagrid71-client-openshift>Internal error occurred: registry.redhat.io/jboss-datagrid-7/datagrid71-client-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datagrid71-client-openshift><imagestream/jboss-amq-62>Internal error occurred: registry.redhat.io/jboss-amq-6/amq62-openshift:1.1: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-amq-62><imagestream/jboss-datagrid65-client-openshift>Internal error occurred: registry.redhat.io/jboss-datagrid-6/datagrid65-client-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datagrid65-client-openshift><imagestream/redhat-openjdk18-openshift>Internal error occurred: registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/redhat-openjdk18-openshift><imagestream/redis>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/redis-32-rhel7:latest" not found<imagestream/redis><imagestream/jboss-datagrid65-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datagrid-6/datagrid65-openshift:1.2" not found<imagestream/jboss-datagrid65-openshift><imagestream/jboss-amq-63>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-amq-6/amq63-openshift:1.0" not found<imagestream/jboss-amq-63><imagestream/jboss-webserver31-tomcat8-openshift>Internal error occurred: registry.redhat.io/jboss-webserver-3/webserver31-tomcat8-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver31-tomcat8-openshift><imagestream/postgresql>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/postgresql-10-rhel7:latest" not found<imagestream/postgresql><imagestream/jboss-webserver30-tomcat8-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-webserver-3/webserver30-tomcat8-openshift:1.1" not found<imagestream/jboss-webserver30-tomcat8-openshift><imagestream/jboss-webserver50-tomcat9-openshift>Internal error occurred: registry.redhat.io/jboss-webserver-5/webserver50-tomcat9-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver50-tomcat9-openshift><imagestream/jboss-eap71-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-eap-7/eap71-openshift:1.1" not found<imagestream/jboss-eap71-openshift><imagestream/ruby>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/ruby-23-rhel7:latest" not found<imagestream/ruby><imagestream/jboss-processserver64-openshift>Internal error occurred: registry.redhat.io/jboss-processserver-6/processserver64-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-processserver64-openshift><imagestream/mongodb>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/mongodb-32-rhel7:latest" not found<imagestream/mongodb><imagestream/jboss-fuse70-karaf-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/fuse7/fuse-karaf-openshift:1.0" not found<imagestream/jboss-fuse70-karaf-openshift><imagestream/python>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/python-27-rhel7:latest" not found<imagestream/python><imagestream/openjdk-11-rhel7>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/openjdk/openjdk-11-rhel7:1.0" not found<imagestream/openjdk-11-rhel7><imagestream/jboss-datavirt64-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datavirt-6/datavirt64-openshift:1.0" not found<imagestream/jboss-datavirt64-openshift><imagestream/rhpam74-smartrouter-openshift>Internal error occurred: registry.redhat.io/rhpam-7/rhpam74-smartrouter-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhpam74-smartrouter-openshift><imagestream/mysql>Internal error occurred: registry.redhat.io/rhscl/mysql-57-rhel7:latest: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/mysql><imagestream/jboss-datavirt64-driver-openshift>Internal error occurred: registry.redhat.io/jboss-datavirt-6/datavirt64-driver-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-datavirt64-driver-openshift><imagestream/modern-webapp>Internal error occurred: registry.redhat.io/rhoar-nodejs-tech-preview/rhoar-nodejs-10-webapp:latest: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/modern-webapp><imagestream/jboss-webserver31-tomcat7-openshift>Internal error occurred: registry.redhat.io/jboss-webserver-3/webserver31-tomcat7-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-webserver31-tomcat7-openshift><imagestream/jboss-decisionserver64-openshift>Internal error occurred: registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-decisionserver64-openshift><imagestream/rhdm74-kieserver-openshift>Internal error occurred: registry.redhat.io/rhdm-7/rhdm74-kieserver-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhdm74-kieserver-openshift><imagestream/jboss-datagrid73-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datagrid-7/datagrid73-openshift:1.0" not found<imagestream/jboss-datagrid73-openshift><imagestream/jboss-webserver30-tomcat7-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-webserver-3/webserver30-tomcat7-openshift:1.1" not found<imagestream/jboss-webserver30-tomcat7-openshift><imagestream/perl>Internal error occurred: registry.redhat.io/rhscl/perl-524-rhel7:latest: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/perl><imagestream/rhpam74-businesscentral-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhpam-7/rhpam74-businesscentral-openshift:1.0" not found<imagestream/rhpam74-businesscentral-openshift><imagestream/jboss-eap70-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-eap-7/eap70-openshift:1.3" not found<imagestream/jboss-eap70-openshift><imagestream/rhdm74-optaweb-employee-rostering-openshift>Internal error occurred: registry.redhat.io/rhdm-7-tech-preview/rhdm74-optaweb-employee-rostering-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/rhdm74-optaweb-employee-rostering-openshift><imagestream/redhat-sso71-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/redhat-sso-7/sso71-openshift:1.0" not found<imagestream/redhat-sso71-openshift><imagestream/jboss-fuse70-java-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/fuse7/fuse-java-openshift:1.0" not found<imagestream/jboss-fuse70-java-openshift><imagestream/jboss-eap72-openshift>Internal error occurred: registry.redhat.io/jboss-eap-7/eap72-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-eap72-openshift><imagestream/rhpam74-kieserver-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhpam-7/rhpam74-kieserver-openshift:1.0" not found<imagestream/rhpam74-kieserver-openshift><imagestream/redhat-sso72-openshift>Internal error occurred: registry.redhat.io/redhat-sso-7/sso72-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/redhat-sso72-openshift><imagestream/jboss-datagrid72-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datagrid-7/datagrid72-openshift:1.0" not found<imagestream/jboss-datagrid72-openshift><imagestream/jboss-fuse70-console>Internal error occurred: registry.redhat.io/fuse7/fuse-console:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-fuse70-console><imagestream/jboss-fuse70-eap-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/fuse7/fuse-eap-openshift:1.0" not found<imagestream/jboss-fuse70-eap-openshift><imagestream/nodejs>Internal error occurred: registry.redhat.io/rhoar-nodejs/nodejs-10:latest: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/nodejs><imagestream/redhat-sso70-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/redhat-sso-7/sso70-openshift:1.3" not found<imagestream/redhat-sso70-openshift><imagestream/redhat-sso73-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/redhat-sso-7/sso73-openshift:1.0" not found<imagestream/redhat-sso73-openshift><imagestream/httpd>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/httpd-24-rhel7:latest" not found<imagestream/httpd><imagestream/mariadb>Internal error occurred: registry.redhat.io/rhscl/mariadb-102-rhel7:latest: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/mariadb><imagestream/fis-java-openshift>Internal error occurred: registry.redhat.io/jboss-fuse-6/fis-java-openshift:1.0: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/fis-java-openshift><imagestream/php>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhscl/php-70-rhel7:latest" not found<imagestream/php><imagestream/jboss-eap64-openshift>Internal error occurred: registry.redhat.io/jboss-eap-6/eap64-openshift:1.1: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/jboss-eap64-openshift><imagestream/nginx>Internal error occurred: registry.redhat.io/rhscl/nginx-110-rhel7:latest: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)<imagestream/nginx><imagestream/rhdm74-decisioncentral-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhdm-7/rhdm74-decisioncentral-openshift:1.0" not found<imagestream/rhdm74-decisioncentral-openshift><imagestream/golang>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/devtools/go-toolset-rhel7:1.11.5" not found<imagestream/golang><imagestream/jboss-datagrid71-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-datagrid-7/datagrid71-openshift:1.0" not found<imagestream/jboss-datagrid71-openshift><imagestream/rhpam74-businesscentral-monitoring-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rhpam-7/rhpam74-businesscentral-monitoring-openshift:1.0" not found<imagestream/rhpam74-businesscentral-monitoring-openshift><imagestream/fis-karaf-openshift>dockerimage.image.openshift.io "jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000/jboss-fuse-6/fis-karaf-openshift:1.0" not found<imagestream/fis-karaf-openshift>' reason: 'dotnet apicurito-ui dotnet-runtime fuse-apicurito-generator fuse7-console apicast-gateway eap-cd-openshift fuse7-eap-openshift fuse7-java-openshift fuse7-karaf-openshift java jboss-datagrid71-client-openshift jboss-amq-62 jboss-datagrid65-client-openshift redhat-openjdk18-openshift redis jboss-datagrid65-openshift jboss-amq-63 jboss-webserver31-tomcat8-openshift postgresql jboss-webserver30-tomcat8-openshift jboss-webserver50-tomcat9-openshift jboss-eap71-openshift ruby jboss-processserver64-openshift mongodb jboss-fuse70-karaf-openshift python openjdk-11-rhel7 jboss-datavirt64-openshift rhpam74-smartrouter-openshift mysql jboss-datavirt64-driver-openshift modern-webapp jboss-webserver31-tomcat7-openshift jboss-decisionserver64-openshift rhdm74-kieserver-openshift jboss-datagrid73-openshift jboss-webserver30-tomcat7-openshift perl rhpam74-businesscentral-openshift jboss-eap70-openshift rhdm74-optaweb-employee-rostering-openshift redhat-sso71-openshift jboss-fuse70-java-openshift jboss-eap72-openshift rhpam74-kieserver-openshift redhat-sso72-openshift jboss-datagrid72-openshift jboss-fuse70-console jboss-fuse70-eap-openshift nodejs redhat-sso70-openshift redhat-sso73-openshift httpd mariadb fis-java-openshift php jboss-eap64-openshift nginx rhdm74-decisioncentral-openshift golang jboss-datagrid71-openshift rhpam74-businesscentral-monitoring-openshift fis-karaf-openshift ' status: "True" type: ImportImageErrorsExist - lastTransitionTime: "2019-11-18T02:10:15Z" lastUpdateTime: "2019-11-18T02:10:15Z" status: "False" type: ImageChangesInProgress - lastTransitionTime: "2019-11-18T02:16:02Z" lastUpdateTime: "2019-11-18T02:16:02Z" status: "False" type: SamplesExist - lastTransitionTime: "2019-11-18T02:16:05Z" lastUpdateTime: "2019-11-18T02:16:05Z" status: "False" type: RemovePending - lastTransitionTime: "2019-11-18T01:53:34Z" lastUpdateTime: "2019-11-18T01:53:34Z" status: "False" type: MigrationInProgress managementState: Removed samplesRegistry: jialiu43-dis2.mirror-registry.qe.gcp.devcluster.openshift.com:5000 version: 4.3.0-0.nightly-2019-11-17-224250 # oc get co openshift-samples NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE openshift-samples 4.3.0-0.nightly-2019-11-17-224250 True False False 2m49s 3. # oc get is ruby -n openshift Error from server (NotFound): imagestreams.image.openshift.io "ruby" not found
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062