Description of problem: When create IPI on AWS private + disconnected cluster, console operator is reporting OCDownloadsSyncDegraded which will leads to installation failure. normal IPI on AWS works fine Version-Release number of selected component (if applicable): 4.5.0-0.nightly-2020-05-12-224129 How reproducible: Always at least on 4.5.0-0.nightly-2020-05-12-224129 Steps to Reproduce: 1. create IPI on AWS private + disconnected cluster with 4.5.0-0.nightly-2020-05-12-224129, check progress Actual results: 1. Cluster operator console is reporting a failure: OCDownloadsSyncDegraded: ConsoleCLIDownload.console.openshift.io \"oc-cli-downloads\" is invalid: spec.links.href: Invalid value: \"\": spec.links.href in body should match '^https://'" .... level=debug msg="Warning: Applied changes may be incomplete" level=debug level=debug msg="The plan was created with the -target option in effect, so some changes" level=debug msg="requested in the configuration may have been ignored and the output values may" level=debug msg="not be fully updated. Run the following command to verify that no other" level=debug msg="changes are pending:" level=debug msg=" terraform plan" level=debug msg="\t" level=debug msg="Note that the -target option is not suitable for routine use, and is provided" level=debug msg="only for exceptional situations such as recovering from errors or mistakes, or" level=debug msg="when Terraform specifically suggests to use it as part of an error message." level=debug level=debug level=debug msg="Destroy complete! Resources: 12 destroyed." level=debug msg="Fetching Install Config..." level=debug msg="Loading Install Config..." level=debug msg=" Loading SSH Key..." level=debug msg=" Loading Base Domain..." level=debug msg=" Loading Platform..." level=debug msg=" Loading Cluster Name..." level=debug msg=" Loading Base Domain..." level=debug msg=" Loading Platform..." level=debug msg=" Loading Pull Secret..." level=debug msg=" Loading Platform..." level=debug msg="Using Install Config loaded from state file" level=debug msg="Reusing previously-fetched Install Config" level=info msg="Waiting up to 30m0s for the cluster at https://api.qe-yapei451.qe.devcluster.openshift.com:6443 to initialize..." level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 69% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: downloading update" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 2% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 8% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 11% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 14% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 16% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 66% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 67% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 68% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 76% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 83% complete" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 84% complete" level=debug msg="Still waiting for the cluster to initialize: Cluster operator console is reporting a failure: OCDownloadsSyncDegraded: ConsoleCLIDownload.console.openshift.io \"oc-cli-downloads\" is invalid: spec.links.href: Invalid value: \"\": spec.links.href in body should match '^https://'" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.5.0-0.nightly-2020-05-12-224129: 85% complete" level=debug msg="Still waiting for the cluster to initialize: Cluster operator console is reporting a failure: OCDownloadsSyncDegraded: ConsoleCLIDownload.console.openshift.io \"oc-cli-downloads\" is invalid: spec.links.href: Invalid value: \"\": spec.links.href in body should match '^https://'" Expected results: 1. cluster can be successfully created and console operator should report no errors Additional info:
# oc get route downloads -n openshift-console -o json | jq '.spec' { "host": "downloads-openshift-console.apps.qe-lzha.qe.devcluster.openshift.com", "port": { "targetPort": "http" }, "tls": { "insecureEdgeTerminationPolicy": "Redirect", "termination": "edge" }, "to": { "kind": "Service", "name": "downloads", "weight": 100 }, "wildcardPolicy": "None" } # oc get route downloads -n openshift-console -o json | jq '.status' {} # oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication Unknown Unknown True 6h52m cloud-credential 4.5.0-0.nightly-2020-05-12-224129 True False False 6h16m cluster-autoscaler 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m config-operator 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m console 4.5.0-0.nightly-2020-05-12-224129 Unknown True True 6h45m csi-snapshot-controller dns 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m etcd 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m image-registry False True False 6h46m ingress False True True 6h46m insights 4.5.0-0.nightly-2020-05-12-224129 True False False 6h46m kube-apiserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m kube-controller-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m kube-scheduler 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m kube-storage-version-migrator 4.5.0-0.nightly-2020-05-12-224129 False False False 6h53m machine-api 4.5.0-0.nightly-2020-05-12-224129 True False False 6h42m machine-approver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h50m machine-config 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m marketplace 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m monitoring False True True 6h41m network 4.5.0-0.nightly-2020-05-12-224129 True False False 6h53m node-tuning 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m openshift-apiserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h47m openshift-controller-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m openshift-samples 4.5.0-0.nightly-2020-05-12-224129 True False False 6h36m operator-lifecycle-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m operator-lifecycle-manager-catalog 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m operator-lifecycle-manager-packageserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m service-ca 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m storage 4.5.0-0.nightly-2020-05-12-224129 True False False 6h46m # oc describe co console Name: console Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-05-13T03:14:22Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:status: .: f:extension: Manager: cluster-version-operator Operation: Update Time: 2020-05-13T03:14:23Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:relatedObjects: f:versions: Manager: console Operation: Update Time: 2020-05-13T03:27:49Z Resource Version: 14090 Self Link: /apis/config.openshift.io/v1/clusteroperators/console UID: 49c6c1bc-5af4-41df-bc9c-3a714783b9c0 Spec: Status: Conditions: Last Transition Time: 2020-05-13T03:27:49Z Message: OCDownloadsSyncDegraded: ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://' Reason: OCDownloadsSync_FailedCreate Status: True Type: Degraded Last Transition Time: 2020-05-13T03:25:51Z Message: DefaultRouteSyncProgressing: default route is not available at canonical host [] OAuthClientSyncProgressing: waiting on route host Reason: DefaultRouteSync_FailedDefaultRouteHost::OAuthClientSync_FailedHost Status: True Type: Progressing Last Transition Time: 2020-05-13T03:25:49Z Reason: NoData Status: Unknown Type: Available Last Transition Time: 2020-05-13T03:25:49Z Reason: AsExpected Status: True Type: Upgradeable Extension: <nil> Related Objects: Group: operator.openshift.io Name: cluster Resource: consoles Group: config.openshift.io Name: cluster Resource: consoles Group: config.openshift.io Name: cluster Resource: infrastructures Group: config.openshift.io Name: cluster Resource: proxies Group: oauth.openshift.io Name: console Resource: oauthclients Group: Name: openshift-console-operator Resource: namespaces Group: Name: openshift-console Resource: namespaces Group: Name: console-public Namespace: openshift-config-managed Resource: configmaps Versions: Name: operator Version: 4.5.0-0.nightly-2020-05-12-224129 Events: <none>
So it appears the ConsoleCLIDownload CRD needs an admitted `downloads` route in the `openshift-console` namespace. In this case the route's status is empty. The issue is most probably connected to the ingress operator which in this particular case is degraded as well.
Moving to the routing component since the underlying cause is the degraded ingress operator.
@Yadan, I used your cluster-1 to look into the issue. The ingress operator was reporting Available=False because the managed ingress controller pods were not getting scheduled. Your cluster is a 3-node master-only cluster and ingress controller pods only run on worker nodes by default [1]: $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-48-210.us-east-2.compute.internal Ready master 42h v1.18.2 ip-10-0-70-58.us-east-2.compute.internal Ready master 42h v1.18.2 ip-10-0-71-73.us-east-2.compute.internal Ready master 42h v1.18.2 According to the cluster's install-config, it appears the desired state is a 9-node (3 master/6 compute) cluster: $ oc get cm/cluster-config-v1 -n kube-system -o yaml <SNIP> compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 6 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: aws: amiID: ami-070c6257b10036038 rootVolume: iops: 0 size: 120 type: gp2 type: m4.xlarge zones: - us-east-2b - us-east-2a replicas: 3 You need to add at least 2 compute nodes to the cluster or make the masters schedulable by setting mastersSchedulable: true: $ oc get schedulers.config/cluster -o json | jq '.spec.mastersSchedulable' false After updating: $ oc get schedulers.config/cluster -o json | jq '.spec.mastersSchedulable' true cluster ingress looks good: $ oc get co/ingress NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE ingress 4.5.0-0.nightly-2020-05-12-224129 True False False 62m [1] https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L175-L176