Bug 1835153
Summary: | console operator is reporting OCDownloadsSyncDegraded | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Yadan Pei <yapei> |
Component: | Networking | Assignee: | Daneyon Hansen <dhansen> |
Networking sub component: | router | QA Contact: | Hongan Li <hongli> |
Status: | CLOSED NOTABUG | Docs Contact: | |
Severity: | high | ||
Priority: | unspecified | CC: | aos-bugs, dhansen, jhadvig, jokerman, spadgett, yapei |
Version: | 4.5 | ||
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-05-14 22:19:42 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Yadan Pei
2020-05-13 09:35:44 UTC
# oc get route downloads -n openshift-console -o json | jq '.spec' { "host": "downloads-openshift-console.apps.qe-lzha.qe.devcluster.openshift.com", "port": { "targetPort": "http" }, "tls": { "insecureEdgeTerminationPolicy": "Redirect", "termination": "edge" }, "to": { "kind": "Service", "name": "downloads", "weight": 100 }, "wildcardPolicy": "None" } # oc get route downloads -n openshift-console -o json | jq '.status' {} # oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication Unknown Unknown True 6h52m cloud-credential 4.5.0-0.nightly-2020-05-12-224129 True False False 6h16m cluster-autoscaler 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m config-operator 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m console 4.5.0-0.nightly-2020-05-12-224129 Unknown True True 6h45m csi-snapshot-controller dns 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m etcd 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m image-registry False True False 6h46m ingress False True True 6h46m insights 4.5.0-0.nightly-2020-05-12-224129 True False False 6h46m kube-apiserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m kube-controller-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m kube-scheduler 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m kube-storage-version-migrator 4.5.0-0.nightly-2020-05-12-224129 False False False 6h53m machine-api 4.5.0-0.nightly-2020-05-12-224129 True False False 6h42m machine-approver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h50m machine-config 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m marketplace 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m monitoring False True True 6h41m network 4.5.0-0.nightly-2020-05-12-224129 True False False 6h53m node-tuning 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m openshift-apiserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h47m openshift-controller-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m openshift-samples 4.5.0-0.nightly-2020-05-12-224129 True False False 6h36m operator-lifecycle-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m operator-lifecycle-manager-catalog 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m operator-lifecycle-manager-packageserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m service-ca 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m storage 4.5.0-0.nightly-2020-05-12-224129 True False False 6h46m # oc describe co console Name: console Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2020-05-13T03:14:22Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:status: .: f:extension: Manager: cluster-version-operator Operation: Update Time: 2020-05-13T03:14:23Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:relatedObjects: f:versions: Manager: console Operation: Update Time: 2020-05-13T03:27:49Z Resource Version: 14090 Self Link: /apis/config.openshift.io/v1/clusteroperators/console UID: 49c6c1bc-5af4-41df-bc9c-3a714783b9c0 Spec: Status: Conditions: Last Transition Time: 2020-05-13T03:27:49Z Message: OCDownloadsSyncDegraded: ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://' Reason: OCDownloadsSync_FailedCreate Status: True Type: Degraded Last Transition Time: 2020-05-13T03:25:51Z Message: DefaultRouteSyncProgressing: default route is not available at canonical host [] OAuthClientSyncProgressing: waiting on route host Reason: DefaultRouteSync_FailedDefaultRouteHost::OAuthClientSync_FailedHost Status: True Type: Progressing Last Transition Time: 2020-05-13T03:25:49Z Reason: NoData Status: Unknown Type: Available Last Transition Time: 2020-05-13T03:25:49Z Reason: AsExpected Status: True Type: Upgradeable Extension: <nil> Related Objects: Group: operator.openshift.io Name: cluster Resource: consoles Group: config.openshift.io Name: cluster Resource: consoles Group: config.openshift.io Name: cluster Resource: infrastructures Group: config.openshift.io Name: cluster Resource: proxies Group: oauth.openshift.io Name: console Resource: oauthclients Group: Name: openshift-console-operator Resource: namespaces Group: Name: openshift-console Resource: namespaces Group: Name: console-public Namespace: openshift-config-managed Resource: configmaps Versions: Name: operator Version: 4.5.0-0.nightly-2020-05-12-224129 Events: <none> So it appears the ConsoleCLIDownload CRD needs an admitted `downloads` route in the `openshift-console` namespace. In this case the route's status is empty. The issue is most probably connected to the ingress operator which in this particular case is degraded as well. Moving to the routing component since the underlying cause is the degraded ingress operator. @Yadan, I used your cluster-1 to look into the issue. The ingress operator was reporting Available=False because the managed ingress controller pods were not getting scheduled. Your cluster is a 3-node master-only cluster and ingress controller pods only run on worker nodes by default [1]: $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-48-210.us-east-2.compute.internal Ready master 42h v1.18.2 ip-10-0-70-58.us-east-2.compute.internal Ready master 42h v1.18.2 ip-10-0-71-73.us-east-2.compute.internal Ready master 42h v1.18.2 According to the cluster's install-config, it appears the desired state is a 9-node (3 master/6 compute) cluster: $ oc get cm/cluster-config-v1 -n kube-system -o yaml <SNIP> compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 6 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: aws: amiID: ami-070c6257b10036038 rootVolume: iops: 0 size: 120 type: gp2 type: m4.xlarge zones: - us-east-2b - us-east-2a replicas: 3 You need to add at least 2 compute nodes to the cluster or make the masters schedulable by setting mastersSchedulable: true: $ oc get schedulers.config/cluster -o json | jq '.spec.mastersSchedulable' false After updating: $ oc get schedulers.config/cluster -o json | jq '.spec.mastersSchedulable' true cluster ingress looks good: $ oc get co/ingress NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE ingress 4.5.0-0.nightly-2020-05-12-224129 True False False 62m [1] https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L175-L176 |