Bug 1835153
| Summary: | console operator is reporting OCDownloadsSyncDegraded | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Yadan Pei <yapei> |
| Component: | Networking | Assignee: | Daneyon Hansen <dhansen> |
| Networking sub component: | router | QA Contact: | Hongan Li <hongli> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | aos-bugs, dhansen, jhadvig, jokerman, spadgett, yapei |
| Version: | 4.5 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-05-14 22:19:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Yadan Pei
2020-05-13 09:35:44 UTC
# oc get route downloads -n openshift-console -o json | jq '.spec'
{
"host": "downloads-openshift-console.apps.qe-lzha.qe.devcluster.openshift.com",
"port": {
"targetPort": "http"
},
"tls": {
"insecureEdgeTerminationPolicy": "Redirect",
"termination": "edge"
},
"to": {
"kind": "Service",
"name": "downloads",
"weight": 100
},
"wildcardPolicy": "None"
}
# oc get route downloads -n openshift-console -o json | jq '.status'
{}
# oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication Unknown Unknown True 6h52m
cloud-credential 4.5.0-0.nightly-2020-05-12-224129 True False False 6h16m
cluster-autoscaler 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m
config-operator 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m
console 4.5.0-0.nightly-2020-05-12-224129 Unknown True True 6h45m
csi-snapshot-controller
dns 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m
etcd 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m
image-registry False True False 6h46m
ingress False True True 6h46m
insights 4.5.0-0.nightly-2020-05-12-224129 True False False 6h46m
kube-apiserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m
kube-controller-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m
kube-scheduler 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m
kube-storage-version-migrator 4.5.0-0.nightly-2020-05-12-224129 False False False 6h53m
machine-api 4.5.0-0.nightly-2020-05-12-224129 True False False 6h42m
machine-approver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h50m
machine-config 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m
marketplace 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m
monitoring False True True 6h41m
network 4.5.0-0.nightly-2020-05-12-224129 True False False 6h53m
node-tuning 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m
openshift-apiserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h47m
openshift-controller-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h45m
openshift-samples 4.5.0-0.nightly-2020-05-12-224129 True False False 6h36m
operator-lifecycle-manager 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m
operator-lifecycle-manager-catalog 4.5.0-0.nightly-2020-05-12-224129 True False False 6h51m
operator-lifecycle-manager-packageserver 4.5.0-0.nightly-2020-05-12-224129 True False False 6h48m
service-ca 4.5.0-0.nightly-2020-05-12-224129 True False False 6h52m
storage 4.5.0-0.nightly-2020-05-12-224129 True False False 6h46m
# oc describe co console
Name: console
Namespace:
Labels: <none>
Annotations: <none>
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2020-05-13T03:14:22Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:status:
.:
f:extension:
Manager: cluster-version-operator
Operation: Update
Time: 2020-05-13T03:14:23Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:relatedObjects:
f:versions:
Manager: console
Operation: Update
Time: 2020-05-13T03:27:49Z
Resource Version: 14090
Self Link: /apis/config.openshift.io/v1/clusteroperators/console
UID: 49c6c1bc-5af4-41df-bc9c-3a714783b9c0
Spec:
Status:
Conditions:
Last Transition Time: 2020-05-13T03:27:49Z
Message: OCDownloadsSyncDegraded: ConsoleCLIDownload.console.openshift.io "oc-cli-downloads" is invalid: spec.links.href: Invalid value: "": spec.links.href in body should match '^https://'
Reason: OCDownloadsSync_FailedCreate
Status: True
Type: Degraded
Last Transition Time: 2020-05-13T03:25:51Z
Message: DefaultRouteSyncProgressing: default route is not available at canonical host []
OAuthClientSyncProgressing: waiting on route host
Reason: DefaultRouteSync_FailedDefaultRouteHost::OAuthClientSync_FailedHost
Status: True
Type: Progressing
Last Transition Time: 2020-05-13T03:25:49Z
Reason: NoData
Status: Unknown
Type: Available
Last Transition Time: 2020-05-13T03:25:49Z
Reason: AsExpected
Status: True
Type: Upgradeable
Extension: <nil>
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: consoles
Group: config.openshift.io
Name: cluster
Resource: consoles
Group: config.openshift.io
Name: cluster
Resource: infrastructures
Group: config.openshift.io
Name: cluster
Resource: proxies
Group: oauth.openshift.io
Name: console
Resource: oauthclients
Group:
Name: openshift-console-operator
Resource: namespaces
Group:
Name: openshift-console
Resource: namespaces
Group:
Name: console-public
Namespace: openshift-config-managed
Resource: configmaps
Versions:
Name: operator
Version: 4.5.0-0.nightly-2020-05-12-224129
Events: <none>
So it appears the ConsoleCLIDownload CRD needs an admitted `downloads` route in the `openshift-console` namespace. In this case the route's status is empty. The issue is most probably connected to the ingress operator which in this particular case is degraded as well. Moving to the routing component since the underlying cause is the degraded ingress operator. @Yadan, I used your cluster-1 to look into the issue. The ingress operator was reporting Available=False because the managed ingress controller pods were not getting scheduled. Your cluster is a 3-node master-only cluster and ingress controller pods only run on worker nodes by default [1]:
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-48-210.us-east-2.compute.internal Ready master 42h v1.18.2
ip-10-0-70-58.us-east-2.compute.internal Ready master 42h v1.18.2
ip-10-0-71-73.us-east-2.compute.internal Ready master 42h v1.18.2
According to the cluster's install-config, it appears the desired state is a 9-node (3 master/6 compute) cluster:
$ oc get cm/cluster-config-v1 -n kube-system -o yaml
<SNIP>
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform: {}
replicas: 6
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
aws:
amiID: ami-070c6257b10036038
rootVolume:
iops: 0
size: 120
type: gp2
type: m4.xlarge
zones:
- us-east-2b
- us-east-2a
replicas: 3
You need to add at least 2 compute nodes to the cluster or make the masters schedulable by setting mastersSchedulable: true:
$ oc get schedulers.config/cluster -o json | jq '.spec.mastersSchedulable'
false
After updating:
$ oc get schedulers.config/cluster -o json | jq '.spec.mastersSchedulable'
true
cluster ingress looks good:
$ oc get co/ingress
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
ingress 4.5.0-0.nightly-2020-05-12-224129 True False False 62m
[1] https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L175-L176
|