Bug 1971832
| Summary: | Upgrade from 4.7.16-x86_64 to 4.8.0-0.nightly-2021-06-11-053039 failed with ingress degraded on canary checks | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mike Fiedler <mifiedle> | ||||
| Component: | Networking | Assignee: | aos-network-edge-staff <aos-network-edge-staff> | ||||
| Networking sub component: | router | QA Contact: | Hongan Li <hongli> | ||||
| Status: | CLOSED DUPLICATE | Docs Contact: | |||||
| Severity: | high | ||||||
| Priority: | unspecified | CC: | aos-bugs, mmasters | ||||
| Version: | 4.8 | ||||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-06-15 16:07:29 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
*** This bug has been marked as a duplicate of bug 1961506 *** |
Created attachment 1791080 [details] must-gather after failed upgrade - possibly truncated. Description of problem: must-gather is attached but it looks too small, possibly truncated. Upgrade from 4.7.16-x86_64 to 4.8.0-0.nightly-2021-06-11-053039 failed with ingress stuck degraded - failing on canary route checks. [2021-06-11T18:20:47.414Z] Message: The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing) [2021-06-11T18:19:37.256Z] Post action: #oc get co:NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE [2021-06-11T18:19:37.256Z] authentication 4.8.0-0.nightly-2021-06-11-053039 False False True 141m [2021-06-11T18:19:37.256Z] baremetal 4.8.0-0.nightly-2021-06-11-053039 True False False 4h37m [2021-06-11T18:19:37.256Z] cloud-credential 4.8.0-0.nightly-2021-06-11-053039 True False False 4h41m [2021-06-11T18:19:37.256Z] cluster-autoscaler 4.8.0-0.nightly-2021-06-11-053039 True False False 4h36m [2021-06-11T18:19:37.256Z] config-operator 4.8.0-0.nightly-2021-06-11-053039 True False False 4h37m [2021-06-11T18:19:37.256Z] console 4.8.0-0.nightly-2021-06-11-053039 False False False 140m [2021-06-11T18:19:37.256Z] csi-snapshot-controller 4.8.0-0.nightly-2021-06-11-053039 True False False 3h28m [2021-06-11T18:19:37.256Z] dns 4.7.16 True False False 4h37m [2021-06-11T18:19:37.256Z] etcd 4.8.0-0.nightly-2021-06-11-053039 True False False 4h36m [2021-06-11T18:19:37.256Z] image-registry 4.8.0-0.nightly-2021-06-11-053039 True False False 4h25m [2021-06-11T18:19:37.256Z] ingress 4.8.0-0.nightly-2021-06-11-053039 True False True 148m [2021-06-11T18:19:37.256Z] insights 4.8.0-0.nightly-2021-06-11-053039 True False False 4h30m [2021-06-11T18:19:37.256Z] kube-apiserver 4.8.0-0.nightly-2021-06-11-053039 True False False 4h34m [2021-06-11T18:19:37.256Z] kube-controller-manager 4.8.0-0.nightly-2021-06-11-053039 True False False 4h35m [2021-06-11T18:19:37.256Z] kube-scheduler 4.8.0-0.nightly-2021-06-11-053039 True False False 4h33m [2021-06-11T18:19:37.256Z] kube-storage-version-migrator 4.8.0-0.nightly-2021-06-11-053039 True False False 3h27m [2021-06-11T18:19:37.256Z] machine-api 4.8.0-0.nightly-2021-06-11-053039 True False False 4h27m [2021-06-11T18:19:37.256Z] machine-approver 4.8.0-0.nightly-2021-06-11-053039 True False False 4h36m [2021-06-11T18:19:37.256Z] machine-config 4.7.16 True False False 4h35m [2021-06-11T18:19:37.256Z] marketplace 4.8.0-0.nightly-2021-06-11-053039 True False False 3h27m [2021-06-11T18:19:37.256Z] monitoring 4.8.0-0.nightly-2021-06-11-053039 True False False 144m [2021-06-11T18:19:37.256Z] network 4.8.0-0.nightly-2021-06-11-053039 True False False 4h37m [2021-06-11T18:19:37.256Z] node-tuning 4.8.0-0.nightly-2021-06-11-053039 True False False 148m [2021-06-11T18:19:37.256Z] openshift-apiserver 4.8.0-0.nightly-2021-06-11-053039 True False False 3h27m [2021-06-11T18:19:37.256Z] openshift-controller-manager 4.8.0-0.nightly-2021-06-11-053039 True False False 4h30m [2021-06-11T18:19:37.256Z] openshift-samples 4.8.0-0.nightly-2021-06-11-053039 True False False 149m [2021-06-11T18:19:37.256Z] operator-lifecycle-manager 4.8.0-0.nightly-2021-06-11-053039 True False False 4h36m [2021-06-11T18:19:37.256Z] operator-lifecycle-manager-catalog 4.8.0-0.nightly-2021-06-11-053039 True False False 4h36m [2021-06-11T18:19:37.256Z] operator-lifecycle-manager-packageserver 4.8.0-0.nightly-2021-06-11-053039 True False False 3h28m [2021-06-11T18:19:37.256Z] service-ca 4.8.0-0.nightly-2021-06-11-053039 True False False 4h37m [2021-06-11T18:19:37.256Z] storage 4.8.0-0.nightly-2021-06-11-053039 True False False 3h27m If there is a different root cause behind the canary checks failing, please help transfer. OpenShift release version: 4.8.0-0.nightly-2021-06-11-053039 Cluster Platform: 30_UPI on AWS with RHEL 7.9 & FIPS on & OVN How reproducible: unknown. part of a ci run