1971832 – Upgrade from 4.7.16-x86_64 to 4.8.0-0.nightly-2021-06-11-053039 failed with ingress degraded on canary checks

Bug 1971832 - Upgrade from 4.7.16-x86_64 to 4.8.0-0.nightly-2021-06-11-053039 failed with ingress degraded on canary checks

Summary: Upgrade from 4.7.16-x86_64 to 4.8.0-0.nightly-2021-06-11-053039 failed with i...

Keywords:
Status:	CLOSED DUPLICATE of bug 1961506
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	aos-network-edge-staff
QA Contact:	Hongan Li
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-06-14 20:04 UTC by Mike Fiedler
Modified:	2022-08-04 22:32 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-06-15 16:07:29 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
must-gather after failed upgrade - possibly truncated. (16.70 MB, application/gzip) 2021-06-14 20:04 UTC, Mike Fiedler	no flags	Details
View All

Description Mike Fiedler 2021-06-14 20:04:26 UTC

Created attachment 1791080 [details]
must-gather after failed upgrade - possibly truncated.

Description of problem:

must-gather is attached but it looks too small, possibly truncated.

Upgrade from 4.7.16-x86_64 to 4.8.0-0.nightly-2021-06-11-053039 failed with ingress stuck degraded - failing on canary route checks.

[2021-06-11T18:20:47.414Z]     Message:               The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

[2021-06-11T18:19:37.256Z] Post action: #oc get co:NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
[2021-06-11T18:19:37.256Z] authentication                             4.8.0-0.nightly-2021-06-11-053039   False       False         True       141m
[2021-06-11T18:19:37.256Z] baremetal                                  4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h37m
[2021-06-11T18:19:37.256Z] cloud-credential                           4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h41m
[2021-06-11T18:19:37.256Z] cluster-autoscaler                         4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h36m
[2021-06-11T18:19:37.256Z] config-operator                            4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h37m
[2021-06-11T18:19:37.256Z] console                                    4.8.0-0.nightly-2021-06-11-053039   False       False         False      140m
[2021-06-11T18:19:37.256Z] csi-snapshot-controller                    4.8.0-0.nightly-2021-06-11-053039   True        False         False      3h28m
[2021-06-11T18:19:37.256Z] dns                                        4.7.16                              True        False         False      4h37m
[2021-06-11T18:19:37.256Z] etcd                                       4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h36m
[2021-06-11T18:19:37.256Z] image-registry                             4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h25m
[2021-06-11T18:19:37.256Z] ingress                                    4.8.0-0.nightly-2021-06-11-053039   True        False         True       148m
[2021-06-11T18:19:37.256Z] insights                                   4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h30m
[2021-06-11T18:19:37.256Z] kube-apiserver                             4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h34m
[2021-06-11T18:19:37.256Z] kube-controller-manager                    4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h35m
[2021-06-11T18:19:37.256Z] kube-scheduler                             4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h33m
[2021-06-11T18:19:37.256Z] kube-storage-version-migrator              4.8.0-0.nightly-2021-06-11-053039   True        False         False      3h27m
[2021-06-11T18:19:37.256Z] machine-api                                4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h27m
[2021-06-11T18:19:37.256Z] machine-approver                           4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h36m
[2021-06-11T18:19:37.256Z] machine-config                             4.7.16                              True        False         False      4h35m
[2021-06-11T18:19:37.256Z] marketplace                                4.8.0-0.nightly-2021-06-11-053039   True        False         False      3h27m
[2021-06-11T18:19:37.256Z] monitoring                                 4.8.0-0.nightly-2021-06-11-053039   True        False         False      144m
[2021-06-11T18:19:37.256Z] network                                    4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h37m
[2021-06-11T18:19:37.256Z] node-tuning                                4.8.0-0.nightly-2021-06-11-053039   True        False         False      148m
[2021-06-11T18:19:37.256Z] openshift-apiserver                        4.8.0-0.nightly-2021-06-11-053039   True        False         False      3h27m
[2021-06-11T18:19:37.256Z] openshift-controller-manager               4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h30m
[2021-06-11T18:19:37.256Z] openshift-samples                          4.8.0-0.nightly-2021-06-11-053039   True        False         False      149m
[2021-06-11T18:19:37.256Z] operator-lifecycle-manager                 4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h36m
[2021-06-11T18:19:37.256Z] operator-lifecycle-manager-catalog         4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h36m
[2021-06-11T18:19:37.256Z] operator-lifecycle-manager-packageserver   4.8.0-0.nightly-2021-06-11-053039   True        False         False      3h28m
[2021-06-11T18:19:37.256Z] service-ca                                 4.8.0-0.nightly-2021-06-11-053039   True        False         False      4h37m
[2021-06-11T18:19:37.256Z] storage                                    4.8.0-0.nightly-2021-06-11-053039   True        False         False      3h27m

If there is a different root cause behind the canary checks failing, please help transfer.


OpenShift release version: 4.8.0-0.nightly-2021-06-11-053039


Cluster Platform: 30_UPI on AWS with RHEL 7.9 & FIPS on & OVN


How reproducible: unknown.  part of a ci run

Comment 1 Miciah Dashiel Butler Masters 2021-06-15 16:07:29 UTC


*** This bug has been marked as a duplicate of bug 1961506 ***

Note You need to log in before you can comment on or make changes to this bug.