Bug 2003445

Summary:	OCP Upgrade failed 4.5 --> 4.6: the cluster operator authentication is degraded
Product:	OpenShift Container Platform	Reporter:	Aleksandra Malykhin <amalykhi>
Component:	Networking	Assignee:	Alexander Constantinescu <aconstan>
Networking sub component:	openshift-sdn	QA Contact:	zhaozhanqi <zzhao>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	aos-bugs, jokerman, sdasu, vpickard, vrutkovs, yboaron
Version:	4.6
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-10-12 08:10:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1885867

Description Aleksandra Malykhin 2021-09-12 10:34:18 UTC

Description of problem:
OCP Upgrade failed 4.5 ->4.6 the cluster operator authentication is degraded


Version-Release number of selected component (if applicable):

Original image:  4.5.0-0.nightly-2021-09-07-164108
Upgrade to: 4.6.0-0.nightly-2021-09-09-165319
Both Connected and Disconnected environment

How reproducible:
5/5

Steps to Reproduce:
1. Deploy OCP4.5 cluster (I used 4.5.0-0.nightly-2021-09-07-164108)
2.After deployment completed successfully, upgrade to 4.6.0-0.nightly-2021-09-09-165319

Actual results:
[kni@provisionhost-0-0 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2021-09-07-164108   True        True          93m     Unable to apply 4.6.0-0.nightly-2021-09-09-165319: the cluster operator authentication is degraded


[kni@provisionhost-0-0 ~]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
...
authentication                             4.6.0-0.nightly-2021-09-09-165319   True        True          True       43m

console                                    4.6.0-0.nightly-2021-09-09-165319   False       True          False      49m

dns                                        4.6.0-0.nightly-2021-09-09-165319   True        False         True       49m

machine-config                             4.5.0-0.nightly-2021-09-07-164108   False       True          True       33m


Expected results:
Cluster was upgraded successfully. All operators are available and not degraded


Additional info:
must-gather see in the next comment

Comment 2 sdasu 2021-09-14 16:14:49 UTC

Doesn't seem to be an issue with the Baremetal IPI upgrade.

Comment 3 Vadim Rutkovsky 2021-09-14 16:30:33 UTC

The upgrade is still progressing - Working towards 4.6.0-0.nightly-2021-09-09-165319: 78% complete, it started at 2021-09-12T08:54:05Z and last updated at 2021-09-12T10:39:49Z - its been less than two hours.

Does upgrade get stalled if its given some more time - i.e. 2 or 4 hours?

Comment 4 Aleksandra Malykhin 2021-09-14 16:43:20 UTC

Yes, Vadim. I've left it for the weekend and it was still in the same stage. I just did the last rerun to be sure it is reproducible

Comment 5 Vadim Rutkovsky 2021-09-14 17:27:35 UTC

Ah, I see

DNS is reporting degraded, as dns-default-44fg9 cannot be created. Seems SDN cannot properly create network for it:

2327:2021-09-12T10:51:24.353522298Z I0912 10:51:24.353285  282721 cni.go:157] [openshift-dns/dns-default-44fg9] CNI request &{ADD openshift-dns dns-default-44fg9 a63c617bbdd6ddadf264efebc2ec204a16f8f5efaf8c6b0adf04cad59fd8e492 /var/run/netns/7454471f-ab4a-4f1b-88d3-7e2aca2c9b4d eth0 0xc002077300}, result "", err failed to configure pod interface: timed out waiting for pod flows for pod: dns-default-44fg9, error: timed out waiting for the condition
2328:2021-09-12T10:51:24.37001122Z I0912 10:51:24.369941  282721 cniserver.go:147] Waiting for DEL result for pod openshift-dns/dns-default-44fg9
2329:2021-09-12T10:51:24.37001122Z I0912 10:51:24.369980  282721 cni.go:147] [openshift-dns/dns-default-44fg9] dispatching pod network request &{DEL openshift-dns dns-default-44fg9 a63c617bbdd6ddadf264efebc2ec204a16f8f5efaf8c6b0adf04cad59fd8e492 /var/run/netns/7454471f-ab4a-4f1b-88d3-7e2aca2c9b4d eth0 0xc001e4a000}
2330:2021-09-12T10:51:24.41211863Z I0912 10:51:24.412015  282721 cni.go:157] [openshift-dns/dns-default-44fg9] CNI request &{DEL openshift-dns dns-default-44fg9 a63c617bbdd6ddadf264efebc2ec204a16f8f5efaf8c6b0adf04cad59fd8e492 /var/run/netns/7454471f-ab4a-4f1b-88d3-7e2aca2c9b4d eth0 0xc001e4a000}, result "", err <nil>
2331:2021-09-12T10:51:24.473234591Z I0912 10:51:24.472997  282721 cniserver.go:147] Waiting for DEL result for pod openshift-dns/dns-default-44fg9
2332:2021-09-12T10:51:24.473234591Z I0912 10:51:24.473031  282721 cni.go:147] [openshift-dns/dns-default-44fg9] dispatching pod network request &{DEL openshift-dns dns-default-44fg9 a63c617bbdd6ddadf264efebc2ec204a16f8f5efaf8c6b0adf04cad59fd8e492 /var/run/netns/7454471f-ab4a-4f1b-88d3-7e2aca2c9b4d eth0 0xc002088100}
2333:2021-09-12T10:51:24.506958553Z I0912 10:51:24.506840  282721 cni.go:157] [openshift-dns/dns-default-44fg9] CNI request &{DEL openshift-dns dns-default-44fg9 a63c617bbdd6ddadf264efebc2ec204a16f8f5efaf8c6b0adf04cad59fd8e492 /var/run/netns/7454471f-ab4a-4f1b-88d3-7e2aca2c9b4d eth0 0xc002088100}, result "", err <nil>
2334:2021-09-12T10:51:25.490224567Z I0912 10:51:25.490058  282721 cniserver.go:147] Waiting for ADD result for pod openshift-dns/dns-default-44fg9
2335:2021-09-12T10:51:25.490224567Z I0912 10:51:25.490094  282721 cni.go:147] [openshift-dns/dns-default-44fg9] dispatching pod network request &{ADD openshift-dns dns-default-44fg9 8562773f2f01730406c3823ccde546f4aea34851226a6064d48d070f902cbcb9 /var/run/netns/b1df92f8-79a7-45ec-8178-beaea90e152b eth0 0xc002000700}

Moving this to SDN

Comment 6 Alexander Constantinescu 2021-10-12 08:10:38 UTC


*** This bug has been marked as a duplicate of bug 1958390 ***