Bug 2034413

Summary:	cloud-network-config-controller fails to init with secret "cloud-credentials" not found in manual credential mode
Product:	OpenShift Container Platform	Reporter:	Patrick Dillon <padillon>
Component:	Networking	Assignee:	Casey Callendrello <cdc>
Networking sub component:	ovn-kubernetes	QA Contact:	Anurag saxena <anusaxen>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	unspecified	CC:	ffernand, huirwang, lwan, maxu, yunjiang, zzhao
Version:	4.10	Keywords:	TestBlocker
Target Milestone:	---	Flags:	cdc: needinfo-
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-10 16:35:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Patrick Dillon 2021-12-20 20:41:07 UTC

Description of problem: On Azure Stack Hub (but I suspect all platforms that use Manual Credentials mode), the cloud-network-config-controller does not init because it cannot find the cloud-credentials secret:


  Warning  FailedMount      4m16s (x23 over 34m)  kubelet            MountVolume.SetUp failed for volume "cloud-provider-secret" : secret "cloud-credentials" not found
[root@f9d1b7830e4f /]# oc logs cloud-network-config-controller-d6564f6f-g6m2l -n openshift-cloud-network-config-controller
Error from server (BadRequest): container "controller" in pod "cloud-network-config-controller-d6564f6f-g6m2l" is waiting to start: ContainerCreating

I see the credentialsrequest is created in the CNO: https://github.com/openshift/cluster-network-operator/blob/83eec9e14b01ede61754634ac527d9d5e6e02bf8/bindata/cloud-network-config-controller/003-credentialrequest.yaml#L42-L56 

But it is not extracted from the release image when I run
oc adm release extract "$release_image" --credentials-requests --cloud=azure --to=credentials-request

I suspect that is, perhaps, because it appears to be a template, but I am not sure.

How reproducible: I suspect this would always be reproducible on all platforms (AWS, GCP, Azure) when running in manual mode. I have only tested on Azure Stack, and we noticed the issue because it always runs in manual mode.


Steps to Reproduce:
Run an install in manual mode. If you want help running in manual mode let me know. But I suspect we can pinpoint the failure to the extraction of credentialsrequest from the release image. It would be easier to start there rather than a full install.


Actual results:
Error message above
No credentials request when extracting with oc adm release extract "$release_image" --credentials-requests --cloud=azure --to=credentials-request
network operator is stuck:   Deployment "openshift-cloud-network-config-controller/cloud-network-config-controller" is waiting for other operators to become ready



Expected results:
network operator starts


Additional info:

Comment 2 Alexander Constantinescu 2022-01-03 10:00:05 UTC

*** Bug 2035906 has been marked as a duplicate of this bug. ***

Comment 8 zhaozhanqi 2022-01-12 03:07:41 UTC

*** Bug 2039286 has been marked as a duplicate of this bug. ***

Comment 10 MayXu 2022-01-14 03:44:29 UTC

$ oc adm release extract registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-14-015144 --credentials-requests -a pull-secret --cloud azure
created 0000_50_cluster-network-operator_02-cncc-credentials.yaml

$cat 0000_50_cluster-network-operator_02-cncc-credentials.yaml
......
 secretRef:
    name: cloud-credentials
    namespace: openshift-cloud-network-config-controller

Comment 11 MayXu 2022-01-14 03:54:01 UTC

checked 4.10.0-0.nightly-2022-01-14-015144 with --cloud aws and --cloud gcp 
openshift-cloud-network-config-controller/cloud-credentials created

Comment 12 zhaozhanqi 2022-01-14 08:38:43 UTC

Move this bug to verified base on comment 10 and 11

Comment 15 errata-xmlrpc 2022-03-10 16:35:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056