Bug 2034413 - cloud-network-config-controller fails to init with secret "cloud-credentials" not found in manual credential mode
Summary: cloud-network-config-controller fails to init with secret "cloud-credentials"...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.10
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.10.0
Assignee: Casey Callendrello
QA Contact: Anurag saxena
: 2035906 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2021-12-20 20:41 UTC by Patrick Dillon
Modified: 2022-03-10 16:35 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2022-03-10 16:35:34 UTC
Target Upstream Version:
cdc: needinfo-

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-network-operator pull 1277 0 None open BUG 2034413: cncc: create Cloud CredentialsRequest in /manifests 2022-01-12 13:08:43 UTC
Red Hat Product Errata RHSA-2022:0056 0 None Closed OCP4 nodes don't have network 2022-04-12 13:41:19 UTC

Description Patrick Dillon 2021-12-20 20:41:07 UTC
Description of problem: On Azure Stack Hub (but I suspect all platforms that use Manual Credentials mode), the cloud-network-config-controller does not init because it cannot find the cloud-credentials secret:

  Warning  FailedMount      4m16s (x23 over 34m)  kubelet            MountVolume.SetUp failed for volume "cloud-provider-secret" : secret "cloud-credentials" not found
[root@f9d1b7830e4f /]# oc logs cloud-network-config-controller-d6564f6f-g6m2l -n openshift-cloud-network-config-controller
Error from server (BadRequest): container "controller" in pod "cloud-network-config-controller-d6564f6f-g6m2l" is waiting to start: ContainerCreating

I see the credentialsrequest is created in the CNO: https://github.com/openshift/cluster-network-operator/blob/83eec9e14b01ede61754634ac527d9d5e6e02bf8/bindata/cloud-network-config-controller/003-credentialrequest.yaml#L42-L56 

But it is not extracted from the release image when I run
oc adm release extract "$release_image" --credentials-requests --cloud=azure --to=credentials-request

I suspect that is, perhaps, because it appears to be a template, but I am not sure.

How reproducible: I suspect this would always be reproducible on all platforms (AWS, GCP, Azure) when running in manual mode. I have only tested on Azure Stack, and we noticed the issue because it always runs in manual mode.

Steps to Reproduce:
Run an install in manual mode. If you want help running in manual mode let me know. But I suspect we can pinpoint the failure to the extraction of credentialsrequest from the release image. It would be easier to start there rather than a full install.

Actual results:
Error message above
No credentials request when extracting with oc adm release extract "$release_image" --credentials-requests --cloud=azure --to=credentials-request
network operator is stuck:   Deployment "openshift-cloud-network-config-controller/cloud-network-config-controller" is waiting for other operators to become ready

Expected results:
network operator starts

Additional info:

Comment 2 Alexander Constantinescu 2022-01-03 10:00:05 UTC
*** Bug 2035906 has been marked as a duplicate of this bug. ***

Comment 8 zhaozhanqi 2022-01-12 03:07:41 UTC
*** Bug 2039286 has been marked as a duplicate of this bug. ***

Comment 10 MayXu 2022-01-14 03:44:29 UTC
$ oc adm release extract registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-14-015144 --credentials-requests -a pull-secret --cloud azure
created 0000_50_cluster-network-operator_02-cncc-credentials.yaml

$cat 0000_50_cluster-network-operator_02-cncc-credentials.yaml
    name: cloud-credentials
    namespace: openshift-cloud-network-config-controller

Comment 11 MayXu 2022-01-14 03:54:01 UTC
checked 4.10.0-0.nightly-2022-01-14-015144 with --cloud aws and --cloud gcp 
openshift-cloud-network-config-controller/cloud-credentials created

Comment 12 zhaozhanqi 2022-01-14 08:38:43 UTC
Move this bug to verified base on comment 10 and 11

Comment 15 errata-xmlrpc 2022-03-10 16:35:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.