Description of problem: The current CredentialsRequests configuration is invalid for IBM Cloud required permissions. Since IBM Cloud uses Manual Credentials mode, the machine-api-operator credentials/permissions generation is broken and causes IPI deployments to fail when the operator cannot progress. Version-Release number of selected component (if applicable): 4.10 How reproducible: Occurs during IPI deployments, requiring the generation of Service Id's with IAM permissions for IBM Cloud, from CredentialsRequests. Using an IPI deployment and ccoctl, generate the Service Id with IAM permissions for IBM Cloud machine-api-operator with the current CredentialsRequests. The Service Id that is created does not have valid permissions required for the MAPI oeprator. Steps to Reproduce: 1. During IPI deployment, generate the Service Id's for operators via 'ccoctl ibmcloud create-service-id', using the available CredentialsRequests, specifically the machine-api-operator 2. Continue IPI deployment, with the generated Secrets for Service Id's (specifically for machine-api-operator), and MAPI operator will fail to progress due to invalid IAM permissions Actual results: IPI deployment fails when machine-api-operator cannot successfully progress, due to invalid IAM permissions assigned during the 'ccoctl ibmcloud create-service-id' generation. Expected results: Fully successful IPI deployment using 'ccoctl ibmcloud create-service-id' for setting up correct Service Id's and IAM permissions for operators Additional info: IBM Cloud is working on a fix to assign the correct IAM permissions to address this isssue.
[QA Summary] [Version] ~~~ $ oc version Client Version: 4.10.0-0.nightly-2021-10-16-173656 $ ./openshift-install version ./openshift-install 4.10.0-0.nightly-2021-10-16-173656 built from commit 95361b7f82a6539d78c170c6677de3fac776bb8d release image registry.ci.openshift.org/ocp/release@sha256:ad3e0e971d2df07c7013925f59a9113603f7fea1eef2fc18dec2d7e740bbeb1f release architecture amd64 ~~~ [Environment] ~~~ $ CCO_IMAGE=$(oc adm release info -a pull-secret --image-for='cloud-credential-operator' registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-10-16-173656) $ oc adm release extract -a pull-secret --credentials-requests --cloud=ibmcloud registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-10-16-173656 --to test42/cco-creds/ $ cat test42/cco-creds/0000_30_machine-api-operator_00_credentials-request.yaml --- apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: annotations: include.release.openshift.io/self-managed-high-availability: "true" labels: controller-tools.k8s.io: "1.0" name: openshift-machine-api-ibmcloud namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: IBMCloudProviderSpec policies: - attributes: - name: serviceName value: is roles: - crn:v1:bluemix:public:iam::::role:Operator - crn:v1:bluemix:public:iam::::role:Editor - crn:v1:bluemix:public:iam::::role:Viewer - attributes: - name: resourceType value: resource-group roles: - crn:v1:bluemix:public:iam::::role:Viewer secretRef: name: ibmcloud-credentials namespace: openshift-machine-api $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a pull-secret $ chmod +x ccoctl $ export IC_API_KEY='xxx' ~~~ [Results] ~~~ $ ./ccoctl ibmcloud create-service-id --name="pamoedo-test" --credentials-requests-dir="test42/cco-creds" --output-dir="test42/cco-mnfst" 2021/10/18 14:50:52 Created IAM Access Policy: ... 2021/10/18 14:50:57 Saved credentials configuration to: test42/cco-mnfst/manifests/openshift-cloud-controller-manager-ibm-cloud-credentials-credentials.yaml 2021/10/18 14:50:57 Saved credentials configuration to: test42/cco-mnfst/manifests/openshift-machine-api-ibmcloud-credentials-credentials.yaml 2021/10/18 14:50:57 Saved credentials configuration to: test42/cco-mnfst/manifests/openshift-image-registry-installer-cloud-credentials-credentials.yaml 2021/10/18 14:50:57 Saved credentials configuration to: test42/cco-mnfst/manifests/openshift-ingress-operator-cloud-credentials-credentials.yaml $ cat test42/cco-mnfst/manifests/openshift-machine-api-ibmcloud-credentials-credentials.yaml apiVersion: v1 kind: Secret metadata: creationTimestamp: null name: ibmcloud-credentials namespace: openshift-machine-api stringData: ibm-credentials.env: |- IBMCLOUD_AUTHTYPE=iam IBMCLOUD_APIKEY=xxx ibmcloud_api_key: xxx type: Opaque $ ibmcloud iam service-ids Getting all services IDs bound to current account as pamoedom... OK ID Name Created At Last Updated Description Locked ServiceId-615d1473-611a-455c-b453-bd5052871fdc pamoedo-test-openshift-cloud-controller-manager-ibm-cloud-credentials 2021-10-18T12:50+0000 2021-10-18T12:50+0000 false ServiceId-2bc09210-c9cb-489c-8ab6-edd1651ea2f2 pamoedo-test-openshift-image-registry-installer-cloud-credentials 2021-10-18T12:50+0000 2021-10-18T12:50+0000 false ServiceId-59332d85-21d9-4be4-bc9f-a0079e20d146 pamoedo-test-openshift-ingress-operator-cloud-credentials 2021-10-18T12:50+0000 2021-10-18T12:50+0000 false ServiceId-af774c81-dc06-48f5-b474-c9e745c06f4a pamoedo-test-openshift-machine-api-ibmcloud-credentials 2021-10-18T12:50+0000 2021-10-18T12:50+0000 false ~~~ *** PASSED ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056