Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2012228

Summary:	ibmcloud: credentialsrequests invalid for machine-api-operator: resource-group
Product:	OpenShift Container Platform	Reporter:	Christopher J Schaefer <cschaefe>
Component:	Cloud Compute	Assignee:	Joel Speed <jspeed>
Cloud Compute sub component:	Other Providers	QA Contact:	Pedro Amoedo <pamoedom>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	mimccune, pamoedom
Version:	4.10
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-10 16:18:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Christopher J Schaefer 2021-10-08 15:29:40 UTC

Description of problem:
The current CredentialsRequests configuration is invalid for IBM Cloud required permissions. Since IBM Cloud uses Manual Credentials mode, the machine-api-operator credentials/permissions generation is broken and causes IPI deployments to fail when the operator cannot progress.

Version-Release number of selected component (if applicable):
4.10

How reproducible:
Occurs during IPI deployments, requiring the generation of Service Id's with IAM permissions for IBM Cloud, from CredentialsRequests.

Using an IPI deployment and ccoctl, generate the Service Id with IAM permissions for IBM Cloud machine-api-operator with the current CredentialsRequests. The Service Id that is created does not have valid permissions required for the MAPI oeprator.

Steps to Reproduce:
1. During IPI deployment, generate the Service Id's for operators via 'ccoctl ibmcloud create-service-id', using the available CredentialsRequests, specifically the machine-api-operator
2. Continue IPI deployment, with the generated Secrets for Service Id's (specifically for machine-api-operator), and MAPI operator will fail to progress due to invalid IAM permissions

Actual results:
IPI deployment fails when machine-api-operator cannot successfully progress, due to invalid IAM permissions assigned during the 'ccoctl ibmcloud create-service-id' generation.

Expected results:
Fully successful IPI deployment using 'ccoctl ibmcloud create-service-id' for setting up correct Service Id's and IAM permissions for operators

Additional info:
IBM Cloud is working on a fix to assign the correct IAM permissions to address this isssue.

Comment 4 Pedro Amoedo 2021-10-18 12:59:48 UTC

[QA Summary]

[Version]

~~~
$ oc version
Client Version: 4.10.0-0.nightly-2021-10-16-173656

$ ./openshift-install version
./openshift-install 4.10.0-0.nightly-2021-10-16-173656
built from commit 95361b7f82a6539d78c170c6677de3fac776bb8d
release image registry.ci.openshift.org/ocp/release@sha256:ad3e0e971d2df07c7013925f59a9113603f7fea1eef2fc18dec2d7e740bbeb1f
release architecture amd64
~~~

[Environment]

~~~
$ CCO_IMAGE=$(oc adm release info -a pull-secret --image-for='cloud-credential-operator' registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-10-16-173656)
$ oc adm release extract -a pull-secret --credentials-requests --cloud=ibmcloud registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-10-16-173656 --to test42/cco-creds/

$ cat test42/cco-creds/0000_30_machine-api-operator_00_credentials-request.yaml 
---
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  annotations:
    include.release.openshift.io/self-managed-high-availability: "true"
  labels:
    controller-tools.k8s.io: "1.0"
  name: openshift-machine-api-ibmcloud
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: IBMCloudProviderSpec
    policies:
    - attributes:
      - name: serviceName
        value: is
      roles:
      - crn:v1:bluemix:public:iam::::role:Operator
      - crn:v1:bluemix:public:iam::::role:Editor
      - crn:v1:bluemix:public:iam::::role:Viewer
    - attributes:
      - name: resourceType
        value: resource-group
      roles:
      - crn:v1:bluemix:public:iam::::role:Viewer
  secretRef:
    name: ibmcloud-credentials
    namespace: openshift-machine-api

$ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a pull-secret
$ chmod +x ccoctl
$ export IC_API_KEY='xxx'
~~~

[Results]

~~~
$ ./ccoctl ibmcloud create-service-id --name="pamoedo-test" --credentials-requests-dir="test42/cco-creds" --output-dir="test42/cco-mnfst"
2021/10/18 14:50:52 Created IAM Access Policy:
...
2021/10/18 14:50:57 Saved credentials configuration to: test42/cco-mnfst/manifests/openshift-cloud-controller-manager-ibm-cloud-credentials-credentials.yaml
2021/10/18 14:50:57 Saved credentials configuration to: test42/cco-mnfst/manifests/openshift-machine-api-ibmcloud-credentials-credentials.yaml
2021/10/18 14:50:57 Saved credentials configuration to: test42/cco-mnfst/manifests/openshift-image-registry-installer-cloud-credentials-credentials.yaml
2021/10/18 14:50:57 Saved credentials configuration to: test42/cco-mnfst/manifests/openshift-ingress-operator-cloud-credentials-credentials.yaml

$ cat test42/cco-mnfst/manifests/openshift-machine-api-ibmcloud-credentials-credentials.yaml 
apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: null
  name: ibmcloud-credentials
  namespace: openshift-machine-api
stringData:
  ibm-credentials.env: |-
    IBMCLOUD_AUTHTYPE=iam
    IBMCLOUD_APIKEY=xxx
  ibmcloud_api_key: xxx
type: Opaque

$ ibmcloud iam service-ids
Getting all services IDs bound to current account as pamoedom...
OK
ID                                               Name                                                                    Created At              Last Updated            Description   Locked   
ServiceId-615d1473-611a-455c-b453-bd5052871fdc   pamoedo-test-openshift-cloud-controller-manager-ibm-cloud-credentials   2021-10-18T12:50+0000   2021-10-18T12:50+0000                 false   
ServiceId-2bc09210-c9cb-489c-8ab6-edd1651ea2f2   pamoedo-test-openshift-image-registry-installer-cloud-credentials       2021-10-18T12:50+0000   2021-10-18T12:50+0000                 false   
ServiceId-59332d85-21d9-4be4-bc9f-a0079e20d146   pamoedo-test-openshift-ingress-operator-cloud-credentials               2021-10-18T12:50+0000   2021-10-18T12:50+0000                 false   
ServiceId-af774c81-dc06-48f5-b474-c9e745c06f4a   pamoedo-test-openshift-machine-api-ibmcloud-credentials                 2021-10-18T12:50+0000   2021-10-18T12:50+0000                 false
~~~

*** PASSED ***

Comment 7 errata-xmlrpc 2022-03-10 16:18:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056