Version: $ openshift-install version 4.9 Platform: ibmcloud Please specify: IPI What happened? During an attempt to use IPI on IBM Cloud, a failure occurred during the manifest creation survey while the installer was attempting to list possible Resource Groups in IBM Cloud. It appears the ability to use IBM Cloud Service API Keys rather than a User API Key for the installer does not properly map back to an IBM Cloud Account. ``` time="2021-08-12T14:31:36Z" level=debug msg="OpenShift Installer unreleased-master-4911-g3f526c477abeb6da54c451b847a64d66de94be62" time="2021-08-12T14:31:36Z" level=debug msg="Built from commit 3f526c477abeb6da54c451b847a64d66de94be62" time="2021-08-12T14:31:36Z" level=debug msg="Fetching Master Machines..." time="2021-08-12T14:31:36Z" level=debug msg="Loading Master Machines..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Cluster ID..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Install Config..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading SSH Key..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Base Domain..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Platform..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Cluster Name..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Base Domain..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Platform..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Networking..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Platform..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Pull Secret..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Platform..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Platform Credentials Check..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Install Config..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Install Config..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Image..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Install Config..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Master Ignition Config..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Install Config..." time="2021-08-12T14:31:36Z" level=debug msg=" Loading Root CA..." time="2021-08-12T14:31:36Z" level=debug msg=" Fetching Cluster ID..." time="2021-08-12T14:31:36Z" level=debug msg=" Fetching Install Config..." time="2021-08-12T14:31:36Z" level=debug msg=" Fetching SSH Key..." time="2021-08-12T14:31:36Z" level=debug msg=" Generating SSH Key..." time="2021-08-12T14:31:40Z" level=debug msg=" Fetching Base Domain..." time="2021-08-12T14:31:40Z" level=debug msg=" Fetching Platform..." time="2021-08-12T14:31:40Z" level=debug msg=" Generating Platform..." time="2021-08-12T14:32:02Z" level=fatal msg="failed to fetch Master Machines: failed to fetch dependency of \"Master Machines\": failed to fetch dependency of \"Cluster ID\": failed to fetch dependency of \"Install Config\": failed to fetch dependency of \"Base Domain\": failed to generate asset \"Platform\": failed to list resource groups: Can not get resource groups without account id in parameter by service id token" ``` What did you expect to happen? The installer should be able to use an API Key generated for an IBM Cloud Service Id and map it to the proper IBM Cloud Account. How to reproduce it (as minimally and precisely as possible)? $ export IC_API_KEY=<service-id-api-key> $ openshift-installer create manifests --dir ibmcloud-cluster Select `ibmcloud` from survey Anything else we need to know? IBM Cloud is investigating the potential fix required to support a Service Id API Key.
Hi Jeremiah, I can see the related PR (openshift/installer/pull/5177) already merged but this BZ is still in MODIFIED state, can you please double check its status? I think it should be "ON_QA". Regarding the patch itself here you have my analysis: [Local compiled version before the patch was included] ~~~ $ export IC_API_KEY='<ServiceID_key>' $ ./openshift-install-local version ./openshift-install-local unreleased-master-4947-gf26cc8e8ede74378c2452634800f6f40c6f1de6e built from commit f26cc8e8ede74378c2452634800f6f40c6f1de6e release image registry.ci.openshift.org/origin/release:4.8 $ ./openshift-install-local create cluster --dir test16/ --log-level debug DEBUG OpenShift Installer unreleased-master-4947-gf26cc8e8ede74378c2452634800f6f40c6f1de6e DEBUG Built from commit f26cc8e8ede74378c2452634800f6f40c6f1de6e DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.ibmcloud.resourceGroupName: Internal error: Can not get resource groups without account id in parameter by service id token ~~~ *** RESULT: Fails as expected. [Local compiled version after the patch was included but using default ServiceID with default "Public Access" access group] ~~~ $ export IC_API_KEY='<ServiceID_key>' $ ./openshift-install-local version ./openshift-install-local unreleased-master-4954-g1d21be17667df4266c3def1de353488009881905 built from commit 1d21be17667df4266c3def1de353488009881905 release image registry.ci.openshift.org/origin/release:4.8 release architecture amd64 $ ./openshift-install-local create cluster --dir test16/ --log-level debug DEBUG OpenShift Installer unreleased-master-4954-g1d21be17667df4266c3def1de353488009881905 DEBUG Built from commit 1d21be17667df4266c3def1de353488009881905 DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.ibmcloud.resourceGroupName: Internal error: You are not authorized to use this API ~~~ *** RESULT: insufficent permissions for the ServiceID. [Local compiled version after the patch was included but assigning extra "Power Users" access group to the ServiceID] ~~~ $ ./openshift-install-local create cluster --dir test16/ --log-level debug DEBUG OpenShift Installer unreleased-master-4954-g1d21be17667df4266c3def1de353488009881905 DEBUG Built from commit 1d21be17667df4266c3def1de353488009881905 DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Using Install Config loaded from target directory [...] DEBUG module.image.ibm_is_image.image: Creating... ERROR ERROR Error: [DEBUG] Image creation err The request is not authorized to access the Cloud Object Storage resource. ~~~ *** RESULT: the patch works as expected but the installation fails due to another problem with "ibm_iam_authorization_policy" already reported via BZ#1992777. [Destroy operation] ~~~ $ ./openshift-install-local destroy cluster --dir test16/ --log-level debug DEBUG OpenShift Installer unreleased-master-4954-g1d21be17667df4266c3def1de353488009881905 DEBUG Built from commit 1d21be17667df4266c3def1de353488009881905 DEBUG Listing virtual service instances DEBUG Listing IAM authorizations DEBUG Listing virtual service instances DEBUG Listing COS instances WARNING Unable to determine IAM policy match. Failed to obtain COS instance ID. Can not get resource groups without account id in parameter by service id token DEBUG Listing load balancers DEBUG Listing subnets DEBUG Deleting subnet "pamoedo-test-mh2b6-subnet-control-plane-eu-de-1" DEBUG Deleting subnet "pamoedo-test-mh2b6-subnet-control-plane-eu-de-2" DEBUG Deleting subnet "pamoedo-test-mh2b6-subnet-compute-eu-de-2" DEBUG Deleting subnet "pamoedo-test-mh2b6-subnet-compute-eu-de-3" DEBUG Deleting subnet "pamoedo-test-mh2b6-subnet-control-plane-eu-de-3" DEBUG Deleting subnet "pamoedo-test-mh2b6-subnet-compute-eu-de-1" DEBUG Subnets: 6 items pending DEBUG Listing subnets INFO Deleted subnet "pamoedo-test-mh2b6-subnet-control-plane-eu-de-3" INFO Deleted subnet "pamoedo-test-mh2b6-subnet-compute-eu-de-1" INFO Deleted subnet "pamoedo-test-mh2b6-subnet-control-plane-eu-de-1" INFO Deleted subnet "pamoedo-test-mh2b6-subnet-control-plane-eu-de-2" INFO Deleted subnet "pamoedo-test-mh2b6-subnet-compute-eu-de-2" INFO Deleted subnet "pamoedo-test-mh2b6-subnet-compute-eu-de-3" DEBUG Listing public gateways INFO Skipping deletion of security groups with generated VPC DEBUG Listing images DEBUG Deleting public gateway "pamoedo-test-mh2b6-public-gateway-eu-de-3" DEBUG Deleting public gateway "pamoedo-test-mh2b6-public-gateway-eu-de-2" DEBUG Deleting public gateway "pamoedo-test-mh2b6-public-gateway-eu-de-1" DEBUG Public Gateways: 3 items pending DEBUG Listing public gateways INFO Deleted public gateway "pamoedo-test-mh2b6-public-gateway-eu-de-3" INFO Deleted public gateway "pamoedo-test-mh2b6-public-gateway-eu-de-2" INFO Deleted public gateway "pamoedo-test-mh2b6-public-gateway-eu-de-1" DEBUG Listing floating IPs DEBUG Listing VPCs DEBUG Deleting VPC "pamoedo-test-mh2b6-vpc" DEBUG VPCs: 1 items pending DEBUG Listing VPCs INFO Deleted VPC "pamoedo-test-mh2b6-vpc" DEBUG Listing DNS records DEBUG Listing COS instances INFO Skipping deletion of user-provided resource group pamoedom-rg DEBUG Cloud Object Storage Instances: Can not get resource groups without account id in parameter by service id token DEBUG Listing COS instances DEBUG Cloud Object Storage Instances: Can not get resource groups without account id in parameter by service id token DEBUG Listing COS instances DEBUG Cloud Object Storage Instances: Can not get resource groups without account id in parameter by service id token DEBUG Listing COS instances [...] ~~~ *** RESULT: Fails, this patch needs to contemplate destroy operation also. SUMMARY: FAILED
[QA Summary] [Version] ~~~ $ ./openshift-install-local-bz1993207 version ./openshift-install-local-bz1993207 unreleased-master-4958-g2a6d2db0f8363f5307c7c637cf2b0af625cd3dd5 built from commit 2a6d2db0f8363f5307c7c637cf2b0af625cd3dd5 release image registry.ci.openshift.org/origin/release:4.8 release architecture amd64 $ git --no-pager log --oneline --first-parent origin/master -3 2a6d2db0f (HEAD -> master, upstream/master, origin/master, origin/HEAD) Merge pull request #5181 from hasueki/ibm-fix-service-id-destroy 4050517ee Merge pull request #5120 from eslutsky/bump-cluster-api-provider 1d21be176 Merge pull request #5177 from hasueki/ibm-fix-service-id ~~~ [Parameters] ~~~ $ cat test19/install-config.yaml.bak apiVersion: v1 baseDomain: ibmcloud.qe.devcluster.openshift.com compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 0 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: ibmcloud: type: bx2-8x32 replicas: 3 metadata: creationTimestamp: null name: pamoedo-test networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: ibmcloud: cisInstanceCRN: 'crn:v1:bluemix:public:internet-svcs:...' region: eu-de resourceGroupName: pamoedom-rg publish: External pullSecret: '{"auths": ...}' sshKey: 'ssh-rsa AAAA...' $ export IC_API_KEY='<ServiceID_Key>' $ DIG=$(skopeo inspect --authfile <authfile> docker://registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-08-31-123131 | grep Digest | cut -d'"' -f4) $ export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.ci.openshift.org/ocp/release@${DIG} ~~~ [Results] -Create Cluster- ~~~ $ ./openshift-install-local-bz1993207 create cluster --dir test19/ --log-level debug DEBUG OpenShift Installer unreleased-master-4958-g2a6d2db0f8363f5307c7c637cf2b0af625cd3dd5 DEBUG Built from commit 2a6d2db0f8363f5307c7c637cf2b0af625cd3dd5 DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Using Install Config loaded from target directory [...] DEBUG module.image.ibm_is_image.image: Creating... ERROR ERROR Error: [DEBUG] Image creation err The request is not authorized to access the Cloud Object Storage resource. ~~~ NOTE: Installation aborts due to BZ#1992777 but "openshift/installer/pull/5177" does the expected job with the ServiceID. -Destroy Cluster- ~~~ $ ./openshift-install-local-bz1993207 destroy cluster --dir test19/ --log-level debug DEBUG OpenShift Installer unreleased-master-4958-g2a6d2db0f8363f5307c7c637cf2b0af625cd3dd5 DEBUG Built from commit 2a6d2db0f8363f5307c7c637cf2b0af625cd3dd5 DEBUG Listing virtual service instances DEBUG Listing IAM authorizations DEBUG Listing virtual service instances DEBUG Listing COS instances DEBUG Deleting IAM authorization "a213ad97-1381-4a02-bde0-2b2d0e0ee883" DEBUG IAM Authorizations: 1 items pending DEBUG Listing IAM authorizations INFO Deleted IAM authorization "a213ad97-1381-4a02-bde0-2b2d0e0ee883" DEBUG Listing load balancers DEBUG Deleting load balancer "pamoedo-test-bnk6s-kubernetes-api-private" DEBUG Deleting load balancer "pamoedo-test-bnk6s-kubernetes-api-public" DEBUG Load Balancers: 2 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-private" to delete DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 2 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-private" to delete DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 2 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-private" to delete DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 2 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-private" to delete DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 2 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-private" to delete DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 2 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-private" to delete DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 2 items pending DEBUG Listing load balancers INFO Deleted load balancer "pamoedo-test-bnk6s-kubernetes-api-private" DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 1 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 1 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 1 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 1 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 1 items pending DEBUG Listing load balancers DEBUG Waiting for load balancer "pamoedo-test-bnk6s-kubernetes-api-public" to delete DEBUG Load Balancers: 1 items pending DEBUG Listing load balancers INFO Deleted load balancer "pamoedo-test-bnk6s-kubernetes-api-public" DEBUG Listing subnets DEBUG Deleting subnet "pamoedo-test-bnk6s-subnet-compute-eu-de-2" DEBUG Deleting subnet "pamoedo-test-bnk6s-subnet-control-plane-eu-de-2" DEBUG Deleting subnet "pamoedo-test-bnk6s-subnet-control-plane-eu-de-3" DEBUG Deleting subnet "pamoedo-test-bnk6s-subnet-compute-eu-de-3" DEBUG Deleting subnet "pamoedo-test-bnk6s-subnet-control-plane-eu-de-1" DEBUG Deleting subnet "pamoedo-test-bnk6s-subnet-compute-eu-de-1" DEBUG Subnets: 6 items pending DEBUG Listing subnets INFO Deleted subnet "pamoedo-test-bnk6s-subnet-control-plane-eu-de-1" INFO Deleted subnet "pamoedo-test-bnk6s-subnet-compute-eu-de-1" INFO Deleted subnet "pamoedo-test-bnk6s-subnet-compute-eu-de-2" INFO Deleted subnet "pamoedo-test-bnk6s-subnet-control-plane-eu-de-2" INFO Deleted subnet "pamoedo-test-bnk6s-subnet-control-plane-eu-de-3" INFO Deleted subnet "pamoedo-test-bnk6s-subnet-compute-eu-de-3" DEBUG Listing images DEBUG Listing public gateways INFO Skipping deletion of security groups with generated VPC DEBUG Deleting public gateway "pamoedo-test-bnk6s-public-gateway-eu-de-2" DEBUG Deleting public gateway "pamoedo-test-bnk6s-public-gateway-eu-de-3" DEBUG Deleting public gateway "pamoedo-test-bnk6s-public-gateway-eu-de-1" DEBUG Public Gateways: 3 items pending DEBUG Listing public gateways INFO Deleted public gateway "pamoedo-test-bnk6s-public-gateway-eu-de-2" INFO Deleted public gateway "pamoedo-test-bnk6s-public-gateway-eu-de-3" INFO Deleted public gateway "pamoedo-test-bnk6s-public-gateway-eu-de-1" DEBUG Listing floating IPs DEBUG Listing VPCs DEBUG Deleting VPC "pamoedo-test-bnk6s-vpc" DEBUG VPCs: 1 items pending DEBUG Listing VPCs INFO Deleted VPC "pamoedo-test-bnk6s-vpc" INFO Skipping deletion of user-provided resource group pamoedom-rg DEBUG Listing DNS records DEBUG Listing COS instances DEBUG Deleting COS instance "pamoedo-test-bnk6s-cos" DEBUG Deleting DNS record "api.pamoedo-test.ibmcloud.qe.devcluster.openshift.com" DEBUG Deleting DNS record "api-int.pamoedo-test.ibmcloud.qe.devcluster.openshift.com" DEBUG DNS Records: 2 items pending DEBUG Cloud Object Storage Instances: 1 items pending DEBUG Listing DNS records DEBUG Listing COS instances INFO Deleted DNS record "api-int.pamoedo-test.ibmcloud.qe.devcluster.openshift.com" INFO Deleted DNS record "api.pamoedo-test.ibmcloud.qe.devcluster.openshift.com" INFO Deleted COS instance "pamoedo-test-bnk6s-cos" DEBUG Purging asset "Metadata" from disk DEBUG Purging asset "Master Ignition Customization Check" from disk DEBUG Purging asset "Worker Ignition Customization Check" from disk DEBUG Purging asset "Terraform Variables" from disk DEBUG Purging asset "Kubeconfig Admin Client" from disk DEBUG Purging asset "Kubeadmin Password" from disk DEBUG Purging asset "Certificate (journal-gatewayd)" from disk INFO Time elapsed: 4m9s ~~~ *** PASSED ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759