Bug 2066463

Summary:	[IBMCloud] failed to list DNS zones: Exactly one of ApiKey or RefreshToken must be specified
Product:	OpenShift Container Platform	Reporter:	Christopher J Schaefer <cschaefe>
Component:	Installer	Assignee:	Nobody <nobody>
Installer sub component:	openshift-installer	QA Contact:	MayXu <maxu>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	maxu, pamoedom
Version:	4.11	Keywords:	TestBlocker
Target Milestone:	---
Target Release:	4.11.0
Hardware:	x86_64
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 10:55:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Christopher J Schaefer 2022-03-21 20:44:08 UTC

Version:

$ openshift-install version
4.11

Platform:
ibmcloud


Please specify:
IPI

What happened?
installer failed when attempting to create an install-config or manifests on IBM Cloud

# bin/openshift-install create install-config --dir test-4.11-create-config/
? SSH Public Key /root/.ssh/id_rsa.pub
? Platform ibmcloud
? Region us-east
FATAL failed to fetch Install Config: failed to fetch dependency of "Install Config": failed to generate asset "Base Domain": could not retrieve base domains: failed to list DNS zones: Exactly one of ApiKey or RefreshToken must be specified. 


What did you expect to happen?
Successfully create a new install-config.yaml or manifests using an existing install-config.yaml for IBM Cloud


How to reproduce it (as minimally and precisely as possible)?

1. export IC_API_KEY=foo
2. openshift-install create install-config --dir my-cluster

Anything else we need to know?
IBM Cloud is working on a solution, the cause appears to be due to the PowerVS enablement updating IBM go-sdk's without making changes to the x86 (VPC) code path.
https://github.com/openshift/installer/pull/5612
https://github.com/openshift/installer/pull/5723

Testing is currently underway to verify the patch works as intended.

Comment 1 Christopher J Schaefer 2022-03-22 19:52:20 UTC

Further investigation has determined a change in the IBM `go-sdk-core` code has now implemented validation for any new "service" call, which creates and sets a RefreshToken.
https://github.com/IBM/go-sdk-core/blob/main/v5/core/iam_authenticator.go#L279-L282

This is causing errors when the provided APIKey and created RefreshToken both exist for the re-used authenticator, and validation is performed for each new service.
https://github.com/openshift/installer/blob/0614a6d63ac4c195c0f0d007802bcb282d3055e4/pkg/asset/installconfig/ibmcloud/client.go#L85-L91
https://github.com/openshift/installer/blob/0614a6d63ac4c195c0f0d007802bcb282d3055e4/pkg/asset/installconfig/ibmcloud/client.go#L245-L251


The solution is to create new authenticator for each IBM service, to prevent hitting the validation issue.


Work to refactor the existing patch is underway by IBM, followed by testing.

Comment 2 MayXu 2022-03-24 06:57:41 UTC

Pre-Merge test done
install succeed with 4.11.0-0.ci.test-2022-03-24-025203-ci-ln-k2dlztb-latest

Comment 3 MayXu 2022-03-24 10:03:52 UTC

with the Pre-Merge version 
fail to destroy the cluster 
DEBUG OpenShift Installer 4.11.0-0.ci.test-2022-03-24-025203-ci-ln-k2dlztb-latest 
DEBUG Built from commit 5651f0285bb3fa1b01e30f11e719cd1fc0c8fcc3 
FATAL Failed to destroy cluster: Exactly one of ApiKey or RefreshToken must be specified.

Comment 4 MayXu 2022-03-25 07:20:05 UTC

with the Pre-Merge version 
4.11.0-0.ci.test-2022-03-25-035056-ci-ln-1s08czk-latest
create cluster and destroy both succeed.

Comment 6 MayXu 2022-03-28 09:29:20 UTC

verified on 4.11.0-0.ci-2022-03-28-043617 PASS

Comment 9 errata-xmlrpc 2022-08-10 10:55:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069