Bug 2099401 - [IBMCloud] Client does not set region endpoint for InstallConfig
Summary: [IBMCloud] Client does not set region endpoint for InstallConfig
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.11
Hardware: x86_64
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.12.0
Assignee: OCP Installer
QA Contact: MayXu
Mike Pytlak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-20 21:17 UTC by Christopher J Schaefer
Modified: 2023-01-17 19:50 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, regardless of which IBM region was specified in the installation configuration file when creating manifests, the installation program set the region to `us-south`. This update resolves the issue and the installation program correctly specifies the region in the manifests. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2099401[*BZ#2099401*])
Clone Of:
Environment:
Last Closed: 2023-01-17 19:50:08 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 6046 0 None open Bug 2099401: IBMCloud: Set regional URL for ibmcloud client 2022-06-24 14:11:17 UTC
Red Hat Product Errata RHSA-2022:7399 0 None None None 2023-01-17 19:50:25 UTC

Description Christopher J Schaefer 2022-06-20 21:17:15 UTC
Version:
4.11 (expect fix to land for 4.12, with 4.11 being Tech Preview)


Platform:
ibmcloud


Please specify:
IPI


What happened?
The generated IBM Client for the InstallConfig (specifically for Metadata usage, looking up additional resource details), does not set the IBM Cloud Regional endpoint, and defaults to us-south.

Any specified or desired resources in other regions cannot be found, e.g. Dedicated Hosts, BYON, etc.


time="2022-06-20T21:04:20Z" level=debug msg="OpenShift Installer unreleased-master-6051-ge96b5145285ae158c4b1757adf48c72fab13cdca"
time="2022-06-20T21:04:20Z" level=debug msg="Built from commit e96b5145285ae158c4b1757adf48c72fab13cdca"
time="2022-06-20T21:04:20Z" level=debug msg="Fetching Master Machines..."
time="2022-06-20T21:04:20Z" level=debug msg="Loading Master Machines..."
time="2022-06-20T21:04:20Z" level=debug msg="  Loading Cluster ID..."
time="2022-06-20T21:04:20Z" level=debug msg="    Loading Install Config..."
time="2022-06-20T21:04:20Z" level=debug msg="      Loading SSH Key..."
time="2022-06-20T21:04:20Z" level=debug msg="      Loading Base Domain..."
time="2022-06-20T21:04:20Z" level=debug msg="        Loading Platform..."
time="2022-06-20T21:04:20Z" level=debug msg="      Loading Cluster Name..."
time="2022-06-20T21:04:20Z" level=debug msg="        Loading Base Domain..."
time="2022-06-20T21:04:20Z" level=debug msg="        Loading Platform..."
time="2022-06-20T21:04:20Z" level=debug msg="      Loading Networking..."
time="2022-06-20T21:04:20Z" level=debug msg="        Loading Platform..."
time="2022-06-20T21:04:20Z" level=debug msg="      Loading Pull Secret..."
time="2022-06-20T21:04:20Z" level=debug msg="      Loading Platform..."
time="2022-06-20T21:04:23Z" level=debug msg="    Using Install Config loaded from target directory"
time="2022-06-20T21:04:23Z" level=debug msg="  Loading Platform Credentials Check..."
time="2022-06-20T21:04:23Z" level=debug msg="    Loading Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="  Loading Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="  Loading Image..."
time="2022-06-20T21:04:23Z" level=debug msg="    Loading Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="  Loading Master Ignition Config..."
time="2022-06-20T21:04:23Z" level=debug msg="    Loading Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="    Loading Root CA..."
time="2022-06-20T21:04:23Z" level=debug msg="  Fetching Cluster ID..."
time="2022-06-20T21:04:23Z" level=debug msg="    Fetching Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="    Reusing previously-fetched Install Config"
time="2022-06-20T21:04:23Z" level=debug msg="  Generating Cluster ID..."
time="2022-06-20T21:04:23Z" level=debug msg="  Fetching Platform Credentials Check..."
time="2022-06-20T21:04:23Z" level=debug msg="    Fetching Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="    Reusing previously-fetched Install Config"
time="2022-06-20T21:04:23Z" level=debug msg="  Generating Platform Credentials Check..."
time="2022-06-20T21:04:23Z" level=debug msg="  Fetching Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="  Reusing previously-fetched Install Config"
time="2022-06-20T21:04:23Z" level=debug msg="  Fetching Image..."
time="2022-06-20T21:04:23Z" level=debug msg="    Fetching Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="    Reusing previously-fetched Install Config"
time="2022-06-20T21:04:23Z" level=debug msg="  Generating Image..."
time="2022-06-20T21:04:23Z" level=debug msg="  Fetching Master Ignition Config..."
time="2022-06-20T21:04:23Z" level=debug msg="    Fetching Install Config..."
time="2022-06-20T21:04:23Z" level=debug msg="    Reusing previously-fetched Install Config"
time="2022-06-20T21:04:23Z" level=debug msg="    Fetching Root CA..."
time="2022-06-20T21:04:23Z" level=debug msg="    Generating Root CA..."
time="2022-06-20T21:04:23Z" level=debug msg="  Generating Master Ignition Config..."
time="2022-06-20T21:04:23Z" level=debug msg="Generating Master Machines..."
time="2022-06-20T21:04:24Z" level=fatal msg="failed to fetch Master Machines: failed to generate asset \"Master Machines\": getting subnet existing-network-1-us-east-1: Not Found"


What did you expect to happen?
installer would properly set the Regional IBM Cloud client endpoint for looking up resources/services/etc.


How to reproduce it (as minimally and precisely as possible)?

1. Create install-config.yaml for non us-south region
2. Specify dedicated hosts (or use BYON code branch, for 4.12) in that region
3. Create manifests


Anything else we need to know?
IBM Cloud will be submitting a PR to address the dedicated host issues and BYON issues, by setting the Regional endpoint.

Comment 1 Christopher J Schaefer 2022-06-20 21:26:53 UTC
I had been testing these changes, to add BYON support, for 4.12, when I discovered this issue
https://github.com/cjschaef/installer/commit/867242923619ede2d703a53450c5beca7ac7cb0c

I also was notified that the current DedicatedHosts support, for 4.11, also suffers from this issue when using a different region other than us-south.

I have a PR open to address the DedicatedHosts issue, but one that also addresses any other resources (such as BYON resources) for future support as well.
https://github.com/openshift/installer/pull/6032

Comment 6 MayXu 2022-07-07 02:59:25 UTC
fail to reproduced this issue with dedicated host profile. 
test with 4.11 on eu-de region, test PASS. 
instal-config.yaml:
controlPlane:
   architecture: amd64
   hyperthreading: Enabled
   name: master
   platform:
     ibmcloud:
       type: cx2-8x16
       zones:
       - eu-de-1
       - eu-de-3
       dedicatedHosts:
       - profile: cx2-host-152x304
       - profile: cx2-host-152x304
   replicas: 3
 compute:
 - architecture: amd64
   hyperthreading: Enabled
   name: worker
   platform:
     ibmcloud:
       type: bx2d-4x16
       zones:
       - eu-de-1
       dedicatedHosts:
       - profile: bx2d-host-152x608
   replicas: 2

Comment 8 Christopher J Schaefer 2022-08-09 18:36:38 UTC
After additional discussions with IBM Cloud team, I have determined this issue was found when performing testing in a specific account, for a specific zone, with a unique host profile.

Since, this path does not sound like it can be easily replicated, I will say the affect on DedicatedHosts was invalid.

However, to verify the fix applied for this issue, in relation to the BYON support,


The following steps can be taken.

I have reverted the patch to the release-4.12 branch of the installer, making an additional commit to remove the BYON changes which also set the Regional URL, available via
https://github.com/cjschaef/installer/compare/3febdea734bb18ba2eeb9ff799bd95fe6ca095b4...bz_2099401-verify

Then, building a custom installer off this branch, I used a install-config.yaml with BYON resources not in us-south

platform:
  ibmcloud:
    region: us-east
    resourceGroupName: existing-network-1
    vpcName: existing-network-1-vpc
    controlPlaneSubnets:
    - existing-network-1-us-east-1
    - existing-network-1-us-east-2
    - existing-network-1-us-east-3
    computeSubnets:
    - existing-compute-network-1-us-east-1
    - existing-compute-network-1-us-east-2
    - existing-compute-network-1-us-east-3


This results in the following failure, when the installer attempts to look for BYON resources only in us-south (default) and not in the proper region.

# go/src/github.com/openshift/installer/bin/openshift-install version; go/src/github.com/openshift/installer/bin/openshift-install create manifests --dir bz_2099401/bz2099401-c2
go/src/github.com/openshift/installer/bin/openshift-install unreleased-master-6302-g70c40257000a8d5924d1ad32e26dca60a23df70f
built from commit 70c40257000a8d5924d1ad32e26dca60a23df70f
release image registry.ci.openshift.org/origin/release:4.12
release architecture amd64
ERROR failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: platform.ibmcloud.vpcName: Not found: "existing-network-1-vpc"

The same install-config.yaml can be tested using master or release-4.12 as it is today, to verify validation of BYON resources during manifest generation.

# go/src/github.com/openshift/installer/bin/openshift-install version; go/src/github.com/openshift/installer/bin/openshift-install create manifests --dir bz_2099401/bz2099401-c2
go/src/github.com/openshift/installer/bin/openshift-install unreleased-master-6300-g3febdea734bb18ba2eeb9ff799bd95fe6ca095b4
built from commit 3febdea734bb18ba2eeb9ff799bd95fe6ca095b4
release image registry.ci.openshift.org/origin/release:4.12
release architecture amd64
INFO Consuming Install Config from target directory 
INFO Manifests created in: bz_2099401/bz2099401-c2/manifests and bz_2099401/bz2099401-c2/openshift


Please use this process to verify the Regional URL bug has been resolved now, using partial BYON support for manifest generation.
Full BYON support for IPI will be available with https://github.com/openshift/installer/pull/6183, if that's necessary.

Comment 9 Christopher J Schaefer 2022-08-10 18:28:54 UTC
I made a draft PR
https://github.com/openshift/installer/pull/6214

with the necessary BYON change to use for verifying this bug
https://github.com/openshift/installer/pull/6214/commits/e7d4ddc5c7b5a0a7346ff4dfef603c504df3c862

and slight modifications in order to build the binary
https://github.com/openshift/installer/pull/6214/commits/33d24c9c8e64843e629c706da97a5cece375b8db


# bin/openshift-install create manifests --dir bz_2099401/failure-1/
ERROR failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: platform.ibmcloud.vpcName: Not found: "us-east-existing-network-2-vpc" 


# cat bz_2099401/failure-1/install-config.yaml | yq '.platform'
ibmcloud:
  region: us-east
  resourceGroupName: us-east-existing-network-2
  vpcName: us-east-existing-network-2-vpc
  controlPlaneSubnets:
    - us-east-existing-network-2-cp-wdc-1
    - us-east-existing-network-2-cp-wdc-2
    - us-east-existing-network-2-cp-wdc-3
  computeSubnets:
    - us-east-existing-network-2-compute-wdc-1
    - us-east-existing-network-2-compute-wdc-2
    - us-east-existing-network-2-compute-wdc-3


Otherwise, given the complexity to verify this and the fact that the original BZ report was under the assumption that this affected DedicatedHosts (and not just BYON work) was incorrect. This could be deemed invalid/bad, if that's desired.

Comment 10 MayXu 2022-08-11 09:06:24 UTC
1. with the build quay.io/openshift-release-dev/ocp-release:4.11.0-x86_64,https://github.com/openshift/installer/pull/6214 registry.build05.ci.openshift.org/ci-ln-4v7zymt/release:latest 
$bin/openshift-install create manifests
reproduce the bug correctly. 

2. with the build https://github.com/openshift/installer/pull/6183 based on the master, registry.build05.ci.openshift.org/ci-ln-2yxrzyk/release:latest 
$bin/openshift-install create manifests succeed. create the cluster succeed. 
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  ibmcloud:
    computeSubnets:
    - maxu-ibm-v429h-subnet-compute-eu-gb-1
    - maxu-ibm-v429h-subnet-compute-eu-gb-2
    - maxu-ibm-v429h-subnet-compute-eu-gb-3
    controlPlaneSubnets:
    - maxu-ibm-v429h-subnet-control-plane-eu-gb-1
    - maxu-ibm-v429h-subnet-control-plane-eu-gb-2
    - maxu-ibm-v429h-subnet-control-plane-eu-gb-3
    region: eu-gb
    resourceGroupName: maxu-ibm-v429h
    vpcName: maxu-ibm-v429h-vpc

Comment 13 errata-xmlrpc 2023-01-17 19:50:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399


Note You need to log in before you can comment on or make changes to this bug.