Hide Forgot
Version: 4.11 (expect fix to land for 4.12, with 4.11 being Tech Preview) Platform: ibmcloud Please specify: IPI What happened? The generated IBM Client for the InstallConfig (specifically for Metadata usage, looking up additional resource details), does not set the IBM Cloud Regional endpoint, and defaults to us-south. Any specified or desired resources in other regions cannot be found, e.g. Dedicated Hosts, BYON, etc. time="2022-06-20T21:04:20Z" level=debug msg="OpenShift Installer unreleased-master-6051-ge96b5145285ae158c4b1757adf48c72fab13cdca" time="2022-06-20T21:04:20Z" level=debug msg="Built from commit e96b5145285ae158c4b1757adf48c72fab13cdca" time="2022-06-20T21:04:20Z" level=debug msg="Fetching Master Machines..." time="2022-06-20T21:04:20Z" level=debug msg="Loading Master Machines..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Cluster ID..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Install Config..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading SSH Key..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Base Domain..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Platform..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Cluster Name..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Base Domain..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Platform..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Networking..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Platform..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Pull Secret..." time="2022-06-20T21:04:20Z" level=debug msg=" Loading Platform..." time="2022-06-20T21:04:23Z" level=debug msg=" Using Install Config loaded from target directory" time="2022-06-20T21:04:23Z" level=debug msg=" Loading Platform Credentials Check..." time="2022-06-20T21:04:23Z" level=debug msg=" Loading Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Loading Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Loading Image..." time="2022-06-20T21:04:23Z" level=debug msg=" Loading Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Loading Master Ignition Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Loading Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Loading Root CA..." time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Cluster ID..." time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Reusing previously-fetched Install Config" time="2022-06-20T21:04:23Z" level=debug msg=" Generating Cluster ID..." time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Platform Credentials Check..." time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Reusing previously-fetched Install Config" time="2022-06-20T21:04:23Z" level=debug msg=" Generating Platform Credentials Check..." time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Reusing previously-fetched Install Config" time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Image..." time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Reusing previously-fetched Install Config" time="2022-06-20T21:04:23Z" level=debug msg=" Generating Image..." time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Master Ignition Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Install Config..." time="2022-06-20T21:04:23Z" level=debug msg=" Reusing previously-fetched Install Config" time="2022-06-20T21:04:23Z" level=debug msg=" Fetching Root CA..." time="2022-06-20T21:04:23Z" level=debug msg=" Generating Root CA..." time="2022-06-20T21:04:23Z" level=debug msg=" Generating Master Ignition Config..." time="2022-06-20T21:04:23Z" level=debug msg="Generating Master Machines..." time="2022-06-20T21:04:24Z" level=fatal msg="failed to fetch Master Machines: failed to generate asset \"Master Machines\": getting subnet existing-network-1-us-east-1: Not Found" What did you expect to happen? installer would properly set the Regional IBM Cloud client endpoint for looking up resources/services/etc. How to reproduce it (as minimally and precisely as possible)? 1. Create install-config.yaml for non us-south region 2. Specify dedicated hosts (or use BYON code branch, for 4.12) in that region 3. Create manifests Anything else we need to know? IBM Cloud will be submitting a PR to address the dedicated host issues and BYON issues, by setting the Regional endpoint.
I had been testing these changes, to add BYON support, for 4.12, when I discovered this issue https://github.com/cjschaef/installer/commit/867242923619ede2d703a53450c5beca7ac7cb0c I also was notified that the current DedicatedHosts support, for 4.11, also suffers from this issue when using a different region other than us-south. I have a PR open to address the DedicatedHosts issue, but one that also addresses any other resources (such as BYON resources) for future support as well. https://github.com/openshift/installer/pull/6032
fail to reproduced this issue with dedicated host profile. test with 4.11 on eu-de region, test PASS. instal-config.yaml: controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: ibmcloud: type: cx2-8x16 zones: - eu-de-1 - eu-de-3 dedicatedHosts: - profile: cx2-host-152x304 - profile: cx2-host-152x304 replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: ibmcloud: type: bx2d-4x16 zones: - eu-de-1 dedicatedHosts: - profile: bx2d-host-152x608 replicas: 2
After additional discussions with IBM Cloud team, I have determined this issue was found when performing testing in a specific account, for a specific zone, with a unique host profile. Since, this path does not sound like it can be easily replicated, I will say the affect on DedicatedHosts was invalid. However, to verify the fix applied for this issue, in relation to the BYON support, The following steps can be taken. I have reverted the patch to the release-4.12 branch of the installer, making an additional commit to remove the BYON changes which also set the Regional URL, available via https://github.com/cjschaef/installer/compare/3febdea734bb18ba2eeb9ff799bd95fe6ca095b4...bz_2099401-verify Then, building a custom installer off this branch, I used a install-config.yaml with BYON resources not in us-south platform: ibmcloud: region: us-east resourceGroupName: existing-network-1 vpcName: existing-network-1-vpc controlPlaneSubnets: - existing-network-1-us-east-1 - existing-network-1-us-east-2 - existing-network-1-us-east-3 computeSubnets: - existing-compute-network-1-us-east-1 - existing-compute-network-1-us-east-2 - existing-compute-network-1-us-east-3 This results in the following failure, when the installer attempts to look for BYON resources only in us-south (default) and not in the proper region. # go/src/github.com/openshift/installer/bin/openshift-install version; go/src/github.com/openshift/installer/bin/openshift-install create manifests --dir bz_2099401/bz2099401-c2 go/src/github.com/openshift/installer/bin/openshift-install unreleased-master-6302-g70c40257000a8d5924d1ad32e26dca60a23df70f built from commit 70c40257000a8d5924d1ad32e26dca60a23df70f release image registry.ci.openshift.org/origin/release:4.12 release architecture amd64 ERROR failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: platform.ibmcloud.vpcName: Not found: "existing-network-1-vpc" The same install-config.yaml can be tested using master or release-4.12 as it is today, to verify validation of BYON resources during manifest generation. # go/src/github.com/openshift/installer/bin/openshift-install version; go/src/github.com/openshift/installer/bin/openshift-install create manifests --dir bz_2099401/bz2099401-c2 go/src/github.com/openshift/installer/bin/openshift-install unreleased-master-6300-g3febdea734bb18ba2eeb9ff799bd95fe6ca095b4 built from commit 3febdea734bb18ba2eeb9ff799bd95fe6ca095b4 release image registry.ci.openshift.org/origin/release:4.12 release architecture amd64 INFO Consuming Install Config from target directory INFO Manifests created in: bz_2099401/bz2099401-c2/manifests and bz_2099401/bz2099401-c2/openshift Please use this process to verify the Regional URL bug has been resolved now, using partial BYON support for manifest generation. Full BYON support for IPI will be available with https://github.com/openshift/installer/pull/6183, if that's necessary.
I made a draft PR https://github.com/openshift/installer/pull/6214 with the necessary BYON change to use for verifying this bug https://github.com/openshift/installer/pull/6214/commits/e7d4ddc5c7b5a0a7346ff4dfef603c504df3c862 and slight modifications in order to build the binary https://github.com/openshift/installer/pull/6214/commits/33d24c9c8e64843e629c706da97a5cece375b8db # bin/openshift-install create manifests --dir bz_2099401/failure-1/ ERROR failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: platform.ibmcloud.vpcName: Not found: "us-east-existing-network-2-vpc" # cat bz_2099401/failure-1/install-config.yaml | yq '.platform' ibmcloud: region: us-east resourceGroupName: us-east-existing-network-2 vpcName: us-east-existing-network-2-vpc controlPlaneSubnets: - us-east-existing-network-2-cp-wdc-1 - us-east-existing-network-2-cp-wdc-2 - us-east-existing-network-2-cp-wdc-3 computeSubnets: - us-east-existing-network-2-compute-wdc-1 - us-east-existing-network-2-compute-wdc-2 - us-east-existing-network-2-compute-wdc-3 Otherwise, given the complexity to verify this and the fact that the original BZ report was under the assumption that this affected DedicatedHosts (and not just BYON work) was incorrect. This could be deemed invalid/bad, if that's desired.
1. with the build quay.io/openshift-release-dev/ocp-release:4.11.0-x86_64,https://github.com/openshift/installer/pull/6214 registry.build05.ci.openshift.org/ci-ln-4v7zymt/release:latest $bin/openshift-install create manifests reproduce the bug correctly. 2. with the build https://github.com/openshift/installer/pull/6183 based on the master, registry.build05.ci.openshift.org/ci-ln-2yxrzyk/release:latest $bin/openshift-install create manifests succeed. create the cluster succeed. networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: ibmcloud: computeSubnets: - maxu-ibm-v429h-subnet-compute-eu-gb-1 - maxu-ibm-v429h-subnet-compute-eu-gb-2 - maxu-ibm-v429h-subnet-compute-eu-gb-3 controlPlaneSubnets: - maxu-ibm-v429h-subnet-control-plane-eu-gb-1 - maxu-ibm-v429h-subnet-control-plane-eu-gb-2 - maxu-ibm-v429h-subnet-control-plane-eu-gb-3 region: eu-gb resourceGroupName: maxu-ibm-v429h vpcName: maxu-ibm-v429h-vpc
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399