Version: 4.10 Platform: ibmcloud Please specify: IPI What happened? Using a customized install-config.yaml with IPI and IBM Cloud, the installer does not use certain values from the install-config.yaml and uses default values. For instance, the machine type will always be set to the default value. What did you expect to happen? The installer would use the custom values set in the install-config.yaml for IBM Cloud IPI deploys. How to reproduce it (as minimally and precisely as possible)? Create an install-config.yaml for IBM Cloud, and add blocks with customized values, such as the compute platform type block 1. openshift-install create install-config 2. Add a 'type' to the compute.platform.ibmcloud (or control-plane) block. 3. openshift-install create manifests Anything else we need to know? IBM Cloud is working on a fix for this bug currently.
Hi Christopher, I'm using latest accepted 4.10 nightly version (4.10.0-0.nightly-2021-10-05-151518) to test the PR but it fails with a SIGSEGV error, not sure if related with this change or a previous one, can you please double check? ~~~ $ ./openshift-install version ./openshift-install 4.10.0-0.nightly-2021-10-05-151518 built from commit 90d215780753b3e456ad434e9ac963da178f456d release image registry.ci.openshift.org/ocp/release@sha256:5f5fe78285d4eed84f05e018b578d165ec579c553160095c9ec7122e35528d7f release architecture amd64 $ ./openshift-install create cluster --dir test30/ --log-level debug DEBUG OpenShift Installer 4.10.0-0.nightly-2021-10-05-151518 DEBUG Built from commit 90d215780753b3e456ad434e9ac963da178f456d DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Using Install Config loaded from target directory DEBUG Loading Install Config... DEBUG Loading Bootstrap Ignition Config... DEBUG Loading Ironic bootstrap credentials... DEBUG Loading CVO Ignore... DEBUG Loading Common Manifests... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading Ingress Config... DEBUG Loading Install Config... DEBUG Loading DNS Config... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Install Config... DEBUG Loading Infrastructure Config... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading Cloud Provider Config... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Additional Trust Bundle Config... DEBUG Loading Install Config... DEBUG Loading Network Config... DEBUG Loading Install Config... DEBUG Loading Network CRDs... DEBUG Loading Proxy Config... DEBUG Loading Install Config... DEBUG Loading Network Config... DEBUG Loading Scheduler Config... DEBUG Loading Install Config... DEBUG Loading Image Content Source Policy... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Certificate (mcs)... DEBUG Loading Root CA... DEBUG Loading Install Config... DEBUG Loading CVOOverrides... DEBUG Loading KubeCloudConfig... DEBUG Loading KubeSystemConfigmapRootCA... DEBUG Loading MachineConfigServerTLSSecret... DEBUG Loading OpenshiftConfigSecretPullSecret... DEBUG Loading OpenshiftMachineConfigOperator... DEBUG Loading KubevirtInfraNamespace... DEBUG Loading Openshift Manifests... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Kubeadmin Password... DEBUG Loading OpenShift Install (Manifests)... DEBUG Loading CloudCredsSecret... DEBUG Loading KubeadminPasswordSecret... DEBUG Loading RoleCloudCredsSecretReader... DEBUG Loading Baremetal Config CR... DEBUG Loading Image... DEBUG Loading Install Config... DEBUG Loading AzureCloudProviderSecret... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Admin Internal Client... DEBUG Loading Certificate (admin-kubeconfig-client)... DEBUG Loading Certificate (admin-kubeconfig-signer)... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Kubelet... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kubelet-client)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Admin Client (Loopback)... DEBUG Loading Certificate (admin-kubeconfig-client)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Install Config... DEBUG Loading Master Ignition Customization Check... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Master Ignition Config... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Worker Ignition Customization Check... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Worker Ignition Config... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Master Machines... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Install Config... DEBUG Loading Image... DEBUG Loading Master Ignition Config... DEBUG Loading Worker Machines... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Install Config... DEBUG Loading Image... DEBUG Loading Worker Ignition Config... DEBUG Loading Common Manifests... DEBUG Loading Openshift Manifests... DEBUG Loading Proxy Config... DEBUG Loading Certificate (admin-kubeconfig-ca-bundle)... DEBUG Loading Certificate (admin-kubeconfig-signer)... DEBUG Loading Certificate (aggregator)... DEBUG Loading Certificate (aggregator-ca-bundle)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (system:kube-apiserver-proxy)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (system:kube-apiserver-proxy)... DEBUG Loading Certificate (aggregator)... DEBUG Loading Bootstrap SSH Key Pair... DEBUG Loading User-provided Service Account Signing key... DEBUG Loading Cloud Provider CA Bundle... DEBUG Loading Install Config... DEBUG Loading Certificate (journal-gatewayd)... DEBUG Loading Root CA... DEBUG Loading Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-external-lb-server)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-internal-lb-server)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-server)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-service-network-server)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-complete-client-ca-bundle)... DEBUG Loading Certificate (admin-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-client-ca-bundle)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (kube-control-plane-ca-bundle)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-client)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kube-control-plane-ca-bundle)... DEBUG Loading Certificate (kube-control-plane-kube-controller-manager-client)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-control-plane-kube-scheduler-client)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-client-ca-bundle)... DEBUG Loading Certificate (kubelet-client)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (kubelet-serving-ca-bundle)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (mcs)... DEBUG Loading Root CA... DEBUG Loading Key Pair (service-account.pub)... DEBUG Loading Release Image Pull Spec... DEBUG Loading Image... DEBUG Fetching Cluster ID... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Cluster ID... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Bootstrap Ignition Config... DEBUG Fetching Ironic bootstrap credentials... DEBUG Generating Ironic bootstrap credentials... DEBUG Fetching CVO Ignore... DEBUG Fetching Common Manifests... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Ingress Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Ingress Config... DEBUG Fetching DNS Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Platform Credentials Check... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Platform Credentials Check... DEBUG Generating DNS Config... DEBUG Fetching Infrastructure Config... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cloud Provider Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Platform Credentials Check... DEBUG Reusing previously-fetched Platform Credentials Check DEBUG Generating Cloud Provider Config... panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xcfd8c99] goroutine 1 [running]: github.com/openshift/installer/pkg/asset/manifests.(*CloudProviderConfig).Generate(0xc000dda2d0, 0xc000d8fb30, 0x104ea80d, 0x12) /go/src/github.com/openshift/installer/pkg/asset/manifests/cloudproviderconfig.go:180 +0x1599 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c63338, 0xc000dda2d0, 0xc000923db6, 0xa, 0xc000923db6, 0xa) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x7df github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c633c8, 0xc000d20ba0, 0xc000923cd8, 0x8, 0xc000923cd8, 0x8) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c63428, 0xc0014d52c0, 0xc000654c70, 0x6, 0xc000654c70, 0x6) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c62eb8, 0xc000d209a8, 0xc000654c44, 0x4, 0xc000654c44, 0x4) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c62e88, 0xc000bb0790, 0x1047d240, 0x2, 0x1047d240, 0x2) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x7f15462565a0, 0x19c7e4f8, 0x0, 0x0, 0x40b845, 0xebe9d20) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000b312c0, 0x7f15462565a0, 0x19c7e4f8, 0x19c460c0, 0x8, 0x8, 0xed8ef625a, 0x72919d37a15288a4) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4b main.runTargetCmd.func1(0x7ffcf07bd1dd, 0x7, 0xc000622900, 0x0) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:238 +0x12d main.runTargetCmd.func2(0x19c50d20, 0xc000cecfc0, 0x0, 0x4) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:265 +0xb5 github.com/spf13/cobra.(*Command).execute(0x19c50d20, 0xc000cecf80, 0x4, 0x4, 0x19c50d20, 0xc000cecf80) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:854 +0x2c2 github.com/spf13/cobra.(*Command).ExecuteC(0xc000291080, 0xc000dd3de8, 0x1, 0x1) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:958 +0x375 github.com/spf13/cobra.(*Command).Execute(...) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:895 main.installerMain() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:72 +0x2fe main.main() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:50 +0x259 ~~~ NOTE: I have also tried with a local compiled version from latest repository content but same error. Regards.
[UPDATE] Tested also with version "4.10.0-0.ci-2021-10-06-062517", which includes PR#5263, same SIGSEGV error: ~~~ $ ./openshift-install create cluster --dir test31/ --log-level debug DEBUG OpenShift Installer 4.10.0-0.ci-2021-10-06-062517 DEBUG Built from commit 4604929ab0c9948fcde1444bfd4df56d9dcbe934 DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Using Install Config loaded from target directory DEBUG Loading Install Config... DEBUG Loading Bootstrap Ignition Config... DEBUG Loading Ironic bootstrap credentials... DEBUG Loading CVO Ignore... DEBUG Loading Common Manifests... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading Ingress Config... DEBUG Loading Install Config... DEBUG Loading DNS Config... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Install Config... DEBUG Loading Infrastructure Config... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading Cloud Provider Config... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Additional Trust Bundle Config... DEBUG Loading Install Config... DEBUG Loading Network Config... DEBUG Loading Install Config... DEBUG Loading Network CRDs... DEBUG Loading Proxy Config... DEBUG Loading Install Config... DEBUG Loading Network Config... DEBUG Loading Scheduler Config... DEBUG Loading Install Config... DEBUG Loading Image Content Source Policy... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Certificate (mcs)... DEBUG Loading Root CA... DEBUG Loading Install Config... DEBUG Loading CVOOverrides... DEBUG Loading KubeCloudConfig... DEBUG Loading KubeSystemConfigmapRootCA... DEBUG Loading MachineConfigServerTLSSecret... DEBUG Loading OpenshiftConfigSecretPullSecret... DEBUG Loading OpenshiftMachineConfigOperator... DEBUG Loading KubevirtInfraNamespace... DEBUG Loading Openshift Manifests... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Kubeadmin Password... DEBUG Loading OpenShift Install (Manifests)... DEBUG Loading CloudCredsSecret... DEBUG Loading KubeadminPasswordSecret... DEBUG Loading RoleCloudCredsSecretReader... DEBUG Loading Baremetal Config CR... DEBUG Loading Image... DEBUG Loading Install Config... DEBUG Loading AzureCloudProviderSecret... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Admin Internal Client... DEBUG Loading Certificate (admin-kubeconfig-client)... DEBUG Loading Certificate (admin-kubeconfig-signer)... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Kubelet... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kubelet-client)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Admin Client (Loopback)... DEBUG Loading Certificate (admin-kubeconfig-client)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Install Config... DEBUG Loading Master Ignition Customization Check... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Master Ignition Config... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Worker Ignition Customization Check... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Worker Ignition Config... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Master Machines... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Install Config... DEBUG Loading Image... DEBUG Loading Master Ignition Config... DEBUG Loading Worker Machines... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Install Config... DEBUG Loading Image... DEBUG Loading Worker Ignition Config... DEBUG Loading Common Manifests... DEBUG Loading Openshift Manifests... DEBUG Loading Proxy Config... DEBUG Loading Certificate (admin-kubeconfig-ca-bundle)... DEBUG Loading Certificate (admin-kubeconfig-signer)... DEBUG Loading Certificate (aggregator)... DEBUG Loading Certificate (aggregator-ca-bundle)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (system:kube-apiserver-proxy)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (system:kube-apiserver-proxy)... DEBUG Loading Certificate (aggregator)... DEBUG Loading Bootstrap SSH Key Pair... DEBUG Loading User-provided Service Account Signing key... DEBUG Loading Cloud Provider CA Bundle... DEBUG Loading Install Config... DEBUG Loading Certificate (journal-gatewayd)... DEBUG Loading Root CA... DEBUG Loading Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-external-lb-server)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-internal-lb-server)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-server)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-service-network-server)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-complete-client-ca-bundle)... DEBUG Loading Certificate (admin-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-client-ca-bundle)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (kube-control-plane-ca-bundle)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-client)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kube-control-plane-ca-bundle)... DEBUG Loading Certificate (kube-control-plane-kube-controller-manager-client)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-control-plane-kube-scheduler-client)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-client-ca-bundle)... DEBUG Loading Certificate (kubelet-client)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (kubelet-serving-ca-bundle)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (mcs)... DEBUG Loading Root CA... DEBUG Loading Key Pair (service-account.pub)... DEBUG Loading Release Image Pull Spec... DEBUG Loading Image... DEBUG Fetching Cluster ID... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Cluster ID... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Bootstrap Ignition Config... DEBUG Fetching Ironic bootstrap credentials... DEBUG Generating Ironic bootstrap credentials... DEBUG Fetching CVO Ignore... DEBUG Fetching Common Manifests... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Ingress Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Ingress Config... DEBUG Fetching DNS Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Platform Credentials Check... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Platform Credentials Check... DEBUG Generating DNS Config... DEBUG Fetching Infrastructure Config... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cloud Provider Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Platform Credentials Check... DEBUG Reusing previously-fetched Platform Credentials Check DEBUG Generating Cloud Provider Config... panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xcfd8cb9] goroutine 1 [running]: github.com/openshift/installer/pkg/asset/manifests.(*CloudProviderConfig).Generate(0xc001353060, 0xc001331740, 0x104eb80d, 0x12) /go/src/github.com/openshift/installer/pkg/asset/manifests/cloudproviderconfig.go:180 +0x1599 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64bb8, 0xc001353060, 0xc001356f96, 0xa, 0xc001356f96, 0xa) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x7df github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64c48, 0xc001284330, 0xc001356eb8, 0x8, 0xc001356eb8, 0x8) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64ca8, 0xc001011860, 0xc0011d9b10, 0x6, 0xc0011d9b10, 0x6) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64738, 0xc001284180, 0xc0011d9ae4, 0x4, 0xc0011d9ae4, 0x4) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64708, 0xc00127d250, 0x1047e240, 0x2, 0x1047e240, 0x2) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x7ff57c3b9e90, 0x19c7f4f8, 0x0, 0x0, 0x40b845, 0xebead20) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625 github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000b404e0, 0x7ff57c3b9e90, 0x19c7f4f8, 0x19c470c0, 0x8, 0x8, 0xed8ef6606, 0x83d98d966d665e58) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4b main.runTargetCmd.func1(0x7ffe389771dd, 0x7, 0xc000c08918, 0x0) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:238 +0x12d main.runTargetCmd.func2(0x19c51d20, 0xc000aea6c0, 0x0, 0x4) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:265 +0xb5 github.com/spf13/cobra.(*Command).execute(0x19c51d20, 0xc000aea680, 0x4, 0x4, 0x19c51d20, 0xc000aea680) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:854 +0x2c2 github.com/spf13/cobra.(*Command).ExecuteC(0xc0001dd080, 0xc000ce3de8, 0x1, 0x1) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:958 +0x375 github.com/spf13/cobra.(*Command).Execute(...) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:895 main.installerMain() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:72 +0x2fe main.main() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:50 +0x259 ~~~
@pamoedom That crash looks like a separate bug. Can you share the install-config.yaml that you were using? I suspect that the installer is not setting the defaults properly for the machine pools.
On the following lines of code, there is an expectation that the platform is populated for the machine pools. That is not a valid expectation. https://github.com/openshift/installer/blob/6617bc2e334654bb6e85976f049e51fc1c01aa3f/pkg/asset/manifests/cloudproviderconfig.go#L173 https://github.com/openshift/installer/blob/6617bc2e334654bb6e85976f049e51fc1c01aa3f/pkg/asset/manifests/cloudproviderconfig.go#L180
Hi Matthew, I always use the same "install-config.yaml" for my ibmcloud tests and never failed before, the content is the following: ~~~ apiVersion: v1 baseDomain: ibmcloud.qe.devcluster.openshift.com compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 0 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: ibmcloud: type: bx2-8x32 replicas: 3 metadata: creationTimestamp: null name: pamoedo-test networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: ibmcloud: region: eu-de resourceGroupName: pamoedom-rg publish: External pullSecret: ... sshKey: ... ~~~ What do you mean by "there is an expectation that the platform is populated for the machine pools. That is not a valid expectation."? can't we set different machine types per pool?
The code that is causing the crash was added only 6 days ago [1]. Perhaps this is the first time that you have tried an install to IBM cloud since then. > What do you mean by "there is an expectation that the platform is populated for the machine pools. That is not a valid expectation."? can't we set different machine types per pool? I mean that there is a bug in the code. The code is expecting that the machine pool has the ibmcloud field set in the platform field. You should be able to work around this by explicitly setting the ibmcloud field. ~~~~ compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: ibmcloud: {} replicas: 0 ~~~
Thanks Matthew, I understood now your comment, and yes, most probably I haven't tried new builds for more than a week. I will try again with your suggestion, thanks.
Hi again Matthew, your suggestion did the trick and I no longer suffer the SIGSEGV, however, for our IBMCloud account we have to bypass BZ#1992777 issue first in order to see if the changes are having effect on the instances, for that reason I'm usign a local compiled version as follows: ~~~ $ ./openshift-install-local version ./openshift-install-local unreleased-master-5073-ga2e7a78af1da644e2b03ae9c14e8249fcb69abe3-dirty built from commit a2e7a78af1da644e2b03ae9c14e8249fcb69abe3 release image registry.ci.openshift.org/origin/release:4.8 release architecture amd64 ~~~ With this version I can bypass BZ#1992777 and continue the execution but to my surprise I've found a new unexpected issue with remote rules limit per security group: ~~~ ERROR ERROR Error: Error while creating Security Group Rule Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group). ERROR ERROR Adding a rule would exceed the limit of remote rules per security group. Consider creating another security group. ERROR { ERROR "StatusCode": 400, ERROR "Headers": { ERROR "Cache-Control": [ ERROR "max-age=0, no-cache, no-store, must-revalidate" ERROR ], ERROR "Cf-Cache-Status": [ ERROR "DYNAMIC" ERROR ], ERROR "Cf-Ray": [ ERROR "699ffed098146653-MAD" ERROR ], ERROR "Connection": [ ERROR "keep-alive" ERROR ], ERROR "Content-Length": [ ERROR "373" ERROR ], ERROR "Content-Type": [ ERROR "application/json" ERROR ], ERROR "Date": [ ERROR "Wed, 06 Oct 2021 15:41:42 GMT" ERROR ], ERROR "Expect-Ct": [ ERROR "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" ERROR ], ERROR "Expires": [ ERROR "-1" ERROR ], ERROR "Pragma": [ ERROR "no-cache" ERROR ], ERROR "Server": [ ERROR "cloudflare" ERROR ], ERROR "Strict-Transport-Security": [ ERROR "max-age=31536000; includeSubDomains" ERROR ], ERROR "Vary": [ ERROR "Accept-Encoding" ERROR ], ERROR "X-Content-Type-Options": [ ERROR "nosniff" ERROR ], ERROR "X-Request-Id": [ ERROR "2f23ff2f-9acc-447a-86e4-2730be710d37" ERROR ], ERROR "X-Xss-Protection": [ ERROR "1; mode=block" ERROR ] ERROR }, ERROR "Result": { ERROR "errors": [ ERROR { ERROR "code": "over_quota", ERROR "message": "Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group).\n\nAdding a rule would exceed the limit of remote rules per security group. Consider creating another security group.", ERROR "more_info": "https://cloud.ibm.com/docs/vpc?topic=vpc-quotas" ERROR } ERROR ], ERROR "trace": "2f23ff2f-9acc-447a-86e4-2730be710d37" ERROR }, ERROR "RawResult": null ERROR } ERROR ERROR ERROR on ../../../../tmp/openshift-install-network-003889995/vpc/security-groups.tf line 268, in resource "ibm_is_security_group_rule" "control_plane_machine_config_lb_inbound": ERROR 268: resource "ibm_is_security_group_rule" "control_plane_machine_config_lb_inbound" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change ~~~ As per IBM docs here[1], there is a hardcoded limit of 5 remote rules per security group, any recent change in your radar that could be causing this? any available workaround? [1] - https://cloud.ibm.com/docs/vpc?topic=vpc-quotas Regards.
We think this enhancement might have caused the problem, overloading the security group. I'll work with our dev team more and look at getting a separate BZ open to address the issue. https://github.com/openshift/installer/pull/5105/files#diff-b237721806432aac5fdc1b5a78484516c931af5affbbcea351e8824b5f9fcd54
Hi Christopher, this BZ has moved now to "ON_QA" but I can't properly test the changes due to the remote rules limit per SG, please provide me the new BZ info when you have it so we can mark this one as blocked accordingly, thanks.
@
@pamoedom I opened a BZ for the `5 rule limit` error you saw above. https://bugzilla.redhat.com/show_bug.cgi?id=2012233
Thanks Christopher, appreciated, I've set dependency between them.
[QA Summary] [Version] ~~~ $ ./openshift-install-local version ./openshift-install-local unreleased-master-5104-g527c46172e6c4fe28838e1a18e0d758eec61fb02 built from commit 527c46172e6c4fe28838e1a18e0d758eec61fb02 release image registry.ci.openshift.org/origin/release:4.8 release architecture amd64 $ git --no-pager log --oneline --first-parent origin/master | grep "5289\|5263" 527c46172 Merge pull request #5289 from cjschaef/bz_2012233 4604929ab Merge pull request #5263 from cjschaef/bz_2009791 ~~~ [Environment] ~~~ apiVersion: v1 baseDomain: ibmcloud.qe.devcluster.openshift.com compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: ibmcloud: {} replicas: 0 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: ibmcloud: type: bx2-8x32 replicas: 3 metadata: creationTimestamp: null name: pamoedo-test networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: ibmcloud: region: eu-de resourceGroupName: pamoedom-rg publish: External pullSecret: sshKey: ~~~ [Results] ~~~ $ DIGEST=$(skopeo inspect --authfile pull-secret docker://registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-10-13-081040 | grep Digest | cut -d '"' -f4) $ export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.ci.openshift.org/ocp/release@${DIGEST} $ export IC_API_KEY='<obfuscated>' $ ./openshift-install-local create cluster --dir test37/ --log-level debug DEBUG OpenShift Installer unreleased-master-5104-g527c46172e6c4fe28838e1a18e0d758eec61fb02 DEBUG Built from commit 527c46172e6c4fe28838e1a18e0d758eec61fb02 ... DEBUG Apply complete! Resources: 60 added, 0 changed, 0 destroyed. DEBUG DEBUG Outputs: DEBUG DEBUG control_plane_security_group_id_list = [ DEBUG "r010-6a6c0109-3b56-413f-8769-f2e09314654c", DEBUG "r010-8624c529-7c01-4563-ac6e-7bd7d86fc470", DEBUG "r010-b193d9de-0d15-4fa6-bb09-16c4f3093e20", DEBUG "r010-3a5c82e4-5102-4b68-ba04-a46ce6799171", DEBUG ] DEBUG control_plane_subnet_id_list = [ DEBUG "02b7-9db6e4f0-d0d1-4fa9-b9f5-b26afb517e84", DEBUG "02c7-519c5479-1142-47d8-b1bf-71848785c0d6", DEBUG "02d7-1cab091f-888f-45c6-826d-20513c4eb14d", DEBUG ] DEBUG control_plane_subnet_zone_list = [ DEBUG "eu-de-1", DEBUG "eu-de-2", DEBUG "eu-de-3", DEBUG ] ... ERROR Attempted to gather debug logs after installation failure: bootstrap host address and at least one control plane host address must be provided ERROR Bootstrap failed to complete: timed out waiting for the condition ERROR Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane. ERROR Attempted to analyze the debug logs after installation failure: could not open the gather bundle: open : no such file or directory FATAL Bootstrap failed to complete $ ibmcloud is instances ID Name Status Address Floating IP Profile Image VPC Zone Resource group 02b7_70e6a3c7-3353-450f-a86a-5adfde8c884c pamoedo-test37-c6hqt-bootstrap running 10.243.0.4 <obfuscated> bx2-8x32 pamoedo-test37-c6hqt-rhcos pamoedo-test37-c6hqt-vpc eu-de-1 pamoedom-rg 02b7_fe4e87a3-f2e9-43e4-881c-608bc1044327 pamoedo-test37-c6hqt-master-0 running 10.243.0.5 - bx2-8x32 pamoedo-test37-c6hqt-rhcos pamoedo-test37-c6hqt-vpc eu-de-1 pamoedom-rg 02c7_e67be714-b138-4ab5-8b20-a73673b2228b pamoedo-test37-c6hqt-master-1 running 10.243.64.6 - bx2-8x32 pamoedo-test37-c6hqt-rhcos pamoedo-test37-c6hqt-vpc eu-de-2 pamoedom-rg 02d7_6cbdeeec-ce28-4126-99fb-bfb00dc5f9fc pamoedo-test37-c6hqt-master-2 running 10.243.129.6 - bx2-8x32 pamoedo-test37-c6hqt-rhcos pamoedo-test37-c6hqt-vpc eu-de-3 pamoedom-rg ~~~ *** PASSED ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056