Bug 2009791 - Installer: ibmcloud ignores install-config values
Summary: Installer: ibmcloud ignores install-config values
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.10.0
Assignee: aos-install
QA Contact: Pedro Amoedo
URL:
Whiteboard:
Depends On: 2012233
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-01 15:14 UTC by Christopher J Schaefer
Modified: 2022-03-10 16:15 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:14:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5263 0 None open Bug 2009791: Fix IBM Cloud config usage 2021-10-01 19:13:35 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:15:06 UTC

Description Christopher J Schaefer 2021-10-01 15:14:36 UTC
Version:
4.10

Platform:

ibmcloud

Please specify:

IPI

What happened?

Using a customized install-config.yaml with IPI and IBM Cloud, the installer does not use certain values from the install-config.yaml and uses default values.
For instance, the machine type will always be set to the default value.


What did you expect to happen?

The installer would use the custom values set in the install-config.yaml for IBM Cloud IPI deploys.

How to reproduce it (as minimally and precisely as possible)?

Create an install-config.yaml for IBM Cloud, and add blocks with customized values, such as the compute platform type block

1. openshift-install create install-config
2. Add a 'type' to the compute.platform.ibmcloud (or control-plane) block.
3. openshift-install create manifests


Anything else we need to know?

IBM Cloud is working on a fix for this bug currently.

Comment 2 Pedro Amoedo 2021-10-06 09:34:21 UTC
Hi Christopher, I'm using latest accepted 4.10 nightly version (4.10.0-0.nightly-2021-10-05-151518) to test the PR but it fails with a SIGSEGV error, not sure if related with this change or a previous one, can you please double check?

~~~
$ ./openshift-install version
./openshift-install 4.10.0-0.nightly-2021-10-05-151518
built from commit 90d215780753b3e456ad434e9ac963da178f456d
release image registry.ci.openshift.org/ocp/release@sha256:5f5fe78285d4eed84f05e018b578d165ec579c553160095c9ec7122e35528d7f
release architecture amd64

$ ./openshift-install create cluster --dir test30/ --log-level debug
DEBUG OpenShift Installer 4.10.0-0.nightly-2021-10-05-151518 
DEBUG Built from commit 90d215780753b3e456ad434e9ac963da178f456d 
DEBUG Fetching Metadata...                         
DEBUG Loading Metadata...                          
DEBUG   Loading Cluster ID...                      
DEBUG     Loading Install Config...                
DEBUG       Loading SSH Key...                     
DEBUG       Loading Base Domain...                 
DEBUG         Loading Platform...                  
DEBUG       Loading Cluster Name...                
DEBUG         Loading Base Domain...               
DEBUG         Loading Platform...                  
DEBUG       Loading Networking...                  
DEBUG         Loading Platform...                  
DEBUG       Loading Pull Secret...                 
DEBUG       Loading Platform...                    
DEBUG     Using Install Config loaded from target directory 
DEBUG   Loading Install Config...                  
DEBUG   Loading Bootstrap Ignition Config...       
DEBUG     Loading Ironic bootstrap credentials...  
DEBUG     Loading CVO Ignore...                    
DEBUG       Loading Common Manifests...            
DEBUG         Loading Cluster ID...                
DEBUG         Loading Install Config...            
DEBUG         Loading Ingress Config...            
DEBUG           Loading Install Config...          
DEBUG         Loading DNS Config...                
DEBUG           Loading Install Config...          
DEBUG           Loading Cluster ID...              
DEBUG           Loading Platform Credentials Check... 
DEBUG             Loading Install Config...        
DEBUG         Loading Infrastructure Config...     
DEBUG           Loading Cluster ID...              
DEBUG           Loading Install Config...          
DEBUG           Loading Cloud Provider Config...   
DEBUG             Loading Install Config...        
DEBUG             Loading Cluster ID...            
DEBUG             Loading Platform Credentials Check... 
DEBUG           Loading Additional Trust Bundle Config... 
DEBUG             Loading Install Config...        
DEBUG         Loading Network Config...            
DEBUG           Loading Install Config...          
DEBUG           Loading Network CRDs...            
DEBUG         Loading Proxy Config...              
DEBUG           Loading Install Config...          
DEBUG           Loading Network Config...          
DEBUG         Loading Scheduler Config...          
DEBUG           Loading Install Config...          
DEBUG         Loading Image Content Source Policy... 
DEBUG           Loading Install Config...          
DEBUG         Loading Root CA...                   
DEBUG         Loading Certificate (mcs)...         
DEBUG           Loading Root CA...                 
DEBUG           Loading Install Config...          
DEBUG         Loading CVOOverrides...              
DEBUG         Loading KubeCloudConfig...           
DEBUG         Loading KubeSystemConfigmapRootCA... 
DEBUG         Loading MachineConfigServerTLSSecret... 
DEBUG         Loading OpenshiftConfigSecretPullSecret... 
DEBUG         Loading OpenshiftMachineConfigOperator... 
DEBUG         Loading KubevirtInfraNamespace...    
DEBUG       Loading Openshift Manifests...         
DEBUG         Loading Install Config...            
DEBUG         Loading Cluster ID...                
DEBUG         Loading Kubeadmin Password...        
DEBUG         Loading OpenShift Install (Manifests)... 
DEBUG         Loading CloudCredsSecret...          
DEBUG         Loading KubeadminPasswordSecret...   
DEBUG         Loading RoleCloudCredsSecretReader... 
DEBUG         Loading Baremetal Config CR...       
DEBUG         Loading Image...                     
DEBUG           Loading Install Config...          
DEBUG         Loading AzureCloudProviderSecret...  
DEBUG     Loading Install Config...                
DEBUG     Loading Kubeconfig Admin Internal Client... 
DEBUG       Loading Certificate (admin-kubeconfig-client)... 
DEBUG         Loading Certificate (admin-kubeconfig-signer)... 
DEBUG       Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG         Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG         Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG         Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Kubeconfig Kubelet...            
DEBUG       Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-client)... 
DEBUG         Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Kubeconfig Admin Client (Loopback)... 
DEBUG       Loading Certificate (admin-kubeconfig-client)... 
DEBUG       Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Master Ignition Customization Check... 
DEBUG       Loading Install Config...              
DEBUG       Loading Root CA...                     
DEBUG       Loading Master Ignition Config...      
DEBUG         Loading Install Config...            
DEBUG         Loading Root CA...                   
DEBUG     Loading Worker Ignition Customization Check... 
DEBUG       Loading Install Config...              
DEBUG       Loading Root CA...                     
DEBUG       Loading Worker Ignition Config...      
DEBUG         Loading Install Config...            
DEBUG         Loading Root CA...                   
DEBUG     Loading Master Machines...               
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Platform Credentials Check...  
DEBUG       Loading Install Config...              
DEBUG       Loading Image...                       
DEBUG       Loading Master Ignition Config...      
DEBUG     Loading Worker Machines...               
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Platform Credentials Check...  
DEBUG       Loading Install Config...              
DEBUG       Loading Image...                       
DEBUG       Loading Worker Ignition Config...      
DEBUG     Loading Common Manifests...              
DEBUG     Loading Openshift Manifests...           
DEBUG     Loading Proxy Config...                  
DEBUG     Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG       Loading Certificate (admin-kubeconfig-signer)... 
DEBUG     Loading Certificate (aggregator)...      
DEBUG     Loading Certificate (aggregator-ca-bundle)... 
DEBUG       Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG       Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG       Loading Certificate (aggregator)...    
DEBUG     Loading Bootstrap SSH Key Pair...        
DEBUG     Loading User-provided Service Account Signing key... 
DEBUG     Loading Cloud Provider CA Bundle...      
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (journal-gatewayd)... 
DEBUG       Loading Root CA...                     
DEBUG     Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-external-lb-server)... 
DEBUG       Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-internal-lb-server)... 
DEBUG       Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-server)... 
DEBUG       Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-server)... 
DEBUG       Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG       Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG         Loading Certificate (kubelet-signer)... 
DEBUG       Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG         Loading Certificate (kube-control-plane-signer)... 
DEBUG         Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG         Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG         Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG         Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG       Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG         Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG       Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG     Loading Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG       Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG       Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client)...  
DEBUG     Loading Certificate (kubelet-signer)...  
DEBUG     Loading Certificate (kubelet-serving-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-signer)... 
DEBUG     Loading Certificate (mcs)...             
DEBUG     Loading Root CA...                       
DEBUG     Loading Key Pair (service-account.pub)... 
DEBUG     Loading Release Image Pull Spec...       
DEBUG     Loading Image...                         
DEBUG   Fetching Cluster ID...                     
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Cluster ID...                   
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG   Fetching Bootstrap Ignition Config...      
DEBUG     Fetching Ironic bootstrap credentials... 
DEBUG     Generating Ironic bootstrap credentials... 
DEBUG     Fetching CVO Ignore...                   
DEBUG       Fetching Common Manifests...           
DEBUG         Fetching Cluster ID...               
DEBUG         Reusing previously-fetched Cluster ID 
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG         Fetching Ingress Config...           
DEBUG           Fetching Install Config...         
DEBUG           Reusing previously-fetched Install Config 
DEBUG         Generating Ingress Config...         
DEBUG         Fetching DNS Config...               
DEBUG           Fetching Install Config...         
DEBUG           Reusing previously-fetched Install Config 
DEBUG           Fetching Cluster ID...             
DEBUG           Reusing previously-fetched Cluster ID 
DEBUG           Fetching Platform Credentials Check... 
DEBUG             Fetching Install Config...       
DEBUG             Reusing previously-fetched Install Config 
DEBUG           Generating Platform Credentials Check... 
DEBUG         Generating DNS Config...             
DEBUG         Fetching Infrastructure Config...    
DEBUG           Fetching Cluster ID...             
DEBUG           Reusing previously-fetched Cluster ID 
DEBUG           Fetching Install Config...         
DEBUG           Reusing previously-fetched Install Config 
DEBUG           Fetching Cloud Provider Config...  
DEBUG             Fetching Install Config...       
DEBUG             Reusing previously-fetched Install Config 
DEBUG             Fetching Cluster ID...           
DEBUG             Reusing previously-fetched Cluster ID 
DEBUG             Fetching Platform Credentials Check... 
DEBUG             Reusing previously-fetched Platform Credentials Check 
DEBUG           Generating Cloud Provider Config... 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xcfd8c99]

goroutine 1 [running]:
github.com/openshift/installer/pkg/asset/manifests.(*CloudProviderConfig).Generate(0xc000dda2d0, 0xc000d8fb30, 0x104ea80d, 0x12)
	/go/src/github.com/openshift/installer/pkg/asset/manifests/cloudproviderconfig.go:180 +0x1599
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c63338, 0xc000dda2d0, 0xc000923db6, 0xa, 0xc000923db6, 0xa)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x7df
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c633c8, 0xc000d20ba0, 0xc000923cd8, 0x8, 0xc000923cd8, 0x8)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c63428, 0xc0014d52c0, 0xc000654c70, 0x6, 0xc000654c70, 0x6)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c62eb8, 0xc000d209a8, 0xc000654c44, 0x4, 0xc000654c44, 0x4)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x11c62e88, 0xc000bb0790, 0x1047d240, 0x2, 0x1047d240, 0x2)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b312c0, 0x7f15462565a0, 0x19c7e4f8, 0x0, 0x0, 0x40b845, 0xebe9d20)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000b312c0, 0x7f15462565a0, 0x19c7e4f8, 0x19c460c0, 0x8, 0x8, 0xed8ef625a, 0x72919d37a15288a4)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4b
main.runTargetCmd.func1(0x7ffcf07bd1dd, 0x7, 0xc000622900, 0x0)
	/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:238 +0x12d
main.runTargetCmd.func2(0x19c50d20, 0xc000cecfc0, 0x0, 0x4)
	/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:265 +0xb5
github.com/spf13/cobra.(*Command).execute(0x19c50d20, 0xc000cecf80, 0x4, 0x4, 0x19c50d20, 0xc000cecf80)
	/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc000291080, 0xc000dd3de8, 0x1, 0x1)
	/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:895
main.installerMain()
	/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:72 +0x2fe
main.main()
	/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:50 +0x259
~~~

NOTE: I have also tried with a local compiled version from latest repository content but same error.

Regards.

Comment 3 Pedro Amoedo 2021-10-06 09:45:47 UTC
[UPDATE]

Tested also with version "4.10.0-0.ci-2021-10-06-062517", which includes PR#5263, same SIGSEGV error:

~~~
$ ./openshift-install create cluster --dir test31/ --log-level debug
DEBUG OpenShift Installer 4.10.0-0.ci-2021-10-06-062517 
DEBUG Built from commit 4604929ab0c9948fcde1444bfd4df56d9dcbe934 
DEBUG Fetching Metadata...                         
DEBUG Loading Metadata...                          
DEBUG   Loading Cluster ID...                      
DEBUG     Loading Install Config...                
DEBUG       Loading SSH Key...                     
DEBUG       Loading Base Domain...                 
DEBUG         Loading Platform...                  
DEBUG       Loading Cluster Name...                
DEBUG         Loading Base Domain...               
DEBUG         Loading Platform...                  
DEBUG       Loading Networking...                  
DEBUG         Loading Platform...                  
DEBUG       Loading Pull Secret...                 
DEBUG       Loading Platform...                    
DEBUG     Using Install Config loaded from target directory 
DEBUG   Loading Install Config...                  
DEBUG   Loading Bootstrap Ignition Config...       
DEBUG     Loading Ironic bootstrap credentials...  
DEBUG     Loading CVO Ignore...                    
DEBUG       Loading Common Manifests...            
DEBUG         Loading Cluster ID...                
DEBUG         Loading Install Config...            
DEBUG         Loading Ingress Config...            
DEBUG           Loading Install Config...          
DEBUG         Loading DNS Config...                
DEBUG           Loading Install Config...          
DEBUG           Loading Cluster ID...              
DEBUG           Loading Platform Credentials Check... 
DEBUG             Loading Install Config...        
DEBUG         Loading Infrastructure Config...     
DEBUG           Loading Cluster ID...              
DEBUG           Loading Install Config...          
DEBUG           Loading Cloud Provider Config...   
DEBUG             Loading Install Config...        
DEBUG             Loading Cluster ID...            
DEBUG             Loading Platform Credentials Check... 
DEBUG           Loading Additional Trust Bundle Config... 
DEBUG             Loading Install Config...        
DEBUG         Loading Network Config...            
DEBUG           Loading Install Config...          
DEBUG           Loading Network CRDs...            
DEBUG         Loading Proxy Config...              
DEBUG           Loading Install Config...          
DEBUG           Loading Network Config...          
DEBUG         Loading Scheduler Config...          
DEBUG           Loading Install Config...          
DEBUG         Loading Image Content Source Policy... 
DEBUG           Loading Install Config...          
DEBUG         Loading Root CA...                   
DEBUG         Loading Certificate (mcs)...         
DEBUG           Loading Root CA...                 
DEBUG           Loading Install Config...          
DEBUG         Loading CVOOverrides...              
DEBUG         Loading KubeCloudConfig...           
DEBUG         Loading KubeSystemConfigmapRootCA... 
DEBUG         Loading MachineConfigServerTLSSecret... 
DEBUG         Loading OpenshiftConfigSecretPullSecret... 
DEBUG         Loading OpenshiftMachineConfigOperator... 
DEBUG         Loading KubevirtInfraNamespace...    
DEBUG       Loading Openshift Manifests...         
DEBUG         Loading Install Config...            
DEBUG         Loading Cluster ID...                
DEBUG         Loading Kubeadmin Password...        
DEBUG         Loading OpenShift Install (Manifests)... 
DEBUG         Loading CloudCredsSecret...          
DEBUG         Loading KubeadminPasswordSecret...   
DEBUG         Loading RoleCloudCredsSecretReader... 
DEBUG         Loading Baremetal Config CR...       
DEBUG         Loading Image...                     
DEBUG           Loading Install Config...          
DEBUG         Loading AzureCloudProviderSecret...  
DEBUG     Loading Install Config...                
DEBUG     Loading Kubeconfig Admin Internal Client... 
DEBUG       Loading Certificate (admin-kubeconfig-client)... 
DEBUG         Loading Certificate (admin-kubeconfig-signer)... 
DEBUG       Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG         Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG         Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG         Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Kubeconfig Kubelet...            
DEBUG       Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-client)... 
DEBUG         Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Kubeconfig Admin Client (Loopback)... 
DEBUG       Loading Certificate (admin-kubeconfig-client)... 
DEBUG       Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Master Ignition Customization Check... 
DEBUG       Loading Install Config...              
DEBUG       Loading Root CA...                     
DEBUG       Loading Master Ignition Config...      
DEBUG         Loading Install Config...            
DEBUG         Loading Root CA...                   
DEBUG     Loading Worker Ignition Customization Check... 
DEBUG       Loading Install Config...              
DEBUG       Loading Root CA...                     
DEBUG       Loading Worker Ignition Config...      
DEBUG         Loading Install Config...            
DEBUG         Loading Root CA...                   
DEBUG     Loading Master Machines...               
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Platform Credentials Check...  
DEBUG       Loading Install Config...              
DEBUG       Loading Image...                       
DEBUG       Loading Master Ignition Config...      
DEBUG     Loading Worker Machines...               
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Platform Credentials Check...  
DEBUG       Loading Install Config...              
DEBUG       Loading Image...                       
DEBUG       Loading Worker Ignition Config...      
DEBUG     Loading Common Manifests...              
DEBUG     Loading Openshift Manifests...           
DEBUG     Loading Proxy Config...                  
DEBUG     Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG       Loading Certificate (admin-kubeconfig-signer)... 
DEBUG     Loading Certificate (aggregator)...      
DEBUG     Loading Certificate (aggregator-ca-bundle)... 
DEBUG       Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG       Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG       Loading Certificate (aggregator)...    
DEBUG     Loading Bootstrap SSH Key Pair...        
DEBUG     Loading User-provided Service Account Signing key... 
DEBUG     Loading Cloud Provider CA Bundle...      
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (journal-gatewayd)... 
DEBUG       Loading Root CA...                     
DEBUG     Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-external-lb-server)... 
DEBUG       Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-internal-lb-server)... 
DEBUG       Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-server)... 
DEBUG       Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-server)... 
DEBUG       Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG       Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG         Loading Certificate (kubelet-signer)... 
DEBUG       Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG         Loading Certificate (kube-control-plane-signer)... 
DEBUG         Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG         Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG         Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG         Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG       Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG         Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG       Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG     Loading Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG       Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG       Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client)...  
DEBUG     Loading Certificate (kubelet-signer)...  
DEBUG     Loading Certificate (kubelet-serving-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-signer)... 
DEBUG     Loading Certificate (mcs)...             
DEBUG     Loading Root CA...                       
DEBUG     Loading Key Pair (service-account.pub)... 
DEBUG     Loading Release Image Pull Spec...       
DEBUG     Loading Image...                         
DEBUG   Fetching Cluster ID...                     
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Cluster ID...                   
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG   Fetching Bootstrap Ignition Config...      
DEBUG     Fetching Ironic bootstrap credentials... 
DEBUG     Generating Ironic bootstrap credentials... 
DEBUG     Fetching CVO Ignore...                   
DEBUG       Fetching Common Manifests...           
DEBUG         Fetching Cluster ID...               
DEBUG         Reusing previously-fetched Cluster ID 
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG         Fetching Ingress Config...           
DEBUG           Fetching Install Config...         
DEBUG           Reusing previously-fetched Install Config 
DEBUG         Generating Ingress Config...         
DEBUG         Fetching DNS Config...               
DEBUG           Fetching Install Config...         
DEBUG           Reusing previously-fetched Install Config 
DEBUG           Fetching Cluster ID...             
DEBUG           Reusing previously-fetched Cluster ID 
DEBUG           Fetching Platform Credentials Check... 
DEBUG             Fetching Install Config...       
DEBUG             Reusing previously-fetched Install Config 
DEBUG           Generating Platform Credentials Check... 
DEBUG         Generating DNS Config...             
DEBUG         Fetching Infrastructure Config...    
DEBUG           Fetching Cluster ID...             
DEBUG           Reusing previously-fetched Cluster ID 
DEBUG           Fetching Install Config...         
DEBUG           Reusing previously-fetched Install Config 
DEBUG           Fetching Cloud Provider Config...  
DEBUG             Fetching Install Config...       
DEBUG             Reusing previously-fetched Install Config 
DEBUG             Fetching Cluster ID...           
DEBUG             Reusing previously-fetched Cluster ID 
DEBUG             Fetching Platform Credentials Check... 
DEBUG             Reusing previously-fetched Platform Credentials Check 
DEBUG           Generating Cloud Provider Config... 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xcfd8cb9]

goroutine 1 [running]:
github.com/openshift/installer/pkg/asset/manifests.(*CloudProviderConfig).Generate(0xc001353060, 0xc001331740, 0x104eb80d, 0x12)
	/go/src/github.com/openshift/installer/pkg/asset/manifests/cloudproviderconfig.go:180 +0x1599
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64bb8, 0xc001353060, 0xc001356f96, 0xa, 0xc001356f96, 0xa)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x7df
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64c48, 0xc001284330, 0xc001356eb8, 0x8, 0xc001356eb8, 0x8)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64ca8, 0xc001011860, 0xc0011d9b10, 0x6, 0xc0011d9b10, 0x6)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64738, 0xc001284180, 0xc0011d9ae4, 0x4, 0xc0011d9ae4, 0x4)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x11c64708, 0xc00127d250, 0x1047e240, 0x2, 0x1047e240, 0x2)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000b404e0, 0x7ff57c3b9e90, 0x19c7f4f8, 0x0, 0x0, 0x40b845, 0xebead20)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x625
github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000b404e0, 0x7ff57c3b9e90, 0x19c7f4f8, 0x19c470c0, 0x8, 0x8, 0xed8ef6606, 0x83d98d966d665e58)
	/go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4b
main.runTargetCmd.func1(0x7ffe389771dd, 0x7, 0xc000c08918, 0x0)
	/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:238 +0x12d
main.runTargetCmd.func2(0x19c51d20, 0xc000aea6c0, 0x0, 0x4)
	/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:265 +0xb5
github.com/spf13/cobra.(*Command).execute(0x19c51d20, 0xc000aea680, 0x4, 0x4, 0x19c51d20, 0xc000aea680)
	/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001dd080, 0xc000ce3de8, 0x1, 0x1)
	/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:895
main.installerMain()
	/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:72 +0x2fe
main.main()
	/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:50 +0x259
~~~

Comment 4 Matthew Staebler 2021-10-06 13:29:48 UTC
@pamoedom That crash looks like a separate bug. Can you share the install-config.yaml that you were using? I suspect that the installer is not setting the defaults properly for the machine pools.

Comment 5 Matthew Staebler 2021-10-06 13:35:57 UTC
On the following lines of code, there is an expectation that the platform is populated for the machine pools. That is not a valid expectation.

https://github.com/openshift/installer/blob/6617bc2e334654bb6e85976f049e51fc1c01aa3f/pkg/asset/manifests/cloudproviderconfig.go#L173
https://github.com/openshift/installer/blob/6617bc2e334654bb6e85976f049e51fc1c01aa3f/pkg/asset/manifests/cloudproviderconfig.go#L180

Comment 6 Pedro Amoedo 2021-10-06 14:00:23 UTC
Hi Matthew, I always use the same "install-config.yaml" for my ibmcloud tests and never failed before, the content is the following:

~~~
apiVersion: v1
baseDomain: ibmcloud.qe.devcluster.openshift.com
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 0
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    ibmcloud:
      type: bx2-8x32
  replicas: 3
metadata:
  creationTimestamp: null
  name: pamoedo-test
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  ibmcloud:
    region: eu-de
    resourceGroupName: pamoedom-rg
publish: External
pullSecret: ...
sshKey: ...
~~~

What do you mean by "there is an expectation that the platform is populated for the machine pools. That is not a valid expectation."? can't we set different machine types per pool?

Comment 7 Matthew Staebler 2021-10-06 14:24:29 UTC
The code that is causing the crash was added only 6 days ago [1]. Perhaps this is the first time that you have tried an install to IBM cloud since then.


> What do you mean by "there is an expectation that the platform is populated for the machine pools. That is not a valid expectation."? can't we set different machine types per pool?

I mean that there is a bug in the code. The code is expecting that the machine pool has the ibmcloud field set in the platform field. You should be able to work around this by explicitly setting the ibmcloud field.

~~~~
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    ibmcloud: {}
  replicas: 0
~~~

Comment 8 Pedro Amoedo 2021-10-06 14:59:10 UTC
Thanks Matthew, I understood now your comment, and yes, most probably I haven't tried new builds for more than a week.

I will try again with your suggestion, thanks.

Comment 9 Pedro Amoedo 2021-10-06 16:15:19 UTC
Hi again Matthew, your suggestion did the trick and I no longer suffer the SIGSEGV, however, for our IBMCloud account we have to bypass BZ#1992777 issue first in order to see if the changes are having effect on the instances, for that reason I'm usign a local compiled version as follows:

~~~
$ ./openshift-install-local version
./openshift-install-local unreleased-master-5073-ga2e7a78af1da644e2b03ae9c14e8249fcb69abe3-dirty
built from commit a2e7a78af1da644e2b03ae9c14e8249fcb69abe3
release image registry.ci.openshift.org/origin/release:4.8
release architecture amd64
~~~

With this version I can bypass BZ#1992777 and continue the execution but to my surprise I've found a new unexpected issue with remote rules limit per security group:

~~~
ERROR                                              
ERROR Error: Error while creating Security Group Rule Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group). 
ERROR                                              
ERROR Adding a rule would exceed the limit of remote rules per security group. Consider creating another security group. 
ERROR {                                            
ERROR     "StatusCode": 400,                       
ERROR     "Headers": {                             
ERROR         "Cache-Control": [                   
ERROR             "max-age=0, no-cache, no-store, must-revalidate" 
ERROR         ],                                   
ERROR         "Cf-Cache-Status": [                 
ERROR             "DYNAMIC"                        
ERROR         ],                                   
ERROR         "Cf-Ray": [                          
ERROR             "699ffed098146653-MAD"           
ERROR         ],                                   
ERROR         "Connection": [                      
ERROR             "keep-alive"                     
ERROR         ],                                   
ERROR         "Content-Length": [                  
ERROR             "373"                            
ERROR         ],                                   
ERROR         "Content-Type": [                    
ERROR             "application/json"               
ERROR         ],                                   
ERROR         "Date": [                            
ERROR             "Wed, 06 Oct 2021 15:41:42 GMT"  
ERROR         ],                                   
ERROR         "Expect-Ct": [                       
ERROR             "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" 
ERROR         ],                                   
ERROR         "Expires": [                         
ERROR             "-1"                             
ERROR         ],                                   
ERROR         "Pragma": [                          
ERROR             "no-cache"                       
ERROR         ],                                   
ERROR         "Server": [                          
ERROR             "cloudflare"                     
ERROR         ],                                   
ERROR         "Strict-Transport-Security": [       
ERROR             "max-age=31536000; includeSubDomains" 
ERROR         ],                                   
ERROR         "Vary": [                            
ERROR             "Accept-Encoding"                
ERROR         ],                                   
ERROR         "X-Content-Type-Options": [          
ERROR             "nosniff"                        
ERROR         ],                                   
ERROR         "X-Request-Id": [                    
ERROR             "2f23ff2f-9acc-447a-86e4-2730be710d37" 
ERROR         ],                                   
ERROR         "X-Xss-Protection": [                
ERROR             "1; mode=block"                  
ERROR         ]                                    
ERROR     },                                       
ERROR     "Result": {                              
ERROR         "errors": [                          
ERROR             {                                
ERROR                 "code": "over_quota",        
ERROR                 "message": "Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group).\n\nAdding a rule would exceed the limit of remote rules per security group. Consider creating another security group.", 
ERROR                 "more_info": "https://cloud.ibm.com/docs/vpc?topic=vpc-quotas" 
ERROR             }                                
ERROR         ],                                   
ERROR         "trace": "2f23ff2f-9acc-447a-86e4-2730be710d37" 
ERROR     },                                       
ERROR     "RawResult": null                        
ERROR }                                            
ERROR                                              
ERROR                                              
ERROR   on ../../../../tmp/openshift-install-network-003889995/vpc/security-groups.tf line 268, in resource "ibm_is_security_group_rule" "control_plane_machine_config_lb_inbound": 
ERROR  268: resource "ibm_is_security_group_rule" "control_plane_machine_config_lb_inbound" { 
ERROR                                              
ERROR                                              
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change
~~~

As per IBM docs here[1], there is a hardcoded limit of 5 remote rules per security group, any recent change in your radar that could be causing this? any available workaround?

[1] - https://cloud.ibm.com/docs/vpc?topic=vpc-quotas

Regards.

Comment 10 Christopher J Schaefer 2021-10-06 17:32:03 UTC
We think this enhancement might have caused the problem, overloading the security group. I'll work with our dev team more and look at getting a separate BZ open to address the issue.
https://github.com/openshift/installer/pull/5105/files#diff-b237721806432aac5fdc1b5a78484516c931af5affbbcea351e8824b5f9fcd54

Comment 13 Pedro Amoedo 2021-10-07 08:59:19 UTC
Hi Christopher, this BZ has moved now to "ON_QA" but I can't properly test the changes due to the remote rules limit per SG, please provide me the new BZ info when you have it so we can mark this one as blocked accordingly, thanks.

Comment 14 Christopher J Schaefer 2021-10-08 15:41:58 UTC
@

Comment 15 Christopher J Schaefer 2021-10-08 15:43:13 UTC
@pamoedom I opened a BZ for the `5 rule limit` error you saw above.
https://bugzilla.redhat.com/show_bug.cgi?id=2012233

Comment 16 Pedro Amoedo 2021-10-08 15:53:59 UTC
Thanks Christopher, appreciated, I've set dependency between them.

Comment 17 Pedro Amoedo 2021-10-13 14:14:34 UTC
[QA Summary]

[Version]

~~~
$ ./openshift-install-local version
./openshift-install-local unreleased-master-5104-g527c46172e6c4fe28838e1a18e0d758eec61fb02
built from commit 527c46172e6c4fe28838e1a18e0d758eec61fb02
release image registry.ci.openshift.org/origin/release:4.8
release architecture amd64

$ git --no-pager log --oneline --first-parent origin/master | grep "5289\|5263"
527c46172 Merge pull request #5289 from cjschaef/bz_2012233
4604929ab Merge pull request #5263 from cjschaef/bz_2009791
~~~

[Environment]

~~~
apiVersion: v1
baseDomain: ibmcloud.qe.devcluster.openshift.com
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    ibmcloud: {}
  replicas: 0
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    ibmcloud:
      type: bx2-8x32
  replicas: 3
metadata:
  creationTimestamp: null
  name: pamoedo-test
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  ibmcloud:
    region: eu-de
    resourceGroupName: pamoedom-rg
publish: External
pullSecret:
sshKey:
~~~

[Results]

~~~
$ DIGEST=$(skopeo inspect --authfile pull-secret docker://registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-10-13-081040 | grep Digest | cut -d '"' -f4)
$ export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.ci.openshift.org/ocp/release@${DIGEST}
$ export IC_API_KEY='<obfuscated>'

$ ./openshift-install-local create cluster --dir test37/ --log-level debug
DEBUG OpenShift Installer unreleased-master-5104-g527c46172e6c4fe28838e1a18e0d758eec61fb02 
DEBUG Built from commit 527c46172e6c4fe28838e1a18e0d758eec61fb02 
...
DEBUG Apply complete! Resources: 60 added, 0 changed, 0 destroyed. 
DEBUG                                              
DEBUG Outputs:                                     
DEBUG                                              
DEBUG control_plane_security_group_id_list = [     
DEBUG   "r010-6a6c0109-3b56-413f-8769-f2e09314654c", 
DEBUG   "r010-8624c529-7c01-4563-ac6e-7bd7d86fc470", 
DEBUG   "r010-b193d9de-0d15-4fa6-bb09-16c4f3093e20", 
DEBUG   "r010-3a5c82e4-5102-4b68-ba04-a46ce6799171", 
DEBUG ]                                            
DEBUG control_plane_subnet_id_list = [             
DEBUG   "02b7-9db6e4f0-d0d1-4fa9-b9f5-b26afb517e84", 
DEBUG   "02c7-519c5479-1142-47d8-b1bf-71848785c0d6", 
DEBUG   "02d7-1cab091f-888f-45c6-826d-20513c4eb14d", 
DEBUG ]                                            
DEBUG control_plane_subnet_zone_list = [           
DEBUG   "eu-de-1",                                 
DEBUG   "eu-de-2",                                 
DEBUG   "eu-de-3",                                 
DEBUG ]
...
ERROR Attempted to gather debug logs after installation failure: bootstrap host address and at least one control plane host address must be provided 
ERROR Bootstrap failed to complete: timed out waiting for the condition 
ERROR Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane. 
ERROR Attempted to analyze the debug logs after installation failure: could not open the gather bundle: open : no such file or directory 
FATAL Bootstrap failed to complete

$ ibmcloud is instances
ID                                          Name                             Status    Address        Floating IP     Profile    Image                        VPC                        Zone      Resource group   
02b7_70e6a3c7-3353-450f-a86a-5adfde8c884c   pamoedo-test37-c6hqt-bootstrap   running   10.243.0.4     <obfuscated>    bx2-8x32   pamoedo-test37-c6hqt-rhcos   pamoedo-test37-c6hqt-vpc   eu-de-1   pamoedom-rg   
02b7_fe4e87a3-f2e9-43e4-881c-608bc1044327   pamoedo-test37-c6hqt-master-0    running   10.243.0.5     -               bx2-8x32   pamoedo-test37-c6hqt-rhcos   pamoedo-test37-c6hqt-vpc   eu-de-1   pamoedom-rg   
02c7_e67be714-b138-4ab5-8b20-a73673b2228b   pamoedo-test37-c6hqt-master-1    running   10.243.64.6    -               bx2-8x32   pamoedo-test37-c6hqt-rhcos   pamoedo-test37-c6hqt-vpc   eu-de-2   pamoedom-rg   
02d7_6cbdeeec-ce28-4126-99fb-bfb00dc5f9fc   pamoedo-test37-c6hqt-master-2    running   10.243.129.6   -               bx2-8x32   pamoedo-test37-c6hqt-rhcos   pamoedo-test37-c6hqt-vpc   eu-de-3   pamoedom-rg
~~~

*** PASSED ***

Comment 20 errata-xmlrpc 2022-03-10 16:14:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.