Bug 1734460 - [OSP] Floating IP specified with lbFloatingIP in install-config.yaml fails to attach to master node causing install to fail
Summary: [OSP] Floating IP specified with lbFloatingIP in install-config.yaml fails to...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.2.0
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: ---
: 4.6.0
Assignee: Martin André
QA Contact: David Sanz
URL:
Whiteboard:
: 1836098 1844792 (view as bug list)
Depends On:
Blocks: 1846297
TreeView+ depends on / blocked
 
Reported: 2019-07-30 15:11 UTC by Jon Uriarte
Modified: 2023-10-06 18:27 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The Terraform step "openstack_networking_floatingip_associate_v2" did not list all its dependendent steps. Consequence: The required step "openstack_networking_router_interface_v2.nodes_router_interface" was not always executed before the FIP association. This caused a race condition that occasionally failed the Terraform job, especially on overloaded systems. Fix: List the dependant Terraform step as "depends_on" to force the router to be created before the FIP is attached to the port. Result: The Terraform steps run in the correct order, ensuring that the FIP association does not result in a "Resource not found" error.
Clone Of:
Environment:
Last Closed: 2020-10-27 15:54:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 2868 0 None closed Bug 1836098: OpenStack: check that port exists before creating a fip 2021-01-20 02:21:30 UTC
Github openshift installer pull 3734 0 None closed Bug 1734460: OpenStack: Wait for router interface to create FIP 2021-01-20 02:22:10 UTC
Red Hat Knowledge Base (Solution) 5093061 0 None None None 2020-05-20 17:23:03 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 15:54:52 UTC

Internal Links: 1844792

Description Jon Uriarte 2019-07-30 15:11:26 UTC
Description of problem:

During the openshift on openstack install process a floating IP is being attached to the bootstrap node. It's failing to attach the floating IP specified in install-config.yml to the api node and another floating IP from the pool is attached to the bootstrap node.

Taken from openshift-install release-4.2 branch.

How reproducible: always

Steps to Reproduce:
1. Deploy OSP
2. Run openshift-install


Actual results:


DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Creation complete after 54s [id=4d641422-0e3d-4db5-90b1-baf9f031f55e] 
ERROR                                              
ERROR Error: Error associating openstack_networking_floatingip_associate_v2 floating_ip e8b95280-334a-4988-bf52-f491412d6fb0 with port d887f6f0-a695-42ba-8de4-859a60d6d2f6: Resource not found 
ERROR                                              
ERROR   on ../../../../../../tmp/openshift-install-380596170/topology/private-network.tf line 84, in resource "openstack_networking_floatingip_associate_v2" "service_fip": 
ERROR   84: resource "openstack_networking_floatingip_associate_v2" "service_fip" { 
ERROR                                              
ERROR                                              
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply using Terraform 

real	2m13.242s
user	0m40.423s
sys	0m3.888s

Expected results:
No error and floating IP attached to api node.

Comment 1 August Simonelli 2019-07-30 15:22:52 UTC
Further to this we see it happening like this:

We pre-allocate an FIP in openstack as usual:

openstack floating ip create --floating-ip-address IP public

We then add that FIP to lbFloatingIP in install-config.yaml

We then install and this results in the error listed above.

However, if we leave the lbFloatingIP line out of install-config.yaml we get past the error; however we need to manually assign the preallocated FIP to the API node for the install to continue.

It seems like the presence of lbFloatingIP in install-config.yaml is triggering the error but we still need the FIP assigned to allow the install to work.

Comment 2 August Simonelli 2019-07-30 15:23:59 UTC
Additionally, we also see another FIP (NOT the preallocated one) assigned to the bootstrap.

Comment 3 Tomas Sedovic 2019-07-31 15:41:16 UTC
The IP address being assigned to the bootstrap node is now expected. We've added that recently to let the installer gather logs on bootstrap failure. You can safely ignore that.

The fact that you can't use a pre-allocated FIP as `lbFloatingIP` is a genuine issue though. I'm unable to reproduce this locally, but it's something we do want to address.

Could you please paste your install-config.yaml (with the sensitive items such as pull secrets snipped out?

Comment 4 August Simonelli 2019-07-31 17:16:25 UTC
Here's my install from the non-service-vm code which is doing the same.

Failed with

time="2019-07-31T12:23:26-04:00" level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Creation complete after 3m4s [id=69821aff-c950-489f-82bc-5d83d0da78a4]"
time="2019-07-31T12:23:27-04:00" level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [3m0s elapsed]"
time="2019-07-31T12:23:29-04:00" level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Creation complete after 3m2s [id=cab5d901-129c-4ef9-88ec-08413127b92d]"
time="2019-07-31T12:23:29-04:00" level=error
time="2019-07-31T12:23:29-04:00" level=error msg="Error: Error associating openstack_networking_floatingip_associate_v2 floating_ip 4a5af48e-d921-427a-9a72-a6b86a9f8873 with port f8e5f7b4-5525-4254-b936-fbb987f9cf9d: Resource not found"
time="2019-07-31T12:23:29-04:00" level=error
time="2019-07-31T12:23:29-04:00" level=error msg="  on ../../tmp/openshift-install-113850301/topology/private-network.tf line 114, in resource \"openstack_networking_floatingip_associate_v2\" \"api_fip\":"
time="2019-07-31T12:23:29-04:00" level=error msg=" 114: resource \"openstack_networking_floatingip_associate_v2\" \"api_fip\" {"
time="2019-07-31T12:23:29-04:00" level=error
time="2019-07-31T12:23:29-04:00" level=error
time="2019-07-31T12:23:29-04:00" level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform"

but i do see this:

[stack@undercloud ~]$ openstack --os-cloud openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 4a5af48e-d921-427a-9a72-a6b86a9f8873 | 192.168.122.160     | None             | None                                 | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b |
| e36d8556-09ce-4c2f-b8ac-6cde47968d4f | 192.168.122.175     | 10.0.128.16      | cb54a30f-1a0b-49ec-91a9-34d4a83aad5b | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
[stack@undercloud ~]$ openstack --os-cloud openstack server list
+--------------------------------------+---------------------------+--------+-----------------------------------------------------+-------+----------+
| ID                                   | Name                      | Status | Networks                                            | Image | Flavor   |
+--------------------------------------+---------------------------+--------+-----------------------------------------------------+-------+----------+
| 478aca2c-1fcc-45f6-accf-9c9c78638432 | ostest-pw6ng-worker-gvcwk | ACTIVE | ostest-pw6ng-openshift=10.0.128.39                  | rhcos | m1.large |
| cab5d901-129c-4ef9-88ec-08413127b92d | ostest-pw6ng-master-0     | ACTIVE | ostest-pw6ng-openshift=10.0.128.26                  | rhcos | m1.large |
| 69821aff-c950-489f-82bc-5d83d0da78a4 | ostest-pw6ng-bootstrap    | ACTIVE | ostest-pw6ng-openshift=10.0.128.16, 192.168.122.175 | rhcos | m1.large |
+--------------------------------------+---------------------------+--------+-----------------------------------------------------+-------+----------+

and my install-config.yaml is this:

apiVersion: v1
baseDomain: shiftstack.com
clusterID: 8d582648-4ed7-45ca-be5b-c5b1c435e834
compute:
- name: worker
  platform: {}
  replicas: 1
controlPlane:
  name: master
  platform: {}
  replicas: 1
metadata:
  creationTimestamp: null
  name: ostest
Networking:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostSubnetLength: 9
  machineCIDR: 10.0.128.0/17
  serviceCIDR: 172.30.0.0/16
  type: OpenShiftSDN
platform:
  openstack:
    cloud: openstack
    computeFlavor: m1.large
    externalNetwork: external
    lbFloatingIP: 192.168.122.160
    region: regionOne
    trunkSupport: "1"
pullSecret: 'AAAAAAAAAAAAAA'
sshKey: ssh-rsa AAAAAAAAA

Comment 5 August Simonelli 2019-07-31 17:17:50 UTC
A little info about the OSP install I'm using:

#!/bin/bash

exec openstack overcloud deploy \
  --templates /usr/share/openstack-tripleo-heat-templates \
  --timeout 90 \
  --verbose \
  -r /home/stack/templates/roles_data.yaml \
  -e /home/stack/templates/node-count.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
  -e /home/stack/templates/environments/10-ntp.yaml \
  -e /home/stack/templates/environments/11-network.yaml \
  -e /home/stack/templates/environments/20-network-environment.yaml \
  -e /home/stack/templates/environments/24-octavia-timeout.yaml \
  -e /home/stack/templates/environments/25-hostname-map.yaml \
  -e /home/stack/templates/environments/30-ips-from-pool-all.yaml \
  -e /home/stack/templates/environments/35-ceph-config.yaml \
  -e /home/stack/templates/environments/40-storage-config.yaml \
  -e /home/stack/templates/environments/50-vip.yaml \
  -e /home/stack/templates/environments/55-rsvd_host_memory.yaml \
  --log-file /home/stack/overcloud-deploy.log

I can give more info on each yaml etc if/as required.

Comment 6 August Simonelli 2019-07-31 18:02:40 UTC
And this is really odd. 
I did another deployment testing a new value for lbFloatingIP of 192.168.122.156 (i was trying assigning the fip by an admin)
I hit an error suggesting that the FIP could not be seen (which is what i expected as I set it by admin).
But that's not the odd part ... keep reading ...

So on this failed run I noticed the bootstrap got 192.168.122.175

So i deleted my cluster. However I noticed the 192.168.122.175 address was not removed from my allocation pool:

(openstack) [stack@undercloud ~]$ openstack --os-cloud openstack floating ip list
+--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+
| 4a5af48e-d921-427a-9a72-a6b86a9f8873 | 192.168.122.160     | None             | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b |
| 5b64be33-ae69-4500-b462-183e9c86993b | 192.168.122.175     | None             | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b |
+--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+

So I decided to use 192.168.122.175 for my lbFloatingIP value on the next run and guess what? It worked.

(openstack) [stack@undercloud ~]$ openstack --os-cloud openstack server list
+--------------------------------------+------------------------+--------+-----------------------------------------------------+-------+----------+
| ID                                   | Name                   | Status | Networks                                            | Image | Flavor   |
+--------------------------------------+------------------------+--------+-----------------------------------------------------+-------+----------+
| f6c798ed-0567-4cfd-8636-9063e8f7f02e | ostest-mnxzv-master-0  | ACTIVE | ostest-mnxzv-openshift=10.0.128.15, 192.168.122.175 | rhcos | m1.large |
| 9b974ae3-9bf8-441a-90bb-dc76a8009cca | ostest-mnxzv-bootstrap | ACTIVE | ostest-mnxzv-openshift=10.0.128.30, 192.168.122.178 | rhcos | m1.large |
+--------------------------------------+------------------------+--------+-----------------------------------------------------+-------+----------+

It was correctly assigned to the master as I'd expected but as was not working when i allocated it manually before.

I manually allocated using 

openstack floating ip create external

Comment 8 August Simonelli 2019-08-01 14:11:52 UTC
On a recent run with:

(openstack) [stack@undercloud ~]$ openstack floating ip list
+--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+
| 20f4d4af-733c-4c74-9b07-002fb00656fc | 192.168.122.151     | None             | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b |
| 58f9205c-814f-439b-a590-2bdbf9f3213e | 192.168.122.168     | None             | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b |
| 92ad0e4d-2a8b-4687-b0b2-51f1836f8202 | 192.168.122.152     | None             | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b |
+--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+
(openstack) [stack@undercloud ~]$ openstack network list
+--------------------------------------+----------+--------------------------------------+
| ID                                   | Name     | Subnets                              |
+--------------------------------------+----------+--------------------------------------+
| 10529ced-278b-4c95-ae88-5f84a4a0e236 | external | dacf2dce-c4a4-4b14-a3a4-ebe65c06654e |
+--------------------------------------+----------+--------------------------------------+
(openstack) [stack@undercloud ~]$ openstack subnet list
+--------------------------------------+-----------------+--------------------------------------+------------------+
| ID                                   | Name            | Network                              | Subnet           |
+--------------------------------------+-----------------+--------------------------------------+------------------+
| dacf2dce-c4a4-4b14-a3a4-ebe65c06654e | external_subnet | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 192.168.122.0/24 |
+--------------------------------------+-----------------+--------------------------------------+------------------+

I set   lbFloatingIP: "192.168.122.151"

and it failed with:


DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Creation complete after 21s [id=55cc90bd-83f0-4018-acc6-1be6cfe5bf7f]
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Creation complete after 23s [id=cd71ec82-53e7-4fd3-98fe-c62c17fc31dc]
ERROR
ERROR Error: Error associating openstack_networking_floatingip_associate_v2 floating_ip 20f4d4af-733c-4c74-9b07-002fb00656fc with port 27706d75-4122-4ed0-ac69-f9e707a8233b: Resource not found
ERROR
ERROR   on ../../tmp/openshift-install-298254184/topology/private-network.tf line 114, in resource "openstack_networking_floatingip_associate_v2" "api_fip":
ERROR  114: resource "openstack_networking_floatingip_associate_v2" "api_fip" {
ERROR
ERROR
ERROR
ERROR Error: Error creating openstack_networking_floatingip_v2: Resource not found
ERROR
ERROR   on ../../tmp/openshift-install-298254184/topology/private-network.tf line 120, in resource "openstack_networking_floatingip_v2" "bootstrap_fip":
ERROR  120: resource "openstack_networking_floatingip_v2" "bootstrap_fip" {
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply using Terraform

install-config.yaml (generated by create install-config using the openstack option:

apiVersion: v1
baseDomain: shiftstack.com
compute:
- hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3
controlPlane:
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  creationTimestamp: null
  name: ostest
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  openstack:
    apiVIP: 10.0.0.5
    cloud: openstack
    computeFlavor: m1.large
    dnsVIP: 10.0.0.6
    externalNetwork: external
    ingressVIP: 10.0.0.7
    lbFloatingIP: ""
    octaviaSupport: "1"
    region: regionOne
    trunkSupport: "1"
pullSecret: 'XXXXXXX'
sshKey: |
  ssh-rsa XXXXXXX

Comment 9 August Simonelli 2019-08-01 14:26:40 UTC
terraform.tfstate shows
  
  {
      "module": "module.topology",
      "mode": "managed",
      "type": "openstack_networking_floatingip_associate_v2",
      "name": "api_fip",
      "each": "list",
      "provider": "provider.openstack",
      "instances": []
    },
    {
      "module": "module.topology",
      "mode": "managed",
      "type": "openstack_networking_floatingip_v2",
      "name": "bootstrap_fip",
      "each": "list",
      "provider": "provider.openstack",
      "instances": []
    },

Comment 10 rstarr 2019-08-20 19:15:28 UTC
I get similar error as above. Simply changing the format seems to fix this issue for me. 
This fails 
platform:
  openstack:
    cloud: openstack
    computeFlavor: ocp1.node
    externalNetwork: public
    lbFloatingIP: "192.168.247.57"
    octaviaSupport: "1"
    region: regionOne
    trunkSupport: "1"

But this works
platform:
  openstack:
    cloud:            openstack 
    externalNetwork:  public
    region:           regionOne
    computeFlavor:    ocp1.node 
    lbFloatingIP:     "192.168.247.57"
    octaviaSupport: "1"
----
Second observation is that lbFloatingIP doesn't get assigned to the master but does work, it gets assigned to the 10.0.0.5 port in the new machineCIDR network. After deploy api.cluster.domain:6443 seems to work as expected, and if you look at the new network it has a port called clusternam-api-port (10.0.0.5) mapped to lbFloatingIP. But console and *.apps.domain do not work. Adding a floating ip to 10.0.0.7 which is listed as clustername-ingress-port and updating *.apps.domain dns to point the new floating ip resolved the console and apps issue.

Comment 11 Tomas Sedovic 2019-08-26 12:25:19 UTC
Oh, that is strange. Maybe there are some issues with the YAML parser. It would also be interesting to know whether the issues is the spacing between keys and values or the ordering (both have changed).

As for the second observation, that is the current expected behaviour. The ports are managed internally by Keepalived and configuring the wildcard *.apps access is a day two operation (due to IPI preferring minimal configuration -- we have to set the API floating IP up front but the ingress one can be done afterwards).

The upstream docs have more details on both the networking configuration: https://github.com/openshift/installer/blob/master/docs/design/openstack/networking-infrastructure.md

and how to set up external access to the cluster: https://github.com/openshift/installer/tree/master/docs/user/openstack#using-floating-ips.

Comment 13 Tomas Sedovic 2019-09-05 15:36:19 UTC
Moving to 4.3. If I understand it correctly, this only happens when you write the install-config.yaml by hand (which should not be required for the majority of the use cases) and even then there is a way to work around this.

We will need more time to reproduce this and come up with some sanitisation and error reporting, but that will not likely happen in 4.3.

Comment 14 David Sanz 2019-09-11 09:31:25 UTC
API FIP is assigned to a port correctly and APPS FIP is not managed by the installer.

Comment 17 Martin André 2020-03-06 08:55:25 UTC
This seems to be linked to an under spec'ed OpenStack cloud. Closing it as WORKSFORME. Please re-open if this happens again.

Comment 18 Andreas Karis 2020-03-17 16:53:45 UTC
Happened for me, too, on an OCP 4.3 / OSP 13 combination.

I could install OCP on this very system before, but after playing around with it and installing / deleting several times, it started acting up.

I'm 0 interested in troubleshooting this, but adding a me too might help prioritize this here a bit. I did not find any way to resume installation (deleted all resources, created new install config, etc. but it would not work)

---

overcloud) [stack@undercloud-0 openshift]$ ./openshift-install create cluster --dir=install-config/ --log-level=debug
DEBUG OpenShift Installer v4.3.1                   
DEBUG Built from commit 2055609f95b19322ee6cfdd0bea73399297c4a3e 
DEBUG Fetching Terraform Variables...              
DEBUG Loading Terraform Variables...               
DEBUG   Loading Cluster ID...                      
DEBUG     Loading Install Config...                
DEBUG       Loading SSH Key...                     
DEBUG       Using SSH Key loaded from state file   
DEBUG       Loading Base Domain...                 
DEBUG         Loading Platform...                  
DEBUG         Using Platform loaded from state file 
DEBUG       Using Base Domain loaded from state file 
DEBUG       Loading Cluster Name...                
DEBUG         Loading Base Domain...               
DEBUG         Loading Platform...                  
DEBUG       Using Cluster Name loaded from state file 
DEBUG       Loading Pull Secret...                 
DEBUG       Using Pull Secret loaded from state file 
DEBUG       Loading Platform...                    
DEBUG     Loading Install Config from both state file and target directory 
DEBUG     Using Install Config loaded from target directory 
DEBUG   Loading Install Config...                  
DEBUG   Loading Image...                           
DEBUG     Loading Install Config...                
DEBUG   Loading BootstrapImage...                  
DEBUG     Loading Install Config...                
DEBUG   Loading Bootstrap Ignition Config...       
DEBUG     Loading Install Config...                
DEBUG     Loading Kubeconfig Admin Internal Client... 
DEBUG       Loading Certificate (admin-kubeconfig-client)... 
DEBUG         Loading Certificate (admin-kubeconfig-signer)... 
DEBUG       Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG         Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG         Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG         Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG           Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Kubeconfig Kubelet...            
DEBUG       Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-client)... 
DEBUG         Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Kubeconfig Admin Client (Loopback)... 
DEBUG       Loading Certificate (admin-kubeconfig-client)... 
DEBUG       Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Master Machines...               
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Platform Credentials Check...  
DEBUG         Loading Install Config...            
DEBUG       Loading Install Config...              
DEBUG       Loading Image...                       
DEBUG       Loading Master Ignition Config...      
DEBUG         Loading Install Config...            
DEBUG         Loading Root CA...                   
DEBUG     Loading Worker Machines...               
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Platform Credentials Check...  
DEBUG       Loading Install Config...              
DEBUG       Loading Image...                       
DEBUG       Loading Worker Ignition Config...      
DEBUG         Loading Install Config...            
DEBUG         Loading Root CA...                   
DEBUG     Loading Common Manifests...              
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Install Config...              
DEBUG       Loading Ingress Config...              
DEBUG         Loading Install Config...            
DEBUG       Loading DNS Config...                  
DEBUG         Loading Install Config...            
DEBUG         Loading Cluster ID...                
DEBUG         Loading Platform Credentials Check... 
DEBUG       Loading Infrastructure Config...       
DEBUG         Loading Cluster ID...                
DEBUG         Loading Install Config...            
DEBUG         Loading Cloud Provider Config...     
DEBUG           Loading Install Config...          
DEBUG           Loading Cluster ID...              
DEBUG           Loading Platform Credentials Check... 
DEBUG         Loading Additional Trust Bundle Config... 
DEBUG           Loading Install Config...          
DEBUG       Loading Network Config...              
DEBUG         Loading Install Config...            
DEBUG         Loading Network CRDs...              
DEBUG       Loading Proxy Config...                
DEBUG         Loading Install Config...            
DEBUG         Loading Network Config...            
DEBUG       Loading Scheduler Config...            
DEBUG         Loading Install Config...            
DEBUG       Loading Image Content Source Policy... 
DEBUG         Loading Install Config...            
DEBUG       Loading Root CA...                     
DEBUG       Loading Certificate (etcd-signer)...   
DEBUG       Loading Certificate (etcd-ca-bundle)... 
DEBUG         Loading Certificate (etcd-signer)... 
DEBUG       Loading Certificate (etcd-client)...   
DEBUG         Loading Certificate (etcd-signer)... 
DEBUG       Loading Certificate (etcd-metric-ca-bundle)... 
DEBUG         Loading Certificate (etcd-metric-signer)... 
DEBUG       Loading Certificate (etcd-metric-signer)... 
DEBUG       Loading Certificate (etcd-metric-signer-client)... 
DEBUG         Loading Certificate (etcd-metric-signer)... 
DEBUG       Loading Certificate (mcs)...           
DEBUG         Loading Root CA...                   
DEBUG         Loading Install Config...            
DEBUG       Loading CVOOverrides...                
DEBUG       Loading EtcdCAConfigMap...             
DEBUG       Loading EtcdClientSecret...            
DEBUG       Loading EtcdHostServiceEndpoints...    
DEBUG       Loading EtcdHostService...             
DEBUG       Loading EtcdMetricClientSecret...      
DEBUG       Loading EtcdMetricServingCAConfigMap... 
DEBUG       Loading EtcdMetricSignerSecret...      
DEBUG       Loading EtcdNamespace...               
DEBUG       Loading EtcdService...                 
DEBUG       Loading EtcdSignerSecret...            
DEBUG       Loading KubeCloudConfig...             
DEBUG       Loading EtcdServingCAConfigMap...      
DEBUG       Loading KubeSystemConfigmapRootCA...   
DEBUG       Loading MachineConfigServerTLSSecret... 
DEBUG       Loading OpenshiftConfigSecretPullSecret... 
DEBUG       Loading OpenshiftMachineConfigOperator... 
DEBUG     Loading Openshift Manifests...           
DEBUG       Loading Install Config...              
DEBUG       Loading Cluster ID...                  
DEBUG       Loading Kubeadmin Password...          
DEBUG       Loading CloudCredsSecret...            
DEBUG       Loading KubeadminPasswordSecret...     
DEBUG       Loading RoleCloudCredsSecretReader...  
DEBUG       Loading Private Cluster Outbound Service... 
DEBUG     Loading Proxy Config...                  
DEBUG     Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG       Loading Certificate (admin-kubeconfig-signer)... 
DEBUG     Loading Certificate (aggregator)...      
DEBUG     Loading Certificate (aggregator-ca-bundle)... 
DEBUG       Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG       Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (aggregator-signer)... 
DEBUG     Loading Certificate (system:kube-apiserver-proxy)... 
DEBUG       Loading Certificate (aggregator)...    
DEBUG     Loading Certificate (etcd-ca-bundle)...  
DEBUG     Loading Certificate (etcd-metric-ca-bundle)... 
DEBUG     Loading Certificate (etcd-metric-signer)... 
DEBUG     Loading Certificate (etcd-metric-signer-client)... 
DEBUG     Loading Certificate (etcd-signer)...     
DEBUG     Loading Certificate (etcd-client)...     
DEBUG     Loading Certificate (journal-gatewayd)... 
DEBUG       Loading Root CA...                     
DEBUG     Loading Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-external-lb-server)... 
DEBUG       Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-internal-lb-server)... 
DEBUG       Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-server)... 
DEBUG       Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-service-network-server)... 
DEBUG       Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Loading Install Config...              
DEBUG     Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG       Loading Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG         Loading Certificate (kubelet-signer)... 
DEBUG       Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG         Loading Certificate (kube-control-plane-signer)... 
DEBUG         Loading Certificate (kube-apiserver-lb-signer)... 
DEBUG         Loading Certificate (kube-apiserver-localhost-signer)... 
DEBUG         Loading Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG         Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG       Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG         Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG       Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Loading Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Loading Certificate (kube-control-plane-ca-bundle)... 
DEBUG     Loading Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG       Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG       Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kube-control-plane-signer)... 
DEBUG     Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client-ca-bundle)... 
DEBUG     Loading Certificate (kubelet-client)...  
DEBUG     Loading Certificate (kubelet-signer)...  
DEBUG     Loading Certificate (kubelet-serving-ca-bundle)... 
DEBUG       Loading Certificate (kubelet-signer)... 
DEBUG     Loading Certificate (mcs)...             
DEBUG     Loading Root CA...                       
DEBUG     Loading Key Pair (service-account.pub)... 
DEBUG     Loading Release Image Pull Spec...       
DEBUG     Loading Image...                         
DEBUG   Loading Master Ignition Config...          
DEBUG   Loading Master Machines...                 
DEBUG   Loading Worker Machines...                 
DEBUG   Fetching Cluster ID...                     
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Cluster ID...                   
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG   Fetching Image...                          
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating Image...                        
DEBUG   Fetching BootstrapImage...                 
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG   Generating BootstrapImage...               
DEBUG   Fetching Bootstrap Ignition Config...      
DEBUG     Fetching Install Config...               
DEBUG     Reusing previously-fetched Install Config 
DEBUG     Fetching Kubeconfig Admin Internal Client... 
DEBUG       Fetching Certificate (admin-kubeconfig-client)... 
DEBUG         Fetching Certificate (admin-kubeconfig-signer)... 
DEBUG         Generating Certificate (admin-kubeconfig-signer)... 
DEBUG       Generating Certificate (admin-kubeconfig-client)... 
DEBUG       Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG         Fetching Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG           Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG           Generating Certificate (kube-apiserver-localhost-signer)... 
DEBUG         Generating Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG         Fetching Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG           Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG           Generating Certificate (kube-apiserver-service-network-signer)... 
DEBUG         Generating Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG         Fetching Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG           Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG           Generating Certificate (kube-apiserver-lb-signer)... 
DEBUG         Generating Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG       Generating Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG     Generating Kubeconfig Admin Internal Client... 
DEBUG     Fetching Kubeconfig Kubelet...           
DEBUG       Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) 
DEBUG       Fetching Certificate (kubelet-client)... 
DEBUG         Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG         Generating Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG       Generating Certificate (kubelet-client)... 
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG     Generating Kubeconfig Kubelet...         
DEBUG     Fetching Kubeconfig Admin Client (Loopback)... 
DEBUG       Fetching Certificate (admin-kubeconfig-client)... 
DEBUG       Reusing previously-fetched Certificate (admin-kubeconfig-client) 
DEBUG       Fetching Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle) 
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG     Generating Kubeconfig Admin Client (Loopback)... 
DEBUG     Fetching Master Machines...              
DEBUG       Fetching Cluster ID...                 
DEBUG       Reusing previously-fetched Cluster ID  
DEBUG       Fetching Platform Credentials Check... 
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG       Generating Platform Credentials Check... 
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG       Fetching Image...                      
DEBUG       Reusing previously-fetched Image       
DEBUG       Fetching Master Ignition Config...     
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG         Fetching Root CA...                  
DEBUG         Generating Root CA...                
DEBUG       Generating Master Ignition Config...   
DEBUG     Generating Master Machines...            
DEBUG     Fetching Worker Machines...              
DEBUG       Fetching Cluster ID...                 
DEBUG       Reusing previously-fetched Cluster ID  
DEBUG       Fetching Platform Credentials Check... 
DEBUG       Reusing previously-fetched Platform Credentials Check 
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG       Fetching Image...                      
DEBUG       Reusing previously-fetched Image       
DEBUG       Fetching Worker Ignition Config...     
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG         Fetching Root CA...                  
DEBUG         Reusing previously-fetched Root CA   
DEBUG       Generating Worker Ignition Config...   
DEBUG     Generating Worker Machines...            
DEBUG     Fetching Common Manifests...             
DEBUG       Fetching Cluster ID...                 
DEBUG       Reusing previously-fetched Cluster ID  
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG       Fetching Ingress Config...             
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG       Generating Ingress Config...           
DEBUG       Fetching DNS Config...                 
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG         Fetching Cluster ID...               
DEBUG         Reusing previously-fetched Cluster ID 
DEBUG         Fetching Platform Credentials Check... 
DEBUG         Reusing previously-fetched Platform Credentials Check 
DEBUG       Generating DNS Config...               
DEBUG       Fetching Infrastructure Config...      
DEBUG         Fetching Cluster ID...               
DEBUG         Reusing previously-fetched Cluster ID 
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG         Fetching Cloud Provider Config...    
DEBUG           Fetching Install Config...         
DEBUG           Reusing previously-fetched Install Config 
DEBUG           Fetching Cluster ID...             
DEBUG           Reusing previously-fetched Cluster ID 
DEBUG           Fetching Platform Credentials Check... 
DEBUG           Reusing previously-fetched Platform Credentials Check 
DEBUG         Generating Cloud Provider Config...  
DEBUG         Fetching Additional Trust Bundle Config... 
DEBUG           Fetching Install Config...         
DEBUG           Reusing previously-fetched Install Config 
DEBUG         Generating Additional Trust Bundle Config... 
DEBUG       Generating Infrastructure Config...    
DEBUG       Fetching Network Config...             
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG         Fetching Network CRDs...             
DEBUG         Generating Network CRDs...           
DEBUG       Generating Network Config...           
DEBUG       Fetching Proxy Config...               
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG         Fetching Network Config...           
DEBUG         Reusing previously-fetched Network Config 
DEBUG       Generating Proxy Config...             
DEBUG       Fetching Scheduler Config...           
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG       Generating Scheduler Config...         
DEBUG       Fetching Image Content Source Policy... 
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG       Generating Image Content Source Policy... 
DEBUG       Fetching Root CA...                    
DEBUG       Reusing previously-fetched Root CA     
DEBUG       Fetching Certificate (etcd-signer)...  
DEBUG       Generating Certificate (etcd-signer)... 
DEBUG       Fetching Certificate (etcd-ca-bundle)... 
DEBUG         Fetching Certificate (etcd-signer)... 
DEBUG         Reusing previously-fetched Certificate (etcd-signer) 
DEBUG       Generating Certificate (etcd-ca-bundle)... 
DEBUG       Fetching Certificate (etcd-client)...  
DEBUG         Fetching Certificate (etcd-signer)... 
DEBUG         Reusing previously-fetched Certificate (etcd-signer) 
DEBUG       Generating Certificate (etcd-client)... 
DEBUG       Fetching Certificate (etcd-metric-ca-bundle)... 
DEBUG         Fetching Certificate (etcd-metric-signer)... 
DEBUG         Generating Certificate (etcd-metric-signer)... 
DEBUG       Generating Certificate (etcd-metric-ca-bundle)... 
DEBUG       Fetching Certificate (etcd-metric-signer)... 
DEBUG       Reusing previously-fetched Certificate (etcd-metric-signer) 
DEBUG       Fetching Certificate (etcd-metric-signer-client)... 
DEBUG         Fetching Certificate (etcd-metric-signer)... 
DEBUG         Reusing previously-fetched Certificate (etcd-metric-signer) 
DEBUG       Generating Certificate (etcd-metric-signer-client)... 
DEBUG       Fetching Certificate (mcs)...          
DEBUG         Fetching Root CA...                  
DEBUG         Reusing previously-fetched Root CA   
DEBUG         Fetching Install Config...           
DEBUG         Reusing previously-fetched Install Config 
DEBUG       Generating Certificate (mcs)...        
DEBUG       Fetching CVOOverrides...               
DEBUG       Generating CVOOverrides...             
DEBUG       Fetching EtcdCAConfigMap...            
DEBUG       Generating EtcdCAConfigMap...          
DEBUG       Fetching EtcdClientSecret...           
DEBUG       Generating EtcdClientSecret...         
DEBUG       Fetching EtcdHostServiceEndpoints...   
DEBUG       Generating EtcdHostServiceEndpoints... 
DEBUG       Fetching EtcdHostService...            
DEBUG       Generating EtcdHostService...          
DEBUG       Fetching EtcdMetricClientSecret...     
DEBUG       Generating EtcdMetricClientSecret...   
DEBUG       Fetching EtcdMetricServingCAConfigMap... 
DEBUG       Generating EtcdMetricServingCAConfigMap... 
DEBUG       Fetching EtcdMetricSignerSecret...     
DEBUG       Generating EtcdMetricSignerSecret...   
DEBUG       Fetching EtcdNamespace...              
DEBUG       Generating EtcdNamespace...            
DEBUG       Fetching EtcdService...                
DEBUG       Generating EtcdService...              
DEBUG       Fetching EtcdSignerSecret...           
DEBUG       Generating EtcdSignerSecret...         
DEBUG       Fetching KubeCloudConfig...            
DEBUG       Generating KubeCloudConfig...          
DEBUG       Fetching EtcdServingCAConfigMap...     
DEBUG       Generating EtcdServingCAConfigMap...   
DEBUG       Fetching KubeSystemConfigmapRootCA...  
DEBUG       Generating KubeSystemConfigmapRootCA... 
DEBUG       Fetching MachineConfigServerTLSSecret... 
DEBUG       Generating MachineConfigServerTLSSecret... 
DEBUG       Fetching OpenshiftConfigSecretPullSecret... 
DEBUG       Generating OpenshiftConfigSecretPullSecret... 
DEBUG       Fetching OpenshiftMachineConfigOperator... 
DEBUG       Generating OpenshiftMachineConfigOperator... 
DEBUG     Generating Common Manifests...           
DEBUG     Fetching Openshift Manifests...          
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG       Fetching Cluster ID...                 
DEBUG       Reusing previously-fetched Cluster ID  
DEBUG       Fetching Kubeadmin Password...         
DEBUG       Generating Kubeadmin Password...       
DEBUG       Fetching CloudCredsSecret...           
DEBUG       Generating CloudCredsSecret...         
DEBUG       Fetching KubeadminPasswordSecret...    
DEBUG       Generating KubeadminPasswordSecret...  
DEBUG       Fetching RoleCloudCredsSecretReader... 
DEBUG       Generating RoleCloudCredsSecretReader... 
DEBUG       Fetching Private Cluster Outbound Service... 
DEBUG       Generating Private Cluster Outbound Service... 
DEBUG     Generating Openshift Manifests...        
DEBUG     Fetching Proxy Config...                 
DEBUG     Reusing previously-fetched Proxy Config  
DEBUG     Fetching Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG       Fetching Certificate (admin-kubeconfig-signer)... 
DEBUG       Reusing previously-fetched Certificate (admin-kubeconfig-signer) 
DEBUG     Generating Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG     Fetching Certificate (aggregator)...     
DEBUG     Generating Certificate (aggregator)...   
DEBUG     Fetching Certificate (aggregator-ca-bundle)... 
DEBUG       Fetching Certificate (aggregator-signer)... 
DEBUG       Generating Certificate (aggregator-signer)... 
DEBUG     Generating Certificate (aggregator-ca-bundle)... 
DEBUG     Fetching Certificate (system:kube-apiserver-proxy)... 
DEBUG       Fetching Certificate (aggregator-signer)... 
DEBUG       Reusing previously-fetched Certificate (aggregator-signer) 
DEBUG     Generating Certificate (system:kube-apiserver-proxy)... 
DEBUG     Fetching Certificate (aggregator-signer)... 
DEBUG     Reusing previously-fetched Certificate (aggregator-signer) 
DEBUG     Fetching Certificate (system:kube-apiserver-proxy)... 
DEBUG       Fetching Certificate (aggregator)...   
DEBUG       Reusing previously-fetched Certificate (aggregator) 
DEBUG     Generating Certificate (system:kube-apiserver-proxy)... 
DEBUG     Fetching Certificate (etcd-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (etcd-ca-bundle) 
DEBUG     Fetching Certificate (etcd-metric-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (etcd-metric-ca-bundle) 
DEBUG     Fetching Certificate (etcd-metric-signer)... 
DEBUG     Reusing previously-fetched Certificate (etcd-metric-signer) 
DEBUG     Fetching Certificate (etcd-metric-signer-client)... 
DEBUG     Reusing previously-fetched Certificate (etcd-metric-signer-client) 
DEBUG     Fetching Certificate (etcd-signer)...    
DEBUG     Reusing previously-fetched Certificate (etcd-signer) 
DEBUG     Fetching Certificate (etcd-client)...    
DEBUG     Reusing previously-fetched Certificate (etcd-client) 
DEBUG     Fetching Certificate (journal-gatewayd)... 
DEBUG       Fetching Root CA...                    
DEBUG       Reusing previously-fetched Root CA     
DEBUG     Generating Certificate (journal-gatewayd)... 
DEBUG     Fetching Certificate (kube-apiserver-lb-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-lb-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-external-lb-server)... 
DEBUG       Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG     Generating Certificate (kube-apiserver-external-lb-server)... 
DEBUG     Fetching Certificate (kube-apiserver-internal-lb-server)... 
DEBUG       Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG     Generating Certificate (kube-apiserver-internal-lb-server)... 
DEBUG     Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG     Fetching Certificate (kube-apiserver-localhost-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-localhost-server)... 
DEBUG       Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) 
DEBUG     Generating Certificate (kube-apiserver-localhost-server)... 
DEBUG     Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) 
DEBUG     Fetching Certificate (kube-apiserver-service-network-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-service-network-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-service-network-server)... 
DEBUG       Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) 
DEBUG       Fetching Install Config...             
DEBUG       Reusing previously-fetched Install Config 
DEBUG     Generating Certificate (kube-apiserver-service-network-server)... 
DEBUG     Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) 
DEBUG     Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG       Fetching Certificate (admin-kubeconfig-ca-bundle)... 
DEBUG       Reusing previously-fetched Certificate (admin-kubeconfig-ca-bundle) 
DEBUG       Fetching Certificate (kubelet-client-ca-bundle)... 
DEBUG         Fetching Certificate (kubelet-signer)... 
DEBUG         Generating Certificate (kubelet-signer)... 
DEBUG       Generating Certificate (kubelet-client-ca-bundle)... 
DEBUG       Fetching Certificate (kube-control-plane-ca-bundle)... 
DEBUG         Fetching Certificate (kube-control-plane-signer)... 
DEBUG         Generating Certificate (kube-control-plane-signer)... 
DEBUG         Fetching Certificate (kube-apiserver-lb-signer)... 
DEBUG         Reusing previously-fetched Certificate (kube-apiserver-lb-signer) 
DEBUG         Fetching Certificate (kube-apiserver-localhost-signer)... 
DEBUG         Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) 
DEBUG         Fetching Certificate (kube-apiserver-service-network-signer)... 
DEBUG         Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) 
DEBUG       Generating Certificate (kube-control-plane-ca-bundle)... 
DEBUG       Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG         Fetching Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG         Generating Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG       Generating Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG       Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG         Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)... 
DEBUG         Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-signer) 
DEBUG       Generating Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG     Generating Certificate (kube-apiserver-complete-client-ca-bundle)... 
DEBUG     Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-ca-bundle) 
DEBUG     Fetching Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG       Fetching Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer) 
DEBUG     Generating Certificate (kube-apiserver-to-kubelet-client)... 
DEBUG     Fetching Certificate (kube-apiserver-to-kubelet-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer) 
DEBUG     Fetching Certificate (kube-control-plane-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kube-control-plane-ca-bundle) 
DEBUG     Fetching Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG       Fetching Certificate (kube-control-plane-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-control-plane-signer) 
DEBUG     Generating Certificate (kube-control-plane-kube-controller-manager-client)... 
DEBUG     Fetching Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG       Fetching Certificate (kube-control-plane-signer)... 
DEBUG       Reusing previously-fetched Certificate (kube-control-plane-signer) 
DEBUG     Generating Certificate (kube-control-plane-kube-scheduler-client)... 
DEBUG     Fetching Certificate (kube-control-plane-signer)... 
DEBUG     Reusing previously-fetched Certificate (kube-control-plane-signer) 
DEBUG     Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-ca-bundle) 
DEBUG     Fetching Certificate (kubelet-client-ca-bundle)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-client-ca-bundle) 
DEBUG     Fetching Certificate (kubelet-client)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-client) 
DEBUG     Fetching Certificate (kubelet-signer)... 
DEBUG     Reusing previously-fetched Certificate (kubelet-signer) 
DEBUG     Fetching Certificate (kubelet-serving-ca-bundle)... 
DEBUG       Fetching Certificate (kubelet-signer)... 
DEBUG       Reusing previously-fetched Certificate (kubelet-signer) 
DEBUG     Generating Certificate (kubelet-serving-ca-bundle)... 
DEBUG     Fetching Certificate (mcs)...            
DEBUG     Reusing previously-fetched Certificate (mcs) 
DEBUG     Fetching Root CA...                      
DEBUG     Reusing previously-fetched Root CA       
DEBUG     Fetching Key Pair (service-account.pub)... 
DEBUG     Generating Key Pair (service-account.pub)... 
DEBUG     Fetching Release Image Pull Spec...      
DEBUG     Generating Release Image Pull Spec...    
DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:ea7ac3ad42169b39fce07e5e53403a028644810bee9a212e7456074894df40f3 
DEBUG     Fetching Image...                        
DEBUG     Reusing previously-fetched Image         
DEBUG   Generating Bootstrap Ignition Config...    
DEBUG   Fetching Master Ignition Config...         
DEBUG   Reusing previously-fetched Master Ignition Config 
DEBUG   Fetching Master Machines...                
DEBUG   Reusing previously-fetched Master Machines 
DEBUG   Fetching Worker Machines...                
DEBUG   Reusing previously-fetched Worker Machines 
DEBUG Generating Terraform Variables...            
INFO Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.3/43.81.202001142154.0/x86_64/rhcos-43.81.202001142154.0-openstack.x86_64.qcow2.gz?sha256=504b9008adf89bb3d05b75d393e057c6d66ba6c92cf631ca4445d99bbf7e2a57' 
INFO The file was found in cache: /home/stack/.cache/openshift-installer/image_cache/d03e06d2824bac47eaee911dcc5feff9. Reusing... 
DEBUG Creating a Swift container for your bootstrap ignition... 
DEBUG Container cluster-m4bbk was created.         
DEBUG Creating a Swift object in container cluster-m4bbk containing your bootstrap ignition... 
DEBUG The object was created.                      
INFO Consuming Install Config from target directory 
DEBUG Purging asset "Install Config" from disk     
DEBUG Fetching Kubeconfig Admin Client...          
DEBUG Loading Kubeconfig Admin Client...           
DEBUG   Loading Certificate (admin-kubeconfig-client)... 
DEBUG   Loading Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG   Loading Install Config...                  
DEBUG   Fetching Certificate (admin-kubeconfig-client)... 
DEBUG   Reusing previously-fetched Certificate (admin-kubeconfig-client) 
DEBUG   Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... 
DEBUG   Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) 
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG Generating Kubeconfig Admin Client...        
DEBUG Fetching Kubeadmin Password...               
DEBUG Reusing previously-fetched Kubeadmin Password 
DEBUG Fetching Certificate (journal-gatewayd)...   
DEBUG Reusing previously-fetched Certificate (journal-gatewayd) 
DEBUG Fetching Metadata...                         
DEBUG Loading Metadata...                          
DEBUG   Loading Cluster ID...                      
DEBUG   Loading Install Config...                  
DEBUG   Fetching Cluster ID...                     
DEBUG   Reusing previously-fetched Cluster ID      
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG Generating Metadata...                       
DEBUG Fetching Cluster...                          
DEBUG Loading Cluster...                           
DEBUG   Loading Cluster ID...                      
DEBUG   Loading Install Config...                  
DEBUG   Loading Platform Credentials Check...      
DEBUG   Loading Terraform Variables...             
DEBUG   Loading Kubeadmin Password...              
DEBUG   Fetching Cluster ID...                     
DEBUG   Reusing previously-fetched Cluster ID      
DEBUG   Fetching Install Config...                 
DEBUG   Reusing previously-fetched Install Config  
DEBUG   Fetching Platform Credentials Check...     
DEBUG   Reusing previously-fetched Platform Credentials Check 
DEBUG   Fetching Terraform Variables...            
DEBUG   Reusing previously-fetched Terraform Variables 
DEBUG   Fetching Kubeadmin Password...             
DEBUG   Reusing previously-fetched Kubeadmin Password 
DEBUG Generating Cluster...                        
INFO Creating infrastructure resources...         
DEBUG Symlinking plugin terraform-provider-azureprivatedns src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-azureprivatedns" 
DEBUG Symlinking plugin terraform-provider-google src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-google" 
DEBUG Symlinking plugin terraform-provider-ignition src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-ignition" 
DEBUG Symlinking plugin terraform-provider-local src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-local" 
DEBUG Symlinking plugin terraform-provider-openstack src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-openstack" 
DEBUG Symlinking plugin terraform-provider-random src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-random" 
DEBUG Symlinking plugin terraform-provider-aws src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-aws" 
DEBUG Symlinking plugin terraform-provider-azurerm src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-azurerm" 
DEBUG Initializing modules...                      
DEBUG - bootstrap in ../../../tmp/openshift-install-233199779/bootstrap 
DEBUG - masters in ../../../tmp/openshift-install-233199779/masters 
DEBUG - topology in ../../../tmp/openshift-install-233199779/topology 
DEBUG                                              
DEBUG Initializing the backend...                  
DEBUG                                              
DEBUG Initializing provider plugins...             
DEBUG                                              
DEBUG Terraform has been successfully initialized! 
DEBUG                                              
DEBUG You may now begin working with Terraform. Try running "terraform plan" to see 
DEBUG any changes that are required for your infrastructure. All Terraform commands 
DEBUG should now work.                             
DEBUG                                              
DEBUG If you ever set or change modules or backend configuration for Terraform, 
DEBUG rerun this command to reinitialize your working directory. If you forget, other 
DEBUG commands will detect it and remind you to do so if necessary. 
DEBUG module.masters.data.ignition_file.hostname[2]: Refreshing state... 
DEBUG module.masters.data.ignition_file.hostname[1]: Refreshing state... 
DEBUG module.masters.data.ignition_file.hostname[0]: Refreshing state... 
DEBUG module.masters.data.ignition_config.master_ignition_config[1]: Refreshing state... 
DEBUG module.masters.data.ignition_config.master_ignition_config[2]: Refreshing state... 
DEBUG module.masters.data.ignition_config.master_ignition_config[0]: Refreshing state... 
DEBUG module.topology.data.openstack_networking_network_v2.external_network: Refreshing state... 
DEBUG module.masters.data.openstack_compute_flavor_v2.masters_flavor: Refreshing state... 
DEBUG module.bootstrap.data.openstack_compute_flavor_v2.bootstrap_flavor: Refreshing state... 
DEBUG module.topology.openstack_networking_secgroup_v2.master: Creating... 
DEBUG module.topology.openstack_networking_secgroup_v2.worker: Creating... 
DEBUG module.topology.openstack_networking_router_v2.openshift-external-router: Creating... 
DEBUG module.topology.openstack_networking_network_v2.openshift-private: Creating... 
DEBUG openstack_images_image_v2.base_image[0]: Creating... 
DEBUG module.topology.openstack_networking_secgroup_v2.worker: Creation complete after 2s [id=ef2f35cc-663c-49e3-890e-c285b36e531a] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_https: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_ssh: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_icmp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_services_tcp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vrrp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_icmp: Creation complete after 0s [id=7720bcf7-3108-4c70-aaa5-96fa3dd40e0e] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal: Creating... 
DEBUG module.topology.openstack_networking_secgroup_v2.master: Creation complete after 2s [id=e264ce98-5970-4aa8-ab25-7e860fb8eafe] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_http: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_udp: Creation complete after 0s [id=60e7cba0-4dfb-4c14-bcfc-97323e61b4ba] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_geneve: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_https: Creation complete after 1s [id=11c09d77-acf9-40c1-a35c-b9577e6a4b87] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_services_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal: Creation complete after 1s [id=7c36a8a3-a9f0-4cfc-8604-1e6fbfeeafda] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_kubelet_insecure: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_ssh: Creation complete after 1s [id=f2973b7c-c011-4b49-bbd4-ca571185ec60] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_router: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vrrp: Creation complete after 1s [id=c7fe5e19-df49-4425-acf9-462499b6a6ea] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_mdns_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_http: Creation complete after 2s [id=9dbb1d0c-ee5b-4898-b578-33377dc21410] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vxlan: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_services_tcp: Creation complete after 2s [id=149f61f4-dbfd-459f-a676-4f35b3183610] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_controller_manager: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_router: Creation complete after 1s [id=5f04e0f4-8805-4a9c-ab83-6cba620240d8] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_mdns_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vxlan: Creation complete after 0s [id=820241b9-2d87-4cd9-93e5-2f059f6f42d3] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_kubelet_insecure_from_master: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_controller_manager: Creation complete after 0s [id=d35066b9-9ff9-42d0-845e-310fecd30029] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kubelet_secure: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_kubelet_insecure_from_master: Creation complete after 0s [id=7d51b197-79ff-4d11-a47b-f2f755f12084] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_mdns_udp: Creation complete after 0s [id=524acfd1-7847-494c-868c-716d39989c9d] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vxlan_from_worker: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_scheduler_from_worker: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kubelet_secure: Creation complete after 0s [id=b93a5deb-21c4-488e-a6e1-323ace6187b8] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_controller_manager_from_worker: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_scheduler_from_worker: Creation complete after 0s [id=31ad01e7-ddb1-4bfd-ada9-083b7ba6d819] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_etcd: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_kubelet_insecure: Creation complete after 2s [id=c3f3a54a-81ed-449f-b3e3-ad8552d2bac2] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_from_worker: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_geneve: Creation complete after 3s [id=ee762237-b1ff-4a6c-a67b-1de9ac3eb9c5] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vrrp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vxlan_from_worker: Creation complete after 1s [id=2aee6302-374a-4852-83c8-6f48e288db77] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_mcs: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_from_worker: Creation complete after 0s [id=72d74bf2-6083-4637-a6c1-532d4c07a94b] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_scheduler: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_etcd: Creation complete after 1s [id=45bb8d2e-96cf-4779-90f9-49fd77abe369] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_geneve: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vrrp: Creation complete after 0s [id=6691a704-85ad-441c-b00c-5853fc2c2bb6] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_https: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_services_udp: Creation complete after 2s [id=7ce9ab9d-49fc-41fd-b378-906bb9ea2ee6] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_geneve: Creation complete after 0s [id=2fce4947-ff24-4b81-94a3-c37673d285ce] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_mdns_udp: Creation complete after 2s [id=903fc1b1-da2a-42ed-b3ec-63705b3eec59] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ssh: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_services_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_udp: Creation complete after 0s [id=9ce12aac-f838-4fac-ad6e-546658afa3a6] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_dns_tcp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_https: Creation complete after 1s [id=424e04ae-9f45-4536-8297-76b33388ac10] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_geneve_from_master: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_dns_tcp: Creation complete after 1s [id=934ee797-aa33-45ec-b9a9-c36c25f57225] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vxlan: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_geneve_from_master: Creation complete after 0s [id=aab3b700-3df7-4226-9879-fdf39bf61a5e] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_dns_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_mcs: Creation complete after 1s [id=44be6211-2eb9-4ae9-a192-6d99a415ffb9] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_from_worker_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ssh: Creation complete after 1s [id=471312fc-d3a9-4c18-af0c-588b35852ab6] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vxlan_from_master: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_dns_udp: Creation complete after 0s [id=f3ae81d5-ef04-4bfa-b6f4-30fe2c1cce05] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ovndb: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vxlan_from_master: Creation complete after 0s [id=03379b77-844f-4755-8296-0dea0802b5b7] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_icmp: Creating... 
DEBUG module.topology.openstack_networking_network_v2.openshift-private: Creation complete after 6s [id=3f0bf23e-4182-4b66-9bb7-1cf7b3b4cfa1] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_from_master_udp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_from_worker_udp: Creation complete after 1s [id=46e32b82-1b07-4014-bd1f-5bd3e8206c19] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kubelet_secure_from_worker: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_icmp: Creation complete after 1s [id=591ba1ef-82a7-4552-850d-986ef48d2d78] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ovndb_from_worker: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_from_master_udp: Creation complete after 1s [id=458e88ab-f189-4b24-9bb7-0e397f30e376] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_services_tcp: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ovndb: Creation complete after 1s [id=a5fc88c3-2065-4d43-9f8a-af1bb114ab92] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_geneve_from_worker: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vxlan: Creation complete after 1s [id=0e591b5f-b4c7-45af-bda5-d597a22a8482] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_from_master: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_geneve_from_worker: Creation complete after 0s [id=b0517ae4-3dd3-4cd3-aace-613b79d61b6a] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_from_master: Creation complete after 1s [id=ac7d35a6-e522-4f59-a254-2c9885230453] 
DEBUG module.topology.openstack_networking_subnet_v2.nodes: Creating... 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ovndb_from_worker: Creation complete after 1s [id=c9c71786-975f-48df-82d5-fe397bed8bc1] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_services_udp: Creation complete after 3s [id=0de9f7f1-a2c6-4c98-89e4-40e0a1b7df2b] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal: Creation complete after 1s [id=4fac13fa-d129-483b-9a1e-aef3077c003a] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_services_tcp: Creation complete after 1s [id=7f6d7e01-4cbd-4302-99bf-00eb861820ee] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_controller_manager_from_worker: Creation complete after 4s [id=2e138fa4-3ad9-4ba6-94ad-75306fb6f64c] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_scheduler: Creation complete after 4s [id=7ac8626d-e9e1-4390-baef-612289faa01f] 
DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kubelet_secure_from_worker: Creation complete after 2s [id=a8f9dd82-1641-44a0-827e-2c9b2463a8f0] 
DEBUG module.topology.openstack_networking_router_v2.openshift-external-router: Still creating... [10s elapsed] 
DEBUG openstack_images_image_v2.base_image[0]: Still creating... [10s elapsed] 
DEBUG module.topology.openstack_networking_subnet_v2.nodes: Creation complete after 6s [id=5bdde3eb-f9b2-4fd8-bc07-32375fad5cd6] 
DEBUG module.topology.openstack_networking_port_v2.ingress_port: Creating... 
DEBUG module.topology.openstack_networking_port_v2.api_port: Creating... 
DEBUG module.bootstrap.openstack_networking_port_v2.bootstrap_port: Creating... 
DEBUG module.topology.openstack_networking_port_v2.masters[2]: Creating... 
DEBUG module.topology.openstack_networking_port_v2.masters[0]: Creating... 
DEBUG module.topology.openstack_networking_port_v2.masters[1]: Creating... 
DEBUG openstack_images_image_v2.base_image[0]: Still creating... [20s elapsed] 
DEBUG module.topology.openstack_networking_router_v2.openshift-external-router: Still creating... [20s elapsed] 
DEBUG module.topology.openstack_networking_router_v2.openshift-external-router: Creation complete after 22s [id=18a9c5da-22ff-46c1-b898-b103921ffda2] 
DEBUG module.topology.openstack_networking_router_interface_v2.nodes_router_interface: Creating... 
DEBUG module.topology.openstack_networking_port_v2.api_port: Creation complete after 9s [id=9cbcab5a-751a-4d83-a8ea-887c4408c79b] 
DEBUG module.topology.openstack_networking_floatingip_associate_v2.api_fip[0]: Creating... 
DEBUG module.topology.openstack_networking_port_v2.masters[2]: Creation complete after 9s [id=ed0eb15b-99a6-4ac6-9d52-928821a7919f] 
DEBUG module.bootstrap.openstack_networking_port_v2.bootstrap_port: Creation complete after 10s [id=02902224-5a91-4bd4-952a-bfda4268d0e4] 
DEBUG module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Creating... 
DEBUG module.topology.openstack_networking_port_v2.masters[1]: Creation complete after 10s [id=5fd4aeff-2b7b-4796-8752-a720a9e215d2] 
DEBUG module.topology.openstack_networking_port_v2.ingress_port: Still creating... [10s elapsed] 
DEBUG module.topology.openstack_networking_port_v2.masters[0]: Still creating... [10s elapsed] 
DEBUG module.topology.openstack_networking_port_v2.ingress_port: Creation complete after 11s [id=51229511-5e1b-4a87-beb3-04494884a2c1] 
DEBUG module.topology.openstack_networking_port_v2.masters[0]: Creation complete after 11s [id=ded7d54d-7019-465c-894f-c3ad9074880d] 
DEBUG module.topology.openstack_networking_trunk_v2.masters[0]: Creating... 
DEBUG module.topology.openstack_networking_trunk_v2.masters[1]: Creating... 
DEBUG module.topology.openstack_networking_trunk_v2.masters[2]: Creating... 
DEBUG openstack_images_image_v2.base_image[0]: Still creating... [30s elapsed] 
DEBUG module.topology.openstack_networking_trunk_v2.masters[1]: Creation complete after 7s [id=fbb4fb13-9ca6-4c34-b6b9-c3baa8890777] 
DEBUG module.topology.openstack_networking_router_interface_v2.nodes_router_interface: Still creating... [10s elapsed] 
DEBUG module.topology.openstack_networking_trunk_v2.masters[2]: Creation complete after 7s [id=e16bf30a-05c8-4c64-9462-327d1269eb63] 
DEBUG module.topology.openstack_networking_trunk_v2.masters[0]: Creation complete after 7s [id=a8e23e70-bec8-4f13-a6df-832652544859] 
DEBUG module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Still creating... [10s elapsed] 
DEBUG module.topology.openstack_networking_router_interface_v2.nodes_router_interface: Creation complete after 14s [id=4d6e25df-45b6-479f-a935-3e1e6d91551b] 
DEBUG module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Creation complete after 14s [id=df770468-140f-476a-9439-ab34ceb2c545] 
DEBUG openstack_images_image_v2.base_image[0]: Still creating... [40s elapsed] 
DEBUG openstack_images_image_v2.base_image[0]: Creation complete after 45s [id=86a2b3f5-4964-4d28-bb75-169774b2d07c] 
DEBUG data.openstack_images_image_v2.base_image: Refreshing state... 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Creating... 
DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Creating... 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Creating... 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Creating... 
DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [10s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [10s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [10s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [10s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [20s elapsed] 
DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [20s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [20s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [20s elapsed] 
DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [30s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [30s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [30s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [30s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [40s elapsed] 
DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [40s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [40s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [40s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [50s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [50s elapsed] 
DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [50s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [50s elapsed] 
DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [1m0s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [1m0s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [1m0s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [1m0s elapsed] 
DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Creation complete after 1m6s [id=95608032-ea7d-4f22-aea3-e07d981ff832] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Creation complete after 1m7s [id=5dc8d9e3-adad-478e-8cb7-d84a70354164] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Creation complete after 1m7s [id=89f75363-b807-41bf-b6af-6d9f360284f8] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [1m10s elapsed] 
DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Creation complete after 1m11s [id=38c524ee-2750-4e32-bf68-5b7e05f1d2b9] 
ERROR                                              
ERROR Error: Error associating openstack_networking_floatingip_associate_v2 floating_ip 74d54930-0753-4d82-b7ea-74a06668ebfa with port 9cbcab5a-751a-4d83-a8ea-887c4408c79b: Resource not found 
ERROR                                              
ERROR   on ../../../tmp/openshift-install-233199779/topology/private-network.tf line 122, in resource "openstack_networking_floatingip_associate_v2" "api_fip": 
ERROR  122: resource "openstack_networking_floatingip_associate_v2" "api_fip" { 
ERROR                                              
ERROR                                              
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply using Terraform 

---------------------

(overcloud) [stack@undercloud-0 openshift]$ openstack port list | grep 9cbcab5a-751a-4d83-a8ea-887c4408c79b
| 9cbcab5a-751a-4d83-a8ea-887c4408c79b | cluster-m4bbk-api-port                          | fa:16:3e:c3:8c:6e | ip_address='172.31.0.5', subnet_id='5bdde3eb-f9b2-4fd8-bc07-32375fad5cd6'     | DOWN   |
(overcloud) [stack@undercloud-0 openshift]$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 43ca9393-acbd-4e16-935a-9cce6750cf03 | 172.16.0.205        | None             | None                                 | aa483134-ef6c-4d62-bf51-ad98377669b0 | d69c1c6601c741deaa205fa1a7e9c632 |
| 74d54930-0753-4d82-b7ea-74a06668ebfa | 172.16.0.204        | None             | None                                 | aa483134-ef6c-4d62-bf51-ad98377669b0 | d69c1c6601c741deaa205fa1a7e9c632 |
| df770468-140f-476a-9439-ab34ceb2c545 | 172.16.0.213        | 172.31.0.37      | 02902224-5a91-4bd4-952a-bfda4268d0e4 | aa483134-ef6c-4d62-bf51-ad98377669b0 | d69c1c6601c741deaa205fa1a7e9c632 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
(overcloud) [stack@undercloud-0 openshift]$

Comment 19 Andreas Karis 2020-05-20 17:13:21 UTC
I ran into this again with OCP 4.3.13  (trying to reproduce a customer issue).

Just to explain the aforementioned workaround:

Change install-config.yaml from:
~~~
(...)
platform:
  openstack:
    cloud: openstack
    computeFlavor: m1.openshift
    externalDNS: null
    externalNetwork: provider1
    lbFloatingIP: 172.16.0.206
    octaviaSupport: "1"
    region: ""
    trunkSupport: "1"
publish: External
~~~

To:
~~~
(...)
platform:
  openstack:
    cloud: openstack
    computeFlavor: m1.openshift
    externalDNS: null
    externalNetwork: provider1
    #lbFloatingIP: 172.16.0.206
    octaviaSupport: "1"
    region: ""
    trunkSupport: "1"
publish: External
~~~

Then, run the installation:
~~~
openshift-install create cluster --dir install-config/ --log-level debug
~~~

As soon as terraform finishes and the installer tries to contact the API ...
~~~
DEBUG Still waiting for the Kubernetes API: Get https://api.cluster43.example.com:6443/version?timeout=32s: dial tcp 172.16.0.206:6443: connect: no route to host 
~~~

... run:
~~~
clusterID=$(openstack server list | egrep -o 'cluster43-[0-9a-z]+-bootstrap' | awk -F '-' '{print $2}')
openstack floating ip set --port cluster43-${clusterID}-api-port 172.16.0.206
~~~

Verify:
~~~
(overcloud) [stack@undercloud-0 openshift43]$ openstack floating ip list | grep 172.16.0.206
| f33afe42-1b8f-45c1-9fd0-b36020e10b0c | 172.16.0.206        | 172.31.0.5       | aa3fa224-5a3c-4552-993b-09189cf16e5b | b3979e59-b9cf-4b95-8933-46aedd536192 | ba4092513d9f4969b224a094e9fdf1dd |
(overcloud) [stack@undercloud-0 openshift43]$ openstack port list | grep cluster43-${clusterID}-api-port
| aa3fa224-5a3c-4552-993b-09189cf16e5b | cluster43-zz94k-api-port                                              | fa:16:3e:fd:4c:06 | ip_address='172.31.0.5', subnet_id='3c59d2c8-6588-41e7-92fc-42eb7ccd7502'                          | DOWN   |
~~~

The install should now continue without issues.

Comment 22 Pierre Prinetti 2020-06-08 16:14:55 UTC
*** Bug 1844792 has been marked as a duplicate of this bug. ***

Comment 23 Pierre Prinetti 2020-06-08 16:23:40 UTC
*** Bug 1836098 has been marked as a duplicate of this bug. ***

Comment 25 Martin André 2020-06-11 12:40:13 UTC
Setting status to MODIFIED as the related fix https://github.com/openshift/installer/pull/3734 was merged in master.

Comment 29 David Sanz 2020-06-11 15:42:34 UTC
Verified on 4.6.0-0.nightly-2020-06-11-133825

Comment 31 errata-xmlrpc 2020-10-27 15:54:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.