Bug 1734460
Summary: | [OSP] Floating IP specified with lbFloatingIP in install-config.yaml fails to attach to master node causing install to fail | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Jon Uriarte <juriarte> |
Component: | Installer | Assignee: | Martin André <m.andre> |
Installer sub component: | OpenShift on OpenStack | QA Contact: | David Sanz <dsanzmor> |
Status: | CLOSED ERRATA | Docs Contact: | |
Severity: | medium | ||
Priority: | high | CC: | adhingra, akaris, asimonel, dsanzmor, eduen, fbrychta, mabajodu, m.andre, mfedosin, pprinett, rstarr, tsedovic, vcojot |
Version: | 4.2.0 | Keywords: | Reopened, UpcomingSprint |
Target Milestone: | --- | ||
Target Release: | 4.6.0 | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: |
Cause: The Terraform step "openstack_networking_floatingip_associate_v2" did not list all its dependendent steps.
Consequence: The required step "openstack_networking_router_interface_v2.nodes_router_interface" was not always executed before the FIP association. This caused a race condition that occasionally failed the Terraform job, especially on overloaded systems.
Fix: List the dependant Terraform step as "depends_on" to force the router to be created before the FIP is attached to the port.
Result: The Terraform steps run in the correct order, ensuring that the FIP association does not result in a "Resource not found" error.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2020-10-27 15:54:19 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1846297 |
Description
Jon Uriarte
2019-07-30 15:11:26 UTC
Further to this we see it happening like this: We pre-allocate an FIP in openstack as usual: openstack floating ip create --floating-ip-address IP public We then add that FIP to lbFloatingIP in install-config.yaml We then install and this results in the error listed above. However, if we leave the lbFloatingIP line out of install-config.yaml we get past the error; however we need to manually assign the preallocated FIP to the API node for the install to continue. It seems like the presence of lbFloatingIP in install-config.yaml is triggering the error but we still need the FIP assigned to allow the install to work. Additionally, we also see another FIP (NOT the preallocated one) assigned to the bootstrap. The IP address being assigned to the bootstrap node is now expected. We've added that recently to let the installer gather logs on bootstrap failure. You can safely ignore that. The fact that you can't use a pre-allocated FIP as `lbFloatingIP` is a genuine issue though. I'm unable to reproduce this locally, but it's something we do want to address. Could you please paste your install-config.yaml (with the sensitive items such as pull secrets snipped out? Here's my install from the non-service-vm code which is doing the same. Failed with time="2019-07-31T12:23:26-04:00" level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Creation complete after 3m4s [id=69821aff-c950-489f-82bc-5d83d0da78a4]" time="2019-07-31T12:23:27-04:00" level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [3m0s elapsed]" time="2019-07-31T12:23:29-04:00" level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Creation complete after 3m2s [id=cab5d901-129c-4ef9-88ec-08413127b92d]" time="2019-07-31T12:23:29-04:00" level=error time="2019-07-31T12:23:29-04:00" level=error msg="Error: Error associating openstack_networking_floatingip_associate_v2 floating_ip 4a5af48e-d921-427a-9a72-a6b86a9f8873 with port f8e5f7b4-5525-4254-b936-fbb987f9cf9d: Resource not found" time="2019-07-31T12:23:29-04:00" level=error time="2019-07-31T12:23:29-04:00" level=error msg=" on ../../tmp/openshift-install-113850301/topology/private-network.tf line 114, in resource \"openstack_networking_floatingip_associate_v2\" \"api_fip\":" time="2019-07-31T12:23:29-04:00" level=error msg=" 114: resource \"openstack_networking_floatingip_associate_v2\" \"api_fip\" {" time="2019-07-31T12:23:29-04:00" level=error time="2019-07-31T12:23:29-04:00" level=error time="2019-07-31T12:23:29-04:00" level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform" but i do see this: [stack@undercloud ~]$ openstack --os-cloud openstack floating ip list +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | 4a5af48e-d921-427a-9a72-a6b86a9f8873 | 192.168.122.160 | None | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b | | e36d8556-09ce-4c2f-b8ac-6cde47968d4f | 192.168.122.175 | 10.0.128.16 | cb54a30f-1a0b-49ec-91a9-34d4a83aad5b | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ [stack@undercloud ~]$ openstack --os-cloud openstack server list +--------------------------------------+---------------------------+--------+-----------------------------------------------------+-------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+---------------------------+--------+-----------------------------------------------------+-------+----------+ | 478aca2c-1fcc-45f6-accf-9c9c78638432 | ostest-pw6ng-worker-gvcwk | ACTIVE | ostest-pw6ng-openshift=10.0.128.39 | rhcos | m1.large | | cab5d901-129c-4ef9-88ec-08413127b92d | ostest-pw6ng-master-0 | ACTIVE | ostest-pw6ng-openshift=10.0.128.26 | rhcos | m1.large | | 69821aff-c950-489f-82bc-5d83d0da78a4 | ostest-pw6ng-bootstrap | ACTIVE | ostest-pw6ng-openshift=10.0.128.16, 192.168.122.175 | rhcos | m1.large | +--------------------------------------+---------------------------+--------+-----------------------------------------------------+-------+----------+ and my install-config.yaml is this: apiVersion: v1 baseDomain: shiftstack.com clusterID: 8d582648-4ed7-45ca-be5b-c5b1c435e834 compute: - name: worker platform: {} replicas: 1 controlPlane: name: master platform: {} replicas: 1 metadata: creationTimestamp: null name: ostest Networking: clusterNetworks: - cidr: 10.128.0.0/14 hostSubnetLength: 9 machineCIDR: 10.0.128.0/17 serviceCIDR: 172.30.0.0/16 type: OpenShiftSDN platform: openstack: cloud: openstack computeFlavor: m1.large externalNetwork: external lbFloatingIP: 192.168.122.160 region: regionOne trunkSupport: "1" pullSecret: 'AAAAAAAAAAAAAA' sshKey: ssh-rsa AAAAAAAAA A little info about the OSP install I'm using: #!/bin/bash exec openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --timeout 90 \ --verbose \ -r /home/stack/templates/roles_data.yaml \ -e /home/stack/templates/node-count.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \ -e /home/stack/templates/environments/10-ntp.yaml \ -e /home/stack/templates/environments/11-network.yaml \ -e /home/stack/templates/environments/20-network-environment.yaml \ -e /home/stack/templates/environments/24-octavia-timeout.yaml \ -e /home/stack/templates/environments/25-hostname-map.yaml \ -e /home/stack/templates/environments/30-ips-from-pool-all.yaml \ -e /home/stack/templates/environments/35-ceph-config.yaml \ -e /home/stack/templates/environments/40-storage-config.yaml \ -e /home/stack/templates/environments/50-vip.yaml \ -e /home/stack/templates/environments/55-rsvd_host_memory.yaml \ --log-file /home/stack/overcloud-deploy.log I can give more info on each yaml etc if/as required. And this is really odd. I did another deployment testing a new value for lbFloatingIP of 192.168.122.156 (i was trying assigning the fip by an admin) I hit an error suggesting that the FIP could not be seen (which is what i expected as I set it by admin). But that's not the odd part ... keep reading ... So on this failed run I noticed the bootstrap got 192.168.122.175 So i deleted my cluster. However I noticed the 192.168.122.175 address was not removed from my allocation pool: (openstack) [stack@undercloud ~]$ openstack --os-cloud openstack floating ip list +--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project | +--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+ | 4a5af48e-d921-427a-9a72-a6b86a9f8873 | 192.168.122.160 | None | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b | | 5b64be33-ae69-4500-b462-183e9c86993b | 192.168.122.175 | None | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b | +--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+ So I decided to use 192.168.122.175 for my lbFloatingIP value on the next run and guess what? It worked. (openstack) [stack@undercloud ~]$ openstack --os-cloud openstack server list +--------------------------------------+------------------------+--------+-----------------------------------------------------+-------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------+--------+-----------------------------------------------------+-------+----------+ | f6c798ed-0567-4cfd-8636-9063e8f7f02e | ostest-mnxzv-master-0 | ACTIVE | ostest-mnxzv-openshift=10.0.128.15, 192.168.122.175 | rhcos | m1.large | | 9b974ae3-9bf8-441a-90bb-dc76a8009cca | ostest-mnxzv-bootstrap | ACTIVE | ostest-mnxzv-openshift=10.0.128.30, 192.168.122.178 | rhcos | m1.large | +--------------------------------------+------------------------+--------+-----------------------------------------------------+-------+----------+ It was correctly assigned to the master as I'd expected but as was not working when i allocated it manually before. I manually allocated using openstack floating ip create external On a recent run with: (openstack) [stack@undercloud ~]$ openstack floating ip list +--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project | +--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+ | 20f4d4af-733c-4c74-9b07-002fb00656fc | 192.168.122.151 | None | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b | | 58f9205c-814f-439b-a590-2bdbf9f3213e | 192.168.122.168 | None | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b | | 92ad0e4d-2a8b-4687-b0b2-51f1836f8202 | 192.168.122.152 | None | None | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 0ae48cc6f6a44ee0ac0f1099d8996e9b | +--------------------------------------+---------------------+------------------+------+--------------------------------------+----------------------------------+ (openstack) [stack@undercloud ~]$ openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | 10529ced-278b-4c95-ae88-5f84a4a0e236 | external | dacf2dce-c4a4-4b14-a3a4-ebe65c06654e | +--------------------------------------+----------+--------------------------------------+ (openstack) [stack@undercloud ~]$ openstack subnet list +--------------------------------------+-----------------+--------------------------------------+------------------+ | ID | Name | Network | Subnet | +--------------------------------------+-----------------+--------------------------------------+------------------+ | dacf2dce-c4a4-4b14-a3a4-ebe65c06654e | external_subnet | 10529ced-278b-4c95-ae88-5f84a4a0e236 | 192.168.122.0/24 | +--------------------------------------+-----------------+--------------------------------------+------------------+ I set lbFloatingIP: "192.168.122.151" and it failed with: DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Creation complete after 21s [id=55cc90bd-83f0-4018-acc6-1be6cfe5bf7f] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Creation complete after 23s [id=cd71ec82-53e7-4fd3-98fe-c62c17fc31dc] ERROR ERROR Error: Error associating openstack_networking_floatingip_associate_v2 floating_ip 20f4d4af-733c-4c74-9b07-002fb00656fc with port 27706d75-4122-4ed0-ac69-f9e707a8233b: Resource not found ERROR ERROR on ../../tmp/openshift-install-298254184/topology/private-network.tf line 114, in resource "openstack_networking_floatingip_associate_v2" "api_fip": ERROR 114: resource "openstack_networking_floatingip_associate_v2" "api_fip" { ERROR ERROR ERROR ERROR Error: Error creating openstack_networking_floatingip_v2: Resource not found ERROR ERROR on ../../tmp/openshift-install-298254184/topology/private-network.tf line 120, in resource "openstack_networking_floatingip_v2" "bootstrap_fip": ERROR 120: resource "openstack_networking_floatingip_v2" "bootstrap_fip" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply using Terraform install-config.yaml (generated by create install-config using the openstack option: apiVersion: v1 baseDomain: shiftstack.com compute: - hyperthreading: Enabled name: worker platform: {} replicas: 3 controlPlane: hyperthreading: Enabled name: master platform: {} replicas: 3 metadata: creationTimestamp: null name: ostest networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineCIDR: 10.0.0.0/16 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: openstack: apiVIP: 10.0.0.5 cloud: openstack computeFlavor: m1.large dnsVIP: 10.0.0.6 externalNetwork: external ingressVIP: 10.0.0.7 lbFloatingIP: "" octaviaSupport: "1" region: regionOne trunkSupport: "1" pullSecret: 'XXXXXXX' sshKey: | ssh-rsa XXXXXXX terraform.tfstate shows { "module": "module.topology", "mode": "managed", "type": "openstack_networking_floatingip_associate_v2", "name": "api_fip", "each": "list", "provider": "provider.openstack", "instances": [] }, { "module": "module.topology", "mode": "managed", "type": "openstack_networking_floatingip_v2", "name": "bootstrap_fip", "each": "list", "provider": "provider.openstack", "instances": [] }, I get similar error as above. Simply changing the format seems to fix this issue for me. This fails platform: openstack: cloud: openstack computeFlavor: ocp1.node externalNetwork: public lbFloatingIP: "192.168.247.57" octaviaSupport: "1" region: regionOne trunkSupport: "1" But this works platform: openstack: cloud: openstack externalNetwork: public region: regionOne computeFlavor: ocp1.node lbFloatingIP: "192.168.247.57" octaviaSupport: "1" ---- Second observation is that lbFloatingIP doesn't get assigned to the master but does work, it gets assigned to the 10.0.0.5 port in the new machineCIDR network. After deploy api.cluster.domain:6443 seems to work as expected, and if you look at the new network it has a port called clusternam-api-port (10.0.0.5) mapped to lbFloatingIP. But console and *.apps.domain do not work. Adding a floating ip to 10.0.0.7 which is listed as clustername-ingress-port and updating *.apps.domain dns to point the new floating ip resolved the console and apps issue. Oh, that is strange. Maybe there are some issues with the YAML parser. It would also be interesting to know whether the issues is the spacing between keys and values or the ordering (both have changed). As for the second observation, that is the current expected behaviour. The ports are managed internally by Keepalived and configuring the wildcard *.apps access is a day two operation (due to IPI preferring minimal configuration -- we have to set the API floating IP up front but the ingress one can be done afterwards). The upstream docs have more details on both the networking configuration: https://github.com/openshift/installer/blob/master/docs/design/openstack/networking-infrastructure.md and how to set up external access to the cluster: https://github.com/openshift/installer/tree/master/docs/user/openstack#using-floating-ips. Moving to 4.3. If I understand it correctly, this only happens when you write the install-config.yaml by hand (which should not be required for the majority of the use cases) and even then there is a way to work around this. We will need more time to reproduce this and come up with some sanitisation and error reporting, but that will not likely happen in 4.3. API FIP is assigned to a port correctly and APPS FIP is not managed by the installer. This seems to be linked to an under spec'ed OpenStack cloud. Closing it as WORKSFORME. Please re-open if this happens again. Happened for me, too, on an OCP 4.3 / OSP 13 combination. I could install OCP on this very system before, but after playing around with it and installing / deleting several times, it started acting up. I'm 0 interested in troubleshooting this, but adding a me too might help prioritize this here a bit. I did not find any way to resume installation (deleted all resources, created new install config, etc. but it would not work) --- overcloud) [stack@undercloud-0 openshift]$ ./openshift-install create cluster --dir=install-config/ --log-level=debug DEBUG OpenShift Installer v4.3.1 DEBUG Built from commit 2055609f95b19322ee6cfdd0bea73399297c4a3e DEBUG Fetching Terraform Variables... DEBUG Loading Terraform Variables... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Using SSH Key loaded from state file DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Using Platform loaded from state file DEBUG Using Base Domain loaded from state file DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Using Cluster Name loaded from state file DEBUG Loading Pull Secret... DEBUG Using Pull Secret loaded from state file DEBUG Loading Platform... DEBUG Loading Install Config from both state file and target directory DEBUG Using Install Config loaded from target directory DEBUG Loading Install Config... DEBUG Loading Image... DEBUG Loading Install Config... DEBUG Loading BootstrapImage... DEBUG Loading Install Config... DEBUG Loading Bootstrap Ignition Config... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Admin Internal Client... DEBUG Loading Certificate (admin-kubeconfig-client)... DEBUG Loading Certificate (admin-kubeconfig-signer)... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Kubelet... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kubelet-client)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Loading Install Config... DEBUG Loading Kubeconfig Admin Client (Loopback)... DEBUG Loading Certificate (admin-kubeconfig-client)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Install Config... DEBUG Loading Master Machines... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Install Config... DEBUG Loading Install Config... DEBUG Loading Image... DEBUG Loading Master Ignition Config... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Worker Machines... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Install Config... DEBUG Loading Image... DEBUG Loading Worker Ignition Config... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Common Manifests... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading Ingress Config... DEBUG Loading Install Config... DEBUG Loading DNS Config... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Infrastructure Config... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading Cloud Provider Config... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Platform Credentials Check... DEBUG Loading Additional Trust Bundle Config... DEBUG Loading Install Config... DEBUG Loading Network Config... DEBUG Loading Install Config... DEBUG Loading Network CRDs... DEBUG Loading Proxy Config... DEBUG Loading Install Config... DEBUG Loading Network Config... DEBUG Loading Scheduler Config... DEBUG Loading Install Config... DEBUG Loading Image Content Source Policy... DEBUG Loading Install Config... DEBUG Loading Root CA... DEBUG Loading Certificate (etcd-signer)... DEBUG Loading Certificate (etcd-ca-bundle)... DEBUG Loading Certificate (etcd-signer)... DEBUG Loading Certificate (etcd-client)... DEBUG Loading Certificate (etcd-signer)... DEBUG Loading Certificate (etcd-metric-ca-bundle)... DEBUG Loading Certificate (etcd-metric-signer)... DEBUG Loading Certificate (etcd-metric-signer)... DEBUG Loading Certificate (etcd-metric-signer-client)... DEBUG Loading Certificate (etcd-metric-signer)... DEBUG Loading Certificate (mcs)... DEBUG Loading Root CA... DEBUG Loading Install Config... DEBUG Loading CVOOverrides... DEBUG Loading EtcdCAConfigMap... DEBUG Loading EtcdClientSecret... DEBUG Loading EtcdHostServiceEndpoints... DEBUG Loading EtcdHostService... DEBUG Loading EtcdMetricClientSecret... DEBUG Loading EtcdMetricServingCAConfigMap... DEBUG Loading EtcdMetricSignerSecret... DEBUG Loading EtcdNamespace... DEBUG Loading EtcdService... DEBUG Loading EtcdSignerSecret... DEBUG Loading KubeCloudConfig... DEBUG Loading EtcdServingCAConfigMap... DEBUG Loading KubeSystemConfigmapRootCA... DEBUG Loading MachineConfigServerTLSSecret... DEBUG Loading OpenshiftConfigSecretPullSecret... DEBUG Loading OpenshiftMachineConfigOperator... DEBUG Loading Openshift Manifests... DEBUG Loading Install Config... DEBUG Loading Cluster ID... DEBUG Loading Kubeadmin Password... DEBUG Loading CloudCredsSecret... DEBUG Loading KubeadminPasswordSecret... DEBUG Loading RoleCloudCredsSecretReader... DEBUG Loading Private Cluster Outbound Service... DEBUG Loading Proxy Config... DEBUG Loading Certificate (admin-kubeconfig-ca-bundle)... DEBUG Loading Certificate (admin-kubeconfig-signer)... DEBUG Loading Certificate (aggregator)... DEBUG Loading Certificate (aggregator-ca-bundle)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (system:kube-apiserver-proxy)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (aggregator-signer)... DEBUG Loading Certificate (system:kube-apiserver-proxy)... DEBUG Loading Certificate (aggregator)... DEBUG Loading Certificate (etcd-ca-bundle)... DEBUG Loading Certificate (etcd-metric-ca-bundle)... DEBUG Loading Certificate (etcd-metric-signer)... DEBUG Loading Certificate (etcd-metric-signer-client)... DEBUG Loading Certificate (etcd-signer)... DEBUG Loading Certificate (etcd-client)... DEBUG Loading Certificate (journal-gatewayd)... DEBUG Loading Root CA... DEBUG Loading Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-external-lb-server)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-internal-lb-server)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-localhost-server)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-service-network-server)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Install Config... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-complete-client-ca-bundle)... DEBUG Loading Certificate (admin-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-client-ca-bundle)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (kube-control-plane-ca-bundle)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-apiserver-lb-signer)... DEBUG Loading Certificate (kube-apiserver-localhost-signer)... DEBUG Loading Certificate (kube-apiserver-service-network-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-client)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Loading Certificate (kube-control-plane-ca-bundle)... DEBUG Loading Certificate (kube-control-plane-kube-controller-manager-client)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-control-plane-kube-scheduler-client)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kube-control-plane-signer)... DEBUG Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Loading Certificate (kubelet-client-ca-bundle)... DEBUG Loading Certificate (kubelet-client)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (kubelet-serving-ca-bundle)... DEBUG Loading Certificate (kubelet-signer)... DEBUG Loading Certificate (mcs)... DEBUG Loading Root CA... DEBUG Loading Key Pair (service-account.pub)... DEBUG Loading Release Image Pull Spec... DEBUG Loading Image... DEBUG Loading Master Ignition Config... DEBUG Loading Master Machines... DEBUG Loading Worker Machines... DEBUG Fetching Cluster ID... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Cluster ID... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Image... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Image... DEBUG Fetching BootstrapImage... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating BootstrapImage... DEBUG Fetching Bootstrap Ignition Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Kubeconfig Admin Internal Client... DEBUG Fetching Certificate (admin-kubeconfig-client)... DEBUG Fetching Certificate (admin-kubeconfig-signer)... DEBUG Generating Certificate (admin-kubeconfig-signer)... DEBUG Generating Certificate (admin-kubeconfig-client)... DEBUG Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-localhost-signer)... DEBUG Generating Certificate (kube-apiserver-localhost-signer)... DEBUG Generating Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-service-network-signer)... DEBUG Generating Certificate (kube-apiserver-service-network-signer)... DEBUG Generating Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-lb-signer)... DEBUG Generating Certificate (kube-apiserver-lb-signer)... DEBUG Generating Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Generating Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Kubeconfig Admin Internal Client... DEBUG Fetching Kubeconfig Kubelet... DEBUG Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) DEBUG Fetching Certificate (kubelet-client)... DEBUG Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Generating Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Generating Certificate (kubelet-client)... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Kubeconfig Kubelet... DEBUG Fetching Kubeconfig Admin Client (Loopback)... DEBUG Fetching Certificate (admin-kubeconfig-client)... DEBUG Reusing previously-fetched Certificate (admin-kubeconfig-client) DEBUG Fetching Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle) DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Kubeconfig Admin Client (Loopback)... DEBUG Fetching Master Machines... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Platform Credentials Check... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Platform Credentials Check... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Image... DEBUG Reusing previously-fetched Image DEBUG Fetching Master Ignition Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Root CA... DEBUG Generating Root CA... DEBUG Generating Master Ignition Config... DEBUG Generating Master Machines... DEBUG Fetching Worker Machines... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Platform Credentials Check... DEBUG Reusing previously-fetched Platform Credentials Check DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Image... DEBUG Reusing previously-fetched Image DEBUG Fetching Worker Ignition Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Root CA... DEBUG Reusing previously-fetched Root CA DEBUG Generating Worker Ignition Config... DEBUG Generating Worker Machines... DEBUG Fetching Common Manifests... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Ingress Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Ingress Config... DEBUG Fetching DNS Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Platform Credentials Check... DEBUG Reusing previously-fetched Platform Credentials Check DEBUG Generating DNS Config... DEBUG Fetching Infrastructure Config... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cloud Provider Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Platform Credentials Check... DEBUG Reusing previously-fetched Platform Credentials Check DEBUG Generating Cloud Provider Config... DEBUG Fetching Additional Trust Bundle Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Additional Trust Bundle Config... DEBUG Generating Infrastructure Config... DEBUG Fetching Network Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Network CRDs... DEBUG Generating Network CRDs... DEBUG Generating Network Config... DEBUG Fetching Proxy Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Network Config... DEBUG Reusing previously-fetched Network Config DEBUG Generating Proxy Config... DEBUG Fetching Scheduler Config... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Scheduler Config... DEBUG Fetching Image Content Source Policy... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Image Content Source Policy... DEBUG Fetching Root CA... DEBUG Reusing previously-fetched Root CA DEBUG Fetching Certificate (etcd-signer)... DEBUG Generating Certificate (etcd-signer)... DEBUG Fetching Certificate (etcd-ca-bundle)... DEBUG Fetching Certificate (etcd-signer)... DEBUG Reusing previously-fetched Certificate (etcd-signer) DEBUG Generating Certificate (etcd-ca-bundle)... DEBUG Fetching Certificate (etcd-client)... DEBUG Fetching Certificate (etcd-signer)... DEBUG Reusing previously-fetched Certificate (etcd-signer) DEBUG Generating Certificate (etcd-client)... DEBUG Fetching Certificate (etcd-metric-ca-bundle)... DEBUG Fetching Certificate (etcd-metric-signer)... DEBUG Generating Certificate (etcd-metric-signer)... DEBUG Generating Certificate (etcd-metric-ca-bundle)... DEBUG Fetching Certificate (etcd-metric-signer)... DEBUG Reusing previously-fetched Certificate (etcd-metric-signer) DEBUG Fetching Certificate (etcd-metric-signer-client)... DEBUG Fetching Certificate (etcd-metric-signer)... DEBUG Reusing previously-fetched Certificate (etcd-metric-signer) DEBUG Generating Certificate (etcd-metric-signer-client)... DEBUG Fetching Certificate (mcs)... DEBUG Fetching Root CA... DEBUG Reusing previously-fetched Root CA DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Certificate (mcs)... DEBUG Fetching CVOOverrides... DEBUG Generating CVOOverrides... DEBUG Fetching EtcdCAConfigMap... DEBUG Generating EtcdCAConfigMap... DEBUG Fetching EtcdClientSecret... DEBUG Generating EtcdClientSecret... DEBUG Fetching EtcdHostServiceEndpoints... DEBUG Generating EtcdHostServiceEndpoints... DEBUG Fetching EtcdHostService... DEBUG Generating EtcdHostService... DEBUG Fetching EtcdMetricClientSecret... DEBUG Generating EtcdMetricClientSecret... DEBUG Fetching EtcdMetricServingCAConfigMap... DEBUG Generating EtcdMetricServingCAConfigMap... DEBUG Fetching EtcdMetricSignerSecret... DEBUG Generating EtcdMetricSignerSecret... DEBUG Fetching EtcdNamespace... DEBUG Generating EtcdNamespace... DEBUG Fetching EtcdService... DEBUG Generating EtcdService... DEBUG Fetching EtcdSignerSecret... DEBUG Generating EtcdSignerSecret... DEBUG Fetching KubeCloudConfig... DEBUG Generating KubeCloudConfig... DEBUG Fetching EtcdServingCAConfigMap... DEBUG Generating EtcdServingCAConfigMap... DEBUG Fetching KubeSystemConfigmapRootCA... DEBUG Generating KubeSystemConfigmapRootCA... DEBUG Fetching MachineConfigServerTLSSecret... DEBUG Generating MachineConfigServerTLSSecret... DEBUG Fetching OpenshiftConfigSecretPullSecret... DEBUG Generating OpenshiftConfigSecretPullSecret... DEBUG Fetching OpenshiftMachineConfigOperator... DEBUG Generating OpenshiftMachineConfigOperator... DEBUG Generating Common Manifests... DEBUG Fetching Openshift Manifests... DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Kubeadmin Password... DEBUG Generating Kubeadmin Password... DEBUG Fetching CloudCredsSecret... DEBUG Generating CloudCredsSecret... DEBUG Fetching KubeadminPasswordSecret... DEBUG Generating KubeadminPasswordSecret... DEBUG Fetching RoleCloudCredsSecretReader... DEBUG Generating RoleCloudCredsSecretReader... DEBUG Fetching Private Cluster Outbound Service... DEBUG Generating Private Cluster Outbound Service... DEBUG Generating Openshift Manifests... DEBUG Fetching Proxy Config... DEBUG Reusing previously-fetched Proxy Config DEBUG Fetching Certificate (admin-kubeconfig-ca-bundle)... DEBUG Fetching Certificate (admin-kubeconfig-signer)... DEBUG Reusing previously-fetched Certificate (admin-kubeconfig-signer) DEBUG Generating Certificate (admin-kubeconfig-ca-bundle)... DEBUG Fetching Certificate (aggregator)... DEBUG Generating Certificate (aggregator)... DEBUG Fetching Certificate (aggregator-ca-bundle)... DEBUG Fetching Certificate (aggregator-signer)... DEBUG Generating Certificate (aggregator-signer)... DEBUG Generating Certificate (aggregator-ca-bundle)... DEBUG Fetching Certificate (system:kube-apiserver-proxy)... DEBUG Fetching Certificate (aggregator-signer)... DEBUG Reusing previously-fetched Certificate (aggregator-signer) DEBUG Generating Certificate (system:kube-apiserver-proxy)... DEBUG Fetching Certificate (aggregator-signer)... DEBUG Reusing previously-fetched Certificate (aggregator-signer) DEBUG Fetching Certificate (system:kube-apiserver-proxy)... DEBUG Fetching Certificate (aggregator)... DEBUG Reusing previously-fetched Certificate (aggregator) DEBUG Generating Certificate (system:kube-apiserver-proxy)... DEBUG Fetching Certificate (etcd-ca-bundle)... DEBUG Reusing previously-fetched Certificate (etcd-ca-bundle) DEBUG Fetching Certificate (etcd-metric-ca-bundle)... DEBUG Reusing previously-fetched Certificate (etcd-metric-ca-bundle) DEBUG Fetching Certificate (etcd-metric-signer)... DEBUG Reusing previously-fetched Certificate (etcd-metric-signer) DEBUG Fetching Certificate (etcd-metric-signer-client)... DEBUG Reusing previously-fetched Certificate (etcd-metric-signer-client) DEBUG Fetching Certificate (etcd-signer)... DEBUG Reusing previously-fetched Certificate (etcd-signer) DEBUG Fetching Certificate (etcd-client)... DEBUG Reusing previously-fetched Certificate (etcd-client) DEBUG Fetching Certificate (journal-gatewayd)... DEBUG Fetching Root CA... DEBUG Reusing previously-fetched Root CA DEBUG Generating Certificate (journal-gatewayd)... DEBUG Fetching Certificate (kube-apiserver-lb-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-lb-ca-bundle) DEBUG Fetching Certificate (kube-apiserver-external-lb-server)... DEBUG Fetching Certificate (kube-apiserver-lb-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-lb-signer) DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Certificate (kube-apiserver-external-lb-server)... DEBUG Fetching Certificate (kube-apiserver-internal-lb-server)... DEBUG Fetching Certificate (kube-apiserver-lb-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-lb-signer) DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Certificate (kube-apiserver-internal-lb-server)... DEBUG Fetching Certificate (kube-apiserver-lb-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-lb-signer) DEBUG Fetching Certificate (kube-apiserver-localhost-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle) DEBUG Fetching Certificate (kube-apiserver-localhost-server)... DEBUG Fetching Certificate (kube-apiserver-localhost-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) DEBUG Generating Certificate (kube-apiserver-localhost-server)... DEBUG Fetching Certificate (kube-apiserver-localhost-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) DEBUG Fetching Certificate (kube-apiserver-service-network-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-service-network-ca-bundle) DEBUG Fetching Certificate (kube-apiserver-service-network-server)... DEBUG Fetching Certificate (kube-apiserver-service-network-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Certificate (kube-apiserver-service-network-server)... DEBUG Fetching Certificate (kube-apiserver-service-network-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) DEBUG Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) DEBUG Fetching Certificate (kube-apiserver-complete-client-ca-bundle)... DEBUG Fetching Certificate (admin-kubeconfig-ca-bundle)... DEBUG Reusing previously-fetched Certificate (admin-kubeconfig-ca-bundle) DEBUG Fetching Certificate (kubelet-client-ca-bundle)... DEBUG Fetching Certificate (kubelet-signer)... DEBUG Generating Certificate (kubelet-signer)... DEBUG Generating Certificate (kubelet-client-ca-bundle)... DEBUG Fetching Certificate (kube-control-plane-ca-bundle)... DEBUG Fetching Certificate (kube-control-plane-signer)... DEBUG Generating Certificate (kube-control-plane-signer)... DEBUG Fetching Certificate (kube-apiserver-lb-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-lb-signer) DEBUG Fetching Certificate (kube-apiserver-localhost-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-localhost-signer) DEBUG Fetching Certificate (kube-apiserver-service-network-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-service-network-signer) DEBUG Generating Certificate (kube-control-plane-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Generating Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Generating Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)... DEBUG Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-signer) DEBUG Generating Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Generating Certificate (kube-apiserver-complete-client-ca-bundle)... DEBUG Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-ca-bundle) DEBUG Fetching Certificate (kube-apiserver-to-kubelet-client)... DEBUG Fetching Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer) DEBUG Generating Certificate (kube-apiserver-to-kubelet-client)... DEBUG Fetching Certificate (kube-apiserver-to-kubelet-signer)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer) DEBUG Fetching Certificate (kube-control-plane-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-control-plane-ca-bundle) DEBUG Fetching Certificate (kube-control-plane-kube-controller-manager-client)... DEBUG Fetching Certificate (kube-control-plane-signer)... DEBUG Reusing previously-fetched Certificate (kube-control-plane-signer) DEBUG Generating Certificate (kube-control-plane-kube-controller-manager-client)... DEBUG Fetching Certificate (kube-control-plane-kube-scheduler-client)... DEBUG Fetching Certificate (kube-control-plane-signer)... DEBUG Reusing previously-fetched Certificate (kube-control-plane-signer) DEBUG Generating Certificate (kube-control-plane-kube-scheduler-client)... DEBUG Fetching Certificate (kube-control-plane-signer)... DEBUG Reusing previously-fetched Certificate (kube-control-plane-signer) DEBUG Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-ca-bundle) DEBUG Fetching Certificate (kubelet-client-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kubelet-client-ca-bundle) DEBUG Fetching Certificate (kubelet-client)... DEBUG Reusing previously-fetched Certificate (kubelet-client) DEBUG Fetching Certificate (kubelet-signer)... DEBUG Reusing previously-fetched Certificate (kubelet-signer) DEBUG Fetching Certificate (kubelet-serving-ca-bundle)... DEBUG Fetching Certificate (kubelet-signer)... DEBUG Reusing previously-fetched Certificate (kubelet-signer) DEBUG Generating Certificate (kubelet-serving-ca-bundle)... DEBUG Fetching Certificate (mcs)... DEBUG Reusing previously-fetched Certificate (mcs) DEBUG Fetching Root CA... DEBUG Reusing previously-fetched Root CA DEBUG Fetching Key Pair (service-account.pub)... DEBUG Generating Key Pair (service-account.pub)... DEBUG Fetching Release Image Pull Spec... DEBUG Generating Release Image Pull Spec... DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:ea7ac3ad42169b39fce07e5e53403a028644810bee9a212e7456074894df40f3 DEBUG Fetching Image... DEBUG Reusing previously-fetched Image DEBUG Generating Bootstrap Ignition Config... DEBUG Fetching Master Ignition Config... DEBUG Reusing previously-fetched Master Ignition Config DEBUG Fetching Master Machines... DEBUG Reusing previously-fetched Master Machines DEBUG Fetching Worker Machines... DEBUG Reusing previously-fetched Worker Machines DEBUG Generating Terraform Variables... INFO Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.3/43.81.202001142154.0/x86_64/rhcos-43.81.202001142154.0-openstack.x86_64.qcow2.gz?sha256=504b9008adf89bb3d05b75d393e057c6d66ba6c92cf631ca4445d99bbf7e2a57' INFO The file was found in cache: /home/stack/.cache/openshift-installer/image_cache/d03e06d2824bac47eaee911dcc5feff9. Reusing... DEBUG Creating a Swift container for your bootstrap ignition... DEBUG Container cluster-m4bbk was created. DEBUG Creating a Swift object in container cluster-m4bbk containing your bootstrap ignition... DEBUG The object was created. INFO Consuming Install Config from target directory DEBUG Purging asset "Install Config" from disk DEBUG Fetching Kubeconfig Admin Client... DEBUG Loading Kubeconfig Admin Client... DEBUG Loading Certificate (admin-kubeconfig-client)... DEBUG Loading Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Loading Install Config... DEBUG Fetching Certificate (admin-kubeconfig-client)... DEBUG Reusing previously-fetched Certificate (admin-kubeconfig-client) DEBUG Fetching Certificate (kube-apiserver-complete-server-ca-bundle)... DEBUG Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle) DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Kubeconfig Admin Client... DEBUG Fetching Kubeadmin Password... DEBUG Reusing previously-fetched Kubeadmin Password DEBUG Fetching Certificate (journal-gatewayd)... DEBUG Reusing previously-fetched Certificate (journal-gatewayd) DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Generating Metadata... DEBUG Fetching Cluster... DEBUG Loading Cluster... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading Platform Credentials Check... DEBUG Loading Terraform Variables... DEBUG Loading Kubeadmin Password... DEBUG Fetching Cluster ID... DEBUG Reusing previously-fetched Cluster ID DEBUG Fetching Install Config... DEBUG Reusing previously-fetched Install Config DEBUG Fetching Platform Credentials Check... DEBUG Reusing previously-fetched Platform Credentials Check DEBUG Fetching Terraform Variables... DEBUG Reusing previously-fetched Terraform Variables DEBUG Fetching Kubeadmin Password... DEBUG Reusing previously-fetched Kubeadmin Password DEBUG Generating Cluster... INFO Creating infrastructure resources... DEBUG Symlinking plugin terraform-provider-azureprivatedns src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-azureprivatedns" DEBUG Symlinking plugin terraform-provider-google src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-google" DEBUG Symlinking plugin terraform-provider-ignition src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-ignition" DEBUG Symlinking plugin terraform-provider-local src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-local" DEBUG Symlinking plugin terraform-provider-openstack src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-openstack" DEBUG Symlinking plugin terraform-provider-random src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-random" DEBUG Symlinking plugin terraform-provider-aws src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-aws" DEBUG Symlinking plugin terraform-provider-azurerm src: "/home/stack/openshift/openshift-install" dst: "/tmp/openshift-install-233199779/plugins/terraform-provider-azurerm" DEBUG Initializing modules... DEBUG - bootstrap in ../../../tmp/openshift-install-233199779/bootstrap DEBUG - masters in ../../../tmp/openshift-install-233199779/masters DEBUG - topology in ../../../tmp/openshift-install-233199779/topology DEBUG DEBUG Initializing the backend... DEBUG DEBUG Initializing provider plugins... DEBUG DEBUG Terraform has been successfully initialized! DEBUG DEBUG You may now begin working with Terraform. Try running "terraform plan" to see DEBUG any changes that are required for your infrastructure. All Terraform commands DEBUG should now work. DEBUG DEBUG If you ever set or change modules or backend configuration for Terraform, DEBUG rerun this command to reinitialize your working directory. If you forget, other DEBUG commands will detect it and remind you to do so if necessary. DEBUG module.masters.data.ignition_file.hostname[2]: Refreshing state... DEBUG module.masters.data.ignition_file.hostname[1]: Refreshing state... DEBUG module.masters.data.ignition_file.hostname[0]: Refreshing state... DEBUG module.masters.data.ignition_config.master_ignition_config[1]: Refreshing state... DEBUG module.masters.data.ignition_config.master_ignition_config[2]: Refreshing state... DEBUG module.masters.data.ignition_config.master_ignition_config[0]: Refreshing state... DEBUG module.topology.data.openstack_networking_network_v2.external_network: Refreshing state... DEBUG module.masters.data.openstack_compute_flavor_v2.masters_flavor: Refreshing state... DEBUG module.bootstrap.data.openstack_compute_flavor_v2.bootstrap_flavor: Refreshing state... DEBUG module.topology.openstack_networking_secgroup_v2.master: Creating... DEBUG module.topology.openstack_networking_secgroup_v2.worker: Creating... DEBUG module.topology.openstack_networking_router_v2.openshift-external-router: Creating... DEBUG module.topology.openstack_networking_network_v2.openshift-private: Creating... DEBUG openstack_images_image_v2.base_image[0]: Creating... DEBUG module.topology.openstack_networking_secgroup_v2.worker: Creation complete after 2s [id=ef2f35cc-663c-49e3-890e-c285b36e531a] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_https: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_ssh: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_icmp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_services_tcp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vrrp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_icmp: Creation complete after 0s [id=7720bcf7-3108-4c70-aaa5-96fa3dd40e0e] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal: Creating... DEBUG module.topology.openstack_networking_secgroup_v2.master: Creation complete after 2s [id=e264ce98-5970-4aa8-ab25-7e860fb8eafe] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_http: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_udp: Creation complete after 0s [id=60e7cba0-4dfb-4c14-bcfc-97323e61b4ba] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_geneve: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_https: Creation complete after 1s [id=11c09d77-acf9-40c1-a35c-b9577e6a4b87] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_services_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal: Creation complete after 1s [id=7c36a8a3-a9f0-4cfc-8604-1e6fbfeeafda] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_kubelet_insecure: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_ssh: Creation complete after 1s [id=f2973b7c-c011-4b49-bbd4-ca571185ec60] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_router: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vrrp: Creation complete after 1s [id=c7fe5e19-df49-4425-acf9-462499b6a6ea] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_mdns_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_http: Creation complete after 2s [id=9dbb1d0c-ee5b-4898-b578-33377dc21410] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vxlan: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_services_tcp: Creation complete after 2s [id=149f61f4-dbfd-459f-a676-4f35b3183610] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_controller_manager: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_router: Creation complete after 1s [id=5f04e0f4-8805-4a9c-ab83-6cba620240d8] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_mdns_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vxlan: Creation complete after 0s [id=820241b9-2d87-4cd9-93e5-2f059f6f42d3] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_kubelet_insecure_from_master: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_controller_manager: Creation complete after 0s [id=d35066b9-9ff9-42d0-845e-310fecd30029] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kubelet_secure: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_kubelet_insecure_from_master: Creation complete after 0s [id=7d51b197-79ff-4d11-a47b-f2f755f12084] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_mdns_udp: Creation complete after 0s [id=524acfd1-7847-494c-868c-716d39989c9d] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vxlan_from_worker: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_scheduler_from_worker: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kubelet_secure: Creation complete after 0s [id=b93a5deb-21c4-488e-a6e1-323ace6187b8] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_controller_manager_from_worker: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_scheduler_from_worker: Creation complete after 0s [id=31ad01e7-ddb1-4bfd-ada9-083b7ba6d819] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_etcd: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_kubelet_insecure: Creation complete after 2s [id=c3f3a54a-81ed-449f-b3e3-ad8552d2bac2] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_from_worker: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_geneve: Creation complete after 3s [id=ee762237-b1ff-4a6c-a67b-1de9ac3eb9c5] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vrrp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vxlan_from_worker: Creation complete after 1s [id=2aee6302-374a-4852-83c8-6f48e288db77] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_mcs: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_from_worker: Creation complete after 0s [id=72d74bf2-6083-4637-a6c1-532d4c07a94b] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_scheduler: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_etcd: Creation complete after 1s [id=45bb8d2e-96cf-4779-90f9-49fd77abe369] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_geneve: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vrrp: Creation complete after 0s [id=6691a704-85ad-441c-b00c-5853fc2c2bb6] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_https: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_services_udp: Creation complete after 2s [id=7ce9ab9d-49fc-41fd-b378-906bb9ea2ee6] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_geneve: Creation complete after 0s [id=2fce4947-ff24-4b81-94a3-c37673d285ce] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_mdns_udp: Creation complete after 2s [id=903fc1b1-da2a-42ed-b3ec-63705b3eec59] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ssh: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_services_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_udp: Creation complete after 0s [id=9ce12aac-f838-4fac-ad6e-546658afa3a6] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_dns_tcp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_https: Creation complete after 1s [id=424e04ae-9f45-4536-8297-76b33388ac10] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_geneve_from_master: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_dns_tcp: Creation complete after 1s [id=934ee797-aa33-45ec-b9a9-c36c25f57225] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vxlan: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_geneve_from_master: Creation complete after 0s [id=aab3b700-3df7-4226-9879-fdf39bf61a5e] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_dns_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_mcs: Creation complete after 1s [id=44be6211-2eb9-4ae9-a192-6d99a415ffb9] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_from_worker_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ssh: Creation complete after 1s [id=471312fc-d3a9-4c18-af0c-588b35852ab6] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vxlan_from_master: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_dns_udp: Creation complete after 0s [id=f3ae81d5-ef04-4bfa-b6f4-30fe2c1cce05] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ovndb: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_vxlan_from_master: Creation complete after 0s [id=03379b77-844f-4755-8296-0dea0802b5b7] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_icmp: Creating... DEBUG module.topology.openstack_networking_network_v2.openshift-private: Creation complete after 6s [id=3f0bf23e-4182-4b66-9bb7-1cf7b3b4cfa1] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_from_master_udp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal_from_worker_udp: Creation complete after 1s [id=46e32b82-1b07-4014-bd1f-5bd3e8206c19] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kubelet_secure_from_worker: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_icmp: Creation complete after 1s [id=591ba1ef-82a7-4552-850d-986ef48d2d78] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ovndb_from_worker: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_from_master_udp: Creation complete after 1s [id=458e88ab-f189-4b24-9bb7-0e397f30e376] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_services_tcp: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ovndb: Creation complete after 1s [id=a5fc88c3-2065-4d43-9f8a-af1bb114ab92] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_geneve_from_worker: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_vxlan: Creation complete after 1s [id=0e591b5f-b4c7-45af-bda5-d597a22a8482] DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_from_master: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_geneve_from_worker: Creation complete after 0s [id=b0517ae4-3dd3-4cd3-aace-613b79d61b6a] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.worker_ingress_internal_from_master: Creation complete after 1s [id=ac7d35a6-e522-4f59-a254-2c9885230453] DEBUG module.topology.openstack_networking_subnet_v2.nodes: Creating... DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_ovndb_from_worker: Creation complete after 1s [id=c9c71786-975f-48df-82d5-fe397bed8bc1] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_services_udp: Creation complete after 3s [id=0de9f7f1-a2c6-4c98-89e4-40e0a1b7df2b] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_internal: Creation complete after 1s [id=4fac13fa-d129-483b-9a1e-aef3077c003a] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_services_tcp: Creation complete after 1s [id=7f6d7e01-4cbd-4302-99bf-00eb861820ee] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_controller_manager_from_worker: Creation complete after 4s [id=2e138fa4-3ad9-4ba6-94ad-75306fb6f64c] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kube_scheduler: Creation complete after 4s [id=7ac8626d-e9e1-4390-baef-612289faa01f] DEBUG module.topology.openstack_networking_secgroup_rule_v2.master_ingress_kubelet_secure_from_worker: Creation complete after 2s [id=a8f9dd82-1641-44a0-827e-2c9b2463a8f0] DEBUG module.topology.openstack_networking_router_v2.openshift-external-router: Still creating... [10s elapsed] DEBUG openstack_images_image_v2.base_image[0]: Still creating... [10s elapsed] DEBUG module.topology.openstack_networking_subnet_v2.nodes: Creation complete after 6s [id=5bdde3eb-f9b2-4fd8-bc07-32375fad5cd6] DEBUG module.topology.openstack_networking_port_v2.ingress_port: Creating... DEBUG module.topology.openstack_networking_port_v2.api_port: Creating... DEBUG module.bootstrap.openstack_networking_port_v2.bootstrap_port: Creating... DEBUG module.topology.openstack_networking_port_v2.masters[2]: Creating... DEBUG module.topology.openstack_networking_port_v2.masters[0]: Creating... DEBUG module.topology.openstack_networking_port_v2.masters[1]: Creating... DEBUG openstack_images_image_v2.base_image[0]: Still creating... [20s elapsed] DEBUG module.topology.openstack_networking_router_v2.openshift-external-router: Still creating... [20s elapsed] DEBUG module.topology.openstack_networking_router_v2.openshift-external-router: Creation complete after 22s [id=18a9c5da-22ff-46c1-b898-b103921ffda2] DEBUG module.topology.openstack_networking_router_interface_v2.nodes_router_interface: Creating... DEBUG module.topology.openstack_networking_port_v2.api_port: Creation complete after 9s [id=9cbcab5a-751a-4d83-a8ea-887c4408c79b] DEBUG module.topology.openstack_networking_floatingip_associate_v2.api_fip[0]: Creating... DEBUG module.topology.openstack_networking_port_v2.masters[2]: Creation complete after 9s [id=ed0eb15b-99a6-4ac6-9d52-928821a7919f] DEBUG module.bootstrap.openstack_networking_port_v2.bootstrap_port: Creation complete after 10s [id=02902224-5a91-4bd4-952a-bfda4268d0e4] DEBUG module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Creating... DEBUG module.topology.openstack_networking_port_v2.masters[1]: Creation complete after 10s [id=5fd4aeff-2b7b-4796-8752-a720a9e215d2] DEBUG module.topology.openstack_networking_port_v2.ingress_port: Still creating... [10s elapsed] DEBUG module.topology.openstack_networking_port_v2.masters[0]: Still creating... [10s elapsed] DEBUG module.topology.openstack_networking_port_v2.ingress_port: Creation complete after 11s [id=51229511-5e1b-4a87-beb3-04494884a2c1] DEBUG module.topology.openstack_networking_port_v2.masters[0]: Creation complete after 11s [id=ded7d54d-7019-465c-894f-c3ad9074880d] DEBUG module.topology.openstack_networking_trunk_v2.masters[0]: Creating... DEBUG module.topology.openstack_networking_trunk_v2.masters[1]: Creating... DEBUG module.topology.openstack_networking_trunk_v2.masters[2]: Creating... DEBUG openstack_images_image_v2.base_image[0]: Still creating... [30s elapsed] DEBUG module.topology.openstack_networking_trunk_v2.masters[1]: Creation complete after 7s [id=fbb4fb13-9ca6-4c34-b6b9-c3baa8890777] DEBUG module.topology.openstack_networking_router_interface_v2.nodes_router_interface: Still creating... [10s elapsed] DEBUG module.topology.openstack_networking_trunk_v2.masters[2]: Creation complete after 7s [id=e16bf30a-05c8-4c64-9462-327d1269eb63] DEBUG module.topology.openstack_networking_trunk_v2.masters[0]: Creation complete after 7s [id=a8e23e70-bec8-4f13-a6df-832652544859] DEBUG module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Still creating... [10s elapsed] DEBUG module.topology.openstack_networking_router_interface_v2.nodes_router_interface: Creation complete after 14s [id=4d6e25df-45b6-479f-a935-3e1e6d91551b] DEBUG module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Creation complete after 14s [id=df770468-140f-476a-9439-ab34ceb2c545] DEBUG openstack_images_image_v2.base_image[0]: Still creating... [40s elapsed] DEBUG openstack_images_image_v2.base_image[0]: Creation complete after 45s [id=86a2b3f5-4964-4d28-bb75-169774b2d07c] DEBUG data.openstack_images_image_v2.base_image: Refreshing state... DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Creating... DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Creating... DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Creating... DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Creating... DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [10s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [10s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [10s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [10s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [20s elapsed] DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [20s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [20s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [20s elapsed] DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [30s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [30s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [30s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [30s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [40s elapsed] DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [40s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [40s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [40s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [50s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [50s elapsed] DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [50s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [50s elapsed] DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [1m0s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [1m0s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [1m0s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [1m0s elapsed] DEBUG module.bootstrap.openstack_compute_instance_v2.bootstrap: Creation complete after 1m6s [id=95608032-ea7d-4f22-aea3-e07d981ff832] DEBUG module.masters.openstack_compute_instance_v2.master_conf[1]: Creation complete after 1m7s [id=5dc8d9e3-adad-478e-8cb7-d84a70354164] DEBUG module.masters.openstack_compute_instance_v2.master_conf[0]: Creation complete after 1m7s [id=89f75363-b807-41bf-b6af-6d9f360284f8] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [1m10s elapsed] DEBUG module.masters.openstack_compute_instance_v2.master_conf[2]: Creation complete after 1m11s [id=38c524ee-2750-4e32-bf68-5b7e05f1d2b9] ERROR ERROR Error: Error associating openstack_networking_floatingip_associate_v2 floating_ip 74d54930-0753-4d82-b7ea-74a06668ebfa with port 9cbcab5a-751a-4d83-a8ea-887c4408c79b: Resource not found ERROR ERROR on ../../../tmp/openshift-install-233199779/topology/private-network.tf line 122, in resource "openstack_networking_floatingip_associate_v2" "api_fip": ERROR 122: resource "openstack_networking_floatingip_associate_v2" "api_fip" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply using Terraform --------------------- (overcloud) [stack@undercloud-0 openshift]$ openstack port list | grep 9cbcab5a-751a-4d83-a8ea-887c4408c79b | 9cbcab5a-751a-4d83-a8ea-887c4408c79b | cluster-m4bbk-api-port | fa:16:3e:c3:8c:6e | ip_address='172.31.0.5', subnet_id='5bdde3eb-f9b2-4fd8-bc07-32375fad5cd6' | DOWN | (overcloud) [stack@undercloud-0 openshift]$ openstack floating ip list +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | 43ca9393-acbd-4e16-935a-9cce6750cf03 | 172.16.0.205 | None | None | aa483134-ef6c-4d62-bf51-ad98377669b0 | d69c1c6601c741deaa205fa1a7e9c632 | | 74d54930-0753-4d82-b7ea-74a06668ebfa | 172.16.0.204 | None | None | aa483134-ef6c-4d62-bf51-ad98377669b0 | d69c1c6601c741deaa205fa1a7e9c632 | | df770468-140f-476a-9439-ab34ceb2c545 | 172.16.0.213 | 172.31.0.37 | 02902224-5a91-4bd4-952a-bfda4268d0e4 | aa483134-ef6c-4d62-bf51-ad98377669b0 | d69c1c6601c741deaa205fa1a7e9c632 | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ (overcloud) [stack@undercloud-0 openshift]$ I ran into this again with OCP 4.3.13 (trying to reproduce a customer issue). Just to explain the aforementioned workaround: Change install-config.yaml from: ~~~ (...) platform: openstack: cloud: openstack computeFlavor: m1.openshift externalDNS: null externalNetwork: provider1 lbFloatingIP: 172.16.0.206 octaviaSupport: "1" region: "" trunkSupport: "1" publish: External ~~~ To: ~~~ (...) platform: openstack: cloud: openstack computeFlavor: m1.openshift externalDNS: null externalNetwork: provider1 #lbFloatingIP: 172.16.0.206 octaviaSupport: "1" region: "" trunkSupport: "1" publish: External ~~~ Then, run the installation: ~~~ openshift-install create cluster --dir install-config/ --log-level debug ~~~ As soon as terraform finishes and the installer tries to contact the API ... ~~~ DEBUG Still waiting for the Kubernetes API: Get https://api.cluster43.example.com:6443/version?timeout=32s: dial tcp 172.16.0.206:6443: connect: no route to host ~~~ ... run: ~~~ clusterID=$(openstack server list | egrep -o 'cluster43-[0-9a-z]+-bootstrap' | awk -F '-' '{print $2}') openstack floating ip set --port cluster43-${clusterID}-api-port 172.16.0.206 ~~~ Verify: ~~~ (overcloud) [stack@undercloud-0 openshift43]$ openstack floating ip list | grep 172.16.0.206 | f33afe42-1b8f-45c1-9fd0-b36020e10b0c | 172.16.0.206 | 172.31.0.5 | aa3fa224-5a3c-4552-993b-09189cf16e5b | b3979e59-b9cf-4b95-8933-46aedd536192 | ba4092513d9f4969b224a094e9fdf1dd | (overcloud) [stack@undercloud-0 openshift43]$ openstack port list | grep cluster43-${clusterID}-api-port | aa3fa224-5a3c-4552-993b-09189cf16e5b | cluster43-zz94k-api-port | fa:16:3e:fd:4c:06 | ip_address='172.31.0.5', subnet_id='3c59d2c8-6588-41e7-92fc-42eb7ccd7502' | DOWN | ~~~ The install should now continue without issues. *** Bug 1844792 has been marked as a duplicate of this bug. *** *** Bug 1836098 has been marked as a duplicate of this bug. *** Setting status to MODIFIED as the related fix https://github.com/openshift/installer/pull/3734 was merged in master. Verified on 4.6.0-0.nightly-2020-06-11-133825 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |