Bug 1810018

Summary:	[IPI] [OSP] openshift-install got failed for floatingip creation if rhcos base image uploading take more than 10m0s
Product:	OpenShift Container Platform	Reporter:	weiwei jiang <wjiang>
Component:	Installer	Assignee:	Mike Fedosin <mfedosin>
Installer sub component:	OpenShift on OpenStack	QA Contact:	David Sanz <dsanzmor>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	unspecified
Priority:	unspecified	CC:	adahiya, mfedosin
Version:	4.3.0
Target Milestone:	---
Target Release:	4.3.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-24 14:34:26 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1780534
Bug Blocks:

Description weiwei jiang 2020-03-04 12:17:00 UTC

This bug was initially created as a copy of Bug #1780534

I am copying this bug because: 



Description of problem:
openshift-install got failed if rhcos base image upload take more than 10m0s to upload 
level=debug msg="module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Still creating... [9m40s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m20s elapsed]"
level=debug msg="module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Still creating... [9m50s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m30s elapsed]"
level=debug msg="module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Still creating... [10m0s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m40s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m50s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [11m0s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Creation complete after 11m8s [id=06881570-3772-44b7-a288-025e85eda912]"
level=debug msg="data.openstack_images_image_v2.base_image: Refreshing state..."
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Creating..."
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Creating..."
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Creating..."
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Creating..."
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [10s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [10s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [10s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [10s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [30s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [30s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [30s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [30s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [50s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [50s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [50s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [50s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Creation complete after 1m1s [id=3d0135f5-b182-43e6-a33a-aed42c75ed95]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Creation complete after 1m3s [id=a315e92e-4bd1-469d-b584-29ff35dff579]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Creation complete after 1m3s [id=7edc5a1b-2970-4f75-9f35-613f95800b2d]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [1m10s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Creation complete after 1m13s [id=af06cfb2-37e4-4eb4-857e-93867f3cec74]"
level=error
level=error msg="Error: Error waiting for openstack_networking_floatingip_v2 926cc4a4-7adb-45d9-8f06-df644c8356f8 to become available: timeout while waiting for state to become 'ACTIVE, DOWN' (timeout: 10m0s)"
level=error
level=error msg="  on ../../../../../tmp/openshift-install-994566403/bootstrap/main.tf line 27, in resource \"openstack_networking_floatingip_v2\" \"bootstrap_fip\":"
level=error msg="  27: resource \"openstack_networking_floatingip_v2\" \"bootstrap_fip\" {"
level=error
level=error
level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform"


Version-Release number of the following components:
4.3.0-0.nightly-2019-12-06-014514

How reproducible:
It happen when rhcos base image take more than 10m0s to upload

Steps to Reproduce:
1. Create ipi on osp cluster and make rhcos base image uploading take more than 10m0s
2.
3.

Actual results:
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m20s elapsed]"
level=debug msg="module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Still creating... [9m50s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m30s elapsed]"
level=debug msg="module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Still creating... [10m0s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m40s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m50s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [11m0s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Creation complete after 11m8s [id=06881570-3772-44b7-a288-025e85eda912]"
level=debug msg="data.openstack_images_image_v2.base_image: Refreshing state..."
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Creating..."
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Creating..."
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Creating..."
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Creating..."
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [10s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [10s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [10s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [10s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [30s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [30s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [30s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [30s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [50s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [50s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [50s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [50s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Creation complete after 1m1s [id=3d0135f5-b182-43e6-a33a-aed42c75ed95]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Creation complete after 1m3s [id=a315e92e-4bd1-469d-b584-29ff35dff579]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Creation complete after 1m3s [id=7edc5a1b-2970-4f75-9f35-613f95800b2d]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [1m10s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Creation complete after 1m13s [id=af06cfb2-37e4-4eb4-857e-93867f3cec74]"
level=error
level=error msg="Error: Error waiting for openstack_networking_floatingip_v2 926cc4a4-7adb-45d9-8f06-df644c8356f8 to become available: timeout while waiting for state to become 'ACTIVE, DOWN' (timeout: 10m0s)"
level=error
level=error msg="  on ../../../../../tmp/openshift-install-994566403/bootstrap/main.tf line 27, in resource \"openstack_networking_floatingip_v2\" \"bootstrap_fip\":"
level=error msg="  27: resource \"openstack_networking_floatingip_v2\" \"bootstrap_fip\" {"
level=error
level=error
level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform"

Expected results:
It should not fail

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 4 weiwei jiang 2020-03-16 02:28:49 UTC

Checked with 4.3.0-0.nightly-2020-03-15-112942, now bootstrap fip is created after base_image done


level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [8m30s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [8m40s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [8m50s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [9m0s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [9m10s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [9m20s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [9m30s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [9m40s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [9m50s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m0s elapsed]"
level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m10s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m20s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m30s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m40s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [10m50s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [11m0s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [11m10s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [11m20s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Still creating... [11m30s elapsed]"

level=debug msg="openstack_images_image_v2.base_image[0]: Creation complete after 11m40s [id=89b605f9-9a47-48e9-8606-844a7bb636a1]"
level=debug msg="data.openstack_images_image_v2.base_image: Refreshing state..."

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Creating..."
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Creating..."
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Creating..."
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Creating..."

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [10s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [10s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [10s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [10s elapsed]"

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [20s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [20s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [20s elapsed]"

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [30s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [30s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [30s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [30s elapsed]"

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [40s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [40s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [40s elapsed]"

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [50s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [50s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [50s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [50s elapsed]"

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [1m0s elapsed]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Still creating... [1m0s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [1m0s elapsed]"

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[2]: Creation complete after 1m3s [id=52735ea5-9801-4f8e-950d-23f3d4d4f6c4]"
level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[1]: Creation complete after 1m3s [id=b8d49367-536f-4eee-8c12-891eef17554e]"

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Still creating... [1m10s elapsed]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Still creating... [1m10s elapsed]"

level=debug msg="module.masters.openstack_compute_instance_v2.master_conf[0]: Creation complete after 1m13s [id=0d9e63b6-182b-44ea-a416-e3ddc0f8d8bb]"
level=debug msg="module.bootstrap.openstack_compute_instance_v2.bootstrap: Creation complete after 1m13s [id=0ac209cf-9d5b-4b37-aed6-7fde016fc2e3]"
level=debug msg="module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Creating..."

level=debug msg="module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Still creating... [10s elapsed]"

level=debug msg="module.bootstrap.openstack_networking_floatingip_v2.bootstrap_fip: Creation complete after 11s [id=e6a7ab84-6349-4ee1-9ee3-00361b6bc6ec]"
level=debug
level=debug msg="Apply complete! Resources: 72 added, 0 changed, 0 destroyed."

Comment 6 errata-xmlrpc 2020-03-24 14:34:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0858