Bug 1879891 - [FloatingIPLess] Openshift-install hang at looping to remove custom router
Summary: [FloatingIPLess] Openshift-install hang at looping to remove custom router
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Maysa Macedo
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-17 09:58 UTC by weiwei jiang
Modified: 2020-10-27 16:42 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:41:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4191 0 None closed Bug 1879891: Fix cluster destroy when fip less installation happened 2021-02-12 20:34:47 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:42:12 UTC

Description weiwei jiang 2020-09-17 09:58:59 UTC
Description of problem: 
When trying to destroy a floatingIPLess ocp cluster, it keeps hanging at remove customer router as following: 
09-15 16:25:09  level=debug msg="OpenShift Installer 4.6.0-0.nightly-2020-09-15-005959" 
09-15 16:25:09  level=debug msg="Built from commit 8fea4de6e7c305a80493593637b22435540f5991" 
09-15 16:25:09  level=debug msg="Removing interfaces from custom router" 
 
09-15 16:25:12  level=debug msg="No router found." 
09-15 16:25:12  level=debug msg="Exiting removal of interfaces from custom router" 
 
09-15 16:25:27  level=debug msg="Removing interfaces from custom router" 
 
09-15 16:25:29  level=debug msg="No router found." 
09-15 16:25:29  level=debug msg="Exiting removal of interfaces from custom router" 
 
09-15 16:25:51  level=debug msg="Removing interfaces from custom router" 
09-15 16:25:51  level=debug msg="No router found." 
09-15 16:25:51  level=debug msg="Exiting removal of interfaces from custom router" 
 
 
 
Version-Release number of the following components: 
4.6.0-0.nightly-2020-09-15-005959 
 
How reproducible: 
Always 
 
Steps to Reproduce: 
1. Create a floatingIPless OCP cluster with BYO network 
2. try to destroy the cluster 
3. 
 
Actual results: 
2. It hang at looping to remove customer router 
 
Expected results: 
2. It should destroy the OCP cluster well. 
 
Additional info: 
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Maysa Macedo 2020-09-17 12:18:03 UTC
Was the Network of the machines Subnet tagged with any other tag different than (infraID + "-primaryClusterNetwork")? Was the machine Subnet connected to a router?

It would be good to have access to the ansible-playbook logs.

Comment 4 weiwei jiang 2020-09-21 04:28:45 UTC
Checked with 4.6.0-0.nightly-2020-09-20-184226, moved to verified.

09-21 12:17:22  level=debug msg="OpenShift Installer 4.6.0-0.nightly-2020-09-20-184226"
09-21 12:17:22  level=debug msg="Built from commit 50fb7fc30969f3b347ec2cab99dfde0dfe8f69bf"
09-21 12:17:22  level=debug msg="Removing interfaces from custom router"
09-21 12:17:22  level=debug msg="Deleting openstack routers"
09-21 12:17:22  level=debug msg="Deleting openstack servers"
09-21 12:17:22  level=debug msg="Deleting openstack subnets"
09-21 12:17:22  level=debug msg="Deleting openstack subnet-pools"
09-21 12:17:22  level=debug msg="Deleting OpenStack volumes"
09-21 12:17:22  level=debug msg="Deleting openstack floating ips"
09-21 12:17:22  level=debug msg="Deleting openstack base image"
09-21 12:17:22  level=debug msg="Deleting openstack networks"
09-21 12:17:22  level=debug msg="Deleting openstack ports"
09-21 12:17:22  level=debug msg="Deleting openstack security-groups"
09-21 12:17:22  level=debug msg="Deleting openstack containers"
09-21 12:17:22  level=debug msg="Deleting openstack server groups"
09-21 12:17:22  level=debug msg="Deleting openstack load balancers"
09-21 12:17:22  level=debug msg="Deleting openstack trunks"
09-21 12:17:22  level=debug msg="Exiting deleting openstack subnet-pools"
09-21 12:17:22  level=debug msg="Exiting deleting openstack floating ips"
09-21 12:17:22  level=debug msg="Exiting deleting openstack networks"
09-21 12:17:22  level=debug msg="Exiting deleting openstack subnets"
09-21 12:17:22  level=debug msg="Deleting image: c531cb00-1ba4-437a-8c45-d1f0b09b5d85"
09-21 12:17:22  level=debug msg="Exiting deleting openstack routers"
09-21 12:17:22  level=debug msg="Deleting Trunk \"04610ab4-5903-4b40-a53f-5f7e40305c2f\""
09-21 12:17:22  level=debug msg="Deleting Security Group: \"e3d3b488-1f34-4da3-a9a0-588c33551bcc\""
09-21 12:17:23  level=debug msg="Deleting Trunk \"04610ab4-5903-4b40-a53f-5f7e40305c2f\" failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/04610ab4-5903-4b40-a53f-5f7e40305c2f], but got 409 instead\n{\"NeutronError\": {\"message\": \"Trunk 04610ab4-5903-4b40-a53f-5f7e40305c2f is currently in use.\", \"type\": \"TrunkInUse\", \"detail\": \"\"}}"
09-21 12:17:23  level=debug msg="Exiting deleting openstack trunks"
09-21 12:17:23  level=debug msg="Bulk deleting container \"wj46ios921z-b6nhz-image-registry-viqliphoglawnuymlpkvtakdlmpae\" objects"
09-21 12:17:23  level=debug msg="Deleting container \"wj46ios921z-b6nhz-image-registry-viqliphoglawnuymlpkvtakdlmpae\""
09-21 12:17:23  level=debug msg="Exiting deleting openstack containers"
09-21 12:17:23  level=debug msg="Deleting Security Group \"e3d3b488-1f34-4da3-a9a0-588c33551bcc\" failed with error: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/e3d3b488-1f34-4da3-a9a0-588c33551bcc], but got 409 instead\n{\"NeutronError\": {\"message\": \"Security Group e3d3b488-1f34-4da3-a9a0-588c33551bcc in use.\", \"type\": \"SecurityGroupInUse\", \"detail\": \"\"}}"
09-21 12:17:23  level=debug msg="Exiting deleting openstack security-groups"
09-21 12:17:23  level=debug msg="Deleting Server Group \"4da349ef-bac6-4619-8989-6ce326b8400f\""

09-21 12:17:23  level=debug msg="Exiting deleting openstack server groups"
09-21 12:17:23  level=debug msg="Deleting Port \"294e45ec-25c1-4a30-af43-95383f11615b\""
09-21 12:17:23  level=debug msg="Exiting deleting openstack load balancers"
09-21 12:17:23  level=debug msg="Deleting Port \"294e45ec-25c1-4a30-af43-95383f11615b\" failed with error: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/294e45ec-25c1-4a30-af43-95383f11615b], but got 409 instead\n{\"NeutronError\": {\"message\": \"Port 294e45ec-25c1-4a30-af43-95383f11615b is currently a parent port for trunk fafe5446-3f07-443e-847f-8c818ab5642d.\", \"type\": \"PortInUseAsTrunkParent\", \"detail\": \"\"}}"
09-21 12:17:23  level=debug msg="Exiting deleting openstack ports"
09-21 12:17:24  level=debug msg="Exiting deleting openstack base image"

09-21 12:17:25  level=debug msg="Exiting removal of interfaces from custom router"
09-21 12:17:25  level=debug msg="goroutine deleteSubnetPools complete"
09-21 12:17:25  level=debug msg="goroutine deleteFloatingIPs complete"
09-21 12:17:25  level=debug msg="goroutine deleteNetworks complete"
09-21 12:17:25  level=debug msg="goroutine deleteSubnets complete"
09-21 12:17:25  level=debug msg="goroutine deleteRouters complete"
09-21 12:17:25  level=debug msg="goroutine deleteContainers complete"
09-21 12:17:25  level=debug msg="goroutine deleteLoadBalancers complete"
09-21 12:17:25  level=debug msg="goroutine deleteImages complete"

09-21 12:17:27  level=debug msg="Exiting deleting OpenStack volumes"
09-21 12:17:27  level=debug msg="goroutine deleteVolumes complete"
09-21 12:17:27  level=debug msg="Deleting Server \"814f2d1c-81c8-4587-bc4a-a164e28e9f4f\""

09-21 12:17:28  level=debug msg="Deleting Server \"59365477-33b7-4353-b8bb-79b8464371e5\""
09-21 12:17:28  level=debug msg="Deleting Server \"933d52ec-5a84-47a2-aa36-ac6123a40de5\""
09-21 12:17:28  level=debug msg="Deleting Server \"1113bbea-471c-4cf0-8f07-a18439aa05c8\""
09-21 12:17:28  level=debug msg="Deleting Server \"3ac8251e-e5b9-4a6e-9350-97b2c62163db\""
09-21 12:17:29  level=debug msg="Deleting Server \"9c6d33f6-69a1-4555-aa91-6d54531e2fb1\""
09-21 12:17:29  level=debug msg="Exiting deleting openstack servers"

09-21 12:17:39  level=debug msg="Deleting openstack trunks"
09-21 12:17:39  level=debug msg="Deleting openstack security-groups"
09-21 12:17:39  level=debug msg="Deleting Trunk \"04610ab4-5903-4b40-a53f-5f7e40305c2f\""
09-21 12:17:39  level=debug msg="Deleting Security Group: \"e3d3b488-1f34-4da3-a9a0-588c33551bcc\""
09-21 12:17:39  level=debug msg="Deleting openstack server groups"
09-21 12:17:39  level=debug msg="Deleting Trunk \"2b18cee5-3c7f-495f-bf63-688b2dbd3048\""
09-21 12:17:39  level=debug msg="Deleting Security Group \"e3d3b488-1f34-4da3-a9a0-588c33551bcc\" failed with error: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/e3d3b488-1f34-4da3-a9a0-588c33551bcc], but got 409 instead\n{\"NeutronError\": {\"message\": \"Security Group e3d3b488-1f34-4da3-a9a0-588c33551bcc in use.\", \"type\": \"SecurityGroupInUse\", \"detail\": \"\"}}"
09-21 12:17:39  level=debug msg="Exiting deleting openstack security-groups"
09-21 12:17:39  level=debug msg="Deleting openstack ports"
09-21 12:17:39  level=debug msg="Deleting Trunk \"4d761a8d-7197-42c1-a115-10cf3fe1241c\""
09-21 12:17:39  level=debug msg="Exiting deleting openstack server groups"
09-21 12:17:39  level=debug msg="goroutine deleteServerGroups complete"
09-21 12:17:39  level=debug msg="Deleting Trunk \"ae5bd018-83a7-473e-b9d5-961d06fd58bf\""
09-21 12:17:39  level=debug msg="Deleting Port \"294e45ec-25c1-4a30-af43-95383f11615b\""
09-21 12:17:39  level=debug msg="Deleting Trunk \"b3f53231-05c6-4fcb-b2d3-6cd52c66f316\""
09-21 12:17:39  level=debug msg="Deleting Port \"294e45ec-25c1-4a30-af43-95383f11615b\" failed with error: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/294e45ec-25c1-4a30-af43-95383f11615b], but got 409 instead\n{\"NeutronError\": {\"message\": \"Port 294e45ec-25c1-4a30-af43-95383f11615b is currently a parent port for trunk fafe5446-3f07-443e-847f-8c818ab5642d.\", \"type\": \"PortInUseAsTrunkParent\", \"detail\": \"\"}}"
09-21 12:17:39  level=debug msg="Exiting deleting openstack ports"
09-21 12:17:40  level=debug msg="Deleting Trunk \"fafe5446-3f07-443e-847f-8c818ab5642d\""
09-21 12:17:40  level=debug msg="Exiting deleting openstack trunks"

09-21 12:17:45  level=debug msg="Deleting openstack servers"

09-21 12:17:48  level=debug msg="Exiting deleting openstack servers"
09-21 12:17:48  level=debug msg="goroutine deleteServers complete"

09-21 12:17:58  level=debug msg="Deleting openstack security-groups"
09-21 12:17:58  level=debug msg="Deleting Security Group: \"e3d3b488-1f34-4da3-a9a0-588c33551bcc\""
09-21 12:17:58  level=debug msg="Deleting Security Group \"e3d3b488-1f34-4da3-a9a0-588c33551bcc\" failed with error: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/e3d3b488-1f34-4da3-a9a0-588c33551bcc], but got 409 instead\n{\"NeutronError\": {\"message\": \"Security Group e3d3b488-1f34-4da3-a9a0-588c33551bcc in use.\", \"type\": \"SecurityGroupInUse\", \"detail\": \"\"}}"
09-21 12:17:58  level=debug msg="Exiting deleting openstack security-groups"
09-21 12:17:59  level=debug msg="Deleting openstack ports"
09-21 12:17:59  level=debug msg="Deleting Port \"294e45ec-25c1-4a30-af43-95383f11615b\""
09-21 12:18:00  level=debug msg="Deleting openstack trunks"
09-21 12:18:00  level=debug msg="Exiting deleting openstack trunks"
09-21 12:18:00  level=debug msg="goroutine deleteTrunks complete"

09-21 12:18:03  level=debug msg="Deleting Port \"29e86789-b3e9-42bd-8f50-a0d1d229ff30\""
09-21 12:18:04  level=debug msg="Deleting Port \"5247ea20-c9b1-4077-9a0b-779229985293\""

09-21 12:18:06  level=debug msg="Deleting Port \"8408eebe-2b2f-4084-99af-5f47967c6b3c\""
09-21 12:18:07  level=debug msg="Deleting Port \"b0a1f118-64e0-41f1-83ef-6d036f60baa8\""

09-21 12:18:09  level=debug msg="Deleting Port \"bf1a7a67-bad9-4a41-bdd2-71c46f65fa6b\""

09-21 12:18:11  level=debug msg="Deleting Port \"c01a409c-c7d6-4992-bbbf-9e56009c8408\""

09-21 12:18:13  level=debug msg="Deleting Port \"c295f789-e064-47a4-ac95-c10ee0925997\""
09-21 12:18:14  level=debug msg="Exiting deleting openstack ports"

09-21 12:18:24  level=debug msg="Deleting openstack security-groups"
09-21 12:18:24  level=debug msg="Deleting Security Group: \"e3d3b488-1f34-4da3-a9a0-588c33551bcc\""
09-21 12:18:24  level=debug msg="Deleting Security Group: \"f3839429-d01e-4527-879b-470e7bd341c5\""
09-21 12:18:25  level=debug msg="Exiting deleting openstack security-groups"

09-21 12:18:40  level=debug msg="Deleting openstack ports"
09-21 12:18:40  level=debug msg="Exiting deleting openstack ports"
09-21 12:18:40  level=debug msg="goroutine deletePorts complete"

09-21 12:18:58  level=debug msg="Deleting openstack security-groups"
09-21 12:18:58  level=debug msg="Exiting deleting openstack security-groups"
09-21 12:18:58  level=debug msg="goroutine deleteSecurityGroups complete"
09-21 12:18:58  level=debug msg="Removing tag wj46ios921z-b6nhz-primaryClusterNetwork from openstack networks"

09-21 12:18:59  level=debug msg="Exiting untagging openstack networks"
09-21 12:18:59  level=debug msg="Purging asset \"Metadata\" from disk"
09-21 12:18:59  level=debug msg="Purging asset \"Terraform Variables\" from disk"
09-21 12:18:59  level=debug msg="Purging asset \"Kubeconfig Admin Client\" from disk"
09-21 12:18:59  level=debug msg="Purging asset \"Kubeadmin Password\" from disk"
09-21 12:18:59  level=debug msg="Purging asset \"Certificate (journal-gatewayd)\" from disk"
09-21 12:18:59  level=debug msg="Purging asset \"Cluster\" from disk"
09-21 12:18:59  level=info msg="Time elapsed: 1m38s"

Comment 7 errata-xmlrpc 2020-10-27 16:41:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.