Bug 2065634

Summary:	Unable to delete Port because of not found IP address in Subnet
Product:	Red Hat OpenStack	Reporter:	Maysa Macedo <mdemaced>
Component:	openstack-neutron	Assignee:	Rodolfo Alonso <ralonsoh>
Status:	CLOSED ERRATA	QA Contact:	Eran Kuris <ekuris>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.2 (Train)	CC:	ccamposr, chrisw, jraju, ralonsoh, rcernin, scohen
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-neutron-15.3.5-2.20220406200213.d21e537.el8osttrunk	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	2145202 (view as bug list)		Environment:
Last Closed:	2022-06-22 16:06:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2145202

Description Maysa Macedo 2022-03-18 11:36:34 UTC

Description of problem:
While running a cluster density[1] test on a OpenShift cluster on top of OpenStack with Kuryr, 250 OpenShift Namespaces were created, which means Networks, Subnets were created and attached to the Router, tests were run and then Namespace deletion was attempted, which would delete all the OpenStack resources using that Subnet. Multiple resources were unable to be cleaned up because Kuryr attempted to delete the Ports on the Subnets and Neutron complained about IP address not existent in the Subnet.

(ostest) [stack@undercloud ~]$ openstack port delete 0d694b3e-61e3-4f67-afd6-1663b00429e4
Failed to delete port with name or ID '0d694b3e-61e3-4f67-afd6-1663b00429e4': HttpException: 500: Server Error for url: http://x.x.x.x:9696/v2.0/ports/0d694b3e-61e3-4f67-afd6-1663b00429e4, Unable to find IP address 10.134.220.100 on subnet f275f96c-335b-43df-af3d-db550fbe2e8f
1 of 1 ports failed to delete.

(ostest) [stack@undercloud ~]$ openstack port show  0d694b3e-61e3-4f67-afd6-1663b00429e4
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                             |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                |
| allowed_address_pairs   |                                                                                                                                                                   |
| binding_host_id         |                                                                                                                                                                   |
| binding_profile         |                                                                                                                                                                   |
| binding_vif_details     |                                                                                                                                                                   |
| binding_vif_type        | unbound                                                                                                                                                           |
| binding_vnic_type       | normal                                                                                                                                                            |
| created_at              | 2022-03-17T22:41:58Z                                                                                                                                              |
| data_plane_status       | None                                                                                                                                                              |
| description             |                                                                                                                                                                   |
| device_id               |                                                                                                                                                                   |
| device_owner            | compute:kuryr                                                                                                                                                     |
| dns_assignment          | fqdn='host-10-134-220-100.rdu2.scalelab.redhat.com.', hostname='host-10-134-220-100', ip_address='10.134.220.100'                                                 |
| dns_domain              |                                                                                                                                                                   |
| dns_name                |                                                                                                                                                                   |
| extra_dhcp_opts         |                                                                                                                                                                   |
| fixed_ips               | ip_address='10.134.220.100', subnet_id='f275f96c-335b-43df-af3d-db550fbe2e8f'                                                                                     |
| id                      | 0d694b3e-61e3-4f67-afd6-1663b00429e4                                                                                                                              |
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='c926ab6e08134fa4b76de3563c604f73', project.name='ostest', region_name='regionOne', zone= |
| mac_address             | fa:16:3e:9e:5c:d9                                                                                                                                                 |
| name                    |                                                                                                                                                                   |
| network_id              | a75efd0a-b0d0-420f-981d-da5e1c23bfa4                                                                                                                              |
| port_security_enabled   | True                                                                                                                                                              |
| project_id              | c926ab6e08134fa4b76de3563c604f73                                                                                                                                  |
| propagate_uplink_status | None                                                                                                                                                              |
| qos_policy_id           | None                                                                                                                                                              |
| resource_request        | None                                                                                                                                                              |
| revision_number         | 1                                                                                                                                                                 |
| security_group_ids      | 3b7baa4f-d9ad-4fcc-8321-ccf0889d7ee6                                                                                                                              |
| status                  | DOWN                                                                                                                                                              |
| tags                    |                                                                                                                                                                   |
| trunk_details           | None                                                                                                                                                              |
| updated_at              | 2022-03-17T22:41:59Z                                                                                                                                              |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(ostest) [stack@undercloud ~]$ openstack subnet show f275f96c-335b-43df-af3d-db550fbe2e8f
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                             |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_pools  | 10.134.220.2-10.134.221.254                                                                                                                                       |
| cidr              | 10.134.220.0/23                                                                                                                                                   |
| created_at        | 2022-03-17T13:33:07Z                                                                                                                                              |
| description       |                                                                                                                                                                   |
| dns_nameservers   |                                                                                                                                                                   |
| enable_dhcp       | False                                                                                                                                                             |
| gateway_ip        | 10.134.220.1                                                                                                                                                      |
| host_routes       |                                                                                                                                                                   |
| id                | f275f96c-335b-43df-af3d-db550fbe2e8f                                                                                                                              |
| ip_version        | 4                                                                                                                                                                 |
| ipv6_address_mode | None                                                                                                                                                              |
| ipv6_ra_mode      | None                                                                                                                                                              |
| location          | cloud='', project.domain_id=, project.domain_name='Default', project.id='c926ab6e08134fa4b76de3563c604f73', project.name='ostest', region_name='regionOne', zone= |
| name              | ns/fbf00256-f63f-4b79-bcc8-874d3528cb80-225-subnet                                                                                                                |
| network_id        | a75efd0a-b0d0-420f-981d-da5e1c23bfa4                                                                                                                              |
| prefix_length     | None                                                                                                                                                              |
| project_id        | c926ab6e08134fa4b76de3563c604f73                                                                                                                                  |
| revision_number   | 1                                                                                                                                                                 |
| segment_id        | None                                                                                                                                                              |
| service_types     |                                                                                                                                                                   |
| subnetpool_id     | adea6e07-cf1f-49c7-9b1c-7932ae441887                                                                                                                              |
| tags              | openshiftClusterID=ostest-kkmms                                                                                                                                   |
| updated_at        | 2022-03-17T13:33:15Z                                                                                                                                              |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+



[1] https://github.com/cloud-bulldozer

Version-Release number of selected component (if applicable):

[heat-admin@controller-0 ~]$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 16.2.2 (Train)
How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Rodolfo Alonso 2022-03-21 17:36:59 UTC

Hello:

This is a problem with the IPAM module, [1] and OVN. When a port is created in "create_port_bulk", we first create the IPAM allocations [1], to save time and DB accesses.

However, if one port fails in the driver call, the port is deleted. In this port deletion, the IPAM reservation is deleted too. At the end of "create_port_bulk", the previously created IPAM reservations are deleted too (again). This is what is triggering the error in the Neutron server [2]. The DB now has a port register, a IP allocation register but not an IPAM allocation register. That prevents from deleting manually this port (error 500).

I'll open a LP bug to solve this in U/S and I'll rebase the fix up to Train (that will be rebased into 16.2).

Regards.

[1]https://review.opendev.org/q/I8877c658446fed155130add6f1c69f2772113c27

Comment 4 Rodolfo Alonso 2022-03-21 17:40:05 UTC

I forgot to add link [2]: http://pastebin.test.redhat.com/1038612

Comment 16 errata-xmlrpc 2022-06-22 16:06:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4793

Comment 22 Red Hat Bugzilla 2023-09-18 04:33:48 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days