Description of problem: While running a cluster density[1] test on a OpenShift cluster on top of OpenStack with Kuryr, 250 OpenShift Namespaces were created, which means Networks, Subnets were created and attached to the Router, tests were run and then Namespace deletion was attempted, which would delete all the OpenStack resources using that Subnet. Multiple resources were unable to be cleaned up because Kuryr attempted to delete the Ports on the Subnets and Neutron complained about IP address not existent in the Subnet. (ostest) [stack@undercloud ~]$ openstack port delete 0d694b3e-61e3-4f67-afd6-1663b00429e4 Failed to delete port with name or ID '0d694b3e-61e3-4f67-afd6-1663b00429e4': HttpException: 500: Server Error for url: http://x.x.x.x:9696/v2.0/ports/0d694b3e-61e3-4f67-afd6-1663b00429e4, Unable to find IP address 10.134.220.100 on subnet f275f96c-335b-43df-af3d-db550fbe2e8f 1 of 1 ports failed to delete. (ostest) [stack@undercloud ~]$ openstack port show 0d694b3e-61e3-4f67-afd6-1663b00429e4 +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2022-03-17T22:41:58Z | | data_plane_status | None | | description | | | device_id | | | device_owner | compute:kuryr | | dns_assignment | fqdn='host-10-134-220-100.rdu2.scalelab.redhat.com.', hostname='host-10-134-220-100', ip_address='10.134.220.100' | | dns_domain | | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='10.134.220.100', subnet_id='f275f96c-335b-43df-af3d-db550fbe2e8f' | | id | 0d694b3e-61e3-4f67-afd6-1663b00429e4 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='c926ab6e08134fa4b76de3563c604f73', project.name='ostest', region_name='regionOne', zone= | | mac_address | fa:16:3e:9e:5c:d9 | | name | | | network_id | a75efd0a-b0d0-420f-981d-da5e1c23bfa4 | | port_security_enabled | True | | project_id | c926ab6e08134fa4b76de3563c604f73 | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 1 | | security_group_ids | 3b7baa4f-d9ad-4fcc-8321-ccf0889d7ee6 | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2022-03-17T22:41:59Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (ostest) [stack@undercloud ~]$ openstack subnet show f275f96c-335b-43df-af3d-db550fbe2e8f +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | allocation_pools | 10.134.220.2-10.134.221.254 | | cidr | 10.134.220.0/23 | | created_at | 2022-03-17T13:33:07Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 10.134.220.1 | | host_routes | | | id | f275f96c-335b-43df-af3d-db550fbe2e8f | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='c926ab6e08134fa4b76de3563c604f73', project.name='ostest', region_name='regionOne', zone= | | name | ns/fbf00256-f63f-4b79-bcc8-874d3528cb80-225-subnet | | network_id | a75efd0a-b0d0-420f-981d-da5e1c23bfa4 | | prefix_length | None | | project_id | c926ab6e08134fa4b76de3563c604f73 | | revision_number | 1 | | segment_id | None | | service_types | | | subnetpool_id | adea6e07-cf1f-49c7-9b1c-7932ae441887 | | tags | openshiftClusterID=ostest-kkmms | | updated_at | 2022-03-17T13:33:15Z | +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [1] https://github.com/cloud-bulldozer Version-Release number of selected component (if applicable): [heat-admin@controller-0 ~]$ cat /etc/rhosp-release Red Hat OpenStack Platform release 16.2.2 (Train) How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Hello: This is a problem with the IPAM module, [1] and OVN. When a port is created in "create_port_bulk", we first create the IPAM allocations [1], to save time and DB accesses. However, if one port fails in the driver call, the port is deleted. In this port deletion, the IPAM reservation is deleted too. At the end of "create_port_bulk", the previously created IPAM reservations are deleted too (again). This is what is triggering the error in the Neutron server [2]. The DB now has a port register, a IP allocation register but not an IPAM allocation register. That prevents from deleting manually this port (error 500). I'll open a LP bug to solve this in U/S and I'll rebase the fix up to Train (that will be rebased into 16.2). Regards. [1]https://review.opendev.org/q/I8877c658446fed155130add6f1c69f2772113c27
I forgot to add link [2]: http://pastebin.test.redhat.com/1038612
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:4793
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days