Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2244365

Summary: Stale ipamallocation entry remains when bulk port creation fails
Product: Red Hat OpenStack Reporter: yatanaka
Component: openstack-neutronAssignee: Rodolfo Alonso <ralonsoh>
Status: CLOSED ERRATA QA Contact: Eran Kuris <ekuris>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 17.1 (Wallaby)CC: chrisw, lmartins, mariel, ralonsoh, rsafrono, scohen
Target Milestone: z4Keywords: Triaged
Target Release: 17.1   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-18.6.1-17.1.20240822200817.85ff760.el9ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-11-21 09:38:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yatanaka 2023-10-16 07:47:11 UTC
Description of problem:

We can create multiple neutron ports using the following API:
- https://docs.openstack.org/api-ref/network/v2/index.html?expanded=bulk-create-ports-detail#bulk-create-ports

When one of the ports fails to be created, other ports won't be created as well.
In the following example, "10.0.0.100" is correct IP address while "10.0.1.100" is wrong IP address.
"10.0.1.100" is outside of the subnet and the port creation of this IP address should fail.

~~~
$ token=`openstack token issue -c id -f value`
$ curl -i https://192.168.24.2:13696/v2.0/ports -X POST -H "Content-Type: text/html; charset=UTF-8" -H "X-Auth-Token: $token" -d \
'{
    "ports": [
        {
            "name": "sample_port_1",
            "network_id": "078c5aef-0112-42bf-979e-812560a36846",
            "fixed_ips": [{"ip_address": "10.0.0.100"}] <==============(*) collect IP address
        },
        {
            "name": "sample_port_1",
            "network_id": "078c5aef-0112-42bf-979e-812560a36846",
            "fixed_ips": [{"ip_address": "10.0.1.100"}] <==============(*) wrong IP address. This port will fail to be created.
        }
    ]
}'
HTTP/1.1 400 Bad Request
content-type: application/json
content-length: 166
x-openstack-request-id: req-7693bc70-4745-40e2-91dd-6d4b22d3d548
date: Mon, 16 Oct 2023 05:41:02 GMT

{"NeutronError": {"type": "InvalidIpForNetwork", "message": "IP address 10.0.1.100 is not a valid IP for any of the subnets on the specified network.", "detail": ""}}

  ===> Because 10.0.1.100 is outside of the subnet, bulk port creation fails with 400 error.


(undercloud) [stack@undercloud-0 ~]$ openstack port list |grep -e 10.0.0.100 -e 10.0.1.100 |wc -l 
0

  ===> If one of the ports fails to be created, other ports is not created as well.
~~~


However, only ipamallocations entry is created in the neutron database.

~~~
(undercloud) [stack@undercloud-0 ~]$ sudo podman exec -u root mysql mysql -B ovs_neutron -u root -e "select * from ipamallocations where ip_address = '10.0.0.100'\G;"
*************************** 1. row ***************************
    ip_address: 10.0.0.100
        status: ALLOCATED
ipam_subnet_id: bfa51848-0c4a-4cc6-80a3-3cf1960f0773
~~~


Because the ipamallocations entry exists, subsequent port creation with the IP address will fail.

~~~
(undercloud) [stack@undercloud-0 ~]$ openstack port create --network 078c5aef-0112-42bf-979e-812560a36846 --fixed-ip ip-address=10.0.0.100 test_port
ConflictException: 409: Client Error for url: https://192.168.24.2:13696/v2.0/ports, IP address 10.0.0.100 already allocated in subnet 18844f4f-7d83-4467-88c2-1b7385ac34b5
~~~

Because the port associated with the IP address doesn't exist, we cannot delete the port.
I think we have no choice other than modifying the ipamallocations table manually.
As a workaround, would it be valid and safe to modify the ipamallocations manually?

Because `openstack overcloud node provision` command calls the same API to create a bulk of ports, we can hit the same issue on `openstack overcloud node provision`.


Version-Release number of selected component (if applicable):
RHOSP 17.1

How reproducible:
Steps to Reproduce:
1. Get Keystone token
2. Call the API to create bulk of ports.
   In this API call, one of the create ports should fail while other ports should succeed.
3. The API call fails.
4. No ports are created, but wrong ipamallocations entries are created in the neutron DB.


Actual results:
Wrong ipamallocations entries are created

Expected results:
Wrong ipamallocations entries are not created


Additional information:
The following Launchpad bug says "if one port fails in the driver call, the port is deleted. In this port deletion, the IPAM reservation is deleted too."
https://bugs.launchpad.net/neutron/+bug/1965807
However, actually the IPAM reservation is not deleted.
Is the change of the above Launchpad bug related to this issue?

Comment 1 Rodolfo Alonso 2023-10-17 09:38:23 UTC
Hello:

I've created an U/S bug: https://bugs.launchpad.net/neutron/+bug/2039550. In the description of this bug you can find a reproducer (based on the description of this BZ).

This issue is reproducible in any version, from 17.x (Wallaby) up to master.

Regards.

Comment 7 Roman Safronov 2024-05-19 14:14:14 UTC
Verified on RHOS-17.1-RHEL-9-20240516.n.1 with python3-neutron-18.6.1-17.1.20231025110810.el9ost.noarch

Verified according to the reproduction scenario specified in the description above.
After following the instructions wrong ipamallocations entries were not created and it was possible to create a port afterwards.

Comment 18 errata-xmlrpc 2024-11-21 09:38:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHOSP 17.1.4 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:9974