Bug 1815590

We see that something has been changed from Mitaka/Newton release in Queens, how ports are assigned.
We have a HEAT template that is compatible with Mitaka and later. That is why it does not have the loop on fixed IP assigning to instances.

I will attach this template here also, so everyone could test on other releases, does it work there.
Versions I currently know it is working:
Mitaka (9.0) - with max_fixed_ips = 200 changed in /etc/neuton/neuton.conf
Mitaka (Mirantis Legacy 9.2) - with max_fixed_ips = 200 changed in /etc/neuton/neuton.conf
Newton (RHOSP10) - with max_fixed_ips = 200 changed in /etc/neuton/neuton.conf

Stopped working on:
Queens (RHOSP13) - with error in this bug.

Have not checked on other versions, do not have such.

Created attachment 1672587 [details]
Heat template we execute

Creates pports with several IP's, some ports are 75, 110, 180 fixed IP addresses. Then creates around 22 instances, with those ports attached.

I have been looking into this.  Heat does not seem to have the concept of a "bulk port" request, where it sends a single request to neutron that has multiple requests in it, which can then be handled by Neutron in a more efficient manner.  Instead, it is firing off one request per port, and then neutron server is racing for the IP address allocation.  If you do that enough times, and rapidly enough, then this is exactly the kind of error you can get.  

We did improve the port allocation code in the Stein cycle so that it uses random sampling of the available IP addresses so that IPs will be requested from random points in the range of available IP addresses.  I just backported that back to Queens; see upstream change 715188.  That change has the best chance for us to optimize Neutron's allocation method to be resilient against this kind of scenario.  If you are requesting a number of IP addresses that approximates the size of the pool of available IPs on the subnet, however, there is still a chance you could hit this issue.  The only other optimization I can think of at this time we could do would be to get Heat to support bulk port requests, but I believe that would be an RFE on the Heat side that would take some time to accomplish.  So I believe the change linked above is the best chance to address this issue.

is it already in the repo?

Ruslanas it is in the upstream repo - see https://review.opendev.org/715188 - and just merged downstream.  But it is not in a released version of either, but it's definitely in the git repo now.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2724