Description of problem:
We have instances failing to boot because users are putting very large
subnets (e.g., /15) into allowed_address_pairs in Neutron. It appears
that we're hitting some sort of 64K limit in the ipset implementation.
The users putting subnets into allowed_address_pairs, so it seems that
they should be stored in ipset as a subnet as well, not individual
This results in errors like from an instance booting:
2015-09-25 17:17:21.418 51840 INFO heat.engine.stack [-] Stack CREATE
FAILED (VMG-NEC2-OAMA-24ilgto3rllb): Resource CREATE failed: Error:
Server VMG-NEC2-OAM-A delete failed: (500) Build of instance
6031be29-f194-43fb-b05b-92e0c2d7cd18 aborted: Failed to allocate the
network(s), not rescheduling.
On the compute host, the neutron logs contain items like:
2015-09-25 12:04:13.129 58378 ERROR neutron.agent.linux.utils
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'ipset', 'add', '-exist', 'IPv4c3742bfb-7f15-4c92-b', '192.168.10.0/15']
Exit code: 1
Stderr: 'ipset v6.19: Hash is full, cannot add more elements\n'
We can see that the ipset hash is indeed near a 64K limit, if that is
[root@host nova]# ipset list | wc -l
We can see that the sets have a limit of 65536:
Header: family inet6 hashsize 1024 maxelem 65536
Version-Release number of selected component (if applicable): OSP6
100% in larger deployments
Steps to Reproduce:
Upstream bug: https://bugs.launchpad.net/neutron/+bug/1439817
Upstream code commit for OSP6: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0325b98c54af276064e367db603bfeb525bbb790
Hotfix worked for my client where it was applied.
Checked with openstack-neutron-2014.2.3-19.el7ost.noarch
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.