Bug 1266977 - ipset - Hash is full, cannot add more elements
ipset - Hash is full, cannot add more elements
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
6.0 (Juno)
x86_64 Linux
urgent Severity urgent
: async
: 6.0 (Juno)
Assigned To: Miguel Angel Ajo
Itzik Brown
: Triaged, ZStream
Depends On:
  Show dependency treegraph
Reported: 2015-09-28 13:30 EDT by Benjamin Schmaus
Modified: 2016-04-26 10:49 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-neutron-2014.2.3-16.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-10-15 12:10:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1439817 None None None Never
OpenStack gerrit 173753 None None None Never

  None (edit)
Description Benjamin Schmaus 2015-09-28 13:30:10 EDT
Description of problem:

We have instances failing to boot because users are putting very large
subnets (e.g., /15) into allowed_address_pairs in Neutron.  It appears
that we're hitting some sort of 64K limit in the ipset implementation.

The users putting subnets into allowed_address_pairs, so it seems that
they should be stored in ipset as a subnet as well, not individual

This results in errors like from an instance booting:
2015-09-25 17:17:21.418 51840 INFO heat.engine.stack [-] Stack CREATE
FAILED (VMG-NEC2-OAMA-24ilgto3rllb): Resource CREATE failed: Error:
Server VMG-NEC2-OAM-A delete failed: (500) Build of instance
6031be29-f194-43fb-b05b-92e0c2d7cd18 aborted: Failed to allocate the
network(s), not rescheduling.

On the compute host, the neutron logs contain items like:
2015-09-25 12:04:13.129 58378 ERROR neutron.agent.linux.utils
[req-c8c2e2e2-1cc5-4fab-b1b6-b84f78043459 None]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'ipset', 'add', '-exist', 'IPv4c3742bfb-7f15-4c92-b', '']
Exit code: 1
Stdout: ''
Stderr: 'ipset v6.19: Hash is full, cannot add more elements\n'

We can see that the ipset hash is indeed near a 64K limit, if that is
[root@host nova]# ipset list | wc -l

We can see that the sets have a limit of 65536:
Name: IPv6f60f0540-fa6c-4a8d-9
Type: hash:ip
Revision: 1
Header: family inet6 hashsize 1024 maxelem 65536

Version-Release number of selected component (if applicable): OSP6

How reproducible:
100% in larger deployments

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Upstream bug: https://bugs.launchpad.net/neutron/+bug/1439817

Upstream code commit for OSP6: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0325b98c54af276064e367db603bfeb525bbb790
Comment 12 Benjamin Schmaus 2015-10-09 11:21:03 EDT
Hotfix worked for my client where it was applied.
Comment 16 Itzik Brown 2015-10-15 07:06:20 EDT
Checked with openstack-neutron-2014.2.3-19.el7ost.noarch
Comment 18 errata-xmlrpc 2015-10-15 12:10:20 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.