Bug 1266974 - ipset - Hash is full, cannot add more elements
ipset - Hash is full, cannot add more elements
Status: CLOSED DUPLICATE of bug 1266975
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
6.0 (Juno)
x86_64 Linux
unspecified Severity urgent
: ---
: 8.0 (Liberty)
Assigned To: lpeer
Ofer Blaut
Depends On:
  Show dependency treegraph
Reported: 2015-09-28 13:29 EDT by Benjamin Schmaus
Modified: 2016-04-26 09:52 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-10-07 09:58:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Benjamin Schmaus 2015-09-28 13:29:08 EDT
Description of problem:

We have instances failing to boot because users are putting very large
subnets (e.g., /15) into allowed_address_pairs in Neutron.  It appears
that we're hitting some sort of 64K limit in the ipset implementation.

The users putting subnets into allowed_address_pairs, so it seems that
they should be stored in ipset as a subnet as well, not individual

This results in errors like from an instance booting:
2015-09-25 17:17:21.418 51840 INFO heat.engine.stack [-] Stack CREATE
FAILED (VMG-NEC2-OAMA-24ilgto3rllb): Resource CREATE failed: Error:
Server VMG-NEC2-OAM-A delete failed: (500) Build of instance
6031be29-f194-43fb-b05b-92e0c2d7cd18 aborted: Failed to allocate the
network(s), not rescheduling.

On the compute host, the neutron logs contain items like:
2015-09-25 12:04:13.129 58378 ERROR neutron.agent.linux.utils
[req-c8c2e2e2-1cc5-4fab-b1b6-b84f78043459 None]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'ipset', 'add', '-exist', 'IPv4c3742bfb-7f15-4c92-b', '']
Exit code: 1
Stdout: ''
Stderr: 'ipset v6.19: Hash is full, cannot add more elements\n'

We can see that the ipset hash is indeed near a 64K limit, if that is
[root@host nova]# ipset list | wc -l

We can see that the sets have a limit of 65536:
Name: IPv6f60f0540-fa6c-4a8d-9
Type: hash:ip
Revision: 1
Header: family inet6 hashsize 1024 maxelem 65536

Version-Release number of selected component (if applicable): OSP6

How reproducible:
100% in larger deployments

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Upstream bug: https://bugs.launchpad.net/neutron/+bug/1439817

Upstream code commit for OSP6: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0325b98c54af276064e367db603bfeb525bbb790
Comment 2 Jakub Libosvar 2015-10-07 09:58:49 EDT

*** This bug has been marked as a duplicate of bug 1266975 ***

Note You need to log in before you can comment on or make changes to this bug.