Bug 1266975

Summary:	ipset - Hash is full, cannot add more elements
Product:	Red Hat OpenStack	Reporter:	Benjamin Schmaus <bschmaus>
Component:	openstack-neutron	Assignee:	lpeer <lpeer>
Status:	CLOSED ERRATA	QA Contact:	Toni Freger <tfreger>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	6.0 (Juno)	CC:	amuller, chrisw, nyechiel, tfreger, yeylon
Target Milestone:	---	Keywords:	Triaged, ZStream
Target Release:	6.0 (Juno)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-neutron-2014.2.3-16.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-02-08 14:17:44 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Benjamin Schmaus 2015-09-28 17:29:32 UTC

Description of problem:

We have instances failing to boot because users are putting very large
subnets (e.g., /15) into allowed_address_pairs in Neutron.  It appears
that we're hitting some sort of 64K limit in the ipset implementation.

The users putting subnets into allowed_address_pairs, so it seems that
they should be stored in ipset as a subnet as well, not individual
addresses.

This results in errors like from an instance booting:
2015-09-25 17:17:21.418 51840 INFO heat.engine.stack [-] Stack CREATE
FAILED (VMG-NEC2-OAMA-24ilgto3rllb): Resource CREATE failed: Error:
Server VMG-NEC2-OAM-A delete failed: (500) Build of instance
6031be29-f194-43fb-b05b-92e0c2d7cd18 aborted: Failed to allocate the
network(s), not rescheduling.


On the compute host, the neutron logs contain items like:
2015-09-25 12:04:13.129 58378 ERROR neutron.agent.linux.utils
[req-c8c2e2e2-1cc5-4fab-b1b6-b84f78043459 None]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf',
'ipset', 'add', '-exist', 'IPv4c3742bfb-7f15-4c92-b', '192.168.10.0/15']
Exit code: 1
Stdout: ''
Stderr: 'ipset v6.19: Hash is full, cannot add more elements\n'


We can see that the ipset hash is indeed near a 64K limit, if that is
problematic:
[root@host nova]# ipset list | wc -l
65647


We can see that the sets have a limit of 65536:
Name: IPv6f60f0540-fa6c-4a8d-9
Type: hash:ip
Revision: 1
Header: family inet6 hashsize 1024 maxelem 65536



Version-Release number of selected component (if applicable): OSP6


How reproducible:
100% in larger deployments

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Upstream bug: https://bugs.launchpad.net/neutron/+bug/1439817

Upstream code commit for OSP6: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0325b98c54af276064e367db603bfeb525bbb790

Comment 2 Jakub Libosvar 2015-10-07 13:58:49 UTC

*** Bug 1266974 has been marked as a duplicate of this bug. ***

Comment 5 Toni Freger 2016-01-21 09:08:18 UTC

Have tested on osp6 with 2 compute nodes.
openstack-neutron-2014.2.3-32.el7ost.noarch


neutron subnet (192.168.10.0/15) created without errors:
  allocation_pools  | {"start": "192.168.0.2", "end": "192.169.255.254"} |
| cidr              | 192.168.0.0/15 

30 instances booted, no errors on openwswitch log of both computes were found.

Comment 7 errata-xmlrpc 2016-02-08 14:17:44 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0130.html