Bug 1711520 - IPSet List - Size in Memory Wildly Inconsistent
Summary: IPSet List - Size in Memory Wildly Inconsistent
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.6
Hardware: x86_64
OS: Linux
low
low
Target Milestone: rc
: 7.8
Assignee: Stefano Brivio
QA Contact: yiche
Marc Muehlfeld
URL:
Whiteboard:
Depends On:
Blocks: 1723958 1714111
TreeView+ depends on / blocked
 
Reported: 2019-05-18 09:17 UTC by NOYB
Modified: 2020-03-31 19:20 UTC (History)
7 users (show)

Fixed In Version: kernel-3.10.0-1093.el7
Doc Type: Bug Fix
Doc Text:
.The `ipset list` command reports consistent memory for `hash` set types When you add entries to a `hash` set type, the `ipset` utility must resize the in-memory representation to for new entries by allocating an additional memory block. Previously, `ipset` set the total per-set allocated size to only the size of the new block instead of adding the value to the current in-memory size. As a consequence, the `ip list` command reported an inconsistent memory size. With this update, `ipset` correctly calculates the in-memory size. As a result, the `ipset list` command now displays the correct in-memory size of the set, and the output matches the actual allocated memory for `hash` set types.
Clone Of:
: 1714111 (view as bug list)
Environment:
Last Closed: 2020-03-31 19:18:03 UTC
Target Upstream Version:


Attachments (Terms of Use)
Reproducer: create two hash:ip sets, add entries, swap (863 bytes, application/x-shellscript)
2019-05-26 15:41 UTC, Stefano Brivio
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:1016 None None None 2020-03-31 19:20:33 UTC

Description NOYB 2019-05-18 09:17:42 UTC
Description of problem:
Size in memory reported by ipset list <list_name> changes wildly.  Sometimes it will be around 10K and sometimes around 90K.

Version-Release number of selected component (if applicable):
ipset v6.38, protocol version: 6

How reproducible:
Don't have a good handle on that.  Maybe 25%.  Don't know how to consistently trigger it.

Steps to Reproduce:
1. Create a nearly identical list but with a few new entries.
2. Swap with existing list.
3. ipset list <list_name>

Actual results:
Sometimes a seemingly wildly incorrect size in memory will be reported for the new list.

Expected results:
Correct size in memory reported.

Additional info:
Sometimes the reported size in memory is way out of line with what is expected.  Other times it seems reasonable.
Like say size in memory of 8184 for an inet list of 4345 entries.
Then other times the size in memory reported will by about 95224 or so for nearly the same list after swap with just a few more entries (4360 entries).

A script is running every 15 minutes that creates a new list with some additional entries (CIDR and single IP address) sorted via sort -V.  The new list is then swapped with the existing list.  Sometimes the size in memory reported is about 10K and other times it is about 90K.  Even though the number of entries has changed very little.

Comment 2 Stefano Brivio 2019-05-26 15:41:58 UTC
Created attachment 1573620 [details]
Reproducer: create two hash:ip sets, add entries, swap

Hi NOYB,

(In reply to NOYB from comment #0)
> Sometimes the reported size in memory is way out of line with what is
> expected.  Other times it seems reasonable.
> Like say size in memory of 8184 for an inet list of 4345 entries.
> Then other times the size in memory reported will by about 95224 or so for
> nearly the same list after swap with just a few more entries (4360 entries).
> 
> A script is running every 15 minutes that creates a new list with some
> additional entries (CIDR and single IP address) sorted via sort -V.  The new
> list is then swapped with the existing list.  Sometimes the size in memory
> reported is about 10K and other times it is about 90K.  Even though the
> number of entries has changed very little.

Thanks for reporting this. The issue can be reproduced rather easily with two
hash:ip sets and the number of entries you provided, see the attached
reproducer.

I identified an issue in the kernel implementation of ipset, that leads to an
inconsistent in-memory size reporting caused by an unexpected reset of
memory accounting during hash table resize with allocation of new array blocks.

I'm about to send a patch to fix this to the upstream ipset maintainer. Can
you please give me a real name so that I can credit you for reporting this
issue? I can also use "NOYB" and the email address you provided, or omit this
altogether. Please let me know.

Comment 3 NOYB 2019-05-26 19:15:24 UTC
Thanks for your work on this.  Glad you are able to easily reproduce the issue and provide a fix.
For credit NOYB and email address is fine.
Thanks

Comment 4 yiche 2019-05-27 03:17:26 UTC
Thank you NOYB for report this bug.
Thank Stefano Brivio to provides such good reproducer.
Set qa_ack +

Comment 5 Stefano Brivio 2019-05-27 14:47:58 UTC
Patch upstream at:

    http://patchwork.ozlabs.org/patch/1105608/

Comment 7 Jan Stancek 2019-09-17 07:34:54 UTC
Patch(es) committed on kernel-3.10.0-1093.el7

Comment 12 errata-xmlrpc 2020-03-31 19:18:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1016


Note You need to log in before you can comment on or make changes to this bug.