RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2016822 - firewalld hang/is really slow loading big ip set
Summary: firewalld hang/is really slow loading big ip set
Keywords:
Status: CLOSED DUPLICATE of bug 2043289
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: firewalld
Version: 8.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Eric Garver
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-24 09:49 UTC by rubus
Modified: 2022-02-11 19:23 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-02-11 19:23:11 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
IP set to load (1.99 MB, application/x-xz)
2021-10-24 09:49 UTC, rubus 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github firewalld firewalld issues 881 0 None closed Firewall-cmd is extremely slow when populating large ipsets 2021-12-20 19:10:14 UTC
Red Hat Bugzilla 1908127 1 unspecified CLOSED nftable segmentation fault with big ip set 2023-06-09 08:50:55 UTC
Red Hat Issue Tracker RHELPLAN-100561 0 None None None 2021-10-24 09:56:29 UTC

Description rubus 2021-10-24 09:49:00 UTC 
Created attachment 1836494 [details]
IP set to load

Created attachment 1836494 [details]
IP set to load

Description of problem:
Try to load big ipset (1.1M records) via firewall-cmd.

# /usr/bin/time firewall-cmd --permanent --ipset=test_list --add-entries-from-file=bad_actors_set

hang or is super slow:
ps -aux | grep firewall-cmd
root        5920  0.0  0.0   4240   912 pts/0    S+   Oct22   0:00 /usr/bin/time firewall-cmd --permanent --ipset=test_list --add-entries-from-file=bad_actors_set
root        5921 99.9  1.0 328420 176164 pts/0   R+   Oct22 2314:17 /usr/libexec/platform-python -s /usr/bin/firewall-cmd --permanent --ipset=test_list --add-entries-from-file=bad_actors_set

2314:17 minutes -> ~38h and still counting.

process 5921 use 100% CPU all the time.


Version-Release number of selected component (if applicable):
firewalld.noarch      0.9.3-7.el8      @rhel-8-for-x86_64-baseos-beta-rpms
nftables.x86_64       1:0.9.3-21.el8   @rhel-8-for-x86_64-baseos-beta-rpms

How reproducible:
Try to load ip set to firewalld

Steps to Reproduce:
1. Download and install RHEL 8.5 beta
2. Download ip set (attachment)
3. xz -dk bad_actors_set.xz
4. firewall-cmd --permanent --new-ipset=test_list --type=hash:net --option=hashsize=65536 --option=maxelem=4000000
4. /usr/bin/time firewall-cmd --permanent --ipset=test_list --add-entries-from-file=bad_actors_set

Actual results:
waiting -> inf?

Expected results:
success after seconds 


Additional info:
- nft load this set in ~8s with fix 1908127
- nft Segmentation fault (in current version)
- dmesg is clear.
Comment 1 Eric Garver 2021-10-26 13:30:18 UTC 
AFAICS, this is fixed by the patch listed in bug 1908127 comment 13. In the description you state with the fix from bug 1908127 it loads in ~8s.

I don't think there is anything to do on the firewalld side, can we mark this as a duplicate of bug 1908127?
Comment 2 rubus 2021-10-26 18:34:21 UTC 
I am afraid that it's not the same problem.

On VM I install rpm  with fix from bug 1908127 comment 8 and disable audit for nft `auditctl -A exclude,never -F msgtype=NETFILTER_CFG`.

Now I can load set directly to nft in ~8s.

But when I try to load the same set of IPs (only change is to format it in direct way) it hang/work really slow more then 2h of waiting and nothing.

@
Comment 3 rubus 2021-10-26 19:02:13 UTC 
@Eric Garver do
 
  firewall-cmd --permanent --ipset=test_list --add-entries-from-file=bad_actors_set 

call nftables before run firewall-cmd --reload?

Without fix from bug 1908127 for this IP set nft crash in seconds so firewall-cmd should end with error.

In this scenario I should see crash in seconds or with fix success. And I think it should be after reloading firewall not during adding set from file. 

One more thing firewall-cmd use CPU not nft.

Do I miss something?


comment 2: I load IP set with nft (success) and ran firewalld (fail) on the same vm with nft fix installed.

firewalld.noarch        0.9.3-7.el8                         @rhel-8-for-x86_64-baseos-beta-rpms
nftables.x86_64         1:0.9.3-18.el8_4.huge_set_segfault  @@commandline                      
python3-nftables.x86_64 1:0.9.3-18.el8_4.huge_set_segfault  @@commandline
Comment 4 Eric Garver 2021-10-26 19:23:57 UTC 
My bad. I thought you were saying that with the nftables fix the command goes down to 8 seconds. I guess I read too fast.

I'll leave this report open.
Comment 5 Eric Garver 2021-12-20 19:10:15 UTC 
Upstream changes that make a big improvement:

  114936c71ab1 ("test(ipset): huge set of entries benchmark")
  7f5b736378c0 ("fix(ipset): reduce cost of entry overlap detection")

Even with these patches loading 1.1M ipset entries will be slow.

Firewalld checks for overlapping entries. Without overlap detection applying the ipset may fail in the backends.

  5b4e8918715a ("fix(ipset): disallow overlapping entries")
Comment 7 Eric Garver 2022-02-11 19:23:11 UTC 

*** This bug has been marked as a duplicate of bug 2043289 ***
Note You need to log in before you can comment on or make changes to this bug.