Bug 2047275

+++ This bug was initially created as a clone of Bug #2038818 +++

Description of problem:
I have large saved hash:ip set (>200000 entries).
systemctl restart ipset works more than 10 minutes and timed out.

--- Additional comment from Phil Sutter on 2022-01-20 13:37:12 UTC ---

OK, so I missed the fact that the problematic code does not run for
unreferenced sets and also that elements are prefixed "add" in save files - I
can only attribute this to coffee deficit. :(

Looking at the code, I found the whole checksum approach a bit too much, so I
got rid of it. Here's the diff:

------------------------------------8<-----------------------------------------
--- a/ipset.start-stop
+++ b/ipset.start-stop
@@ -235,41 +235,23 @@ load() {
                return
        fi
 
-       # Find a salt for md5sum that makes names of saved sets unique
-       salt=0
-       while true; do
-               unique=1
-               IFS="
-"
-               for set in $(${IPSET_BIN} list -n -t); do
-                       if grep -q "^create $(echo "${salt}${set}" | md5sum | head -c31) " "${
merged}"; then
-                               unique=0
-                               break
-                       fi
-               done
-               unset IFS
-               [ ${unique} -eq 1 ] && break
-               salt=$((salt + 1))
+       # Find a free prefix for temporary set names
+       pfx="__tmp$((idx = 0))_"
+       while ${IPSET_BIN} list -n -t | grep -q "^${pfx}"; do
+               pfx="__tmp$((idx += 1))_"
        done
 
        # Add sets, mangling names for conflicting sets
-       mangled="$(mktemp -q /tmp/ipset.XXXXXX)"
-       CLEAN_FILES="${CLEAN_FILES} ${mangled}"
-       chmod 600 "${mangled}"
-
-       awk '/^(add|create) ('"${conflicts}"')/ { printf "%s ",$1; system("echo '${salt}'" $2 " | md5sum | head -c31"); $1=""; $2=""; print; next} {print}' "${merged}" > "${mangled}"
-       if ! ipset_restore "${mangled}"; then
+       sed -i -E -e "s/^(add|create) (${conflicts}) /\1 ${pfx}\2 /g" "${merged}"
+       if ! ipset_restore "${merged}"; then
                err "Failed to restore configured sets"
                exit 1
        fi
 
-       rm "${mangled}"
-       CLEAN_FILES="${CLEAN_FILES%* ${mangled}}"
-
        # Swap and delete old sets
        IFS='|'
        for set in ${conflicts}; do
-               mangled="$(echo "${salt}${set}" | md5sum | head -c31)"
+               mangled="${pfx}${set}"
------------------------------------8<-----------------------------------------

Please give it a try if you find time to.

Stefano, what's your take on this? Was there a specific reason requiring the
checksumming in the first place?

--- Additional comment from  on 2022-01-21 14:46:15 UTC ---

Hi.

I tested the patch in a VM.

[root@ol8-test ~]# ipset list -t
Name: test1
Type: hash:ip
Revision: 4
Header: family inet hashsize 131072 maxelem 300000
Size in memory: 5235048
References: 1
Number of entries: 214372

Name: test2
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 168
References: 1
Number of entries: 1

Name: test3
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 216
References: 0
Number of entries: 2

[root@ol8-test ~]# iptables -S FORWARD
-P FORWARD ACCEPT
-A FORWARD -m set --match-set test1 dst 
-A FORWARD -m set --match-set test2 dst

[root@ol8-test ~]# /usr/libexec/ipset/ipset.start-stop save

[root@ol8-test ~]# time /usr/libexec/ipset/ipset.start-stop start

real    18m15,710s
user    7m17,382s
sys     13m0,186s


[root@ol8-test ipset]# cd /usr/libexec/ipset

[root@ol8-test ipset]# patch -p1 < /root/bug2038818.patch 
patching file ipset.start-stop

[root@ol8-test ipset]# time /usr/libexec/ipset/ipset.start-stop start

real    0m2,250s
user    0m1,153s
sys     0m0,929s

[root@ol8-test ipset]# ipset list -t
Name: test1
Type: hash:ip
Revision: 4
Header: family inet hashsize 131072 maxelem 300000
Size in memory: 5217768
References: 1
Number of entries: 214372

Name: test2
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 168
References: 1
Number of entries: 1

Name: test3
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 216
References: 0
Number of entries: 2


[root@ol8-test ipset]# time systemctl restart ipset

real    0m2,213s
user    0m0,011s
sys     0m0,013s

It seems you can change the status to fixed/verified. )

Thanks a lot!