Bug 2228945

* Thu Aug 03 2023 Open vSwitch CI <ovs-ci> - 2.17.0-97
- Merging upstream branch-2.17 [RH git: 22854be7fc]
    Commit list:
    fe98b0c1f9 ovsdb-tool: Fix json leak while showing clustered log.
    44722bbda3 ovsdb-server: Fix excessive memory usage on DB open. (#2228464)
    9db221fcd8 tests: Add ovsdb execution cases for set size constraints.
    3cfe388cb8 ovsdb: relay: Fix handling of XOR updates with size constraints.
    f4d15497f8 ovsdb: file: Fix diff application to a default column value.
    7864ed557e ovsdb: file: Fix inability to read diffs that violate type size.
    97d91ad2d6 ovs-tcpdump: Clear auto-assigned ipv6 address of mirror port.

(In reply to ovs-bugzilla from comment #0)
> +++ This bug was initially created as a clone of Bug #2228464 +++
> 
> In a clustered database mode, after restart, ovsdb-server may consume
> a lot of memory during initial database file read:
> 
>   |00004|memory|INFO|95650400 kB peak resident set size after 96.9 seconds
>   |00005|memory|INFO|atoms:3083346 cells:1838767 monitors:0 raft-log:123309
> txn-history:123307 txn-history-atoms:1647022868
> 
> In the example above, it's a Northbound OVN database with 123K small
> transaction
> records in the file.  The total file size is just 150 MB.  Fully compacted -
> 80 MB.
> During the initial database file read, ovsdb-server allocates 95 GB of RAM to
> store these transactions in a history.  History will be drained right after
> the
> initial read is completed, but the memory may not be returned to the system
> until the next compaction (glibc behavior).  The process may just be killed
> before finishing the read if there is no enough memory in the system.
> 
> How to reproduce:
> 
> 1. Create OVN setup with clustered databases.
> 2. Execute 100K small Northbound database updates in a short time, but make
>    sure that ovsdb-server didn't compact it.
Hi Ilya,
 what command could meet this requirements? any accurate reproducer?

(In reply to Jianlin Shi from comment #4)
> (In reply to ovs-bugzilla from comment #0)
> > +++ This bug was initially created as a clone of Bug #2228464 +++
> > 
> > In a clustered database mode, after restart, ovsdb-server may consume
> > a lot of memory during initial database file read:
> > 
> >   |00004|memory|INFO|95650400 kB peak resident set size after 96.9 seconds
> >   |00005|memory|INFO|atoms:3083346 cells:1838767 monitors:0 raft-log:123309
> > txn-history:123307 txn-history-atoms:1647022868
> > 
> > In the example above, it's a Northbound OVN database with 123K small
> > transaction
> > records in the file.  The total file size is just 150 MB.  Fully compacted -
> > 80 MB.
> > During the initial database file read, ovsdb-server allocates 95 GB of RAM to
> > store these transactions in a history.  History will be drained right after
> > the
> > initial read is completed, but the memory may not be returned to the system
> > until the next compaction (glibc behavior).  The process may just be killed
> > before finishing the read if there is no enough memory in the system.
> > 
> > How to reproduce:
> > 
> > 1. Create OVN setup with clustered databases.
> > 2. Execute 100K small Northbound database updates in a short time, but make
> >    sure that ovsdb-server didn't compact it.
> Hi Ilya,
>  what command could meet this requirements? any accurate reproducer?

I seem to be able to reproduce the issue by adding and removing the same
logical switch in OVN's Northbound database many times until the database
file size on disk grows just below 10 MB.  10 MB is a minimal threshold
for database compaction, so do not cross that line.  For me it took about
13K add-del cycles.  After that you may restart the ovsdb-server to see
the issue.

tested with following script:

1. create cluster:

node1:
ctl_cmd="/usr/share/ovn/scripts/ovn-ctl"

ip_s=1.1.7.3
ip_c1=1.1.7.20
ip_c2=1.1.7.12
$ctl_cmd --db-nb-addr=$ip_s --db-nb-create-insecure-remote=yes \
        --db-sb-addr=$ip_s --db-sb-create-insecure-remote=yes \
        --db-nb-cluster-local-addr=$ip_s --db-sb-cluster-local-addr=$ip_s \
        --ovn-northd-nb-db=tcp:$ip_s:6641,tcp:$ip_c1:6641,tcp:$ip_c2:6641 \
        --ovn-northd-sb-db=tcp:$ip_s:6642,tcp:$ip_c1:6642,tcp:$ip_c2:6642 start_northd 
node2:
ctl_cmd="/usr/share/ovn/scripts/ovn-ctl"

ip_s=1.1.7.3
ip_c1=1.1.7.20
ip_c2=1.1.7.12
$ctl_cmd --db-nb-addr=$ip_c1 --db-nb-create-insecure-remote=yes \
        --db-sb-addr=$ip_c1 --db-sb-create-insecure-remote=yes \
        --db-nb-cluster-local-addr=$ip_c1 --db-sb-cluster-local-addr=$ip_c1 \
        --db-nb-cluster-remote-addr=$ip_s --db-sb-cluster-remote-addr=$ip_s \
        --ovn-northd-nb-db=tcp:$ip_s:6641,tcp:$ip_c1:6641,tcp:$ip_c2:6641 \
        --ovn-northd-sb-db=tcp:$ip_s:6642,tcp:$ip_c1:6642,tcp:$ip_c2:6642 start_northd
node3:
ctl_cmd="/usr/share/ovn/scripts/ovn-ctl"

ip_s=1.1.7.3
ip_c1=1.1.7.20
ip_c2=1.1.7.12
$ctl_cmd --db-nb-addr=$ip_c2 --db-nb-create-insecure-remote=yes \
        --db-sb-addr=$ip_c2 --db-sb-create-insecure-remote=yes \
        --db-nb-cluster-local-addr=$ip_c2 --db-sb-cluster-local-addr=$ip_c2 \
        --db-nb-cluster-remote-addr=$ip_s --db-sb-cluster-remote-addr=$ip_s \
        --ovn-northd-nb-db=tcp:$ip_s:6641,tcp:$ip_c1:6641,tcp:$ip_c2:6641 \
        --ovn-northd-sb-db=tcp:$ip_s:6642,tcp:$ip_c1:6642,tcp:$ip_c2:6642 start_northd

2. add and delete lsp
for i in {1..10000}
do
        echo $i
        ovn-nbctl ls-add ls1
        ovn-nbctl ls-del ls1
done
3. restart cluster on the leader:
ctl_cmd="/usr/share/ovn/scripts/ovn-ctl"

ip_s=1.1.7.3
ip_c1=1.1.7.20
ip_c2=1.1.7.12
$ctl_cmd --db-nb-addr=$ip_s --db-nb-create-insecure-remote=yes \
        --db-sb-addr=$ip_s --db-sb-create-insecure-remote=yes \
        --db-nb-cluster-local-addr=$ip_s --db-sb-cluster-local-addr=$ip_s \
        --ovn-northd-nb-db=tcp:$ip_s:6641,tcp:$ip_c1:6641,tcp:$ip_c2:6641 \
        --ovn-northd-sb-db=tcp:$ip_s:6642,tcp:$ip_c1:6642,tcp:$ip_c2:6642 restart_northd

4. check the memory peak in ovsdb-server-nb.log:

result on openvswitch2.17-92:
2023-09-21T09:34:59.951Z|00014|memory|INFO|38044 kB peak resident set size after 10.1 seconds 

[root@dell-per730-03 ovs2.17-92]# rpm -qa | grep -E "openvswitch2.17|ovn23.06"                        
ovn23.06-23.06.1-11.el9fdp.x86_64
ovn23.06-central-23.06.1-11.el9fdp.x86_64
ovn23.06-host-23.06.1-11.el9fdp.x86_64
openvswitch2.17-2.17.0-92.el9fdp.x86_64

result on openvswitch2.17-97:

2023-09-21T09:25:39.463Z|00014|memory|INFO|29336 kB peak resident set size after 10.2 seconds

[root@dell-per730-03 ovs2.17-97]# rpm -qa | grep -E "openvswitch2.17|ovn23.06"
ovn23.06-23.06.1-11.el9fdp.x86_64
ovn23.06-central-23.06.1-11.el9fdp.x86_64
ovn23.06-host-23.06.1-11.el9fdp.x86_64
openvswitch2.17-2.17.0-97.el9fdp.x86_64

tested with another script:

addr="10.1.1.1"
for i in {0..255}
do
        for j in {0..45}
        do
                addr="$addr,1.1.$i.$j"
        done
done
addr_uuid=$(ovn-nbctl --no-leader create Address_Set name=test addresses=\"$addr\")
echo $addr_uuid
for i in {0..255}
do
        addr=1.1.$i.46
        for j in {47..255}
        do
                addr="$addr,1.1.$i.$j"
        done
        ovn-nbctl add Address_Set $addr_uuid addresses $addr
done
for i in {0..5000}
do
        ovn-nbctl add Address_Set $addr_uuid addresses 10.1.1.2
        ovn-nbctl remove Address_Set $addr_uuid addresses 10.1.1.2
done

then restart the cluster, get the following result:

on openvswitch2.17-92:

[root@dell-per730-03 ovs2.17-92]# rpm -qa | grep -E "openvswitch2.17|ovn23.06"
ovn23.06-23.06.1-11.el9fdp.x86_64
ovn23.06-central-23.06.1-11.el9fdp.x86_64
ovn23.06-host-23.06.1-11.el9fdp.x86_64
openvswitch2.17-2.17.0-92.el9fdp.x86_64

[root@dell-per730-03 ovs2.17-92]# grep peak ovsdb-server-nb.log
2023-09-22T09:21:05.275Z|00004|memory|INFO|17051776 kB peak resident set size after 25.8 seconds

on openvswitch2.17-97:

[root@dell-per730-03 ovs2.17-92]# rpm -qa | grep -E "openvswitch2.17|ovn23.06"                        
ovn23.06-23.06.1-11.el9fdp.x86_64
ovn23.06-central-23.06.1-11.el9fdp.x86_64                                                             
ovn23.06-host-23.06.1-11.el9fdp.x86_64                                                                
openvswitch2.17-2.17.0-97.el9fdp.x86_64 

[root@dell-per730-03 ovs2.17-92]# grep peak /var/log/ovn/ovsdb-server-nb.log
2023-09-22T09:56:45.205Z|00004|memory|INFO|31908 kB peak resident set size after 25.0 seconds

set VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openvswitch2.17 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:5817