Bug 1908774 - [SCALE] enable OVN DB memory trimming on compaction
Summary: [SCALE] enable OVN DB memory trimming on compaction
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Tim Rozet
QA Contact: Ross Brattain
URL:
Whiteboard:
Depends On: 1888829
Blocks: 1943310
TreeView+ depends on / blocked
 
Reported: 2020-12-17 15:14 UTC by Tim Rozet
Modified: 2021-07-27 22:36 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1943310 (view as bug list)
Environment:
Last Closed: 2021-07-27 22:35:34 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 438 0 None closed Downstream merge 2-22-21 2021-03-03 17:09:39 UTC
Github ovn-org ovn-kubernetes pull 1920 0 None open Enable DB memory trimming on compaction 2021-02-01 18:54:58 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:36:03 UTC

Description Tim Rozet 2020-12-17 15:14:01 UTC 
Description of problem:
With the changes introduced in https://bugzilla.redhat.com/show_bug.cgi?id=1888829#c16 we can now free up some of the memory usage. We need to enable this for scale improvements.
Comment 3 Ross Brattain 2021-03-09 01:39:32 UTC 
Verified that memory trimming is enabled on 4.8.0-0.nightly-2021-03-08-092651

observed "memory trimming after compaction enabled."  and "raft-log:" stat seems to decrease after compaction.

log_ovnkube-master-85nsf_rbrattai-o48v13-r4swd-master-1
179:2021-03-08T13:47:31Z|00014|memory|INFO|16968 kB peak resident set size after 10.0 seconds
180:2021-03-08T13:47:31Z|00015|memory|INFO|cells:20 monitors:0
181:2021-03-08T13:47:32Z|00016|ovsdb_server|INFO|memory trimming after compaction enabled.
182:2021-03-08T13:47:32Z|00017|ovsdb_server|INFO|memory trimming after compaction enabled.
183:2021-03-08T13:47:46Z|00018|ovsdb_server|INFO|memory trimming after compaction enabled.
203:2021-03-08T13:48:11Z|00038|ovsdb_server|INFO|memory trimming after compaction enabled.
204:2021-03-08T14:12:51Z|00039|memory|INFO|peak resident set size grew 55% in last 1520.7 seconds, from 16968 kB to 26308 kB
205:2021-03-08T14:12:51Z|00040|memory|INFO|cells:4875 monitors:0 raft-connections:4 raft-log:2572
206:2021-03-09T00:56:33Z|00041|memory|INFO|peak resident set size grew 50% in last 38621.2 seconds, from 26308 kB to 39528 kB
207:2021-03-09T00:56:33Z|00042|memory|INFO|cells:4470 monitors:0 raft-connections:4 raft-log:6413
342:2021-03-08T13:48:11Z|00017|ovsdb_server|INFO|memory trimming after compaction enabled.
343:2021-03-08T13:48:11Z|00018|memory|INFO|16776 kB peak resident set size after 10.0 seconds
344:2021-03-08T13:48:11Z|00019|memory|INFO|cells:269 monitors:0 raft-connections:2 raft-log:8
351:2021-03-08T13:56:02Z|00026|memory|INFO|peak resident set size grew 51% in last 470.2 seconds, from 16776 kB to 25284 kB
352:2021-03-08T13:56:02Z|00027|memory|INFO|cells:22541 monitors:2 raft-connections:4 raft-log:1169 sessions:1
353:2021-03-08T14:11:22Z|00028|memory|INFO|peak resident set size grew 62% in last 920.5 seconds, from 25284 kB to 40860 kB
354:2021-03-08T14:11:22Z|00029|memory|INFO|cells:38677 monitors:2 raft-connections:4 raft-log:2196 sessions:1
355:2021-03-08T17:29:59Z|00030|memory|INFO|peak resident set size grew 50% in last 11916.5 seconds, from 40860 kB to 61324 kB
356:2021-03-08T17:29:59Z|00031|memory|INFO|cells:32102 monitors:2 raft-connections:4 raft-log:4729 sessions:1

log_ovnkube-master-5ml8s_rbrattai-o48v13-r4swd-master-0
86:2021-03-08T13:47:59Z|00007|ovsdb_server|INFO|memory trimming after compaction enabled.
105:2021-03-08T13:48:06Z|00026|memory|INFO|10720 kB peak resident set size after 10.0 seconds
106:2021-03-08T13:48:06Z|00027|memory|INFO|cells:132 monitors:3 raft-connections:4 raft-log:16 sessions:4
107:2021-03-08T13:50:56Z|00028|memory|INFO|peak resident set size grew 51% in last 170.1 seconds, from 10720 kB to 16156 kB
108:2021-03-08T13:50:56Z|00029|memory|INFO|cells:2632 monitors:3 raft-connections:4 raft-log:729 sessions:9
109:2021-03-08T14:11:27Z|00030|memory|INFO|peak resident set size grew 54% in last 1230.6 seconds, from 16156 kB to 24848 kB
110:2021-03-08T14:11:27Z|00031|memory|INFO|cells:5365 monitors:3 raft-connections:4 raft-log:2142 sessions:9
111:2021-03-08T21:32:38Z|00032|memory|INFO|peak resident set size grew 50% in last 26470.9 seconds, from 24848 kB to 37272 kB
112:2021-03-08T21:32:38Z|00033|memory|INFO|cells:4470 monitors:3 raft-connections:4 raft-log:5469 sessions:9
247:2021-03-08T13:47:59Z|00011|ovsdb_server|INFO|memory trimming after compaction enabled.
256:2021-03-08T13:48:04Z|00020|memory|INFO|9560 kB peak resident set size after 10.0 seconds
257:2021-03-08T13:48:04Z|00021|memory|INFO|cells:39 monitors:1 raft-connections:2 raft-log:7 sessions:2
269:2021-03-08T13:49:14Z|00033|memory|INFO|peak resident set size grew 72% in last 70.0 seconds, from 9560 kB to 16484 kB
270:2021-03-08T13:49:14Z|00034|memory|INFO|cells:9486 monitors:3 raft-connections:4 raft-log:221 sessions:6
280:2021-03-08T13:55:54Z|00044|memory|INFO|peak resident set size grew 51% in last 400.1 seconds, from 16484 kB to 24896 kB
281:2021-03-08T13:55:54Z|00045|memory|INFO|cells:22505 monitors:3 raft-connections:4 raft-log:1159 sessions:6
291:2021-03-08T14:10:55Z|00055|memory|INFO|peak resident set size grew 51% in last 900.4 seconds, from 24896 kB to 37476 kB
292:2021-03-08T14:10:55Z|00056|memory|INFO|cells:36937 monitors:4 raft-connections:4 raft-log:2018 sessions:7
317:2021-03-08T14:59:26Z|00081|memory|INFO|peak resident set size grew 50% in last 2911.4 seconds, from 37476 kB to 56220 kB
318:2021-03-08T14:59:26Z|00082|memory|INFO|cells:32462 monitors:4 raft-connections:4 raft-log:4081 sessions:7

log_ovnkube-master-rkb22_rbrattai-o48v13-r4swd-master-2
17958:2021-03-08T13:47:58Z|00014|memory|INFO|18780 kB peak resident set size after 10.0 seconds
17959:2021-03-08T13:47:58Z|00015|memory|INFO|cells:20 monitors:0
17960:2021-03-08T13:47:59Z|00016|ovsdb_server|INFO|memory trimming after compaction enabled.
17961:2021-03-08T13:48:00Z|00017|ovsdb_server|INFO|memory trimming after compaction enabled.
17982:2021-03-08T13:48:14Z|00038|ovsdb_server|INFO|memory trimming after compaction enabled.
17983:2021-03-08T13:48:40Z|00039|ovsdb_server|INFO|memory trimming after compaction enabled.
17984:2021-03-08T14:14:09Z|00040|memory|INFO|peak resident set size grew 51% in last 1570.7 seconds, from 18780 kB to 28324 kB
17985:2021-03-08T14:14:09Z|00041|memory|INFO|cells:4630 monitors:0 raft-connections:4 raft-log:2845
18125:2021-03-08T13:48:39Z|00022|memory|INFO|18832 kB peak resident set size after 10.0 seconds
18126:2021-03-08T13:48:39Z|00023|memory|INFO|cells:4858 monitors:0 raft-connections:4 raft-log:106
18127:2021-03-08T13:48:40Z|00024|ovsdb_server|INFO|memory trimming after compaction enabled.
18128:2021-03-08T13:57:19Z|00025|memory|INFO|peak resident set size grew 50% in last 520.3 seconds, from 18832 kB to 28272 kB
18129:2021-03-08T13:57:19Z|00026|memory|INFO|cells:24394 monitors:2 raft-connections:4 raft-log:1261 sessions:2
18130:2021-03-08T14:11:50Z|00027|memory|INFO|peak resident set size grew 51% in last 870.5 seconds, from 28272 kB to 42600 kB
18131:2021-03-08T14:11:50Z|00028|memory|INFO|cells:35061 monitors:2 raft-connections:4 raft-log:2390 sessions:4
18132:2021-03-08T18:59:10Z|00029|memory|INFO|peak resident set size grew 51% in last 17240.4 seconds, from 42600 kB to 64160 kB
18133:2021-03-08T18:59:10Z|00030|memory|INFO|cells:32694 monitors:2 raft-connections:4 raft-log:5105 sessions:4
Comment 6 errata-xmlrpc 2021-07-27 22:35:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438
Note You need to log in before you can comment on or make changes to this bug.