Bug 1770953
| Summary: | ovn-controller consumes to much memory and eventually core dumps | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Numan Siddique <nusiddiq> |
| Component: | ovn2.11 | Assignee: | Dumitru Ceara <dceara> |
| Status: | CLOSED ERRATA | QA Contact: | Jianlin Shi <jishi> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | FDP 20.A | CC: | amuller, apevec, ctrautma, dceara, dhill, ekuris, ffernand, jlibosva, kfida, lhh, majopela, scohen |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | ovn2.11-2.11.1-15.el7fdn | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1770295 | Environment: | |
| Last Closed: | 2020-01-21 17:02:44 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1766410, 1770295 | ||
| Bug Blocks: | |||
|
Description
Numan Siddique
2019-11-11 14:53:12 UTC
run stress test with following script:
systemctl start ovn-northd
systemctl start openvswitch
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set Open_vSwitch . external-ids:system-id=hv0 external-ids:ovn-remote=tcp:20.0.30.26:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.30.26
systemctl start ovn-controller
ctl_pid=$(ps aux | grep ovn-controller | grep -v grep | awk '{print $2}')
free -m | tee memory.log
pmap $ctl_pid >> memory.log
echo "start"
for i in {11..99}
do
for j in {11..13}
do
#free -m
ovn-nbctl ls-add s${j}w$i
ovn-nbctl lsp-add s${j}w$i a${j}p${i}1
ovn-nbctl lsp-set-addresses a${j}p${i}1 "00:00:00:$j:$i:02 192.$i.$j.2"
ovn-nbctl lsp-add s${j}w$i a${j}p${i}2
ovn-nbctl lsp-set-addresses a${j}p${i}2 "00:00:00:$j:$i:03 192.$i.$j.3"
ovn-nbctl lsp-add s${j}w$i a${j}p${i}3
ovn-nbctl lsp-set-addresses a${j}p${i}3 "00:00:00:$j:$i:04 192.$i.$j.4"
ovs-vsctl add-port br-int a${j}p${i}1 -- set Interface a${j}p${i}1 type=internal external_ids:iface-id=a${j}p${i}1
ovs-vsctl add-port br-int a${j}p${i}2 -- set Interface a${j}p${i}2 type=internal external_ids:iface-id=a${j}p${i}2
ovs-vsctl add-port br-int a${j}p${i}3 -- set Interface a${j}p${i}3 type=internal external_ids:iface-id=a${j}p${i}3
ovn-nbctl create Address_Set name=set$i$j "addresses=192.$i.$j.2,192.$i.$j.3,192.$i.$j.4"
ovn-nbctl pg-add pg${i}$j a${j}p${i}1 a${j}p${i}2 a${j}p${i}3
ovn-nbctl acl-add s${j}w$i to-lport 1000 "inport==@pg${i}${j} && ip4 && ip4.src==\$set$i$j" allow
#ovs-ofctl dump-flows br-int | wc -l
#sleep 5
#ovs-ofctl dump-flows br-int | wc -l
#free -m
#ovs-ofctl dump-flows br-int | wc -l | tee -a memory.log
#echo "i:$i, j:$j" >> memory.log
#free -m | tee -a memory.log
done
done
sleep 5
ovs-ofctl dump-flows br-int | wc -l | tee -a memory.log
echo "finish add ports" >> memory.log
free -m | tee -a memory.log
pmap $ctl_pid >> memory.log
echo "start del ports"
for i in {11..99}
do
for j in {11..13}
do
ovn-nbctl ls-del s${j}w$i
done
done
sleep 5
ovs-ofctl dump-flows br-int | wc -l | tee -a memory.log
echo "finish del ports" >> memory.log
free -m | tee -a memory.log
pmap $ctl_pid >> memory.log
on ovn2.11.1-13:
total used free shared buff/cache available
Mem: 63686 1218 61631 63 837 61961
Swap: 31999 0 31999
103151: ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info ---
no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controo
ller.pid --detach
......
total 281120K
23810
finish add ports
total used free shared buff/cache available
Mem: 63686 1424 61371 69 890 61748
Swap: 31999 0 31999
103151: ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info ---
no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controo
ller.pid --detach
......
total 328256K
6
finish del ports
total used free shared buff/cache available
Mem: 63686 1428 61364 69 892 61744
Swap: 31999 0 31999
103151: ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info ---
no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controo
ller.pid --detach
......
total 327832K
on ovn2.11.1-15:
total used free shared buff/cache available
Mem: 63686 1194 61651 69 840 61981
Swap: 31999 0 31999
110033: ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info ---
no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controo
ller.pid --detach
......
total 281120K
23810
finish add ports
total used free shared buff/cache available
Mem: 63686 1402 61394 69 889 61771
Swap: 31999 0 31999
110033: ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info ---
no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controo
ller.pid --detach
......
total 316448K
6
finish del ports
total used free shared buff/cache available
Mem: 63686 1405 61387 69 892 61767
Swap: 31999 0 31999
110033: ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info ---
no-chdir --log-file=/var/log/openvswitch/ovn-controller.log --pidfile=/var/run/openvswitch/ovn-controo
ller.pid --detach
......
total 316184K
<==== memory for ovn-controller on 15 is much less than that on 13.
set VERIFIED per above test.
@Dumitru, how do you think?
(In reply to Jianlin Shi from comment #5) > > <==== memory for ovn-controller on 15 is much less than that on 13. > > set VERIFIED per above test. > > @Dumitru, how do you think? Looks good to me. Regards, Dumitru Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0190 |