Bug 1678197
| Summary: | memory leak in openvswitch python wrapper for RHOSP13 | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Pierre Amadio <pamadio> | |
| Component: | openvswitch | Assignee: | Open vSwitch development team <ovs-team> | |
| Status: | CLOSED ERRATA | QA Contact: | Candido Campos <ccamposr> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 13.0 (Queens) | CC: | aguetta, amcleod, amuller, apevec, broskos, ccamposr, ccopello, chrisw, dalvarez, dhill, ebarrera, marjones, mvalsecc, njohnston, ovs-team, pmorey, rhos-maint, rlapore, sisadoun, slinaber, smykhail, svmichel, tfreger, yocha | |
| Target Milestone: | async | Keywords: | Triaged, ZStream | |
| Target Release: | 13.0 (Queens) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openvswitch-2.9.0-97.el7fdp | Doc Type: | Bug Fix | |
| Doc Text: |
Previously, the python-openvswitch library did not de-allocate objects used when communicating with the openvswitch process, causing a memory leak in neutron agents. As a result, Neutron agent memory size increased continuously until reaching a state of memory starvation and the kernel out-of-memory-killer started terminating processes.
With this update, the python-openvswitch library de-allocates memory correctly and the memory consumption for neutron agent processes is stable.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1693430 (view as bug list) | Environment: | ||
| Last Closed: | 2019-03-14 13:28:34 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1693430 | |||
|
Description
Pierre Amadio
2019-02-18 09:49:45 UTC
Escalated due to customer escalation Subject: Escalation and need immediate support Description: Work around also have impact on the environment Expectations: There is major impact to the business please help to provide a Hot fix immediately Escalated due to customer escalation Subject: Escalation and need immediate support Description: Work around also have impact on the environment Expectations: There is major impact to the business please help to provide a Hot fix immediately According to Candido from the verification on latest puddle the problem been solved.
Memory usage of the neutron agents is not increasing over the time.
[root@controller-0 heat-admin]# while true; do top -b -p 117507 -p 117769 -p 118533 -n 1 ; sleep 120 ;done
top - 08:59:36 up 8:53, 1 user, load average: 2.54, 1.77, 1.54
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 18.6 us, 13.6 sy, 0.0 ni, 67.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16266388 total, 203504 free, 7309772 used, 8753112 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 8026376 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117507 42435 20 0 290436 86404 6456 S 0.0 0.5 4:36.48 neutron-dhcp-ag
117769 42435 20 0 293328 87996 6528 S 0.0 0.5 0:23.25 neutron-l3-agen
118533 42435 20 0 328256 113656 7180 S 0.0 0.7 2:11.77 neutron-openvsw
top - 09:01:37 up 8:55, 1 user, load average: 2.05, 1.87, 1.60
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.0 us, 1.7 sy, 0.0 ni, 93.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16266388 total, 238980 free, 7269296 used, 8758112 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 8066968 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117507 42435 20 0 290436 86404 6456 S 0.0 0.5 4:37.57 neutron-dhcp-ag
117769 42435 20 0 293328 87996 6528 S 0.0 0.5 0:23.34 neutron-l3-agen
118533 42435 20 0 328256 113656 7180 S 0.0 0.7 2:12.28 neutron-openvsw
top - 09:03:37 up 8:57, 1 user, load average: 0.99, 1.54, 1.52
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.1 us, 1.7 sy, 0.0 ni, 93.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16266388 total, 236920 free, 7270580 used, 8758888 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 8065708 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117507 42435 20 0 290436 86404 6456 S 0.0 0.5 4:38.66 neutron-dhcp-ag
117769 42435 20 0 293328 87996 6528 S 0.0 0.5 0:23.42 neutron-l3-agen
118533 42435 20 0 328256 113656 7180 S 0.0 0.7 2:12.78 neutron-openvsw
top - 09:05:37 up 8:59, 1 user, load average: 1.49, 1.55, 1.52
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 33.3 us, 40.0 sy, 0.0 ni, 26.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16266388 total, 185996 free, 7320656 used, 8759736 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 8015512 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117507 42435 20 0 290436 86404 6456 S 0.0 0.5 4:39.76 neutron-dhcp-ag
117769 42435 20 0 293328 87996 6528 S 0.0 0.5 0:23.51 neutron-l3-agen
118533 42435 20 0 328256 113656 7180 S 0.0 0.7 2:13.29 neutron-openvsw
top - 09:07:37 up 9:01, 1 user, load average: 1.32, 1.49, 1.50
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 6.6 us, 1.6 sy, 0.0 ni, 91.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16266388 total, 260404 free, 7271696 used, 8734288 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 8064752 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117507 42435 20 0 290436 86404 6456 S 0.0 0.5 4:40.87 neutron-dhcp-ag
117769 42435 20 0 293328 87996 6528 S 0.0 0.5 0:23.60 neutron-l3-agen
118533 42435 20 0 328256 113656 7180 S 0.0 0.7 2:13.79 neutron-openvsw
top - 09:09:37 up 9:03, 1 user, load average: 2.03, 1.57, 1.52
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 9.8 us, 6.6 sy, 0.0 ni, 83.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16266388 total, 259916 free, 7271488 used, 8734984 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 8064728 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117507 42435 20 0 290436 86404 6456 S 0.0 0.5 4:41.95 neutron-dhcp-ag
117769 42435 20 0 293328 87996 6528 S 0.0 0.5 0:23.67 neutron-l3-agen
118533 42435 20 0 328256 113656 7180 S 0.0 0.7 2:14.30 neutron-openvsw
top - 09:11:37 up 9:05, 1 user, load average: 1.58, 1.63, 1.56
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 13.1 us, 18.0 sy, 0.0 ni, 67.2 id, 0.0 wa, 0.0 hi, 1.6 si, 0.0 st
KiB Mem : 16266388 total, 260580 free, 7270488 used, 8735320 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 8065880 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117507 42435 20 0 290436 86404 6456 S 6.7 0.5 4:43.05 neutron-dhcp-ag
117769 42435 20 0 293328 87996 6528 S 0.0 0.5 0:23.76 neutron-l3-agen
118533 42435 20 0 328256 113656 7180 S 0.0 0.7 2:14.91 neutron-openvsw
^C
[root@controller-0 heat-admin]# while true; do top -b -p 117507 -p 117769 -p 118533 -n 1 ; sleep 120 ;done
top - 09:44:29 up 9:37, 1 user, load average: 1.93, 1.61, 1.49
Tasks: 3 total, 0 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 8.2 us, 8.2 sy, 0.0 ni, 83.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16266388 total, 260112 free, 7278244 used, 8728032 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 8057928 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117507 42435 20 0 290436 86404 6456 S 0.0 0.5 5:01.05 neutron-dhcp-ag
117769 42435 20 0 293584 87996 6528 S 0.0 0.5 0:25.13 neutron-l3-agen
118533 42435 20 0 328256 113656 7180 S 0.0 0.7 2:23.30 neutron-openvsw
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0552 |