Bug 1676971

Summary: Open files counts for octavia-api grow dramatically under normal operation
Product: Red Hat OpenStack Reporter: Carlos Goncalves <cgoncalves>
Component: openstack-octaviaAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED ERRATA QA Contact: Bruna Bonguardo <bbonguar>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: bbonguar, ihrachys, lpeer, majopela, rheslop, tvignaud
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-octavia-3.1.0-2.el7ost Doc Type: Bug Fix
Doc Text:
Previously, the octavia-api package would orphan AMQP connections leading to a large number of open files accumulating on the file system. This patch fixes that issue.
 Story Points: ---
Clone Of:
: 1676975 (view as bug list) Environment:
Last Closed: 2019-07-02 19:47:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1676975    

Description Carlos Goncalves 2019-02-13 17:24:32 UTC 
It was discovered that the octavia-api process, that uses the oslo.messaging package, is orphaning amqp connections with no effort to gc old connections. This leads to a large number of open files accumulating rapidly on the host filesystem and an overall negative impact on system load and resource utilization.

Original report: https://storyboard.openstack.org/#!/story/2004993

How reproducible:
100%

Steps to Reproduce:
1. trigger dozens of all sorts of CRUD operations
2. monitor CPU and memory usage

Actual results:
CPU and virtual memory usage of octavia-api container grow with the number of LB control-plane operations.

Expected results:
CPU and memory usage should remain mostly flat with time and operations.

Additional info:
This issue was also observed on Rally runs.
Comment 6 errata-xmlrpc 2019-07-02 19:47:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1680