Bug 1676971 - Open files counts for octavia-api grow dramatically under normal operation
Summary: Open files counts for octavia-api grow dramatically under normal operation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-octavia
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z3
: 14.0 (Rocky)
Assignee: Carlos Goncalves
QA Contact: Bruna Bonguardo
URL:
Whiteboard:
Depends On:
Blocks: 1676975
TreeView+ depends on / blocked
 
Reported: 2019-02-13 17:24 UTC by Carlos Goncalves
Modified: 2019-09-10 14:09 UTC (History)
6 users (show)

Fixed In Version: openstack-octavia-3.1.0-2.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the octavia-api package would orphan AMQP connections leading to a large number of open files accumulating on the file system. This patch fixes that issue.
Clone Of:
: 1676975 (view as bug list)
Environment:
Last Closed: 2019-07-02 19:47:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 636428 0 None None None 2019-02-13 17:24:31 UTC
OpenStack gerrit 638454 0 None None None 2019-05-13 15:53:34 UTC
Red Hat Product Errata RHBA-2019:1680 0 None None None 2019-07-02 19:47:54 UTC
Storyboard 2004993 0 None None None 2019-02-13 17:24:31 UTC

Description Carlos Goncalves 2019-02-13 17:24:32 UTC
It was discovered that the octavia-api process, that uses the oslo.messaging package, is orphaning amqp connections with no effort to gc old connections. This leads to a large number of open files accumulating rapidly on the host filesystem and an overall negative impact on system load and resource utilization.

Original report: https://storyboard.openstack.org/#!/story/2004993

How reproducible:
100%

Steps to Reproduce:
1. trigger dozens of all sorts of CRUD operations
2. monitor CPU and memory usage

Actual results:
CPU and virtual memory usage of octavia-api container grow with the number of LB control-plane operations.

Expected results:
CPU and memory usage should remain mostly flat with time and operations.

Additional info:
This issue was also observed on Rally runs.

Comment 6 errata-xmlrpc 2019-07-02 19:47:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1680


Note You need to log in before you can comment on or make changes to this bug.