Bug 1545939

Summary: neutron lbaas commands take a long time to complete
Product: Red Hat OpenStack Reporter: nalmond
Component: openstack-neutron-lbaasAssignee: Rodolfo Alonso <ralonsoh>
Status: CLOSED ERRATA QA Contact: Alexander Stafeyev <astafeye>
Severity: high Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: amuller, apevec, astafeye, cgoncalves, dalvarez, ebarrera, jschluet, lhh, mburns, molasaga, mschuppe, nyechiel, pcaruana, pmorey, ralonsoh
Target Milestone: z11Keywords: Reopened, Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-lbaas-9.2.2-6.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1546710 1546711 1619178 1648821 (view as bug list) Environment:
Last Closed: 2019-01-08 16:02:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1381612, 1523064, 1546710, 1546711, 1619178, 1648821, 1669986    

Description nalmond 2018-02-15 21:50:53 UTC
Description of problem:
Several lbaas commands are taking longer than expected to complete, this time increases as more loadbalancers or pools are added.

In the production environment, 'neutron lbaas-loadbalancer-list' is taking 1.5 minutes to complete. Running 'neutron lbaas-loadbalancer-show <lb>' on a loadbalancer with 1 pool takes about 1.5 seconds. Running the same command on a loadbalancer with 7 pools takes about 5.5 seconds.

In the neutron debug logs, a large number of mysql queries being run for these operations can be seen.

Version-Release number of selected component (if applicable):
openstack-neutron-lbaas-9.2.1-6.el7ost.noarch

How reproducible:
Consistently, but becomes more apparent with more loadbalancers/pools

Steps to Reproduce:
1. create multiple loadbalancers and pools
2. run lbaas cli commands

Actual results:
Commands take unacceptably long with more loadbalancers present

Expected results:
Commands should complete more quickly even as scale increases

Additional info:
The F5 plugin is being used, but that does not seem to be the culprit at this point.

This looks very similar to https://bugs.launchpad.net/octavia/+bug/1706466, if this is the issue, can this fix be backported?

Comment 17 Eduard Barrera 2018-03-29 13:44:16 UTC
Created attachment 1414775 [details]
benchmarks.

Comment 32 errata-xmlrpc 2018-05-17 15:51:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1603

Comment 46 Nir Magnezi 2018-11-12 12:08:51 UTC
Fixes included here: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=799021