Bug 1466019

Summary:	Seeing DBDeadlock on neutron server log when attempting to create networks and subnets
Product:	Red Hat OpenStack	Reporter:	bigswitch <rhosp-bugs-internal>
Component:	openstack-neutron	Assignee:	Assaf Muller <amuller>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Toni Freger <tfreger>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	10.0 (Newton)	CC:	amuller, chjones, chrisw, jjoyce, jschluet, jtaleric, slinaber, smalleni, srevivo, tvignaud
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-04-24 12:23:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description bigswitch 2017-06-28 17:43:49 UTC

Description of problem:
We are trying to scale RHOSP 10 with 900 projects, one logical router per project, and four network with one subnet each per tenant with a script.
The setup has three controller and one compute node deployed. Each controller has 48 cores and 380gig ram.
The script creates 900 tenants first, than attempt to create 900 logical routers and fail at about 150 logical routers. It than attempts to create 4 networks per tenant and at about 400 networks/subnets it is seeing 504 gateway timeout error. There is also DBDeadlock error in neutron-server log.
Would like to know if there is any settings we can tweak to improve the database performance.

Version-Release number of selected component (if applicable):
RHOSP 10

How reproducible:
Always

Will upload sosreport from three controllers to box.

Comment 1 bigswitch 2017-06-28 17:45:22 UTC

The sosreport is uploaded to box

https://bigswitch.box.com/s/ujvfkhxs7gpsuirr09sujub5w9714nl2

Comment 2 Chris Jones 2017-09-20 15:22:26 UTC

Reassigning to DFG:Networking - this probably isn't related to PIDONE, but please let us know if you disagree.

Comment 3 Assaf Muller 2017-09-27 19:41:23 UTC

Any initial insight here?

Comment 5 Joe Talerico 2017-10-18 11:52:53 UTC

jtaleric  …  var  log  neutron  ls
total 12K
drwxr-xr-x. 23 jtaleric jtaleric 4.0K Jun 27 02:34 ..
drwxr-x---.  2 jtaleric jtaleric 4.0K Jun 28 06:34 .
lrwxrwxrwx.  1 jtaleric jtaleric   76 Jun 28 13:34 dhcp-agent.log -> ../../../sos_strings/openstack_neutron/var.log.neutron.dhcp-agent.log.tailed
 jtaleric  …  var  log  neutron  pwd
/home/jtaleric/tmp/bigswitch/sosreport-overcloud-controller-0.localdomain-20170628102811/var/log/neutron

Reviewing the sos, I do not see the neutron-server logs.

Looking at the mariadb logs, did something shut down db during the test?

170626 23:34:41  InnoDB: Starting shutdown...
170626 23:34:42  InnoDB: Shutdown completed; log sequence number 1598129
170626 23:34:42 [Note] /usr/libexec/mysqld: Shutdown complete

With the BSN Plugin, do you leave l3_ha enabled?

Comment 6 Sai Sindhur Malleni 2019-04-24 12:23:08 UTC

No activity for a long time, closing this. Please re-open if this is still an issue.