Bug 1466019 - Seeing DBDeadlock on neutron server log when attempting to create networks and subnets
Seeing DBDeadlock on neutron server log when attempting to create networks an...
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
10.0 (Newton)
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Assaf Muller
Toni Freger
Depends On:
  Show dependency treegraph
Reported: 2017-06-28 13:43 EDT by bigswitch
Modified: 2018-06-23 21:58 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description bigswitch 2017-06-28 13:43:49 EDT
Description of problem:
We are trying to scale RHOSP 10 with 900 projects, one logical router per project, and four network with one subnet each per tenant with a script.
The setup has three controller and one compute node deployed. Each controller has 48 cores and 380gig ram.
The script creates 900 tenants first, than attempt to create 900 logical routers and fail at about 150 logical routers. It than attempts to create 4 networks per tenant and at about 400 networks/subnets it is seeing 504 gateway timeout error. There is also DBDeadlock error in neutron-server log.
Would like to know if there is any settings we can tweak to improve the database performance.

Version-Release number of selected component (if applicable):

How reproducible:

Will upload sosreport from three controllers to box.
Comment 1 bigswitch 2017-06-28 13:45:22 EDT
The sosreport is uploaded to box

Comment 2 Chris Jones 2017-09-20 11:22:26 EDT
Reassigning to DFG:Networking - this probably isn't related to PIDONE, but please let us know if you disagree.
Comment 3 Assaf Muller 2017-09-27 15:41:23 EDT
Any initial insight here?
Comment 5 Joe Talerico 2017-10-18 07:52:53 EDT
jtaleric  …  var  log  neutron  ls
total 12K
drwxr-xr-x. 23 jtaleric jtaleric 4.0K Jun 27 02:34 ..
drwxr-x---.  2 jtaleric jtaleric 4.0K Jun 28 06:34 .
lrwxrwxrwx.  1 jtaleric jtaleric   76 Jun 28 13:34 dhcp-agent.log -> ../../../sos_strings/openstack_neutron/var.log.neutron.dhcp-agent.log.tailed
 jtaleric  …  var  log  neutron  pwd

Reviewing the sos, I do not see the neutron-server logs.

Looking at the mariadb logs, did something shut down db during the test?

170626 23:34:41  InnoDB: Starting shutdown...
170626 23:34:42  InnoDB: Shutdown completed; log sequence number 1598129
170626 23:34:42 [Note] /usr/libexec/mysqld: Shutdown complete

With the BSN Plugin, do you leave l3_ha enabled?

Note You need to log in before you can comment on or make changes to this bug.