Red Hat Bugzilla – Bug 1466019
Seeing DBDeadlock on neutron server log when attempting to create networks and subnets
Last modified: 2018-01-25 22:01:27 EST
Description of problem:
We are trying to scale RHOSP 10 with 900 projects, one logical router per project, and four network with one subnet each per tenant with a script.
The setup has three controller and one compute node deployed. Each controller has 48 cores and 380gig ram.
The script creates 900 tenants first, than attempt to create 900 logical routers and fail at about 150 logical routers. It than attempts to create 4 networks per tenant and at about 400 networks/subnets it is seeing 504 gateway timeout error. There is also DBDeadlock error in neutron-server log.
Would like to know if there is any settings we can tweak to improve the database performance.
Version-Release number of selected component (if applicable):
Will upload sosreport from three controllers to box.
The sosreport is uploaded to box
Reassigning to DFG:Networking - this probably isn't related to PIDONE, but please let us know if you disagree.
Any initial insight here?
jtaleric … var log neutron ls
drwxr-xr-x. 23 jtaleric jtaleric 4.0K Jun 27 02:34 ..
drwxr-x---. 2 jtaleric jtaleric 4.0K Jun 28 06:34 .
lrwxrwxrwx. 1 jtaleric jtaleric 76 Jun 28 13:34 dhcp-agent.log -> ../../../sos_strings/openstack_neutron/var.log.neutron.dhcp-agent.log.tailed
jtaleric … var log neutron pwd
Reviewing the sos, I do not see the neutron-server logs.
Looking at the mariadb logs, did something shut down db during the test?
170626 23:34:41 InnoDB: Starting shutdown...
170626 23:34:42 InnoDB: Shutdown completed; log sequence number 1598129
170626 23:34:42 [Note] /usr/libexec/mysqld: Shutdown complete
With the BSN Plugin, do you leave l3_ha enabled?