Bug 1251566

Summary: Undercloud mariadb max_connection default is too low
Product: Red Hat OpenStack Reporter: jliberma <jliberma>
Component: instack-undercloudAssignee: James Slagle <jslagle>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: calfonso, dmacpher, ebagdasa, ggillies, kbasil, kprabhak, mburns, morazi, rhel-osp-director-maint
Target Milestone: y1Keywords: Field, ZStream
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: instack-undercloud-2.1.2-24.el7ost Doc Type: Bug Fix
Doc Text:
The director's database (MariaDB) only accept a maximum of 1024 connections. An Undercloud with a high number of CPU cores (typically 24 or more) exhausted these database connections due to the number of OpenStack API workers spawned. This fix configures the Undercloud to accept 4096 connections for MariaDB. All services now connect to MariaDB when needed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-08 12:16:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description jliberma@redhat.com 2015-08-07 18:26:45 UTC
Description of problem:

Related to https://bugzilla.redhat.com/show_bug.cgi?id=1247924#c13

MariaDB on undercloud controller max_connections should be based on core count or connection limits will be exceeded during large, multiple deployments.

Version-Release number of selected component (if applicable):

python-rdomanager-oscplugin-0.0.8-44.el7ost.noarch


How reproducible:


Steps to Reproduce:
1. Deploy undercloud
2. mysql -e "SHOW GLOBAL VARIABLES LIKE 'max_connections'"

Actual results:


Expected results:


Additional info: 

Workaround to change on already deployed undercloud:

[root@rhos0 ~]# mysql -e "SHOW STATUS WHERE variable_name = 'Threads_connected'"
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 519   |
+-------------------+-------+

[root@rhos0 ~]# mysql -e "SET GLOBAL max_connections = 4096"

[root@rhos0 ~]# mysql -e "SHOW GLOBAL VARIABLES LIKE 'max_connections'"
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 4096  |
+-----------------+-------+

Workaround to change for restarts and boot:

[root@rhos0 ~]# sed -i 's/max_connections =.*$/max_connections = 4096/' /etc/my.cnf.d/server.cnf 

[root@rhos0 ~]# grep max_connections /etc/my.cnf.d/server.cnf 
max_connections = 4096

NOTE: Normally HAproxy max_conn should be greater than or equal to MariaDB max_connections, but no HAproxy on undercloud so max_conn does not need to be adjusted.

Comment 4 Marius Cornea 2015-09-07 14:25:08 UTC
[stack@instack ~]$ rpm -qa | grep instack-undercloud
instack-undercloud-2.1.2-25.el7ost.noarch
[stack@instack ~]$ mysql -e "SHOW GLOBAL VARIABLES LIKE 'max_connections'"
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 4096  |
+-----------------+-------+

Comment 5 chris alfonso 2015-09-17 16:44:14 UTC
*** Bug 1263552 has been marked as a duplicate of this bug. ***

Comment 7 errata-xmlrpc 2015-10-08 12:16:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1862