Bug 1323728

Summary: Deployment on barematal failed due to max_connections limit on mysql on the undercloud
Product: Red Hat OpenStack Reporter: Udi Shkalim <ushkalim>
Component: instack-undercloudAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: achernet, dbecker, fj-lsoft-ofuku, fzdarsky, jcoufal, jstransk, lnatapov, mburns, mcornea, morazi, rhel-osp-director-maint, rlandy, sasha, ushkalim
Target Milestone: ga   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: instack-undercloud-2.2.7-3.el7ost Doc Type: Known Issue
Doc Text:
Cause: A bad setting on the undercloud mysql server limits the maximum number of connections Consequence: Most deployments will fail Workaround (if any): set the open_files_limit to -1 in /etc/my.cnf.d/server.cnf Result: deployments should succeed.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-15 14:32:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1214764, 1274548    
Attachments:
Description Flags
mariadb.log none

Description Udi Shkalim 2016-04-04 14:11:22 UTC 
Description of problem:
When trying to deploy 3 controllers 1 compute on a baremetal setup got the error: http://pastebin.test.redhat.com/362092

Got the error: "OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')"  - http://pastebin.test.redhat.com/362093

Got the following on mariadb.log:
160404 15:14:45 [Note] /usr/libexec/mysqld (mysqld 5.5.47-MariaDB) starting as process 108818 ...
160404 15:14:45 [Warning] Changed limits: max_open_files: 1024  max_connections: 214  table_cache: 400

Version-Release number of selected component (if applicable):
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy ospd8 on BM 3 controllers 1 compute
2.
3.

Actual results:
 deployment failed with OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections'

Expected results:
deployment passed

Additional info:
the actual max_connections value was 214.
This might be related to the change that introduced in this puddle - https://github.com/openstack/instack-undercloud/commit/baf3a0b20b7df1545d0697672dd6293fad6e2991
Comment 2 Ronelle Landy 2016-04-04 14:20:30 UTC 
This error is failing our CI.

I see the same value of 214 connections:

[stack@host15 ~]$ mysql -e "SHOW GLOBAL VARIABLES LIKE 'max_connections'"
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 214   |

and yet /etc/my.cnf* files had the correct 4096 value set.
Comment 3 Udi Shkalim 2016-04-04 14:20:50 UTC 
Created attachment 1143368 [details]
mariadb.log
Comment 4 James Slagle 2016-04-04 14:48:18 UTC 
can i get access to the environment, or all undercloud logs somehow (at least the undercloud install log)?

you could also try checking the connections, i know you said it was already set, but i'd like to see the config file:

grep -rin connections /etc/my.cnf* 

or attach the contents of /etc/my.cnf.d/

also, mariadb versions:
rpm -qa | grep maria
Comment 8 Alexander Chuzhoy 2016-04-13 21:31:24 UTC 
Verified:
Environment:
instack-undercloud-2.2.7-4.el7ost.noarch

Successfully deployed 8.0 on BM:
3 controllers, 2 computes, 2 ceph nodes.
Comment 10 errata-xmlrpc 2016-04-15 14:32:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0637.html
Comment 11 Sergey Gotliv 2016-05-02 22:39:46 UTC 
*** Bug 1330803 has been marked as a duplicate of this bug. ***