Bug 1323728 - Deployment on barematal failed due to max_connections limit on mysql on the undercloud
Summary: Deployment on barematal failed due to max_connections limit on mysql on the u...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Linux
unspecified
urgent
Target Milestone: ga
: 8.0 (Liberty)
Assignee: Jiri Stransky
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
: 1330803 (view as bug list)
Depends On:
Blocks: 1214764 1274548
TreeView+ depends on / blocked
 
Reported: 2016-04-04 14:11 UTC by Udi Shkalim
Modified: 2016-05-02 22:39 UTC (History)
14 users (show)

Fixed In Version: instack-undercloud-2.2.7-3.el7ost
Doc Type: Known Issue
Doc Text:
Cause: A bad setting on the undercloud mysql server limits the maximum number of connections Consequence: Most deployments will fail Workaround (if any): set the open_files_limit to -1 in /etc/my.cnf.d/server.cnf Result: deployments should succeed.
Clone Of:
Environment:
Last Closed: 2016-04-15 14:32:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
mariadb.log (4.52 KB, text/plain)
2016-04-04 14:20 UTC, Udi Shkalim
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1565828 0 None None None 2016-04-04 15:00:04 UTC
OpenStack gerrit 301168 0 None MERGED Set mariadb open_files_limit to -1 (unlimited) 2020-07-03 02:41:55 UTC
OpenStack gerrit 301518 0 None MERGED Set mariadb open_files_limit to -1 (unlimited) 2020-07-03 02:41:55 UTC
Red Hat Product Errata RHBA-2016:0637 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 8 director release candidate Bug Fix Advisory 2016-04-15 18:28:05 UTC

Description Udi Shkalim 2016-04-04 14:11:22 UTC
Description of problem:
When trying to deploy 3 controllers 1 compute on a baremetal setup got the error: http://pastebin.test.redhat.com/362092

Got the error: "OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')"  - http://pastebin.test.redhat.com/362093

Got the following on mariadb.log:
160404 15:14:45 [Note] /usr/libexec/mysqld (mysqld 5.5.47-MariaDB) starting as process 108818 ...
160404 15:14:45 [Warning] Changed limits: max_open_files: 1024  max_connections: 214  table_cache: 400

Version-Release number of selected component (if applicable):
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy ospd8 on BM 3 controllers 1 compute
2.
3.

Actual results:
 deployment failed with OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections'

Expected results:
deployment passed

Additional info:
the actual max_connections value was 214.
This might be related to the change that introduced in this puddle - https://github.com/openstack/instack-undercloud/commit/baf3a0b20b7df1545d0697672dd6293fad6e2991

Comment 2 Ronelle Landy 2016-04-04 14:20:30 UTC
This error is failing our CI.

I see the same value of 214 connections:

[stack@host15 ~]$ mysql -e "SHOW GLOBAL VARIABLES LIKE 'max_connections'"
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 214   |

and yet /etc/my.cnf* files had the correct 4096 value set.

Comment 3 Udi Shkalim 2016-04-04 14:20:50 UTC
Created attachment 1143368 [details]
mariadb.log

Comment 4 James Slagle 2016-04-04 14:48:18 UTC
can i get access to the environment, or all undercloud logs somehow (at least the undercloud install log)?

you could also try checking the connections, i know you said it was already set, but i'd like to see the config file:

grep -rin connections /etc/my.cnf* 

or attach the contents of /etc/my.cnf.d/

also, mariadb versions:
rpm -qa | grep maria

Comment 8 Alexander Chuzhoy 2016-04-13 21:31:24 UTC
Verified:
Environment:
instack-undercloud-2.2.7-4.el7ost.noarch

Successfully deployed 8.0 on BM:
3 controllers, 2 computes, 2 ceph nodes.

Comment 10 errata-xmlrpc 2016-04-15 14:32:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0637.html

Comment 11 Sergey Gotliv 2016-05-02 22:39:46 UTC
*** Bug 1330803 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.