Description of problem: The default value for max_connections (150) is too low for an HA environment on OSP6. Testing on a 3 controller environment, with 24 CPUs each, we see more than 300 connections being opened to the database. With the default max_connections value, we get errors like the following: 2015-02-11 11:47:08.349 2770 TRACE nova.servicegroup.drivers.db OperationalError: (OperationalError) (1040, 'Too many connections') None None Some services, like nova-conductor or keystone, spawn by default 1 worker per CPU, and each worker opens its own DB connection. In the above mentioned environment, it means 144 connections just for those two services. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. Create an HA environment with multi-CPU nodes (e.g. 24) 2. From the Horizon UI, try to spawn 10 VMs Actual results: Errors in log files, Galera IP switched over in HAProxy Expected results: No errors. Additional info: Setting max_connections=1024 in /etc/my.cnf.d/galera.cnf works fine for me. Note this requires increasing the file limit for the Galera service, setting "LimitNOFILE=16384" or similar.
It seems that we're setting max_connections=1024 already. We're investigating the LimitNOFILE option, but we don't appear to have a way to set this already. Still investigating...
https://mariadb.com/kb/en/mariadb/server-system-variables/#max_connections It can be set up to 100000. Not sure if it has other impacts.
(In reply to Javier Peña from comment #0) > Description of problem: > The default value for max_connections (150) is too low for an HA environment > on OSP6. Testing on a 3 controller environment, with 24 CPUs each, we see > more than 300 connections being opened to the database. With the default > max_connections value, we get errors like the following: > > 2015-02-11 11:47:08.349 2770 TRACE nova.servicegroup.drivers.db > OperationalError: (OperationalError) (1040, 'Too many connections') None None > > Some services, like nova-conductor or keystone, spawn by default 1 worker > per CPU, and each worker opens its own DB connection. In the above mentioned > environment, it means 144 connections just for those two services. > > Version-Release number of selected component (if applicable): > > > How reproducible: always > > > Steps to Reproduce: > 1. Create an HA environment with multi-CPU nodes (e.g. 24) > 2. From the Horizon UI, try to spawn 10 VMs > > Actual results: > Errors in log files, Galera IP switched over in HAProxy > > Expected results: > No errors. > > Additional info: > Setting max_connections=1024 in /etc/my.cnf.d/galera.cnf works fine for me. > Note this requires increasing the file limit for the Galera service, setting > "LimitNOFILE=16384" or similar. Javier, with a quickstack-deployed setup, I have: /etc/my.cnf.d/server.cnf: max_connections = 1024 Did yours not have this already? As for LimitNOFILE, where exactly did you set that? I am not sure is this is something we can currently do in puppet or not, as I don't know where we want this setting.
Oh, and fwiw, the max_connections parameter is configurable already via quickstack::pacemaker::galera, so I suspect this may boil down to the LimitNOFILE part.
The problem is that even 1024 is too low. Starting a PoC setup with 3 controller nodes and 3 compute nodes already brings you up to > 130 connections (48 of which are nova-conductor from the compute nodes). In a slight medium scale environment, even 1024 are too low. I strongly recommend to: 1) short term solution, max out mariadb setting as pointed out above 2) medium/long term solution, we need to find a way to contain the connections below the max that mariadb allows. I see little point to have each and every nova-conductor creating their own connection to the db.
LimitNOFILE needs to be set on the systemd side, to allow MariaDB to increase its file limit high enough to have max_connections = 1024. With the default setting, it can only raise max_connections to 214 iirc, regardless of the setting in server.cnf. So, the simplest way for this would be to create a file named /etc/systemd/system/mariadb.service.d/limits.conf with the following contents: [Service] LimitNOFILE=16384 Then: systemctl daemon-reload systemctl restart mariadb
(In reply to Fabio Massimo Di Nitto from comment #13) > The problem is that even 1024 is too low. > > Starting a PoC setup with 3 controller nodes and 3 compute nodes already > brings you up to > 130 connections (48 of which are nova-conductor from the > compute nodes). > In my test setup (3 controller nodes, 24 logical CPUs each) I have seen up to ~500 connections. Since in Juno the default is to have one worker per CPU for most services, the number can grow quite rapidly, so yes, we should figure out a good default. Setting LimitNOFILE to 16384 should give us enough room to grow max_open_connections higher if needed. In theory, the number of compute nodes should not change this, unless we are running nova-conductor on each compute node.
After a discussion on IRC with Jason, I realized setting LimitNOFILE on the systemd side will not help in all cases. When running through Pacemaker, MariaDB inherits any ulimit set to Pacemaker, which seems to be higher than the default, allowing max_connections to stay on 1024. This would not help on a single controller deployment where the controller node has a lot of CPUs, though, but it definitely makes this situation less likely to happen. Fabio, do you know how we can set a different ulimit for Pacemaker resources?
(In reply to Javier Peña from comment #16) > After a discussion on IRC with Jason, I realized setting LimitNOFILE on the > systemd side will not help in all cases. When running through Pacemaker, > MariaDB inherits any ulimit set to Pacemaker, which seems to be higher than > the default, allowing max_connections to stay on 1024. > > This would not help on a single controller deployment where the controller > node has a lot of CPUs, though, but it definitely makes this situation less > likely to happen. > > Fabio, do you know how we can set a different ulimit for Pacemaker resources? No, but David would.
(In reply to Fabio Massimo Di Nitto from comment #17) > (In reply to Javier Peña from comment #16) > > After a discussion on IRC with Jason, I realized setting LimitNOFILE on the > > systemd side will not help in all cases. When running through Pacemaker, > > MariaDB inherits any ulimit set to Pacemaker, which seems to be higher than > > the default, allowing max_connections to stay on 1024. > > > > This would not help on a single controller deployment where the controller > > node has a lot of CPUs, though, but it definitely makes this situation less > > likely to happen. > > > > Fabio, do you know how we can set a different ulimit for Pacemaker resources? > > No, but David would. I'd want to do it in the agent through an argument. Unless there is a way to get the mysqld_safe script to set ulimit using a local configuration file, the only way we'll be able to set ulimit is by modifying the galera resource-agent. -- David
Given that we have the settings already available now and are configurable in advanced configuration, I'm moving this out to A2.
(In reply to Mike Burns from comment #19) > Given that we have the settings already available now and are configurable > in advanced configuration, I'm moving this out to A2. so, just to be clear, is the galera resource-agent change is still necessary? If so should this issue be moved to resource-agents and have the priority lowered since there appears to be a work around?
I think we can set the values we need, but we might hit something like this in the future. As I see it in this bz, there are 3 things: 1. ulimit in resource-agent 2. open files limit (-1) 3. max connections (1024) Both 2 and 3 are configurable now in puppet, though we may need to increase defaults at some point for max connections. IIUC, the real max open files is the smaller of the open files limit and the ulimit in the resource-agent. -1 means no limit, so we're bounded by the resource-agent. I think we determined that the current defaults work, so there isn't a time pressure, so I'd leave it to your judgement if you think the RA needs to be changed.
David, I don't think there is any puppet work needed here based on the conversation. There seems to be some need for larger deployments to get a higher ulimit via the RA though, unless I am misunderstanding. I proposed we move this BZ to that component, do you agree? Not sure on release version needed.
For a single controller scenario, if Pacemaker is not in use, we'd still need to increase LimitNOFILE on the systemd side to avoid issues in servers with a large amount of CPUs.
(In reply to Jason Guiditta from comment #23) > David, I don't think there is any puppet work needed here based on the > conversation. There seems to be some need for larger deployments to get a > higher ulimit via the RA though, unless I am misunderstanding. I proposed > we move this BZ to that component, do you agree? Not sure on release > version needed. If there is going to galera resource-agent change, it will involve a new option to the agent specify the ulimit count. This will involve puppet integration will it not? If so, we should clone this bug to the resource-agents for the galera agent changes, and leave this issue for the puppet integration part. -- David
(In reply to David Vossel from comment #25) > (In reply to Jason Guiditta from comment #23) > > David, I don't think there is any puppet work needed here based on the > > conversation. There seems to be some need for larger deployments to get a > > higher ulimit via the RA though, unless I am misunderstanding. I proposed > > we move this BZ to that component, do you agree? Not sure on release > > version needed. > > If there is going to galera resource-agent change, it will involve a new > option to the agent specify the ulimit count. This will involve puppet > integration will it not? If so, we should clone this bug to the > resource-agents for the galera agent changes, and leave this issue for the > puppet integration part. > > -- David Yes, looking at quickstack puppet, we would need to update quickstack::pacemaker::resource::galera to call any such new flag.
(In reply to Javier Peña from comment #24) > For a single controller scenario, if Pacemaker is not in use, we'd still > need to increase LimitNOFILE on the systemd side to avoid issues in servers > with a large amount of CPUs. Even for single controller, we use pacemaker (for all supported deployments). The older, non-pacemaker classes are only there for legacy usage, no new deployment should use them. That said, I think if mysql needed such config, it should be via puppet-mysql module rather than implemented here (though of course we would have to call it, as with pacemaker RA)
(In reply to David Vossel from comment #25) > (In reply to Jason Guiditta from comment #23) > > David, I don't think there is any puppet work needed here based on the > > conversation. There seems to be some need for larger deployments to get a > > higher ulimit via the RA though, unless I am misunderstanding. I proposed > > we move this BZ to that component, do you agree? Not sure on release > > version needed. > > If there is going to galera resource-agent change, it will involve a new > option to the agent specify the ulimit count. This will involve puppet > integration will it not? If so, we should clone this bug to the > resource-agents for the galera agent changes, and leave this issue for the > puppet integration part. > > -- David David, Do you want me to clone this somewhere to track the resource agent change? What timeframe might we be able to get that in? If we moved the bug here to our next release (A3), would that allow enough time? I can't imagine it could be done in the next couple days, scheduling-wise.
deferring until the resource-agents bug is fixed
David, thanks for the good news (and detail how to test)! I am implementing the needed puppet changes now. I am not sure who would know this answer, but I am setting 'reasonable defaults' to what Javier originally mentioned plus what we already had, namely: LimitNOFILE ="16384" max_connections = "1024" open_files_limit = '-1'
Patch posted: https://github.com/redhat-openstack/astapor/pull/490 David, output there, but I think it looks right, given my default: Max open files 16384 16384 files Javier, does this seem reasonable to you as well? This can also now be altered by the user to suit their environment via the limit_no_file parameter in quickstack::pacemaker::galera via the UI or scripts.
(In reply to Jason Guiditta from comment #36) > Patch posted: > > https://github.com/redhat-openstack/astapor/pull/490 > > David, output there, but I think it looks right, given my default: patch looks correct to me. > Max open files 16384 16384 files > > Javier, does this seem reasonable to you as well? This can also now be > altered by the user to suit their environment via the limit_no_file > parameter in quickstack::pacemaker::galera via the UI or scripts.
(In reply to Jason Guiditta from comment #36) > Patch posted: > > https://github.com/redhat-openstack/astapor/pull/490 > > David, output there, but I think it looks right, given my default: > Max open files 16384 16384 files > > Javier, does this seem reasonable to you as well? This can also now be > altered by the user to suit their environment via the limit_no_file > parameter in quickstack::pacemaker::galera via the UI or scripts. Looks good to me. Also, having it as a configurable parameter will allow us to change it in the future if needed.
cwolfe merged
verified on A2. [root@mac441ea1733d43 ~]# cat /proc/$(pidof mysqld)/limits | grep "Max open files" Max open files 16384 16384 files also i managed to create 30 vms on the setup. [root@mac441ea1733d43 ~]# rpm -qa |grep maria mariadb-galera-common-5.5.40-6.el7ost.x86_64 mariadb-galera-server-5.5.40-6.el7ost.x86_64 mariadb-5.5.41-2.el7_0.x86_64 mariadb-libs-5.5.41-2.el7_0.x86_64 rhel-osp-installer-client-0.5.7-1.el7ost.noarch foreman-installer-1.6.0-0.3.RC1.el7ost.noarch openstack-foreman-installer-3.0.17-1.el7ost.noarch rhel-osp-installer-0.5.7-1.el7ost.noarch puppet-3.6.2-2.el7.noarch puppet-server-3.6.2-2.el7.noarch openstack-puppet-modules-2014.2.8-2.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0791.html