Bug 1240826

I changed the max_connections using:

MariaDB [(none)]> set global max_connections =4096;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show variables like "max_connections";
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 4096  |
+-----------------+-------+
1 row in set (0.00 sec)

I tried to stop puppet and change the default in my.cnf and then restart maridb but the value stayed 1024.. also after "set" command the value is not permanent and a restart changed it back to 1024.. So im asking where is the right place to edit the values?

If this is truly against ofi and not osp-d, you can already set this value in quickstack::pacemaker::galera, via the max_connections parameter.  Note that we also have an open_files_limit parameter, which defaults to -1, meaning galera should use whatever the system limit is.  Lastly, there is the limit_no_file param, which is passed into this same class and used bythe pacemaker resource agent to set the system file limit to an appropriate setting for your environment.  This was all fixed in Bug 1159390 in February in version 3.0.8 of ofi and Bug 1191519 in April in version 3.0.17 of ofi.  

I am confused what this bug is asking for exactly beyond that?  Is the request to automatically determine these settings somehow?  I think just being able to set them was deemed sufficient previously, has this changed?

If we are kicking up the number of connections, do we need to look at things like the amount of memory allocated as well ?

Each connection requires a certain amount of memory and can possibly use more than that while running queries.

(In reply to Jason Guiditta from comment #4)
> If this is truly against ofi and not osp-d, you can already set this value
> in quickstack::pacemaker::galera, via the max_connections parameter.  Note
> that we also have an open_files_limit parameter, which defaults to -1,
> meaning galera should use whatever the system limit is.  Lastly, there is
> the limit_no_file param, which is passed into this same class and used bythe
> pacemaker resource agent to set the system file limit to an appropriate
> setting for your environment.  This was all fixed in Bug 1159390 in February
> in version 3.0.8 of ofi and Bug 1191519 in April in version 3.0.17 of ofi.  
> 
> I am confused what this bug is asking for exactly beyond that?  Is the
> request to automatically determine these settings somehow?  I think just
> being able to set them was deemed sufficient previously, has this changed?

The long term is to make it automatically determined, but given that OFI's lifespan is limited at this point, I think it would be better to focus on the dynamic configuration of max connections for mariadb to be in RHEL OSP director only.

I think this bug on OFI should focus on making sure that:
a) the value of max_connections is configurable easily
b) the default value is set to something more suitable for a production 
   deployment, like 4096 or above

Your comment above indicates that the value of max_connections in mariadb uses the same value as open_files_limit, which is defaulted to -1. Does that mean that mariadb is truly configured to have 'unlimited' value for max_connections? Is that even possible?

Also, see my comment in https://bugzilla.redhat.com/show_bug.cgi?id=1240824#c14

We need to also check the HAProxy connections limit, which defaults to 2000. If that is left at the default, you'll likely hit that limit before hitting the database max_connections limit if you have a larger deployment (and the max_connections is set > 2048)

(In reply to Perry Myers from comment #8)
> (In reply to Jason Guiditta from comment #4)
> > If this is truly against ofi and not osp-d, you can already set this value
> > in quickstack::pacemaker::galera, via the max_connections parameter.  Note
> > that we also have an open_files_limit parameter, which defaults to -1,
> > meaning galera should use whatever the system limit is.  Lastly, there is
> > the limit_no_file param, which is passed into this same class and used bythe
> > pacemaker resource agent to set the system file limit to an appropriate
> > setting for your environment.  This was all fixed in Bug 1159390 in February
> > in version 3.0.8 of ofi and Bug 1191519 in April in version 3.0.17 of ofi.  
> > 
> > I am confused what this bug is asking for exactly beyond that?  Is the
> > request to automatically determine these settings somehow?  I think just
> > being able to set them was deemed sufficient previously, has this changed?
> 
> The long term is to make it automatically determined, but given that OFI's
> lifespan is limited at this point, I think it would be better to focus on
> the dynamic configuration of max connections for mariadb to be in RHEL OSP
> director only.
> 
> I think this bug on OFI should focus on making sure that:
> a) the value of max_connections is configurable easily
This is already the case, we have:
quickstack::pacemaker::galera::$max_connections         = "1024"

> b) the default value is set to something more suitable for a production 
>    deployment, like 4096 or above
We can do this, not a problem.

> 
> Your comment above indicates that the value of max_connections in mariadb
> uses the same value as open_files_limit, which is defaulted to -1. Does that
> mean that mariadb is truly configured to have 'unlimited' value for
> max_connections? Is that even possible?

No, I must have explained poorly, I will try again here.
* quickstack::pacemaker::galera::$open_files_limit - this is used to configure galera to use whatever the system limit is for open files allowed with the default value of -1 (distinct from number of connections, which I am sure you know, but just to be clear).  If a specific amount lower than the system setting is desired for galera itself, this can be set toa different number.
* quickstack::pacemaker::galera::$limit_no_file - This is the one that actually _sets_ the system limit for galera, via the pacemaker resource agent.
* quickstack::pacemaker::galera::$max_connections - The number of concurrent connections (vs open files) that galera is configured to allow (again, probably well known, just trynig to call out the distinctions here for clarity).

I will respond separately on your other comment to try to keep this more coherent.

(In reply to Perry Myers from comment #9)
> Also, see my comment in
> https://bugzilla.redhat.com/show_bug.cgi?id=1240824#c14
> 
> We need to also check the HAProxy connections limit, which defaults to 2000.
> If that is left at the default, you'll likely hit that limit before hitting
> the database max_connections limit if you have a larger deployment (and the
> max_connections is set > 2048)

The only connection limit we set is for the global maxconn setting, which is 10000 in ofi[1]. This issue looks to be tracked in a new bug from Ryan [2], so apparently this setting is not enough.  If what Giulio did for tripleo [3] is reasonable, we can do a variation of that for ofi.  Note that Ryan's BZ is against OSP 6, so I am not sure if we might need to clone it to track it into OFI's OSP 7 release separately.  If Giulio's settings are what we want, that is a relatively straight forward addition here as well.  Since our puppet is structured a bit differently that tripleo's, we could also do this by setting the haproxy connection limit for galera to whatever was passed in for max_connections (or some calculation based on that number), preventing an additional parameter being needed for the user to set.

[1] https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/load_balancer/common.pp#L9
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1244929
[3] https://review.openstack.org/#/c/202525/1

(In reply to Jason Guiditta from comment #11)
> (In reply to Perry Myers from comment #9)
> > Also, see my comment in
> > https://bugzilla.redhat.com/show_bug.cgi?id=1240824#c14
> > 
> > We need to also check the HAProxy connections limit, which defaults to 2000.
> > If that is left at the default, you'll likely hit that limit before hitting
> > the database max_connections limit if you have a larger deployment (and the
> > max_connections is set > 2048)
> 
> The only connection limit we set is for the global maxconn setting, which is
> 10000 in ofi[1].

We will probably need to increase this a bit (20000 or 40000) if we increase maxconn for the galera proxy.

> This issue looks to be tracked in a new bug from Ryan [2],
> so apparently this setting is not enough.  If what Giulio did for tripleo
> [3] is reasonable, we can do a variation of that for ofi.  Note that Ryan's
> BZ is against OSP 6, so I am not sure if we might need to clone it to track
> it into OFI's OSP 7 release separately.  If Giulio's settings are what we
> want, that is a relatively straight forward addition here as well.  Since
> our puppet is structured a bit differently that tripleo's, we could also do
> this by setting the haproxy connection limit for galera to whatever was
> passed in for max_connections (or some calculation based on that number),
> preventing an additional parameter being needed for the user to set.

Yes. That is the right idea. Ideally we want the galera proxy's maxconn to be a bit under the database max_connections so that things like clustercheck can always get a connection to the database. For example, if max_connections is 4096 we would want to set maxconn it the galera proxy to ~4000.

If we set the maxconn in the galera proxy to something greater than max_connections, clustercheck will fail to connect if/when the db max_connections is reached and this would trigger a failover. We want to avoid this.

> [1]
> https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/
> quickstack/manifests/load_balancer/common.pp#L9
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1244929
> [3] https://review.openstack.org/#/c/202525/1

Only changing max_connections for galera for this BZ, the other changes are tracked separately:
https://github.com/redhat-openstack/astapor/pull/551

*** Bug 1246084 has been marked as a duplicate of this bug. ***

Merged to master