Description of problem: Seeing this on 3 controller, 30 compute nodes setup. var/log/messages on undercloud is constantly reporting cant connect to mysql server. I am able to ping the mysql server though. messages is attached Sep 14 12:03:00 dell-undercloud ironic-discoverd: AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)") (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-9b9c5e89-aa93-4b74-b510-29cb2ff96a80) Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 ERROR ironic.openstack.common.periodic_task [-] Error during ConductorManager._check_deploy_timeouts: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)") Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2519, in _connection_from_session Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 882, in connection Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 889, in _connection_for_bind Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task conn = engine.contextual_connect(**kw) Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2037, in contextual_connect Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task self._wrap_pool_connect(self.pool.connect, None), Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2076, in _wrap_pool_connect Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2072, in _wrap_pool_connect Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task dbapi_connection = rec.get_connection() Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task self.connection = self.__connect() Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 602, in __connect Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task connection = self.__pool._invoke_creator(self) Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task return dialect.connect(*cargs, **cparams) Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task return self.dbapi.connect(*cargs, **cparams) Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__ Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)") Version-Release number of selected component (if applicable): 7.0 How reproducible: seeing this on a small scaling setup, not seeing this on a setup with three controller and 1 computes Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Please verify the network config is correct and that the socket is listening on 192.0.2.1 as you intend it to be.
the network config is correct. The issue is actually mariadb stopped running on the undercloud for some unknown reason. I restarted mariadb and it comes up fine.
Out of curiosity, what is the memory utilization on the undercloud host? MySQL should just terminate...I'm wondering if oom-killer got it.
I didnt check at the time it died, currently, out of 16gig memory, 11gig is in used [root@dell-undercloud ~]# free -m total used free shared buff/cache available Mem: 15720 11075 167 786 4478 3522 Swap: 7999 425 7574
Okay so a couple things to check here. 1, did you reach the max connections allowed to MySQL, make sure the service is up and listening to the interface you expect, and check the memory allocation if the service dies. If possible do this as soon as you notice it has died. It would be good if you can monitor the mem usage and notify yourself it it gets very low.
Let us know if you continue to see this issue. If we don't hear anything on this by next week, I'll go ahead an close the bug.
If you run into the issue again, go ahead and reopen the bug.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days