Bug 1262971 - messages is reporting cant connect to MySql server on undercloud [NEEDINFO]
messages is reporting cant connect to MySql server on undercloud
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 7.0 (Kilo)
Assigned To: chris alfonso
yeylon@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-14 15:13 EDT by bigswitch
Modified: 2016-04-18 03:01 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-02 12:23:29 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
calfonso: needinfo? (rhosp-bugs-internal)


Attachments (Terms of Use)

  None (edit)
Description bigswitch 2015-09-14 15:13:41 EDT
Description of problem:
Seeing this on 3 controller, 30 compute nodes setup. var/log/messages on undercloud is constantly reporting cant connect to mysql server. I am able to ping the mysql server though.
messages is attached


Sep 14 12:03:00 dell-undercloud ironic-discoverd: AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)") (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-9b9c5e89-aa93-4b74-b510-29cb2ff96a80)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 ERROR ironic.openstack.common.periodic_task [-] Error during ConductorManager._check_deploy_timeouts: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)")
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2519, in _connection_from_session
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 882, in connection
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 889, in _connection_for_bind
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     conn = engine.contextual_connect(**kw)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2037, in contextual_connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     self._wrap_pool_connect(self.pool.connect, None),
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2076, in _wrap_pool_connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2072, in _wrap_pool_connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     dbapi_connection = rec.get_connection()
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     self.connection = self.__connect()
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 602, in __connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     connection = self.__pool._invoke_creator(self)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)")


Version-Release number of selected component (if applicable):
7.0

How reproducible:
seeing this on a small scaling setup, not seeing this on a setup with three controller and 1 computes

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 3 chris alfonso 2015-09-17 12:53:19 EDT
Please verify the network config is correct and that the socket is listening on 192.0.2.1 as you intend it to be.
Comment 4 bigswitch 2015-09-17 14:23:17 EDT
the network config is correct. The issue is actually mariadb stopped running on the undercloud for some unknown reason. I restarted mariadb and it comes up fine.
Comment 5 chris alfonso 2015-09-17 15:53:26 EDT
Out of curiosity, what is the memory utilization on the undercloud host? MySQL should just terminate...I'm wondering if oom-killer got it.
Comment 6 bigswitch 2015-09-17 16:06:05 EDT
I didnt check at the time it died, currently, out of 16gig memory, 11gig is in used

[root@dell-undercloud ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          15720       11075         167         786        4478        3522
Swap:          7999         425        7574
Comment 7 chris alfonso 2015-09-22 12:35:48 EDT
Okay so a couple things to check here. 1, did you reach the max connections allowed to MySQL, make sure the service is up and listening to the interface you expect, and check the memory allocation if the service dies. If possible do this as soon as you notice it has died. It would be good if you can monitor the mem usage and notify yourself it it gets very low.
Comment 8 chris alfonso 2015-09-30 12:49:39 EDT
Let us know if you continue to see this issue. If we don't hear anything on this by next week, I'll go ahead an close the bug.
Comment 9 chris alfonso 2015-10-02 12:23:29 EDT
If you run into the issue again, go ahead and reopen the bug.

Note You need to log in before you can comment on or make changes to this bug.