Bug 1262971

Summary:	messages is reporting cant connect to MySql server on undercloud
Product:	Red Hat OpenStack	Reporter:	bigswitch <rhosp-bugs-internal>
Component:	rhosp-director	Assignee:	chris alfonso <calfonso>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	yeylon <yeylon>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	hbrock, mburns, rhel-osp-director-maint, rhosp-bugs-internal, srevivo
Target Milestone:	---
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-10-02 16:23:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description bigswitch 2015-09-14 19:13:41 UTC

Description of problem:
Seeing this on 3 controller, 30 compute nodes setup. var/log/messages on undercloud is constantly reporting cant connect to mysql server. I am able to ping the mysql server though.
messages is attached


Sep 14 12:03:00 dell-undercloud ironic-discoverd: AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)") (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-9b9c5e89-aa93-4b74-b510-29cb2ff96a80)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 ERROR ironic.openstack.common.periodic_task [-] Error during ConductorManager._check_deploy_timeouts: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)")
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2519, in _connection_from_session
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 882, in connection
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 889, in _connection_for_bind
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     conn = engine.contextual_connect(**kw)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2037, in contextual_connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     self._wrap_pool_connect(self.pool.connect, None),
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2076, in _wrap_pool_connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2072, in _wrap_pool_connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     dbapi_connection = rec.get_connection()
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     self.connection = self.__connect()
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 602, in __connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     connection = self.__pool._invoke_creator(self)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__
Sep 14 12:03:02 dell-undercloud ironic-conductor: 2015-09-14 12:03:02.557 12529 TRACE ironic.openstack.common.periodic_task DBConnectionError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on '192.0.2.1' (111)")


Version-Release number of selected component (if applicable):
7.0

How reproducible:
seeing this on a small scaling setup, not seeing this on a setup with three controller and 1 computes

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 chris alfonso 2015-09-17 16:53:19 UTC

Please verify the network config is correct and that the socket is listening on 192.0.2.1 as you intend it to be.

Comment 4 bigswitch 2015-09-17 18:23:17 UTC

the network config is correct. The issue is actually mariadb stopped running on the undercloud for some unknown reason. I restarted mariadb and it comes up fine.

Comment 5 chris alfonso 2015-09-17 19:53:26 UTC

Out of curiosity, what is the memory utilization on the undercloud host? MySQL should just terminate...I'm wondering if oom-killer got it.

Comment 6 bigswitch 2015-09-17 20:06:05 UTC

I didnt check at the time it died, currently, out of 16gig memory, 11gig is in used

[root@dell-undercloud ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          15720       11075         167         786        4478        3522
Swap:          7999         425        7574

Comment 7 chris alfonso 2015-09-22 16:35:48 UTC

Okay so a couple things to check here. 1, did you reach the max connections allowed to MySQL, make sure the service is up and listening to the interface you expect, and check the memory allocation if the service dies. If possible do this as soon as you notice it has died. It would be good if you can monitor the mem usage and notify yourself it it gets very low.

Comment 8 chris alfonso 2015-09-30 16:49:39 UTC

Let us know if you continue to see this issue. If we don't hear anything on this by next week, I'll go ahead an close the bug.

Comment 9 chris alfonso 2015-10-02 16:23:29 UTC

If you run into the issue again, go ahead and reopen the bug.

Comment 10 Red Hat Bugzilla 2023-09-14 03:05:18 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days